Retiring the RAG pipeline

TerraTitan is the machine that ports AWS CDK L2 constructs into TerraConstructs. The first version was a Mastra workflow: an Upstash vector index over the Terraform AWS provider, an embedding-and-rerank retrieval step, two points where the run suspended and waited for a human, and Gemini generating the result in a single few-shot prompt.

One year later—an eternity in LLM advancement—the replacement has a core operational layer of four workflow scripts, backed by a few small deterministic tools and Markdown rulebooks. There is no vector database, no embedding step, no suspend/resume, and no API keys beyond basic Claude Code harness authentication. Across seven conversion runs, it has ported AWS CDK modules into the library and landed them as merged pull requests.

This article breaks down how we got there: what changed, what the runs measured, and what the workflow still cannot prove.

What the old pipeline did

The old workflow layered setup and static discovery, two blocking human reviews, retrieval-driven mapping, and source/test generation. The human reviews made sense as safety checks, but they also held the whole run open.

When the reranker scored a candidate at 0.7 or below, a human had to choose among the five returned mappings.

Interestingly, the retrieved chunks were never what the generator saw. Once a human confirmed a mapping, the selected type name was used to look up data/reference/merged/provider-aws/<resource>/index.d.ts, and a compressed file containing only type definitions went into the prompt to ground the TypeScript generator. The vector database only bridged the CloudFormation-to-Terraform-provider-AWS mapping semantically.

But once the initial conversion ran, no gate of any kind looped over the outputs. There was no compile check, test run, or fix loop.

This planned expansion stalled when HashiCorp announced the deprecation of Terraform for CDK. That created an opportunity to establish the official, community-maintained CDKTN.io fork. After six months of stabilization and revival efforts under the Open Constructs Foundation, it was time to revisit TerraTitan.

The inversion

The new system's design note captures the goal of the rewrite:

Two things replaced retrieval:

Live types instead of an index. Every agent is pointed at node_modules/@cdktn/provider-aws/lib/<resource>/index.d.ts inside the worktree it is compiling, plus an exact-tag aws-cdk-lib install for the Cfn* definitions. The mapping prompt literally instructs ls $PROVIDER | grep -i $prefix. With Claude Code, agents inspect the files in the active worktree directly. There are no snapshots to refresh, because the ground truth is the same files the compiler will use. The entire staleness class disappears by construction.

An AST scan instead of LLM discovery. Early runs spent LLM tokens working out which files to convert and in what order. tools/cfn-scan.mjs now reads the TypeScript source and writes down those facts deterministically.

Think of it as a dependency-aware checklist. If service.ts imports task-definition.ts, the task definition must exist before the service can import it. The scanner puts task-definition.ts in an earlier conversion wave, then puts service.ts in a later one. Files in the same wave can be converted in parallel because they do not depend on one another.

The scanner also classifies each file so the workflow can choose the right treatment:

• L1_BACKED files use one or more Cfn* resources. They need a real conversion, including a CloudFormation-to-Terraform mapping.
• PURE_L2 files only compose other L2 constructs or helpers. The workflow can copy them nearly verbatim, adjust imports and headers, and send many of them to one lower-effort agent.
• BARREL files mostly re-export other files. They are wired after the destination paths are known.

The scanner emits the Cfn inventory, conversion waves, cross-module prerequisites, and those classifications in about a second. It replaced LLM-based inventory discovery with a parser, leaving the agents to spend their effort on mapping and code changes that require judgment.

What a dynamic workflow looks like

The workflow is a plain JS script. It spawns agents, and it branches on what they return.

phase('Plan')
const plan = await agent(planPrompt, { model: 'opus', schema: PLAN_SCHEMA })

phase('Convert')
for (const ord of orders) {
  const wave = plan.srcFiles.filter((f) => f.order === ord)
  await parallel(batchesOf(wave).map((batch) => () =>
    agent(convertPrompt(batch), { model: 'sonnet', schema: CONVERT_SCHEMA })))
}

Three properties matter more than the syntax.

Models are pinned per agent and never inherited. Sonnet converts and fixes; opus plans, adversarially checks the mappings, verifies independently, and reviews. Mechanical agents — the copy-mode batch, codegen, git add -A — run at low reasoning effort. For first-time Dynamic Workflows adopters, one rule is crucial: never inherit the session model.

Nearly every agent returns structured data that the script can read. plan.srcFiles decides the fan-out shape. plan.integChoice decides which integration test gets ported. verify.violations.length decides whether a fixer agent gets spawned at all. The workflow is dynamic because the graph is computed from what the previous phase found rather than drawn in advance.

Loops are bounded and the verifier is independent. Compile and test retries are capped at six attempts; integration synthesis is capped at three; verification is limited to three rounds with a fixer in between.

Blocking, per-file, mid-run CLI prompts became asynchronous review of committed artifacts—the plan JSON, mapping manifests, reports, and pull-request diffs. The reviewing agent records the sign-offs a human should consider before a PR. The external review of the ECS pull request came back with seven blocking findings, so the review load is real; it simply no longer holds the pipeline open.

Seven runs

The numbers below come from the harness's own run journals.

Runs 1 through 5 all happened inside 21 hours, orchestrated by Fable. Some attempts do not appear in the table because they were misfires corrected two minutes later.

Run 1: never let the generator be its own oracle

The first run reached compile-green and test-green on its own, which felt like a result for about as long as it took the reviewing agent to diff the output against the reference implementation.

The verdict identified a crucial gap in its initial design:

the "passing" state is self-certified: the generator authored the tests from upstream CloudFormation semantics and then made its own output satisfy them, so the checks were blind to the repo's actual invariants.

Concretely: the repo's stack-scoped physical naming was gone, replaced with a plain name. A public handle had been made private. Policy statements grew an attribute they shouldn't have. All five golden snapshot tests had vanished. And one fix-loop agent had "repaired" a failing assertion by removing the resolve call from one side of it, turning it into a tautology.

Fable determined the fix wasn't better tests. It was a structurally independent verifier that reads the files, the snapshots and the git diff against an explicit rule list, plus the rulebook those rules live in. Invariant six in that document reads: tests are not the oracle.

Run 2: each oracle catches what the one below cannot

Run 2 added the independent verify phase, and then went further and deployed the result.

aws_autoscaling_schedule defaults unset min/max/desired values to zero, and reserves −1 as the "don't modify this" sentinel. An undefined passing straight through produced a request AWS rejected at apply time. Jest was green. The convention verifier was green. tofu validate was green. Only a real apply against a real account failed.

That produced the ladder every run since has climbed: jest for shape, independent verification for invariants, tofu validate for provider schema, live apply for provider runtime semantics—and, by run 7, a drift oracle that asserts a second plan comes back empty. In these runs, each added check found failures the earlier checks had not detected.

Run 3: verify the completed artifact

Run 3 was an adjudication: the system converted aws-secretsmanager blind, then diffed the result against a pull request a human had already written, which the agents were forbidden to read.

It won the mapping. Faced with a CloudFormation resource that has no Terraform equivalent, it produced a composition rather than hallucinating a resource that doesn't exist, and independently derived a validation error the human PR only got after review. It lost the API design: its version model made the canonical upstream call pattern throw.

The structural lesson was smaller and more useful. Verification ran before the integration leg, so it reported violations about artifacts that had not been produced yet—and each phantom finding triggered a fix round. Moving integration ahead of verification removed that class of false finding.

Run 4: the miss that located the next optimization

Run 4 swapped LLM inventory work for the AST scanner and measured it.

Run 4 matched the scanner's inventory exactly, included no deprecated members (the first run to pass a gate that Runs 2 and 3 both failed), required no manual fixes afterward, and received a review verdict of no correctness defects found.

The run used 966 tokens per LOC against a 350-token-per-LOC target—about 2.8 times the target.

Both axes moved, in opposite directions. Per-agent cost fell by a third — the tooling worked exactly as intended. Agent count rose 46% on a module half the size, because fan-out was one agent per file and servicediscovery is twelve small files averaging 110 lines where secretsmanager was five averaging 265.

Fixed per-agent overhead (reading conventions + plan + mapping + siblings ≈ 40–70k tokens) dominates for many-small-file modules. LOC-normalization hides granularity effects; per-agent cost is the truer tooling signal.

Both metrics are needed. tok/LOC measures the workload; tok/agent measures the overhead. Report only one and you will misread the experiment.

Run 5: the granularity A/B

Same module, same tag, same inputs, an explicit integrity guard forbidding the agents from reading run 4's output — and only the script changed.

Batching became possible once the planner emitted facts the script could act on. In Run 4 the planner already knew which files were pure-L2 boilerplate; it wrote so in a free-text notes field, where no if statement could reach it. Run 5 promoted loc and copyMode to required schema fields.

Then the converter loop could budget:

for (const ord of orders) {
  const wave = plan.srcFiles
    .filter(f => f.order === ord)

  await parallel(wave.map(f => () =>
    agent(convertOne(f), {
      model: 'sonnet',
      schema: CONVERT_SCHEMA,
    })))
}

const BATCH_LOC = 700
for (const ord of orders) {
  const wave = plan.srcFiles
    .filter(f => f.order === ord)
  const copies = wave.filter(f => f.copyMode)
  const real = wave.filter(f => !f.copyMode)
  const batches = budget(real, BATCH_LOC)
  if (copies.length) batches.push(copies)

  await parallel(batches.map(b => () =>
    agent(convertMany(b), {
      model: 'sonnet',
      schema: CONVERT_SCHEMA,
      ...(b[0].copyMode ? { effort: 'low' } : {}),
    })))
}

Copy-mode files — the ones with no CloudFormation resource behind them — all go to a single agent, with a different prompt and reduced reasoning effort. The rulebook got split too: converters now read a 49-line core sheet instead of the 26 KB full document, which every converter had previously been paying for.

The last change fixed a latent script bug. Run 4's verify schema had only violations[], and its journal records the consequence — verifyPass: true, unresolvedViolations: 3. The opus verifier had returned "pass" while listing three violations, because it had nowhere to put a judgment call. Run 5 added an advisories[] array, tightened the gate to pass && violations.length === 0, and told the verifier which findings belong where. Advisories never trigger a fix round.

Two honest footnotes on that table. The converter phase saved seven agents, not five—verification needed a second round in Run 5 whereas Run 4 passed on the first attempt, because the tightened gate caught two real violations the old gate would have waved through. And the planner also chose a different wave grouping and a different integration test, so a run advertised as controlled had one uncontrolled variable inside it.

Run 4's reviewer reported no correctness defects, but Run 5's A/B diff found one it had missed: a property injection identifier omitted its submodule segment, and the defect had already shipped in a merged pull request. A clean bill of health from the same family of models that wrote the code is weak evidence; this project's own artifacts provide a concrete counterexample.

Runs 6 and 7: what scale and reality added

aws-ecs was 45 files and 37,000 lines, and achieved the best ratio of any run—229 tokens per line— because the fixed overhead finally had something to amortize against. It also found a new failure mode: after two schema-validation failures the review agent returned placeholder junk that passed validation. Every workflow since writes its report to a file first and returns only a pointer.

Then the integration tests were actually deployed, and four construct defects surfaced that jest, snapshots and tofu validate had all missed: scaling policies referencing raw strings instead of resource attributes, so no dependency edge existed and two AWS calls raced; a name-length validator that rejected names generated by the code itself; a volume policy destroyed in parallel with the service that needed it, wedging the task; and a desired_count sentinel that created a service with zero tasks. Run 7 added a drift oracle and immediately caught a block that Terraform would have re-planned forever.

That campaign became port-integ-tests.js, a reusable workflow with a playbook based on failures observed during real deployments.

What generalizes

1. Never let the generator be its own oracle. Compile-green and test-green mean nothing when the same pipeline wrote the tests. You need a verifier that is structurally independent, not merely a second prompt.
2. Structured output is for the script, not the reader. If your control flow needs a fact, put the fact in the schema. Run 5's entire win traces back to promoting two fields out of a prose notes string.
3. Batch to a work budget, not a file count. Fixed per-agent overhead makes one-file-per-agent a tax.
4. Tier by whether the work needs judgment. Model and reasoning effort are per-call. Pin both; inheriting the session model makes cost unpredictable across runs.
5. Give the verifier somewhere to put opinions. With only a blocking channel, an opinionated reviewer either fails the run over a style preference or passes while listing violations.
6. Use tools for deterministic work and prompts for judgment. A 141-line AST script replaced a vector database, an embedding budget, and a manual re-index ritual.
7. Order phases so the verifier sees the finished artifact. Running verification too early creates phantom findings and unnecessary fix rounds.
8. Add checks in order of cost and coverage. Keep the checks that find failures the cheaper ones miss. In these runs, later checks exposed failures earlier checks did not.
9. A/B the pipeline, not just the output. Use the same inputs, change one variable, and prevent agents from reading the previous result. That turns “this feels better” into a measurement.

What this does not prove

Runs 4 and 5 are the only controlled experiment; every other run-to-run delta changes the module and the pipeline at the same time. Module size dominates the headline ratio — fitting tokens against lines across the full-pipeline runs suggests a fixed floor around 2M tokens per run plus roughly 180 per line, which means the 350 tokens-per-line goal was unreachable for small modules no matter what the script did. The floor alone exceeds the entire budget for a 2,500-line module. That regression mixes five pipeline versions, so treat it as a shape, not a constant.

The scope is not like-for-like either: the new pipeline ports source, tests and integration tests, and deploys them. Some of the extra spend produces more artifacts. Wall-clock is not clean—part of Run 5's 25% came from Run 4 hitting two stall-and-retry events. The A/B's quality advantage comes from concrete, checkable findings, but its aggregate verdict is still model judgment.

The honest summary is narrower than the numbers make it look. A vector database was replaced by a grep and an AST scan. Two blocking human gates became reviewable artifacts and PR-level review. Re-conversions showed controlled quality improvements as the new workflow evolved. The old pipeline's generator call was cheap, but the new workflow now reaches tests and live integration validation in less than half a day instead of leaving the remaining iteration to a week of human effort, depending on module size.