Episode 003: Scale, Measure, Optimize

An explainer on why the advantage in AI-assisted development has moved from generating code to measuring it — and why that shift favours the individual builder over the funded team.

Scale is already commoditized. The differentiator is what you do with the evidence.

What's Covered

• Why parallel agents only count as parallel when each one is isolated
• The difference between input metrics and outcome metrics, and why almost everyone measures the wrong one
• How an agent performance score can be built so a new agent structurally cannot post a high number
• Attributing every model call in an application to the stage that fired it
• Running two implementations of the same task and having a blind judge score them
• A one-click cost optimization that can lose you money, and why it ships switched off
• Where this argument stops applying

---

Narration Script

Single narrator. Section headings are structural — they are not read aloud.

Opening

A twelve-person engineering team has something you don't. It isn't talent. It's process.

They have code review that happens whether or not anyone feels like it. They have velocity data nobody had to assemble by hand. They have someone whose entire job is noticing that the last three sprints slipped.

That machinery is most of what funding actually buys. And for the first time, a single developer can build the same machinery — and run it on a cycle the funded team structurally cannot match.

The loop is three steps. Scale. Measure. Optimize.

Most of the industry is selling you the first one. The advantage is in the other two.

Part one: scale

Scaling as an individual developer used to mean working longer hours. It now means something different. Your ceiling moves from how fast you can type to how many independent units of work you can keep in flight at once.

The first has a hard biological limit. The second is an orchestration problem, and orchestration problems have engineering solutions.

The naive version fails immediately. Open five terminals in the same repository, point five agents at it, and you don't get five developers. You get five processes fighting over one working tree. One stashes another's changes. One checks out a different branch mid-edit. The result isn't slow. It's incoherent.

This is why isolation is the load-bearing part of the architecture. CrewDeck scales to twenty concurrent agent sessions, and each task-execution session gets its own git worktree — a separate working directory, on a separate branch, sharing the same underlying repository data. Agent seven cannot see agent three's uncommitted files, so agent seven cannot corrupt them.

Isolation is what makes the number real. Twenty agents in one directory is one agent and nineteen sources of corruption. Twenty agents in twenty worktrees is twenty parallel units of work.

One detail matters more than it looks. Twenty is the ceiling the architecture supports. It is not the value the application ships at. Concurrency is a dial you raise as your process earns it — and that turns out to be the entire argument, built into a settings field.

Here is the part worth sitting with. Scale is directionless. It's a multiplier, and multipliers work on negative numbers too. If your process is sloppy at one pull request a day, running twenty agents gives you twenty times the sloppiness, shipped faster, with more confidence, because now there's a dashboard.

CrewDeck's own repository has merged more than two thousand five hundred pull requests since the middle of January. That's roughly eleven a day, weekends included.

No human being reviews eleven pull requests a day with real attention. So that number is either a genuine multiplier, or it is two and a half thousand opportunities to ship a defect nobody looked at. And you cannot tell which one it is from the number alone.

You need the second step to tell which.

Part two: measure

"Measure your AI usage" is advice that sounds actionable and isn't. The question is what.

The tempting metrics are tokens consumed, prompts sent, lines generated. Those are input metrics. They tell you how hard the machine worked — which is the AI equivalent of evaluating an engineer by keystroke count.

What you want are outcome metrics, attached to a unit you can actually change. There are three such units.

The first unit is the agent.

Every task is assigned to a specialized agent, and every agent accumulates a track record scoped to your project. That record becomes a composite score from zero to one hundred, built from five weighted components.

Success rate, at thirty-five percent — did the task complete and produce a mergeable result. Satisfaction, at twenty-two — your thumbs up or thumbs down on the outcome. Efficiency, at eighteen — how this agent's run duration compares to other agents on the same category of task. Recency, at fifteen, decaying on a thirty-day half-life.

And the fifth one, at ten percent, is the one nobody else has. Override decay.

Override decay is the score watching you overrule it. When the system recommends an agent and you pick a different one, that rejection is recorded, and it costs that agent fifteen points — itself decaying on a thirty-day half-life, so last quarter's rejection has faded and Tuesday's has not.

Every other component asks how the agent did. That one asks whether the human believed us. An agent that keeps getting recommended and keeps getting swapped out has a real problem — and no success-rate figure would ever surface it, because the swapped-out runs never happened.

There's a second design decision worth more than it looks. The composite is multiplied by a confidence factor derived from sample count, and confidence does not reach one hundred percent until an agent has ten recorded runs. An agent with two runs cannot post a high score. It isn't that the number is unreliable and you should mentally discount it — the number is mechanically held down until the evidence exists.

Every spreadsheet anyone has ever built to track this has the opposite bug. Three runs, three successes, this agent is at a hundred percent.

The second unit is the spend.

Every model call the application makes is written to a cost ledger, and every row is tagged with the sub-stage that fired it. Not task execution as one undifferentiated bucket — around thirty-four distinct sub-stages.

Prompt refinement. Architecture validation. Execution itself. Pull request review, pull request improvement, documentation. And then all the passes most tools would never count as spend at all: effort assessment, backlog prioritization, task decomposition, complexity scoring, commit message generation, release notes, compliance scanning, failure coaching.

Failure coaching costs money. If you don't attribute it, it shows up as an unexplained percentage on your bill and you go looking for it in the wrong place. The point of the sub-stage tag is that when cost per task climbs, you can answer which part climbed instead of guessing.

On the outcome side, the dashboard reports cost per merged line of code — joining the cost ledger against real git statistics for the task. Lines added, lines removed, files changed, commits, merge timestamp. Tokens measure effort. Cost per merged line measures result.

The single most useful number on that screen is rework share: the fraction of your in-progress spend that went to retries. One figure, telling you what proportion of your money is being spent doing things twice. A high rework share is almost never a model problem or an agent problem. It's a specification problem — and it stays invisible until somebody adds up the retries.

There is a forecasting layer on top of all this. The application estimates what a task will cost before you run it, then calibrates those estimates against your actual outcomes. It shrinks any factor fitted from too few samples back toward no-correction, so a brand-new project is a strict no-op. It backtests before trusting its own bias correction. It caps every coefficient so one freak run can't drag the model.

It doesn't just forecast. It grades its own forecasts and adjusts.

The third unit is the decision.

This is where measurement stops being a dashboard and becomes an experiment.

Take one task specification and run it twice, as two arms, with different configurations — different agent, different model, different settings. Both arms run in isolation against the same base. Then a judge scores the two resulting diffs against a versioned, weighted rubric.

The judge is a model. And here is what makes this an experiment rather than a demo. The judge is blind.

Its prompt contains the task specification, the rubric, and two diffs labeled arm A and arm B. It does not receive the agent names. It does not receive the model names. It cannot know which arm was the expensive one — so it structurally cannot do the thing every human reviewer does, which is decide the frontier model won before reading the code.

Results feed Elo ratings per agent, updated from head-to-head outcomes, with the cost of each arm attached so you can see what a win cost you. Over time you stop asking whether an agent feels good, and start asking what its record is against the field, on your repository.

So how does anyone verify two and a half thousand pull requests?

They don't, personally. The verification is structural.

Work moves through five stages — to do, in progress, in review, document, done — and the stages aren't decoration. Each one fires a command. In progress runs prompt refinement, then architecture validation, then execution. In review runs a review pass, then an improvement pass against that review. Document runs a documentation pass.

And review and improve are deliberately two separate passes rather than one. The reviewer is explicitly forbidden from implementing its own fixes. It checks the diff against a written standards document, and its job ends when the findings are posted. A different pass picks them up and acts on them.

That separation is the point. A finding has to survive being written down and handed off. If the same context that noticed a problem is also allowed to quietly patch it, you never find out what it caught — and neither does your ledger.

The human sits at the top of that pipeline rather than in the middle of it. Which is exactly what a functioning engineering organization does with a director. You don't read every diff. You build a system that surfaces the diffs that need you.

And you can check whether the system works, because the repository has a floor. Continuous integration enforces coverage thresholds — ninety-three percent of statements, ninety-four percent of lines, eighty-eight percent of branches, ninety percent of functions — across roughly sixteen hundred test files. Two and a half thousand pull requests through a gate like that is a different claim from two and a half thousand pull requests.

Part three: optimize

Optimization without measurement is superstition. It seems to work better when I phrase it this way. Maybe it does. You have no idea.

With the data in hand, the moves become concrete. Here are four of them — and the first one has a trap in it, which is the most honest thing in the product.

Route models per stage. Because every stage is separately instrumented, every stage can separately be assigned a model. The defaults already do this — the documentation pass runs on a small, fast model, because writing docs does not need a frontier one. And there's a one-click action that down-tiers the two planning stages, prompt refinement and architecture validation, the same way. Per project. Reversible.

That sounds like free money. And the codebase explicitly refuses to make it the default.

The reasoning is written into the source: down-tiering prompt refinement trades plan quality for drafting cost — and a weaker plan can raise your downstream execution cost by more than it saved upstream.

The optimization can lose money. There is no way to know which way it goes for your repository without running it and reading the numbers. That is this entire argument, compressed into one product decision. Ship the lever. Refuse to pull it for you. Provide the instrument.

A "save forty percent" button would be a lie in some fraction of cases, and nobody would ever find out which.

Route work by record, not intuition. Once you can see which agents perform on which categories of task in your repository — and once duels give you head-to-head ratings rather than self-reported success rates — assignment stops being a guess. It's the cheapest available win, and most developers never get it, because they never had the data.

Make context explicit instead of rediscovered. This is the cost story nobody warns you about. When a codebase is small, prompts are cheap, because there's very little to read. As the repository grows, every prompt re-triggers the same expensive discovery — the agent goes hunting for the same architectural facts, again and again. Your cost per task climbs while task difficulty stays flat.

That's a context problem, not a model problem, and it has two fixes.

The first is a cache boundary. The context assembled for a run is deliberately partitioned. Sections that don't vary between runs — your conventions file, your cross-repository documentation — go above a frozen marker. Sections that vary per run go below it. The invariant half becomes a cached prefix the provider bills at a fraction of the rate, and the cache reads land in your ledger where you can watch it work.

The second is a shared ledger of established facts. Agents working in isolated worktrees are each a fresh mind, and by default each one re-derives what the last wave already settled. A small shared database at the repository root fixes that: an agent's first action is a warm-start read. Here is the pinned base commit. Here is what was already decided about the files you own. Here is who currently owns what.

Discovery happens once instead of twenty times — and it shows up in your efficiency numbers, so you can confirm it worked instead of assuming.

Fix the input, not just the agent. Decomposition with tracked dependencies is optimization at the front of the pipeline. A poorly scoped task produces a poor result regardless of which agent gets it, and burns the same money doing it. When success rate craters on a specific class of task, the cause is usually upstream of the agent.

And there's one more, which applies the loop to decisions rather than to code. For an architectural choice, multiple specialized agents can each propose an approach, critique the others, and have a moderator synthesize a consensus — which is then written to a markdown file and committed to your repository.

That last detail isn't incidental. The output is a decision record, which means the next agent to touch that subsystem can read why the call was made. Optimization that gets written down compounds. Optimization that lives in your head evaporates.

And disagreement is the real deficit of working alone — more than headcount. When you work solo, nobody tells you the idea is wrong before you've spent four days on it.

Why the loop beats the funding

Here's the thesis, stated plainly. A funded company runs the same loop. But their cycle time is measured in quarters, and yours is measured in afternoons.

They measure too. Their measurement arrives in a quarterly review. Their optimization requires cross-team alignment, a migration plan, and someone's sign-off. Every person added makes that slower, because coordination cost grows faster than headcount.

You open the efficiency panel, see that architecture validation is eating nineteen percent of your in-progress spend, down-tier it on one project, run five tasks, read the delta, and keep it or revert it. Before lunch. No meeting. No consensus. No migration plan.

You are strictly worse on raw capacity, and strictly better on iteration speed. And iteration speed compounds.

Where this doesn't apply

Accuracy beats triumphalism, so let's name the limits.

Fifty engineers give you things no loop replaces. An enterprise sales motion. Support at scale. Ten years of accumulated domain knowledge. Regulatory and compliance depth. The ability to survive one person getting the flu.

And the loop makes you faster at building whatever you already decided to build. If you decided wrong, you now reach the wrong destination in a third of the time.

Velocity is not direction.

Start at two, not twenty

The most common failure is starting at the headline number. Don't.

Start with two agents. One teaches you nothing about orchestration. Twenty teaches you everything simultaneously, at maximum volume, while you're on fire. Two makes the coordination problems visible while they're still cheap to fix.

That's what the concurrency dial is for. Twenty is what the architecture supports once your process deserves it — which is precisely why it isn't the shipping default.

Then measure before you scale further. Let the pipeline run — don't disable the review and improve passes to save money before you know what they're catching. Get an agent past ten runs, so its score carries full confidence and actually means something. Watch your rework share for a week.

Only then add agents. Because now each one inherits a system that works, instead of amplifying one that doesn't.

The order is a dependency chain, not a slogan. You cannot optimize what you haven't measured, and there is nothing worth measuring until you've scaled past what fits in your head. Each step earns the next.

Closing

One last thing, and it's the strategic heart of this.

The reason this window exists at all is that the tooling is lopsided. Enormous investment is going into step one — every vendor alive wants to sell you more agent throughput. Almost nothing is going into step two.

That asymmetry is temporary, and it cuts both ways. Closing the loop is an advantage available right now to anyone willing to do it — and it's one that gets competed away as the category matures. The developers who build measurement discipline now will keep it. The ones waiting for the tools to get easier will start from zero, against people who already have two years of data.

Scale quickly. Measure honestly. Optimize on the evidence.

---

The long-form written version of this material, with the source references behind each figure, is in the companion post Scale, Measure, Optimize: How One Developer Outruns a Funded Team.