Scale, Measure, Optimize

10 minutes
scale measure optimizeagent performance analyticsAI FinOpsagent elo ratingsgit worktree isolationCrewDeck

Scale, Measure, Optimize

Overview

A twelve-person engineering team has something a solo developer doesn't, and it isn't talent — it's process. Review that happens whether or not anyone feels like it. Velocity data nobody had to assemble by hand. Someone whose job is noticing that the last three sprints slipped.

That machinery is most of what funding actually buys. This video argues that a single developer can now build the same machinery, and run it on a cycle a funded team structurally cannot match — because their loop turns over in quarters and yours turns over in afternoons.

The loop has three steps: scale, measure, optimize. Most of the industry is selling the first one. The advantage is in the other two.

Key Topics

  • Isolation is what makes parallelism real — five agents in one working tree is one agent and four sources of corruption; CrewDeck gives each task-execution session its own git worktree, and that is what turns concurrency into throughput
  • Scale is directionless — it's a multiplier, and multipliers work on negative numbers too; 2,500 merged pull requests is either a genuine win or 2,500 chances to ship a defect nobody looked at, and the number alone can't tell you which
  • Input metrics vs. outcome metrics — tokens, prompts, and lines generated measure how hard the machine worked, which is the AI equivalent of counting keystrokes
  • Scoring the agent — a five-component composite (success rate, satisfaction, efficiency, recency, and override decay) multiplied by a confidence factor, so an agent with two runs is mechanically held down rather than merely "unreliable"
  • Override decay — the one component that asks whether the human believed the recommendation, catching failures no success rate can surface
  • Attributing the spend — every model call tagged with the sub-stage that fired it, cost per merged line of code joined against real git statistics, and rework share as the single most useful number on the screen
  • Duels and Elo — running one task specification twice as two arms and having a blind judge score the diffs, so the frontier model can't win before the code is read
  • Structural verification — a five-stage pipeline where review and improve are deliberately separate passes, because a finding has to survive being written down
  • Optimization with a trap in it — per-stage model routing, an explicit cache boundary, a shared ledger of established facts, and the one-click down-tier that can lose you money, which is exactly why it doesn't ship switched on
  • Where the argument stops — fifty engineers still buy things no loop replaces, and velocity is not direction

About This Video

This is the video companion to Episode 003 of the CrewDeck podcast. The full narration script is in Episode 003: Scale, Measure, Optimize, and the long-form written version, with the source references behind each figure, is in Scale, Measure, Optimize: How One Developer Outruns a Funded Team.

If you take one thing away, take the ordering. It's a dependency chain, not a slogan: you cannot optimize what you haven't measured, and there is nothing worth measuring until you've scaled past what fits in your head. Start at two agents, not twenty.