Mar 2, 2026

Outrunning Oversight (Part 3)

Cycle time looked fine for two straight weeks. Escalations quietly tripled in the same window.

Nobody caught it until Friday.

We were only watching one number, and that number is the one most likely to lie to you. Cycle time answers “are we fast.” It says nothing about “did anyone actually understand what shipped.” That second question is the one that matters once an agent can produce more implementation volume in a day than your review rotation can read in a week.

That is the lie cycle time tells you.

By the time that gap shows up in cycle time, it is already expensive. Reviewers rubber-stamping to keep pace. Incidents nobody can explain, because nobody who reviewed the change actually understood it. The whole system’s credibility eroding for reasons that took a week to trace back to their actual source.

That is what pushed us toward measuring the system deliberately instead of glancing at velocity.

Six numbers, one system health check, read together, never alone

Here are the six numbers that actually matter.

Everything in the metrics stack rolls up to six headline numbers, checked weekly on a rolling 4-week average. These targets emerged from our own baseline measurements and risk tolerance, not generic benchmarks:

Metric	Definition	Target	What it is actually watching for
Cycle Time	Spec approved to shipped to production	< 5 days	Speed, and almost nothing else; see below
Escalation Rate	% of slices hitting the 3rd-failure threshold	< 5%	Agent output repeatedly exceeding what a Spec/prompt can constrain
Human Attention Utilization	% of available human time spent on active decisions	40 to 60%	Whether oversight capacity is keeping pace with agent throughput
Token Consumption per Slice	Total input plus output tokens per completed slice	< 150k tokens	Cost and compute efficiency; agents burning tokens on unhealthy code or rework
Goal Adherence Rate	% of slices where Post-Eval passes first attempt and Review approves without major rework	> 85%	Whether agent output actually satisfies Spec acceptance criteria
Codebase Entropy Score	Duplication plus churn ratio measured across agent-generated changes	< 15%	Whether agents are re-writing existing functionality instead of extending/improving

Cycle time by itself is dangerous precisely because it cannot distinguish “fast because the system is healthy” from “fast because review quietly became a formality.” The story above is exactly that.

We are not the only team to hit this. A 2025 controlled trial by Augment Code found that the three most common AI productivity metrics (adoption rate, developer satisfaction, self-reported speed) all produced false positives. The pattern is industry-wide: agent output compresses lead time faster than review capacity scales, and cycle time alone cannot see the gap.

Here’s the one people do not expect: 40 to 60% utilization is the target band, not a floor to push past.

Below 40%, your humans have real slack and you can likely take on more agent throughput. Above 60% for any stretch, reviewers are past their real capacity and something is already giving, whether or not cycle time has noticed.

Push past 60% and something gives. Whether cycle time has noticed yet or not.

That is the earliest hard signal that agent output has outrun oversight capacity. Treat it with more urgency than a cycle time miss.

Read these six together and they answer a single question: are we shipping understood work, or just fast work?

Token consumption and goal adherence: the cost of volume

We track total input plus output tokens per completed slice, rolling 4-week average. Target under 150k tokens. When the number climbs, it usually means one of three things: the codebase is unhealthy, the Spec is ambiguous, or the agent is re-working the same problem repeatedly. CodeScene research shows agents working on unhealthy codebases consume almost 50% more tokens to complete the same tasks. That’s in addition to defect risk.

Goal Adherence Rate measures whether the agent actually achieved what the Spec asked for. Definition: percentage of slices where Post-Eval passes on first attempt and Review approves without major rework. Target > 85%. This is task completion, not token spend. JetBrains Kotlin Benchmark reports Claude Code with Opus 4.7 xhigh resolving 85.71% of tasks in a production-derived benchmark. Below 80% signals issues that need immediate attention.

Read token consumption alongside Goal Adherence. High tokens with low adherence means the agent is working hard and getting it wrong. That’s the most expensive failure mode: you pay for compute and you still pay for human rework.

We measure duplication plus churn ratio across agent-generated changes. Duplication is calculated by comparing blocks of code, lines, or entire files to find syntactic similarities, then dividing duplicated code by total code. Churn tracks how often the same files are touched repeatedly by agents. Combined, they form an entropy signal.

Target < 15%. When entropy climbs, agents are often gaming the system: producing new code that looks productive but creates parallel implementations of existing logic. That creates implicit coupling and bug surface area that will require the same fixes applied to multiple places.

Entropy also correlates with token waste. Unhealthy code burns more tokens and produces more rework. Keeping entropy low means agents build on existing foundations rather than starting from scratch each time.

Two human-capacity signals, read together

Review capacity: Review Approval Rate dropping while Review Throughput climbs is the sharpest signal that reviewers are underwater. Slices are taking longer to get through Review and passing less often on the first try.

This is what volume pressure looks like before it turns into a rubber stamp.

Spec quality: Intent Debt Rate is a secondary signal, not a headline metric. It measures the share of Post-Eval failures traced back to an ambiguous Spec (target under 20%). A rising rate means specs are producing harder-to-verify output, which pressures the review bench downstream.

It is real but narrower than the three headline numbers above. A clear spec makes the reviewer’s job faster; an ambiguous one forces them to reverse-engineer intent from the diff.

Read these two signals together and you get a diagnosis:

Review Approval falling, Intent Debt stable: the specs are fine, but you have too much volume for your reviewers. Fix: smaller slices, tighter WIP limits, more reviewers.
Intent Debt rising, Review Approval stable: your specs are getting worse, but reviewers are still catching it. Fix: invest in spec quality upstream before reviewers start burning out on bad targets.
Both rising: the system is breaking on both sides. Specs are ambiguous and reviewers are drowning. This is the pattern that leads to rubber-stamped reviews shipping unaccountable code. Fix: slow agent throughput immediately and address both spec quality and review capacity.

Agent layer: Agent Utilization pinned near 100% is not, by itself, good news.

High utilization next to falling review approval is not efficiency. It’s a warning.

Two human-capacity signals, read together

Reading the headline numbers alongside the human/agent split from Part 1 is what actually locates a problem. The Human Layer has two capacity signals that tell you about the two sides of the bottleneck:

Cycle time includes time spent actually doing the work and time spent waiting between steps.

A ticket sitting in “Post-Eval passed” for six hours before anyone picks up the review is not work, it is tax. That tax has a name: Orchestration Latency, and it is tracked explicitly rather than absorbed silently into cycle time.

That is process waste. Pure and simple.

Orchestration Latency = (sum of all handoff latencies) / (total cycle time)
Target: < 20% of cycle time

Five handoffs get timestamped automatically, each with its own SLA:

Handoff	SLA
Spec approved to Slicing starts	< 4h
Pre-Eval passed to Agent starts	< 15m
Agent complete to Post-Eval starts	< 30m
Post-Eval passed to Review starts	< 15m
Review approved to Integration starts	< 15m

Each transition fires a Jira automation rule the moment the status changes, so the latency is a timestamp diff computed automatically, not a manual log entry someone forgets to make. If a handoff sits past its SLA, the column owner gets alerted immediately. If it breaches twice in the same week, it escalates to the Delivery Coach.

Handoff latency is pure process waste, not comprehension time. Cutting it makes the system faster without asking anyone to review more carelessly.

That is exactly why it is the easiest lever to pull, and exactly why it is not a substitute for watching Utilization and Review Approval Rate. You can drive orchestration latency to zero and still have a comprehension problem underneath it.

Metrics data flow

The pipeline turns daily slice logs into actionable signals:

Daily log (one row per slice)
|
+---> Weekly Summary (rolling 4-week averages)
|
+---> Alerts & Trends (rule-driven: SLA breach, cycle-time spike,
|       pass-rate drop, escalation rate)
|
+---> Capacity View (utilization per role: QA, Integration, etc.)

What the dashboard actually looks like

Those three feeds are also the three views the team actually opens on Friday.

Weekly Summary smooths noise so a one-bad-day spike does not trigger a panic. Alerts & Trends is where root-causing happens once something crosses threshold. Capacity View is the one place utilization gets broken down by role instead of averaged into a single number that hides who is actually underwater (QA at 75%, Integration at 35%, say).

Friday’s 30-minute review runs straight off this: fifteen minutes on the Weekly Summary, ten minutes chasing down any active alert’s root cause, five minutes picking exactly one change to try next week.

Not five changes, one, because a dashboard with five simultaneous experiments running against it tells you nothing about which one actually worked. And a “we shipped more this week” result is never celebrated on its own. It is only good news if Utilization, Review Approval Rate, Intent Debt Rate, Token Consumption, Goal Adherence, and Codebase Entropy all came along for the ride.

I started measuring handoff latency on a Tuesday. By Thursday I had found three hours of pure wait time hiding inside our cycle numbers.

Nobody was slacking; the tickets were just sitting in the wrong columns. Fixing the handoff rules shaved a full day off cycle time without asking any reviewer to read faster.

That is the kind of signal these metrics give you when you actually look at them. It cost us an afternoon of digging and paid for itself in the first week. [Up next: end of series]