Back to: Part 1 (Two Backlogs, One System)

The riskiest slice we ever shipped got a four-item checklist. The safest one got six.

We had it backwards for the first few weeks.

A one-line doc fix and an auth-flow change went through the same review process, so the doc fix waited behind the carefully reviewed auth change. It wasn’t a discipline problem or a routing problem.

It was a capacity problem in disguise: we had a fixed amount of reviewer attention per day and no way to spend it deliberately, so it went to whatever landed first, not to what risk demanded.

Part 1 covered why human and agent work need separate layers: agent throughput scales, human review capacity doesn’t. This post covers the mechanism that makes fixed capacity sufficient: the five points where a person has to make a call no matter how good the agent gets, and risk-tiered review that spends scarce attention where it matters.

Codacy found (source) that roughly a quarter of pull requests in studied organizations were reviewed by AI agents, while fewer than one percent were opened fully autonomously. Most of the risk still runs through code a human is nominally accountable for, even when the review itself is automated.

---

The five gates, at a glance

No matter how fast agents get, five decisions stay human. Here they are, each with an owner, a question, and an SLA:

Gate	Who	Question	SLA
Spec Approval	Product Lead / Spec Owner	Is the Spec complete enough to slice?	3 days
Pre-Eval	Spec Owner + QA	Is this slice ready for an agent to pick up?	4 h
Post-Eval	QA / Test Lead	Did it meet criteria, and if not, why?	2 h
Review	Tech Lead	Do I understand this well enough to be accountable for it in production?	6 h (medium risk) / 4 h (high risk)
Integration	Integrator / DevOps	Can we merge and deploy cleanly?	2 h

Each has concrete failure criteria, which is what makes the SLA credible instead of aspirational.

Spec Approval isn’t “does this look done.” It’s a checklist: clear problem statement, three or more testable acceptance criteria, three or more documented edge cases, a concrete example, unambiguous output format.

A clear spec is the first line of defense for downstream reviewers: it gives them a fixed, small target to check output against instead of an open-ended “does this look right.” Without one, even careful reviewers reverse-engineer intent from the diff, which takes longer, creates more errors, and burns scarce attention at higher cost.

That is the point. A clear spec is the first line of defense for the person who has to read the diff.

Post-Eval deserves a closer look because “it failed” isn’t the end. It triggers root-cause routing:

Post-Eval Failed
|- Intent Error   (Spec was ambiguous)     -> back to Spec Owner
|- Prompt Error   (prompt was unclear)     -> Prompt Refinement, then rerun
|- Eval Error     (test itself is wrong)   -> back to QA to fix the test
`- Impl Error     (agent execution wrong)  -> rerun the agent

This routing keeps a bad prompt from becoming a full Spec rewrite, and keeps a genuinely ambiguous Spec from getting patched over with a clever prompt that papers over the real gap.

It also surfaces spec-quality problems early: routing Intent Errors back to the Spec Owner signals that the spec needs work, not that it was bad luck. Fixing specs upstream is what saves review capacity downstream.

---

Review is where the actual capacity constraint lives

Here’s where the real constraint lives.

Every slice gets a risk tier when its Spec is approved, and the tier decides how much attention it can consume:

Tier	Examples	Reviewer	Checklist depth	SLA
Low	Docs, simple refactors	Any senior engineer	4-item spot-check	6 h
Medium	Feature additions, schema changes	Tech Lead	4-item standard review	6 h
High	Security, core logic	Tech Lead + Architect	6-item deep review	4 h
Escalation	3rd Post-Eval failure on the same slice	Tech Lead + Product	4-item escalation review	2 h

The Spec Owner assigns the tier at approval. Whoever picks up the agent work confirms it. It can go up if the slice turns out riskier than expected, never quietly down.

The decision tree is mechanical:

Start
  Touches security, auth, or secrets?
    Yes -> HIGH
    No  -> Changes core business logic or data model?
      Yes -> HIGH
      No  -> Impacts performance or scalability?
        Yes -> HIGH
        No  -> Changes a public API or contract?
          Yes -> MEDIUM
          No  -> Adds a new dependency or infrastructure?
            Yes -> MEDIUM
            No  -> Docs, tests, or config only?
              Yes -> LOW
              No  -> DEFAULT MEDIUM

Here’s what a high-risk checklist asks for versus a low-risk one. Low-risk review: does this diff make sense, match the Spec, have no obvious bugs? Four items, anyone senior. Deep-review time here is wasted capacity, not diligence.

High-risk review adds correctness across edge cases, a security pass, a performance check, architectural fit, and operational readiness. Six items, and it needs the Tech Lead and an architect.

That asymmetry is the mechanism.

It isn’t leniency. It’s triage.

Every minute not spent double-checking a doc fix is a minute available for the auth-change-level review. As agent output grows, that reallocation is what keeps Review from becoming a bottleneck or theater. It protects the Tech Lead’s finite hours for review that needs a human brain fully engaged.

Every review ends in one of three outcomes. Teams often forget the middle one:

• Ship: checklist passes clean, merge within the hour.
• Canary: checklist passes with a documented minor concern (new dependency, needs monitoring). Deploy to a slice of traffic first, watch 24 hours, then go wide.
• Block: a checklist item fails. The blocker must be specific and actionable. “Missing error handling for the empty-cart case” works. “Doesn’t look right” doesn’t. Vague blocks move ambiguity downstream instead of solving it.

Canary exists so reviewers don’t have to choose between blocking mostly-fine work and rubber-stamping something with a minor open question.

Without it, minor concerns get waved through, and that is the exact failure mode this gate prevents. That distinction matters more as more of the review process becomes agent-assisted: whatever tooling helps triage or draft a review, someone still has to be the accountable reviewer of record for what ships.

---

Escalation is a designed outcome, not a fire alarm

If a slice fails Post-Eval three times, that isn’t a signal to keep retrying with slightly different prompts. It’s an automatic trigger.

Three failures is the signal to stop retrying and start thinking.

An Escalation Review gets created, assigned to the Tech Lead and Product Manager together, with a 2-hour SLA to decide:

• Ship: failures were transient. The work is actually correct but had excessive tool usage.
• Spec Rewrite: the underlying intent is missing or wrong.
• Prompt Overhaul: the agent can do the work but needs a fundamentally different prompt.
• Block: the slice needs redesign before any agent can touch it.

This formalizes the conversation that used to happen in ad-hoc Slack threads, and gives it a clear owner and deadline.

---

Together, these five gates turn a fixed reviewer budget into a steering wheel. They let us spend scarce attention where risk lives, keeping the review pipeline from choking as agent output rises.

I still think about that first week, when the doc fix sat behind the auth change. It wasn’t a big deal in isolation, but it was the first signal that our review process wasn’t calibrated to the new reality. Once we started tiering, the difference was immediate.

The Tech Lead stopped feeling guilty about skipping the easy reviews. The easy reviews stopped feeling guilty about being skipped.

None of this worked because I was clever about it. It worked because the team actually used the tiers instead of working around them.

Gates and tiers tell you where a human has to make a call. Part 3 covers the numbers that tell you whether all five gates are still keeping pace with agent output, or whether one of them has quietly started to lag.

Up next: Part 3 (Measuring the Machine)