Back to: Intro

It is 8:50 a.m. and your incident-triage agent is doing its job. A pager fired at 8:41: the checkout service’s p99 latency crossed its alert threshold. The workflow has five steps: assess the alert, investigate the cause, contain the blast radius, verify the metric actually recovered, and notify the on-call. You wrote those five steps into the prompt.

The agent is a single loop. Each turn, it looks at what it has done so far and decides what to do next. It assesses the alert. It investigates the cause, and the logs point at a deploy that shipped at 8:35. It forms a hypothesis: that deploy is the cause. It rolls the deploy back. The rollback tool returns success.

On the next turn, the agent decides the task is complete. It writes “Rollback applied. Incident resolved.” and stops. It never ran the verification step. It never re-read the metric. The rollback had a warm-up window, and the service was still serving stale responses at 9 a.m. with p99 still climbing. Nobody was told, because the agent had already declared the incident over.

The agent had the satisfaction of a job well done. The checkout service had a morning of slow requests. They were not the same thing.

The agent was not broken. It was doing exactly what you built: it was left to decide what to do next, and it decided. The problem is that “the workflow is over” was never a decision the model should have made. It was a decision a machine should have made, from a declared list of states.

The fix everyone reaches for is “make the agent more careful.” A better prompt that says “always verify before you stop.” A more capable model. A longer context window so it remembers the steps. None of those touch the actual failure. The failure is that control flow lived in the model.

---

The failure already has a name

In every system we have ever made reliable, the thing doing the work does not own the control flow. A worker thread does not decide which task to run next; the scheduler does. A compiler does not decide its own parse order at runtime; the grammar does. A program running on a virtual machine is data; the fetch-decode-execute loop is the machine, and the program never gets to choose the next instruction. The next step is always decided by a deterministic structure, not by the payload.

This failure is the violation of that rule: the payload owns the control flow. The model was the payload, and it was also the scheduler. It decided what to do next, and it decided when it was done. That is the same failure as a worker thread that picks its own tasks, or a program that rewrites its own instruction pointer. We have a name for the fix, and it is old: separate the contribution from the control flow. The worker contributes a value. The machine decides what that value means for the next step.

That separation has a name in the frameworks I have been building with. It is the Redux and xstate lineage: a component dispatches an action (it contributes data), and a reducer decides the new state (it owns the control flow). The component never mutates state directly, and it never decides the next state. An actor sends an event to a state machine, and the machine decides the transition. The contributor and the controller are different things, and the controller is deterministic.

That is the principle of this article: the LLM should not own control flow. Put nondeterministic reasoning inside deterministic orchestration, policy, and state-management boundaries. The model is the contributor. The machine is the controller. And the machine is a finite state machine with a reducer, a policy, a state store, guardrails, and tool boundaries. The next section maps each of those.

---

The mapping

Here is the table I keep coming back to. The left side is a layer of the deterministic shell. The right side is what that layer decides, and what the model is reduced to doing inside it.

Deterministic shell layer	What it decides (and what the model is reduced to)
Control flow (state machine / reducer)	The current state and the legal next state. Transitions are declared edges; the model contributes a value that a guard routes on, never a next step.
Policy (what is allowed)	Which tools, states, and agents are in scope, and the budgets (max exchanges, max attempts). The model works inside the allowlist; it does not choose scope. This is also where model routing and fallback live: which model handles this state, and which backup model takes over when the primary is rate-limited, degraded, or down.
State (what is true)	The durable, typed context. Single source of truth, journaled, re-hydrated every turn. The model reads and contributes to it; it holds no private copy.
Guardrails (what the model may not do)	The hard limits: undeclared edges cannot fire, undeclared tools cannot be called, out-of-type values are rejected. A guard that fails simply does not fire. A guard-rejection rate that climbs is also a monitoring signal: it usually means a model or prompt change shipped that the eval suite did not catch.
Tool boundaries (what the model may call)	A fixed, typed interface. The model sees only the tools the current state exposes, and calls only those.

I would argue that every row in that table is a real incident waiting to happen if you have not thought about it. And every row has a known answer. You are not inventing a new kind of control. You are putting the control where it has always belonged: in the machine, not the payload.

The honest caveat is that the contributor on the right side is not a pure function. It is a model. That changes some of the semantics, and it is worth being precise about. But the shape of the separation is the same, and the shape is what fails.

---

What the architecture looks like

Here is the centerpiece: the LLM enclosed by the deterministic shell. Read it as a picture of containment. The model is in the middle. The five layers wrap it. And the only way the model reaches the outside world is through the tool boundary, which the policy scopes and the guardrails gate.

The undirected links are the claim: the five layers wrap the model. The single directed path out (LLM to tool boundary to outside world) is the only door. The model cannot step over the shell. It can only hand a value to the state store, call a tool that the policy has allowed, and let the reducer decide what that means for the next state.

Three properties fall out of this shape, and they are the same properties that make a state machine reliable:

• Control flow is declared, not decided. The current state and the legal next states are data in the machine. The model contributes a value; a guard routes on it. There is no “next step” the model can choose, because there is no such thing as a next step in the model’s vocabulary. The next step is an edge, and edges are declared.
• Legality is structural, not evaluated at fire time. An edge that is not in the YAML cannot fire. A tool that is not in the allowlist cannot be called. There is no permission check for the model to pass or fail; there is only the surface the machine has built. The model works inside a fixed, typed interface, not an open shell.
• State is external and the model is re-hydrated. The model does not hold the workflow in its head. It reads the store and contributes to it. Every turn, the runtime injects the current context and the current state’s objective, and the model starts from there. The “what is true” question has one answer, and it is not the model’s memory.

Run the 8:50 failure through this architecture and the early “task complete” stops being possible. Here is the corrected control flow, because the ordering is the whole trick: the model contributes, the reducer routes, and the machine decides when it is done.

The model does real work, but it never appears in the “decide what is next” role. At the agent-driven states it contributes values that take judgment: it assesses the alert, reads the logs and forms a hypothesis, and chooses the containment. That is the open-ended reasoning the shell is built to contain. But it never chooses what to do next, or when it is done. The machine’s job is everything else: it routes on declared edges, it makes verify a required state the model cannot skip, and it reaches done as a terminal state, not a verdict the model writes.

If you want the loop as plain code, here is the shape of it. This is the reducer, and it is the entire “shell” in miniature:

def advance(state, context, fsm):
    # The machine, not the model, decides what happens next.
    for _ in range(MICROSTEP_CAP):
        current = fsm.states[state]
        if current.terminal:
            return state
        # 1. Eligibility: are all required context fields present?
        if not all(has(context, r.name) for r in current.requires):
            return park(state, context)        # wait for a contribution
        # 2. Routing: the first declared edge whose guards all pass fires.
        edge = next((e for e in current.transitions
                     if guards_pass(e, context)), None)
        if edge is None:
            return park(state, context)
        # 3. Fire: run the onTransition actions (literal writes only), then move.
        for action in edge.on_transition:
            context = assign(context, action)
        state = edge.toState
    return state

Notice what is not in that loop: no “what should the model do next,” no “is the model allowed to stop,” no “ask the model if it is done.” The loop parks when it is waiting for a contribution, and it fires the first eligible edge when it is not. The model is called to produce a value for the current state’s objective, and that is the sum of its role in the control flow.

I covered the implementation of this in depth in Binding LLMs to Finite State Machines Using the Reactive Reducer Pattern. This article is the principle that design is built on.

---

An incident-triage workflow that survives its own shortcuts

Let me show the whole thing as an Aegis FSM, because this is the format I use for agent workflows. It is a YAML file, and it is small on purpose. The interesting parts are the three agent-driven states where the model does real work (assess, investigate, contain), the verify state that the model used to skip, the re-entrant loop that bounds the re-checks, and the fact that done is only reachable after verify passes.

fsm:
  name: incident-triage
  version: 1
  initialState: assess

  context:
    alertSeverity:     { type: enum, values: [critical, major, minor] }
    rootCause:         { type: string }
    containment:       { type: enum, values: [rollback, scale_out, restart, none] }
    containmentOutcome: { type: enum, values: [succeeded, failed] }
    verifyOutcome:     { type: enum, values: [recovered, not_recovered] }
    checks:            { type: list, of: string }   # the tool appends one entry per verify pass; the list is the loop counter
    notifyRef:         { type: string }

  agents:
    ops:
      canTransition: [assess, investigate, contain]
      guidance: full

  states:
    assess:
      description: "Assess the alert. Contribute the alertSeverity."
      requires:
        - { name: alertSeverity, from: agent }
      transitions:
        - { toState: investigate }

    investigate:
      description: "Investigate the cause. Read the logs and contribute the rootCause."
      requires:
        - { name: rootCause, from: agent }
      transitions:
        - { toState: contain }

    contain:
      description: >-
        Choose and apply the containment. The model picks the containment
        (rollback, scale out, restart); the tool applies it. The model
        chooses the value; the machine owns the effect.
      requires:
        - { name: containment, from: agent }
        - { name: containmentOutcome, from: { tool: opsTool } }
      transitions:
        - toState: verify
          guards: [{ backend: cel, expression: "context.containmentOutcome == 'succeeded'" }]
        - { toState: escalate, guards: [{ backend: cel, expression: "context.containmentOutcome == 'failed'" }] }

    verify:
      description: >-
        Verify the metric actually recovered. This is the step the model used to skip.
        It is now a required state: done is only reachable after it passes.
        Each pass appends to checks so the loop is bounded and observable.
      requires:
        - { name: verifyOutcome, from: { tool: opsTool } }
      transitions:
        - toState: notify
          guards: [{ backend: cel, expression: "context.verifyOutcome == 'recovered'" }]
        - toState: verify   # re-enter: metric not recovered yet, re-check
          guards: [{ backend: cel, expression: "context.verifyOutcome == 'not_recovered' && size(context.checks) < 3" }]
          onTransition:
            - { assign: { verifyOutcome: "" } }   # consume the trigger so the next pass re-collects a fresh outcome
        - toState: escalate
          guards: [{ backend: cel, expression: "context.verifyOutcome == 'not_recovered' && size(context.checks) >= 3" }]

    notify:
      description: >-
        Notify the on-call that the metric recovered. Journaled with a key
        so a replay does not notify twice.
      requires:
        - { name: notifyRef, from: { tool: opsTool } }
      transitions:
        - { toState: done }

    done:     { description: "Incident contained and verified recovered. Reference: {{ context.notifyRef }}.", terminal: true }
    escalate: { description: "Containment or verification failed. A human takes over with the full history.", terminal: true }

  policy:
    maxExchangesPerStage: 20
    maxExchangesPerAgentPerStage: 10
    checkpointEvery: 2

A few things to notice:

• The model contributes a value, not a next step. In assess, investigate, and contain, the model does real work: it judges the alert, reads the logs and forms a hypothesis, and picks the containment. That is the open-ended reasoning the shell is built to contain. But it does not choose to investigate, or to contain, or to verify, or to stop. Those are edges the machine fires. The “Incident resolved” the model wrote at 8:50 is not a thing it can write, because done is a terminal state the machine reaches, not a verdict the model contributes.
• The skipped step is now a required state. verify is a state with a requirement, and done is only reachable from notify, which is only reachable from verify. The model cannot skip it, because there is no edge from contain to done. The only paths out of contain are to verify or to escalate. That is the whole fix for the 8:50 failure: the step the model used to skip is now structurally required.
• The re-check loop is bounded and consumes its own trigger. checks is a list in context: the tool appends one entry per verify pass, so the counter is the data. The budget is size(context.checks) < 3. The re-entry edge back to verify clears verifyOutcome in an onTransition assign, so the next pass re-collects a fresh outcome instead of routing on a stale one. When the budget is spent, the machine does not guess. It hands off to a human with the full history.
• The effect is journaled before it fires. notify writes notifyRef with a key before it pings the on-call. If the process dies after the tool returns but before the terminal state, the replay sees the journaled reference and does not notify twice. That is the idempotency-key pattern from Part 1, and it is the same trick that keeps the on-call from getting paged twice for the same incident.

---

Where the analogy breaks down

I want to be honest about the limits of this design, because a good principle taken too far is worse than no principle at all. The deterministic shell is the right shape, but it is not a free lunch, and some of the assumptions do not hold.

The shell is only as good as the guards you wrote. A guard you did not think of is a hole. The shell cannot stop a transition you did not declare illegal, and it cannot catch a value you did not declare a guard for. If you forgot to guard the containment outcome, the shell will apply the wrong containment and never blink. The determinism is real, but it is the determinism of the guards you authored, not of the model. You are writing the safety net, and you are the one who gets to find the holes.

The model can fail semantically inside a perfectly deterministic shell. Right structure, wrong content. The shell guarantees the model cannot skip verify, cannot call a tool it was not given, and cannot declare the workflow done. It does not guarantee the model verifies the right thing, or reads the logs correctly, or writes a value that is true. A deterministic shell around a wrong answer is still a wrong answer, just a well-organized one. The shell catches structural failures (the wrong step, the wrong tool, the wrong order). It does not catch semantic ones (the wrong value). Those need guards you write deliberately, and they need a human in the loop when the stakes are high.

Make the shell too rigid and you have built a brittle state machine, not an agent. The whole point of putting a model in the loop is that it handles the open-ended cases a fixed workflow cannot: the ticket that is not quite the shape you expected, the customer who asks for something you did not declare a state for. If the shell is so rigid that every surprise is an escalation, you have not built an agent. You have built a form with a language model filling in the blanks, and you have paid the model’s cost to do what a form could have done for free. The boundary has to leave room for the model to be useful, or the model is not doing anything.

The boundary between “the model’s job” and “the machine’s job” is a design decision, not a free lunch. Every workflow has a line: here is what the model contributes, and here is what the machine decides. Draw that line too far toward the machine and the model is a token-expensive lookup. Draw it too far toward the model and you are back to the 8:50 failure. The line is not given to you by the framework. You draw it, per workflow, by deciding which states are agent-driven and which are machine-driven, and which values the model is allowed to contribute. That decision is the actual design work, and it is the part that is different for every workflow.

The shell does not make the model smart. It makes the model safe. This is worth saying plainly, because it is where the expectation usually goes wrong. The deterministic shell does not improve the model’s answers. It does not make the verification smarter, or the containment more careful, or the notification more accurate. It makes the model’s failures containable: a wrong value is caught by a guard or a human, a wrong step is impossible, a wrong tool is unavailable. The model is still probabilistic. The shell just means its probability is bounded by a structure you can read, test, and audit.

The shell itself is a deployable artifact, and it needs a release process, not just a design. The FSM, the policy, and the guards are not a diagram you draw once. They are a YAML file (or several) that someone edits, reviews, and ships, the same way anyone ships a change to a service. A change to a guard is a change to production behavior, and it deserves the same discipline: a diff someone reviews, an eval suite it has to pass before it merges, a staged rollout instead of a flip for every workflow at once, and a rollback path when the new guard turns out to be wrong. Skip that process and the shell is still deterministic. It is just deterministically wrong, for everyone, starting the moment you merge.

None of these break the thesis. They sharpen it. The point is that the deterministic shell is the right shape for a production agent, and the shape is what fails. The model is a new kind of contributor, and it needs a few new patterns on top of the old ones. The old patterns still apply. You just have to add a few new ones, and that is what the rest of this series is about.

---

What changes at 10, 100, 10,000

The reason I frame this as an architecture problem is that the failure modes change as the number of concurrent workflows grows. Here is what I have actually seen.

At 10 concurrent workflows, you can babysit the control flow. You can watch the model decide what to do next. You can catch a skipped verify in a code review. You can fix a run that declared itself done early by restarting it and asking it where it is. The system is reliable because you are the reliability layer. That is fine for a prototype and a demo. It is not an architecture.

At 100 concurrent workflows, you stop being the reliability layer. You cannot watch 100 models decide what to do next, let alone catch a subtle skipped step. The shell stops being nice-to-have and starts being load-bearing. The declared edges, the guards, the tool boundaries, and the escalation queue are the only thing standing between you and a hundred early “task complete” verdicts. Cost starts to matter: a model that re-plans its own control flow on every turn is a model that burns tokens deciding things the machine should have decided for free. The question “why did this run skip verify” stops being a question you answer by reading the logs and becomes a query you run against the declared edges. The answer is in the YAML, not the model.

At 10,000 concurrent workflows, the shell is the only thing between you and a population of wrong control-flow decisions. A single missing guard fans out across thousands of runs, so a hole is not an incident, it is a population. You cannot fix it by restarting. You need the shell to be reviewable data (a YAML file in git, with scannable diffs), the guards to be testable, and the tool boundaries to be auditable. You need replay to reconstruct a run and answer “why did this happen” across a thousand of them, and a control plane to roll back a bad version of the workflow. The vocabulary from the mapping table stops being a metaphor and becomes your incident runbook.

Here is the mechanic behind that, in numbers, because “load-bearing” should mean something you can measure. A guard suite of 30 assertions takes a few seconds to run at 10 workflows: nobody notices, and nobody bothers to run it before a change ships. At 100 workflows, that same suite is what stands between a Tuesday-afternoon guard edit and a hundred runs routing on the wrong edge, so it runs in CI, on every change, and it blocks the merge if it fails. At 10,000 workflows, the guard suite is not enough on its own: you also want a canary, a fixed slice of traffic (1 percent, or a single low-stakes workflow type) that gets the new shell first, for an hour, before the rest of the population does. The cost of skipping the canary does not change with scale. What changes is how many runs are wrong before you notice.

The through-line is that the architecture you need at 10,000 is the same architecture you need at 10. The difference is that at 10, you can get away without it. That is the trap: the system works in the demo, the reliability is invisible, and you do not find out what you are missing until the scale makes it visible. Which is to say, until 8:50 a.m. on a Monday.

---

Operating this in production

Everything above is the design. Here is what it takes to actually run it.

The shell ships like code, because it is code, even when it is YAML. A change to a guard, a policy budget, or a model route goes through the same steps a change to a service does: a pull request, a reviewer who is not the author, and an eval suite that has to pass. The eval suite is not optional decoration. It is the thing that catches a guard that is too loose (a value that should have been rejected gets through) or too tight (a legitimate value now gets rejected and every run in that state escalates). Both failures are silent until someone is paging you about a hundred workflows stuck at the same state.

Model routing lives in the policy layer, and it needs a fallback, not just a primary. When the primary model is rate-limited, slow, or returning garbage, the policy should already know which backup model takes over, and the workflow should not need a human to notice and flip a switch. This is the same instinct as a load balancer’s health check. Nobody designs a load balancer with one backend and no failover plan, and a policy layer with one model and no fallback is the same gap wearing a different hat.

Guard-rejection rate is the metric to put on a dashboard, next to the business metrics. Under normal operation it is low and boring. A spike almost always means one of two things: a model or prompt change shipped that the eval suite did not catch, or a real shift in the kind of input arriving (a new customer segment, a new ticket type) that the shell was not designed for. Either way, it is a signal worth paging on, because by the time a human notices “a lot of workflows are stuck,” the guard-rejection dashboard already told you an hour ago.

---

The whole thing, one idea

Step back and look at what actually happened in that 8:50 incident, and what the fix was:

Failure	Systems name	Pattern that fixes it
The model decides the workflow is over	Control flow in the payload	The machine owns the terminal state; done is a declared state, not a model decision
The model skips a required step	Undeclared behavior	The step is a required state; the edge to done is guarded by it
The model re-orders the steps	Emergent control flow	Transitions are declared edges; the model contributes values, not next steps
The model calls a tool it should not	No tool boundary	Tool boundaries: a fixed, typed interface scoped by policy
The model holds a private copy of state	State in the model’s head	The state store is the single source of truth; the model is re-hydrated every turn
A guard you did not write	Missing guard	The shell is only as good as its guards; author them deliberately

No new computer science. No new kind of reliability. The same separation that keeps a worker thread from picking its own tasks, and a program from rewriting its own instruction pointer, keeps an incident-triage agent from deciding its own control flow: a deterministic shell of orchestration, policy, state, guardrails, and tool boundaries around a probabilistic core. The model contributes. The machine decides.

I built the incident-triage workflow above because I had to. The early “Incident resolved” was real, and the fix was not a better prompt. It was a state machine, a set of guards, and a terminal state the model was not allowed to reach on its own. If you are putting agents in front of anything that touches a customer, changes a record, or moves money, start with the mapping table. Find the row that is missing from your architecture, and build that one first.

That is the move for this article. The next one is where the deterministic boundary gets its teeth: why a task list is not a state machine, and why reliable agents need explicit state, guards, and transitions instead of a list of intentions.

Up next: Part 4 (Stop Treating Agent Workflows Like Task Lists)