Back to: Intro

It is 8:48 a.m. and your incident-triage agent is doing its job. A pager fired at 8:41: the checkout service’s p99 latency crossed its alert threshold. You gave the agent a six-step task list: assess the alert, investigate the cause, contain the blast radius, verify the metric actually recovered, notify the on-call, and mark it done. You wrote those six steps into the prompt.

The agent works through the list. It assesses the alert. It investigates the cause, and the logs point at the deploy that shipped at 8:35. It contains the blast radius: it rolls the deploy back, and the rollback tool returns success. It is a single loop: each turn, it looks at what it has done and decides what to do next. It is doing a fine job, and it has no idea where it is.

At 8:48 the runtime gets a deploy. The process restarts. The workflow’s “state” was a growing prompt in process memory and a current-step variable on the heap. No journal. No store. Nothing on disk.

On the next turn, the agent looks at the task list and starts again. It re-assesses the alert. It re-investigates. It re-contains the blast radius, rolling the deploy back again, on top of the rollback it had already done. And when it reaches the end, it declares the incident contained. It never verified the metric actually recovered. It never re-read the p99. The task list had no idea where the run was, and neither did the agent.

The agent finished the list. The on-call got a second rollback they did not ask for, and the metric was never checked. Neither of those is a success.

The agent was not broken. It was doing exactly what you built: it was left to decide what to do next from a list of intentions, and it decided. The problem is that a list of intentions is not a position. It has no guards, no concept of an illegal state, no re-entry, and no durable place to resume after a crash. The failure is that control flow lived in a list of intentions, not in a machine.

---

The failure already has a name

Nobody runs a production system on a to-do list. A worker thread does not decide which task to run next from a list of intentions; the scheduler does. A database does not apply a sequence of writes from a checklist; the transaction log and the isolation level do. A long-running order does not remember which step it is on from a to-do list; the saga does. In every system we have ever made reliable, the thing doing the work does not own the control flow, and the control flow is not a list. It is a machine with a declared set of states and the legal moves between them.

A task list is a plan. A plan is not a position. A plan says what you meant to do. It does not say where you are, what you are allowed to do next, what happens when a step fails, or where you resume after a crash. Those are questions a plan was never designed to answer. We have machines that answer them, and we have been using them for decades: schedulers, state machines, sagas, and workflow engines.

There is a distinction worth being precise about, because it is the whole article. A task list describes intentions: what you meant to do, in the order you meant to do it. A state machine describes legal system behavior: what the system may be in, and what it may do next from there. Those are different questions, and they need different artifacts.

A task list says “do these six things.” A state machine says “the system is in one of these states, and from each state these are the legal moves, guarded by these conditions.” The state machine has three things the task list does not: explicit state (a named position the system is in), guards (conditions that must hold for a move to be legal), and transitions (declared edges from one state to the next). A task list has none of the three. It is a sequence of intentions with no position, no legality, and no recovery.

That is the thesis of this article: a task list describes intentions, and a state machine describes legal system behavior. Reliable agents need explicit state, guards, and transitions, not a list of steps. Part 3 set up the deterministic shell. This article is where the shell gets its teeth: the explicit states, guards, and transitions that make a workflow legal instead of a sequence the model is free to wander through.

---

The mapping

So what happens to the task list? It does not disappear. It gets demoted from the control flow to the description of the states. Each step in the list becomes a state with an objective. The “next” in the list becomes a declared edge. The “done when” becomes a guard. And the crash-recovery position becomes durable state in a store that survives a restart. Here is the table I keep coming back to. The left side is the task list, the intentions. The right side is the state machine, the legal behavior.

Task list (intentions)	FSM (legal behavior)
A step in the list (“verify the metric”)	A state with an objective (verify: re-read the p99 and confirm recovery)
“Then do the next thing”	A declared transition (an edge from verify to notify)
“Done when the metric recovered”	A guard (verifyOutcome == recovered) on the edge out of verify
”If it fails, try again”	A bounded re-entry edge (verify to verify, guarded by size(checks) < 3)
“Where am I after a crash?”	Durable state: (state, context) journaled and re-hydrated on restart
”Is this step allowed now?”	Legality is structural: an edge not in the YAML cannot fire
”When is the workflow over?”	A terminal state the machine reaches, not a verdict the model writes

I would argue that every row in that table is a real incident waiting to happen if you have not thought about it. And every row has a known answer. You are not inventing a new kind of control. You are putting the control where it has always belonged: in the machine, not in the list.

The honest caveat is that the thing filling in the states is not a pure function. It is a model. That changes some of the semantics, and it is worth being precise about. But the shape of the separation is the same, and the shape is what fails.

There is a second, quieter payoff to naming the states, and it has nothing to do with correctness. A named state is a thing you can point a metric at. A task list cannot support this at all, because “item 4 of 6” is not a place, it is a position in a sequence that only means something if nothing has failed. A state is a place. It has a name, an entry count, and a duration. That difference is the whole basis for operating a fleet of these workflows instead of just running one:

What you want to know	Task list (intentions)	FSM (legal behavior)
How many runs are stuck right now, and where	No answer: a list has no position to query	Backlog per state: count(*) where state = 'verify'
Is this step usually fast or slow	No answer: steps are not named, durations are not tracked	Time-in-state: a distribution per state, with a p99 you can alert on
Where should more workers go	No answer: the whole agent is one undifferentiated loop	Worker pool sizing per state: cheap states (assess) need few workers, model-heavy states (verify) need more
Did the new version make things worse	No answer: nothing to diff against	Compare backlog and time-in-state, state by state, before and after a deploy

You cannot build a capacity plan on top of “the agent is somewhere in a six-step list.” You can build one on top of “14 runs are in verify, and the median time-in-state just doubled.” That is not a small difference. It is the difference between an architecture you can operate and one you can only watch.

---

What the architecture looks like

Here is the centerpiece, side by side. On the left, the task list: a straight line of intentions. On the right, the state machine: states with guarded edges, re-entry loops, and an escalation path. Read it as a pair. The left side is what you wrote in the prompt. The right side is what the machine actually runs.

The contrast is the whole point. The task list is a sequence. The state machine is a machine. In the sequence, “verify the metric” is item 4 of 6, and if the process dies after item 3, the sequence has no idea where it is. In the machine, “verify” is a state, and if the process dies after “contain,” the next process reads the store, sees the state is “contain” or “verify,” and resumes there. The sequence restarts from the top. The machine resumes from its position.

Here is the loop the machine runs. It is the reactive reducer: read the current state and context, check whether the state’s requirements are met, route on the first edge whose guards pass, fire the edge’s onTransition actions, and move. When it is waiting for a contribution, it parks. When it reaches a terminal state, it stops.

The model never appears in the “decide what is next” role. It contributes a value, and the machine routes on it. I covered the binding of the model to the state machine in depth in Binding LLMs to Finite State Machines Using the Reactive Reducer Pattern, and the pattern catalog is in The Reactive Reducer Patterns series. This article is where the deterministic shell from Part 3 gets its teeth: the explicit states, guards, and transitions that make a workflow legal.

---

An incident-triage workflow that survives its own task list

Let me show the whole thing as an Aegis FSM, because this is the format I use for agent workflows. It is a YAML file, and it is small on purpose. The interesting parts are the verify state that the task list used to skip, the re-entrant loop that bounds the re-checks, and the fact that done is only reachable after verify passes.

fsm:
  name: incident-triage
  version: 1
  initialState: assess

  context:
    alertSeverity:     { type: enum, values: [critical, major, minor] }
    rootCause:         { type: string }
    containment:       { type: enum, values: [rollback, scale_out, restart, none] }
    containmentOutcome: { type: enum, values: [succeeded, failed] }
    verifyOutcome:     { type: enum, values: [recovered, not_recovered] }
    checks:            { type: list, of: string }   # the tool appends one entry per verify pass; the list is the loop counter
    notifyRef:         { type: string }

  agents:
    ops:
      canTransition: [assess, investigate, contain]
      guidance: full

  states:
    assess:
      description: "Assess the alert. Contribute the alertSeverity."
      requires:
        - { name: alertSeverity, from: agent }
      transitions:
        - { toState: investigate }

    investigate:
      description: "Investigate the cause. Read the logs and contribute the rootCause."
      requires:
        - { name: rootCause, from: agent }
      transitions:
        - { toState: contain }

    contain:
      description: >-
        Choose and apply the containment. The model picks the containment
        (rollback, scale out, restart); the tool applies it. The model
        chooses the value; the machine owns the effect.
      requires:
        - { name: containment, from: agent }
        - { name: containmentOutcome, from: { tool: opsTool } }
      transitions:
        - toState: verify
          guards: [{ backend: cel, expression: "context.containmentOutcome == 'succeeded'" }]
        - { toState: escalate, guards: [{ backend: cel, expression: "context.containmentOutcome == 'failed'" }] }

    verify:
      description: >-
        Verify the metric actually recovered. This is the step the model used to skip.
        It is now a required state: done is only reachable after it passes.
        Each pass appends to checks so the loop is bounded and observable.
      requires:
        - { name: verifyOutcome, from: { tool: opsTool } }
      transitions:
        - toState: notify
          guards: [{ backend: cel, expression: "context.verifyOutcome == 'recovered'" }]
        - toState: verify   # re-enter: metric not recovered yet, re-check
          guards: [{ backend: cel, expression: "context.verifyOutcome == 'not_recovered' && size(context.checks) < 3" }]
          onTransition:
            - { assign: { verifyOutcome: "" } }   # consume the trigger so the next pass re-collects a fresh outcome
        - toState: escalate
          guards: [{ backend: cel, expression: "context.verifyOutcome == 'not_recovered' && size(context.checks) >= 3" }]

    notify:
      description: >-
        Notify the on-call that the metric recovered. Journaled with a key so a
        replay does not notify twice.
      requires:
        - { name: notifyRef, from: { tool: opsTool } }
      transitions:
        - { toState: done }

    done:     { description: "Incident contained and verified recovered. Reference: {{ context.notifyRef }}.", terminal: true }
    escalate: { description: "Containment or verification failed. A human takes over with the full history.", terminal: true }

  policy:
    maxExchangesPerStage: 20
    maxExchangesPerAgentPerStage: 10
    checkpointEvery: 2

A few things to notice:

• The skipped step is now a required state. verify is a state with a requirement, and done is only reachable from notify, which is only reachable from verify. The task list could skip verify and still “finish.” The machine cannot: there is no edge from contain to done that bypasses verify.
• The re-check loop is bounded and consumes its own trigger. checks is a list in context: the tool appends one entry per verify pass, so the counter is the data. The budget is size(context.checks) < 3. The re-entry edge back to verify clears verifyOutcome in an onTransition assign, so the next pass re-collects a fresh outcome instead of routing on a stale one. When the budget is spent, the machine hands off to a human with the full history.
• The double-contain is structurally impossible. contain is a state with a position, and the position is durable. If the process dies after contain, the next process reads the store, sees the state is verify, and resumes there. It does not re-contain. It does not roll the deploy back a second time. The 8:48 failure is structurally impossible, because the machine knows where it is.
• The position is durable. The state and context live in a store that survives a restart. If the process dies after contain, the next process reads the store, sees the state is contain or verify, and resumes there. It does not re-assess. It does not re-investigate. The task list had no position, and that is why it re-ran the whole list from the top.

If you want the loop as plain code, here is the shape of it. This is the reducer, and it is the entire “machine” in miniature:

def resume(workflow_id, store, fsm):
    # The position is data, not a decision. Read it; do not ask the model.
    state, context = store.load(workflow_id)     # (state, context) survives the crash
    for _ in range(MICROSTEP_CAP):
        current = fsm.states[state]
        if current.terminal:
            return state
        # 1. Eligibility: are all required context fields present?
        if not all(has(context, r.name) for r in current.requires):
            return park(workflow_id, store, state, context)   # wait for a contribution
        # 2. Routing: the first declared edge whose guards all pass fires.
        edge = next((e for e in current.transitions
                     if guards_pass(e, context)), None)
        if edge is None:
            return park(workflow_id, store, state, context)
        # 3. Fire: run the onTransition actions (literal writes only), then move.
        for action in edge.on_transition:
            context = assign(context, action)
        state = edge.toState
        store.save(workflow_id, state, context)   # every move is journaled
    return state

Notice what is not in that loop: no “what should the model do next,” no “is the model allowed to stop,” no “ask the model if it is done.” The loop parks when it is waiting for a contribution, and it fires the first eligible edge when it is not. The model is called to produce a value for the current state’s objective, and that is the sum of its role in the control flow. The task list is gone from the control flow. It is now the description of the states.

---

Where the analogy breaks down

I want to be honest about the limits of this design, because a good principle taken too far is worse than no principle at all. The state machine is the right shape, but it is not a free lunch, and some of the assumptions do not hold.

Not every workflow needs a full FSM. A linear script with no branches, no re-entry, and no crash-recovery is fine as a list. If your workflow is “do A, then B, then C” and it never fails in the middle, a list is the right artifact. The state machine earns its keep when there are branches, re-entry, and recovery. Do not model a three-step linear script as a state machine. You will have built a machine to do what a for-loop could do for free.

FSMs are authored, not learned. You must enumerate the states. You cannot learn the state machine from the task list, or from the model, or from the data. If the workflow is open-ended (the model decides which steps to take based on the input), it does not fit a finite state machine, because the states are not fixed. The state machine is for workflows where you can name the states in advance. For open-ended work, the model’s planning is the control flow, and you are back to the Part 3 problem of containing it inside a deterministic boundary.

Guards are only as good as you wrote them. A guard you did not think of is a hole. The state machine cannot stop a transition you did not declare illegal, and it cannot catch a value you did not declare a guard for. If you forgot to guard the metric recovery on the verify, the machine will declare the incident done with the service still degraded and never blink. The legality is real, but it is the legality of the guards you authored, not of the model. This is the modeling-side echo of Part 3’s point: the hole is in the machine you drew, not in the shell you built around it.

A task list is still the right artifact for humans and for the description. The task list does not go away. It is how you describe the workflow to a teammate, to a customer, to yourself in a month. The state machine is the control flow; the task list is the documentation. The mapping table above is the bridge: the list describes the states, and the machine runs them. Confusing the two is the original error. Treating the list as the control flow is the failure. Treating the list as the description of the states is the fix.

Over-modeling produces a brittle state machine. If you model every micro-decision as a state and every micro-condition as a guard, you have not built a machine that runs the workflow. You have built a machine so rigid that every surprise is an escalation, and you have paid the modeling cost to do what a simpler structure could have done. The boundary between “the model’s job” and “the machine’s job” has to leave room for the model to be useful, or the model is not doing anything.

The state machine is also the unit you ship, and it needs a version number. A task list lived in a prompt, so changing it was editing a string. A state machine lives in a YAML file, in git, and changing it is a production deploy: it needs a diff someone reviews, a version number, and a way to canary the new definition on a slice of runs before every in-flight incident picks it up. This is the same discipline Part 3 asked for around the deterministic shell, applied here from the authoring side: the states you named for capacity planning are the same states you diff, review, and roll back when a change to verify’s guard turns out to be wrong.

None of these break the thesis. They sharpen it. The point is that the state machine is the right shape for a production agent workflow, and the shape is what fails. The task list is the description, and the machine is the control flow. You just have to keep them separate, and that is what the rest of this series is about.

---

What changes at 10, 100, 10,000

The reason I frame this as an architecture problem is that the failure modes change as the number of concurrent workflows grows. Here is what I have actually seen.

At 10 concurrent workflows, you can babysit the task list. You can watch the agent work through the six steps. You can catch a skipped verify in a code review. You can fix a run that re-rolled-back the deploy by restarting it and asking it where it is. The system is reliable because you are the reliability layer, and you are the one who notices the list has no position. That is fine for a prototype and a demo. It is not an architecture.

At 100 concurrent workflows, you stop being the reliability layer. You cannot watch 100 agents work through 100 task lists, let alone catch a subtle skipped verify or a double-contain. The state machine stops being nice-to-have and starts being load-bearing. The declared states, the guards, the re-entry loops, and the escalation queue are the only thing standing between you and a hundred incidents that re-rolled-back their own deploy. Cost starts to matter: a model that re-plans its own position on every turn is a model that burns tokens deciding things the machine should have decided for free. The question “where is this workflow” stops being a question you answer by reading the logs and becomes a query you run against the store. The answer is the state, not the list.

At 10,000 concurrent workflows, the state machine is the only thing between you and a population of wrong positions. A single missing guard fans out across thousands of runs, so a hole is not an incident, it is a population. You cannot fix it by restarting. You need the state machine to be reviewable data (a YAML file in git, with scannable diffs), the guards to be testable, and the states to be auditable. You need replay to reconstruct a run and answer “why did this happen” across a thousand of them, and a control plane to roll back a bad version of the workflow. The vocabulary from the mapping table stops being a metaphor and becomes your incident runbook.

Here is the mechanic, concretely, because “the state machine helps at scale” is not a claim you should take on faith. Say verify takes a median of 8 seconds (a metric re-read plus a model turn to judge recovery) and every other state takes under a second (tool calls). At 10 concurrent workflows, that imbalance is invisible: everything finishes before anyone notices which state was slow. At 100 concurrent workflows, verify is the bottleneck: if your worker pool assigns workers uniformly across states instead of weighting toward the slow one, verify’s backlog grows while assess’s workers sit idle, and the fix is a one-line change once you can see it: count(*) where state = 'verify' against count(*) where state = 'assess'. At 10,000 concurrent workflows, that same imbalance, unaddressed, is an unbounded queue: verify’s backlog grows faster than its workers can drain it, and every workflow behind it in line inherits a wait time nobody set on purpose. The cost of the imbalance does not change with scale. What changes is whether the backlog empties before the next spike, or compounds into one.

The through-line is that the architecture you need at 10,000 is the same architecture you need at 10. The difference is that at 10, you can get away without it. That is the trap: the system works in the demo, the reliability is invisible, and you do not find out what you are missing until the scale makes it visible. Which is to say, until 8:48 a.m. on a Monday.

---

Operating this in production

None of the states, guards, or metrics above run themselves. Here is what actually running this looks like day to day.

Alert on backlog per state, not on failure count. A handful of failed runs is Tuesday. A backlog that is growing in verify and not draining is the leading indicator that something changed upstream: a slower model, a flakier metric re-read, a deploy that keeps reverting. By the time failure count moves, the backlog has usually been growing for an hour. Page on the backlog trend, not the failure count.

Dashboard the fleet by state, not by workflow. A dashboard with one row per in-flight incident does not scale past a handful of runs, and it does not tell you anything a single workflow’s logs would not. A dashboard with one row per state (assess: 3 running, 0 stuck, verify: 14 running, 2 stuck for > 10 min) tells you where the fleet’s attention is, and it is the same shape whether you have 10 workflows or 10,000.

Treat a version bump to the FSM like the deploy it is. When you change a guard on verify, that change should ship the way a service change ships: reviewed, versioned, canaried on a slice of new runs, with the old version still resumable for the runs that are already in flight. A workflow that started under version 3 of the FSM should not be silently reinterpreted under version 4’s guards mid-run; that is exactly the kind of implicit state change this whole article argued against.

---

The whole thing, one idea

Step back and look at what actually happened in that 8:48 incident, and what the fix was:

Failure	Systems name	Pattern that fixes it
The agent re-runs an early step after a crash	No durable position	Durable state: (state, context) journaled and re-hydrated on restart
The agent skips a required step	Undeclared behavior	The step is a required state; the edge to done is guarded by it
The agent does a step twice (double-contain)	Duplicate effect	The machine owns the transition; the effect is journaled with a key
The agent re-plans its own position every turn	Control flow in the payload	The reducer routes on declared edges; the model contributes values, not next steps
The agent declares done with steps left	No terminal state	done is a terminal state the machine reaches, not a verdict the model writes
A re-check loop runs forever	Unbounded loop	A bounded re-entry edge guarded by a list-as-counter

No new computer science. No new kind of control. The same patterns that keep a scheduler from re-running a job and a saga from re-applying a write keep an incident-triage agent from re-rolling-back its own deploy: explicit state, guards, transitions, and a machine that drives while the model contributes.

I built the incident-triage workflow above because I had to. The double-rollback was real, and the fix was not a better prompt. It was a state machine, a set of guards, and a terminal state the model was not allowed to reach on its own. The task list is still on the wall. It is how I describe the workflow to the team. But it is no longer the thing that runs. If you are putting agents in front of anything that moves data, changes a record, or touches a customer, start with the mapping table. Find the row that is missing from your architecture, and build that one first.

That is the move for this article. The next one is where the effects get their teeth: you cannot make an LLM run exactly once, but you can make its effects exactly once, and that is the difference between a double-rollback and a single rollback.

Up next: Part 5 (Exactly-Once Effects)