Back to: Intro

It is 8:41 a.m., and your incident-triage agent is doing its job. A pager fired: the checkout service’s p99 latency crossed its alert threshold, six minutes after a deploy shipped. The agent assesses the alert, investigates the cause, and the logs point at the 8:35 deploy. It forms a hypothesis, rolls the deploy back, and the rollback tool returns success. The metric recovers on the next check. The agent notifies the on-call that the incident is contained.

The notification goes out. The on-call’s phone buzzes.

Then the paging API hangs. The agent’s 30-second timeout fires, the agent concludes the call failed, and it retries, this time deriving a fresh notification instead of reusing the first one. The second call succeeds too. The on-call is now paged twice for an incident that is already handled, and the agent has moved on to the next alert, which is also a page it will need to send.

By the time anyone notices, it has done this forty times.

Forty double-pages. The most expensive version of “did it actually work?” I have ever seen, measured in on-call trust instead of dollars.

The agent was not broken. It was doing exactly what you taught it to do: when a call fails, retry. The problem is that “the call failed” was never true. The call had succeeded. The network just did not say so.

And here is the part that stung when I wrote it up for the team: the first fix everyone reaches for is “make the timeout longer.” Longer timeouts, a better model, a prompt that says “please be more careful.” None of those touch the actual failure.

---

The failure already has a name

This exact failure has a name, and it is decades old. In distributed systems it is called duplicate delivery: a request is delivered more than once because the sender could not tell whether the receiver got it. The classic answer is not “deliver exactly once.” That is famously hard to guarantee over an unreliable network. The classic answer is:

Deliver at least once, and make the operation idempotent, so that a duplicate is harmless.

An idempotent operation is one where doing it twice is the same as doing it once. INSERT is not idempotent. UPSERT is. SET x = 5 is. “Page the on-call about incident 4021” is not, until you give it a key that the paging system can use to say “I already sent that page. Here is the original reference.”

The vocabulary for this failure is complete and battle-tested:

• At-least-once delivery is what retries give you by default.
• Idempotency is what makes duplicates safe.
• Effect journaling is how you know what actually happened, when the network will not tell you.
• Compensating actions (the saga pattern) are how you undo a multi-step effect when it only half-happens.
• Backpressure is how you stop the system from generating more work than it can safely handle.

Every one of those terms was invented for a system made of long-lived processes talking over unreliable networks. And that is the thesis of this article: an agent system is one of those systems now.

Notice what the diagram does not contain: a bug. Every step the agent took was rational. The failure is structural. It is the same structure that takes down order systems, payment pipelines, and job schedulers, and it shows up in agent systems for the same reason: retries are cheap, and effects are not.

---

The mapping

If agents are distributed systems, then the mapping should be concrete, not metaphorical. Here is the table I keep coming back to. The left side is a concept you already know from running distributed software. The right side is what it looks like when the “nodes” are agents:

Distributed systems concept	Agent system analogue
Node (long-lived process)	A workflow instance running on an agent runtime
Unreliable network	Model, tool, and agent-to-agent calls that time out or drop
Message loss	A tool call succeeds, but the response never reaches the agent
Duplicate delivery	A retry re-issues an effect that already happened
At-least-once semantics	What agent retries give you by default
Idempotent operation	An effect with a stable key: a page, an email, an order, a ticket
Durable state	Workflow state in a journal, not in a model’s context window
Consistency	”What does the agent believe is true” versus what the journal says
Two-phase commit / saga	A multi-step effect (assess, contain, notify) with compensating actions
Checkpointing	Persisting enough state to resume a run after a restart
Backpressure	Throttling agent concurrency to what models, APIs, and humans can absorb
Coordination / leader election	Which agent is driving the workflow right now, and who may move it
Dead letter queue	Escalation to a human when an agent is stuck or uncertain
Clock and time	Workflows that span hours or days while a human reviews
Observability	Decision provenance: not just what the LLM called, but why the system did what it did
Capacity planning	A concurrency budget: how many workflows may run at once, sized against model throughput, API quotas, and human review capacity
Autoscaling	Worker pool sizing under queue depth: more workers where the backlog grows, not where it is empty
Cost accounting	Token spend as a per-effect line item: what a page, a ticket, or an incident actually costs, not just whether it succeeded
SLO / paging	Who is on call when the escalation queue backs up, and what metric moves before the queue does

I would argue that every row in that table is a real incident waiting to happen if you have not thought about it. And every row has a known answer. That is the good news about this problem: you are not inventing reliability for a new kind of computer. You are applying forty years of reliability engineering to a new kind of node.

The bad news is that the node is probabilistic, stateless, expensive, and occasionally wrong in ways a process never is. We get to that in a minute.

This is the part that makes the rest of the article worth reading.

---

What the architecture looks like

Here is the shape of an incident-triage agent that survives its own retries. The important move is where state and effects live: not in the model, not in the prompt, but in durable structures the model reads from and writes to.

Three properties fall out of this shape, and they are the same properties that make a distributed system reliable:

• State is external and durable. The current state and the context live in a store that survives a restart. The model is re-hydrated from it on every turn. If the process dies mid-notification, the next process reads the journal and knows exactly where it is.
• Effects are journaled and keyed. Every real-world effect (a page, an email, a rollback) is recorded with an idempotency key before it is attempted. A retry reuses the key. The downstream system deduplicates. The duplicate is harmless.
• The machine drives, the model contributes. The agent proposes values and effects. A deterministic state machine decides what is legal, what is pending, and what is done. That separation is the subject of Part 3, but even here you can see it doing work: the retry loop is a declared edge, not a habit.

Run the same failure through this architecture and the second call stops being a problem. The agent records its intent in the effect journal, with the idempotency key, before it calls the paging API. The first call times out. The agent marks the attempt unknown, backs off, and retries with the same key. The paging API sees the key, recognizes that it already sent that page, and returns the original reference instead of sending a second one. The agent records the reference and moves on. One page, two attempts. That is the entire trick, and it is the same trick that payment systems have used for years.

---

An incident-triage workflow that survives its own retries

Let me show the whole thing as an Aegis FSM, because this is the format I use for agent workflows. It is a YAML file, and it is small on purpose. The interesting parts are the derive-key state, the re-entrant notify state, and the fact that the idempotency key is derived once and never changes.

fsm:
  name: incident-triage
  version: 1
  initialState: assess

  context:
    incidentId:        { type: string }
    alertSeverity:     { type: enum, values: [critical, major, minor] }
    rootCause:         { type: string }
    containment:       { type: enum, values: [rollback, scale_out, restart, none] }
    containmentOutcome: { type: enum, values: [succeeded, failed] }
    verifyOutcome:     { type: enum, values: [recovered, not_recovered] }
    checks:            { type: list, of: string }   # the tool appends one entry per verify pass; the list is the loop counter
    notifyKey:         { type: string }
    attempts:          { type: list, of: string }   # the tool records one entry per notify attempt; the list is the retry counter
    notifyOutcome:     { type: enum, values: [succeeded, timed_out] }
    notifyRef:         { type: string }

  agents:
    ops:
      canTransition: [assess, investigate, contain]
      guidance: full

  states:
    assess:
      description: "Assess the alert. Contribute the incident id and the alertSeverity."
      requires:
        - { name: incidentId, from: agent }
        - { name: alertSeverity, from: agent }
      transitions:
        - { toState: investigate }

    investigate:
      description: "Investigate the cause. Read the logs and contribute the rootCause."
      requires:
        - { name: rootCause, from: agent }
      transitions:
        - { toState: contain }

    contain:
      description: >-
        Choose and apply the containment. The model picks the containment
        (rollback, scale out, restart); the tool applies it.
      requires:
        - { name: containment, from: agent }
        - { name: containmentOutcome, from: { tool: opsTool } }
      transitions:
        - toState: verify
          guards: [{ backend: cel, expression: "context.containmentOutcome == 'succeeded'" }]
        - { toState: escalate, guards: [{ backend: cel, expression: "context.containmentOutcome == 'failed'" }] }

    verify:
      description: "Verify the metric actually recovered. Each pass appends to checks so the loop is bounded and observable."
      requires:
        - { name: verifyOutcome, from: { tool: opsTool } }
      transitions:
        - toState: derive-key
          guards: [{ backend: cel, expression: "context.verifyOutcome == 'recovered'" }]
        - toState: verify   # re-enter: metric not recovered yet, re-check
          guards: [{ backend: cel, expression: "context.verifyOutcome == 'not_recovered' && size(context.checks) < 3" }]
          onTransition:
            - { assign: { verifyOutcome: "" } }   # consume the trigger so the next pass re-collects a fresh outcome
        - toState: escalate
          guards: [{ backend: cel, expression: "context.verifyOutcome == 'not_recovered' && size(context.checks) >= 3" }]

    derive-key:
      # No agent, no LLM turn. The key is derived once, from the incident id, so a second run for
      # the same incident derives the same key and the paging system deduplicates across runs.
      requires:
        - name: notifyKey
          from: system
          extract: "'notify-' + context.incidentId"
      transitions:
        - { toState: notify }

    notify:
      description: >-
        Notify the on-call with the idempotency key. A timeout is UNKNOWN, not FAILED:
        back off and try again with the SAME key. A duplicate is harmless.
      requires:
        - { name: notifyOutcome, from: tool }                # always written: succeeded / timed_out
        - { name: notifyRef, from: tool, required: false }   # only on success
      transitions:
        - toState: done
          guards: [{ backend: cel, expression: "context.notifyOutcome == 'succeeded'" }]
        - toState: notify   # re-enter: same key, next attempt
          guards: [{ backend: cel, expression: "context.notifyOutcome == 'timed_out' && size(context.attempts) < 3" }]
          onTransition:
            - { assign: { notifyOutcome: "" } }   # consume the trigger so the next attempt re-collects
        - toState: escalate
          guards: [{ backend: cel, expression: "context.notifyOutcome == 'timed_out' && size(context.attempts) >= 3" }]

    done:     { description: "Incident contained, verified recovered, and the on-call notified. Reference: {{ context.notifyRef }}.", terminal: true }
    escalate: { description: "Could not confirm containment, recovery, or notification. Attempt ledger: {{ context.attempts }}. A human takes over with the full history and the key.", terminal: true }

  policy:
    maxExchangesPerStage: 20
    maxExchangesPerAgentPerStage: 10
    checkpointEvery: 2

A few things to notice:

• The key is data, not a decision. derive-key is a state with no agent in it. The idempotency key is computed by a CEL expression from the incident id, once, and stored in context. Scoping it to the incident rather than the run means a second run that picks up the same incident derives the same key, and the paging system deduplicates across runs, not just within one. The model never gets a chance to invent a new key on a retry. That is the whole idempotency guard.
• A timeout is “unknown,” not “failed.” The notify state says this out loud, because it is the most common authoring mistake I see. The moment you treat a timeout as a failure, you are one retry away from a double page.
• The retry loop is bounded and observable. attempts is a list in context: the tool appends one entry per attempt, so the counter is the data, and it cannot disagree with the ledger it is counting. The budget is size(context.attempts) < 3. The re-entry edge also consumes its own trigger: it clears notifyOutcome on the way in, so the next attempt re-collects a fresh outcome instead of routing on a stale one. When the budget is spent, the machine does not guess. It hands off to a human with the full history and the key, so the human can check the paging system and confirm or complete the notification by hand.
• The effect journal is the source of truth. If the process dies between notify and done, the next process reads the journal, sees the key and the pending attempt, and re-issues the same call. The paging API deduplicates. Nothing is lost, and nothing is doubled.

If you want the retry logic as plain code, here is the shape of it:

def send_notification(run, incident):
    key = run.context.get("notifyKey") or derive_key(incident.id)
    run.context.set("notifyKey", key)              # derived once, reused
    for attempt in range(run.policy.max_attempts):
        try:
            ref = paging.notify(incident.id, "contained", idempotency_key=key)
            run.context.set("notifyRef", ref)
            return "done"
        except TimeoutError:
            sleep(backoff(attempt))                 # same key on the next attempt
    return "escalate"

The except TimeoutError branch is the entire lesson. It does not assume the notification failed. It backs off and tries again with the same key, and the paging system’s answer to the second attempt is either a new reference (the first one really did fail) or the original reference (the first one succeeded). Either way, the on-call is paged exactly once.

---

Where the analogy breaks down

I want to be honest about the limits of this mapping, because a good analogy taken too far is worse than no analogy at all. Agents are distributed systems, but they are a new kind of node, and some of the old assumptions do not hold.

The node is probabilistic, not deterministic. A retry of a pure function returns the same answer. A retry of an LLM call can return a different one. The model can “succeed” at the wrong thing: misread the alert, pick the wrong containment, decide the metric recovered when it did not. In a distributed system, a node that misunderstands the spec is not a thing that happens. In an agent system, it is the main event. That is why the state machine and the guards matter so much: they are the deterministic shell around a probabilistic core, and they are what keep a wrong answer from becoming a wrong effect.

Retries are expensive, and the cost is not just tokens. Retrying a network call costs a few milliseconds and a few bytes. Retrying an agent turn costs tokens, latency, and possibly a human’s attention. A retry storm in a distributed system is an infrastructure problem. A retry storm in an agent system is a line item on your invoice. Backpressure and admission control are not optional here. They are cost control. And the cost has three components that a runbook has to account for separately: the tokens burned by the retry itself, the latency the on-call inherits while the retry runs, and the human attention a spent retry budget eventually consumes at the escalation queue. A retry ceiling (size(context.attempts) < 3) bounds the first two, but nothing in the machine bounds the third unless the escalation queue itself has a capacity plan: a depth limit, a paging threshold, and an owner. A runbook for an agent system needs a per-workflow cost ceiling, not just a retry ceiling. The ceiling is not a nice-to-have metric. It is the difference between a retry storm that costs an afternoon and one that costs a quarter.

The node is stateless by default. A long-lived process keeps its state in memory and on its disk. A model call keeps nothing. Every turn, the model starts from whatever you put in the prompt. Which means all state has to be externalized, re-hydrated, and made consistent on every single turn. The durable state store is not an optimization. It is the only place the state exists at all.

The humans are in the loop, and they are the slowest node. A human review can take hours or days. A distributed system that parks on an external dependency for a week is unusual. An agent workflow that parks on a human for a week is the normal case. The machine has to be able to wait, to persist that wait, and to resume it cleanly. That is a durability requirement, not a convenience.

The “network” has no consistency guarantees. The link between two agents is often a single API call. There is no replication, no quorum, no shared clock. If agent A tells agent B something and the call is dropped, agent B simply never hears it. The coordination has to be rebuilt from durable state, not from the assumption that the message got through.

Failure is semantic, not just mechanical. A process crashes, and you know it crashed. An agent can complete a turn, write a confident-sounding note, and be wrong about the world. “The incident was contained” in the agent’s summary is a claim, not a fact. The effect journal, not the model’s prose, is the thing you check.

None of these break the thesis. They sharpen it. The point is that agents are distributed systems with a new failure class (semantic, probabilistic, expensive) layered on top of the old ones. The old patterns still apply. You just have to add a few new ones, and that is what the rest of this series is about.

---

What changes at 10, 100, 10,000

The reason I frame this as a systems problem is that the failure modes change as the number of concurrent workflows grows. Here is what I have actually seen.

At 10 concurrent workflows, you can babysit. You can read every run. You can catch a duplicate page in a code review. You can fix a stuck agent by restarting it and asking it where it is. The system is reliable because you are the reliability layer. That is fine for a prototype and a demo. It is not an architecture.

At 100 concurrent workflows, you stop being the reliability layer. You cannot read 100 runs a day, let alone catch a subtle duplicate. The journal, the idempotency keys, and the escalation queue stop being nice-to-have and start being load-bearing. Cost starts to matter: a retry storm across 100 workflows is a real number on your invoice, and backpressure becomes something you have to design. The question “what does the agent believe is true” stops being philosophical and becomes an on-call question.

At 10,000 concurrent workflows, the system is a distributed system in every sense that matters. You need admission control, rate limiting, and chaos testing. A single bad state transition fans out across thousands of runs, so a bug is not an incident, it is a population. You cannot fix it by restarting. You need replay to reconstruct a run, observability to answer “why did this happen” across a thousand of them, and a control plane to roll back a bad version of the workflow. The vocabulary from the mapping table stops being a metaphor and becomes your incident runbook.

Here is the mechanic, concretely, because “cost starts to matter” is not a claim you should take on faith. Say an incident-triage workflow averages 40,000 tokens end to end (the assessment turn, the investigation, the containment choice, the retry attempts, the escalation handoff) and your effective blended rate is $10 per million tokens. That is $0.40 per workflow. At 10 concurrent workflows, the daily model spend is a number you can eyeball: a few dollars, and a retry storm is an inconvenience. At 100 concurrent workflows, the daily spend is a number you watch: $40 per 100 workflows on a good day, and a retry storm that doubles the attempt count doubles the line item on the same day. At 10,000 concurrent workflows, the daily spend is a number you budget against: $4,000 on a good day, and the question is no longer “did the retry storm happen” but “which state is burning the tokens, and is the retry budget on that state the right one.” The formula is concurrent workflows x average tokens per workflow x cost per token, and the answer it gives you changes character at each order of magnitude: a rounding error, a dashboard tile, a budget line. The architecture does not change. The way you watch the number does.

The through-line is that the architecture you need at 10,000 is the same architecture you need at 10. The difference is that at 10, you can get away without it. That is the trap: the system works in the demo, the reliability is invisible, and you do not find out what you are missing until the scale makes it visible. Which is to say, until 8:41 a.m. on a Monday.

---

Operating this in production

None of the patterns above run themselves. Here is what actually running this looks like day to day.

Version the FSM the way you version a service. The YAML above carries a version: 1 field, and that is not decoration. It is the unit you deploy, review, and roll back. A change to the notify state’s retry budget is a production change: it needs a diff someone reads, a version number, and a way to say “runs that started under version 1 keep running under version 1’s rules while new runs start under version 2.” In-flight workflows should not be silently reinterpreted by a new definition mid-run. That is the same discipline as any long-lived process, and the FSM is the first place in an agent system where it has a natural home.

Roll out a change without breaking in-flight runs. The savepoint discipline from stream processing applies here: take a checkpoint of what is in flight, ship the new version, and let the in-flight runs finish under the old one while new runs start under the new one. The alternative is a fleet of incident-triage workflows that started with a 3-attempt notify budget and are now being re-evaluated against a 2-attempt budget, with no one able to say which version of the rules produced which behavior. That is not a reliability failure. It is an audit failure, and it is harder to recover from.

Own the escalation queue. A dead letter queue without an owner is a pile. The escalation queue is where every workflow that spent its retry budget ends up, and it is the one place in the system where a human is the next node. It needs a named owner, a paging threshold (a queue depth that moves the on-call, not a failure count that moves the on-call), and a working definition of “stuck” that is a query against the store, not a feeling. The queue is the slowest node in the system, and it is the only node whose capacity is a person.

---

The whole thing, one idea

Step back and look at what actually happened in that 8:41 incident, and what the fix was:

Failure	Systems name	Pattern that fixes it
Retry after a timeout pages twice	Duplicate delivery	Idempotency key on every effect
A dropped response looks like a failure	No distinction between failed and unknown	Timeouts are UNKNOWN, not FAILED; retry, do not assume
The agent forgets what it already tried	State in the model’s head	Durable state: (state, context) journaled and re-hydrated
A multi-step incident half-happens	Partial effect	Saga with compensating actions
A retry storm burns tokens unbounded	No backpressure	Bounded retries plus a capacity plan
Nobody owns the stuck runs	No dead letter queue	An escalation queue with a named owner and a paging threshold

No new computer science. No new kind of reliability. The same patterns that keep a payment pipeline from double-charging a card keep an incident-triage agent from double-paging the on-call: durable state, idempotent effects, bounded retries, and a machine that drives while the model contributes.

I built the incident-triage workflow above because I had to. The double page was real, and the fix was not a better prompt. It was a state machine, a journal, and a key that the model was not allowed to change. If you are putting agents in front of anything that moves money, changes a record, or talks to a human, start with the mapping table. Find the row that is missing from your architecture, and build that one first.

That is the move for this article. The next one is where the patterns get their names from: what Kafka and Flink taught me about building reliable agent runtimes, and why a stream processor is the closest thing to an agent runtime that we already know how to make reliable.

Up next: Part 2 (What Kafka and Flink Taught Me)