Back to: Intro

An incident-triage agent picks up a page at 8:41 a.m. The checkout service’s p99 latency has crossed its alert threshold, six minutes after a deploy shipped. The agent assesses the alert, investigates the cause, and the logs point at the 8:35 deploy. It forms a hypothesis, rolls the deploy back, and the rollback tool returns success. It is on its way into verify, the step that confirms the metric actually recovered.

At 8:48 a.m. the runtime gets a deploy of its own, a routine restart. The process restarts. Every in-flight workflow dies with it, because the workflow’s “state” was a growing prompt in process memory and a current-state variable on the heap. No journal. No store. Nothing on disk.

The on-call notices, eventually, that 31 incidents are gone: the runtime’s logs show the workflows started, and nothing after. When the runtime comes back, it re-runs each one from the top, because nothing was recorded about what it had already done. It re-assesses the alert, re-investigates the cause, and re-contains, which means it rolls back a deploy that had already been rolled back. Then it re-notifies, and for every incident that had already reached notify before the restart, the on-call gets paged a second time for something that was already handled.

The fix everyone reaches for is “make the agent more careful.” A longer context window, a prompt that says “remember where you are,” a fancier model. None of those touch the actual failure. The failure is that the workflow had no state that could survive the process.

I have run streaming systems for a long time, and I have a different fix. It is not new. It is the fix we built into Flink years ago for a problem that is structurally identical. I get to be boring on purpose, and it works.

---

The failure already has a name

In stream processing, this failure has a name: state that lives only in the operator’s memory. If the task manager dies, the operator’s memory dies with it, and the job has to be restarted from the last checkpoint. A Flink job that kept its aggregates in a HashMap instead of managed state was not a job. It was a demo.

The lesson was drilled into us the hard way, because the fix is boring and the failure is not:

• State must be external to the process. It lives in managed state, behind a backend that survives a restart. The operator is re-hydrated from it.
• Every state change is journaled. The job’s progress is a sequence of events you can read back, not a number in a register.
• Recovery is replay, not restart. After a failure, you restore the last good checkpoint and replay the events since it. The job picks up where it left off. It does not start over.
• The operator is a function of its input and its state. Same input, same state, same output. The operator does not “decide” anything outside that function.

Now do the translation. A Flink job is a long-running computation over an unbounded stream of events, with state that must survive failures, progress that must be recoverable, and operators that must be deterministic. An agent workflow is a long-running computation over an unbounded stream of events (alerts, tool results, model turns, human replies), with state that must survive restarts, progress that must be recoverable, and a contributor that must be kept inside deterministic boundaries.

That is the thesis of this article: a stream processor is the closest thing to an agent runtime that we already know how to make reliable. Every mechanism in the left column below exists because a streaming system hit the same wall an agent runtime hits, and every mechanism has a direct translation to the right.

Part 1 argued that agents are distributed systems. This article is the next step: not just which distributed system, but which one. Streaming is the shape that fits, because an agent workflow is a stream of events with state, not a request-response service.

---

The mapping

Here is the table I keep coming back to. The left side is a concept you already know from running Kafka and Flink. The right side is what it looks like when the “job” is an agent workflow:

Stream processing	Agent system analogue
Kafka topic (append-only, partitioned)	The event journal: every trigger, context write, tool call, and effect, in order, per workflow
Producer	Whatever injects events: alerting and monitoring systems, webhooks, timers, other agents
Consumer group	The runtime process driving a workflow. One workflow has one driving consumer at a time
Offset	The journal position. Where a workflow is, exactly
Keyed state	Workflow context, keyed by workflow id. The typed fields the FSM reads and writes
Operator	One step of the workflow: a state, an agent turn, a tool call
Operator chain	The FSM’s declared path through its states
Watermark / event time	The workflow’s own clock: what has happened so far, independent of when the process last looked
Checkpoint	A durable snapshot of (state, context, offset). What a restart restores
Barrier alignment	Ordering a checkpoint around in-flight effects: nothing in flight is lost or doubled
Exactly-once processing	The goal. What idempotent effects plus the journal give you (Part 1’s pattern)
Replay from an offset	Reconstructing a run from its journal: debugging, forking, recovery
Backpressure	Throttling agent concurrency to what models, APIs, and humans can absorb
Dead letter topic	The escalation queue: a workflow handed to a human because containment or verification failed
Savepoint	A deliberate, user-triggered snapshot: upgrade the workflow definition without losing in-flight runs. This is also how you roll a new model or prompt version: take a savepoint, swap the definition, let in-flight runs finish under the old one
Side output	The side channel for events that do not fit the main path: rejected alerts, uncertain hypotheses, audit records
Lag monitoring	The gap between what happened and what the runtime has processed: your queue depth, your stuck-workflow count
Schema evolution	Evolving the workflow’s context and state machine while runs are in flight. The same discipline covers a prompt or model change: the context schema is the contract, and a breaking change is a deploy, not an edit
Consumer lag	The distance between the journal head and the offset a workflow has processed. The metric you page on, the way a data platform pages on consumer lag
Canary / blue-green deploy	Canarying a new model or prompt version on a slice of traffic before it goes to 100 percent, with a rollback path that is a savepoint, not a prayer
Capacity plan	Sizing the worker pool against p99 tool-call latency and model throughput, not against the average

I would argue that every row in that table is either a feature your agent runtime should have or an incident it will produce. And every row has a known answer. You are not inventing reliability for a new kind of computer. You are running a job.

The honest caveat comes at the end of this article, because the operator on the right side is not a pure function. It is a model. That changes some of the semantics, and it is worth being precise about. But the shape of the system is the same, and the shape is what fails.

---

What the architecture looks like

Here is the centerpiece, the side-by-side. On the left, a stream processor. On the right, an agent runtime built the same way. Read it as a pair: every box on the left has its twin on the right, and the twin is what you should be building.

The dotted edges are the claim. The two columns are the same machine with a different operator.

Three properties fall out of this shape, and they are the same properties that make a Flink job reliable:

• The journal is the source of truth, not the process. Every event (an alert arrives, the model contributes a value, a tool returns, an effect fires) is appended before it is acted on. The process is disposable. The journal is not. If the process dies, a new process reads the journal and knows exactly where every workflow is.
• State is keyed and durable. Each workflow’s context lives in a store keyed by workflow id, behind a backend that survives a restart. The model is re-hydrated from it on every turn. A workflow that lands on escalate because containment or verification failed is not lost in a process’s memory. It is a terminal record in the store, with the full history attached, and a human takes over from the data, not from a process that has to still be running.
• Recovery is replay, not restart. After a crash, you do not ask the agent “where were you?” You restore the last snapshot, replay the journal since it, and the workflow resumes at its offset. The replay is deterministic over the journal. The only thing that is not deterministic is the model’s next turn, and that is the one part you keep inside a deterministic boundary (more on that below, and in Part 3).

Run the 8:48 restart through this architecture and the 31 incidents stop being a problem. Here is the recovery as a sequence, because the ordering is the whole trick: the journal and the snapshot are on disk, so the new process does not ask the old one where it was. It reads the disk and reconstructs.

The restart kills the process. The journal is on disk. The 31 workflows are records with offsets: some at contain, some at verify. The new process rehydrates them, re-issues the pending work, and continues. An incident that had already rolled back the deploy does not roll it back again, because the containment effect was journaled with a key before it fired, and the replay sees the key and skips the effect. An incident that had already notified the on-call does not page a second time, for the same reason. That is the idempotency-key pattern from Part 1, and it is the same thing a Flink job does with a barrier: nothing in flight is lost, and nothing in flight is doubled.

---

An incident-triage workflow that survives its own restart

Let me show the whole thing as an Aegis FSM, because this is the format I use for agent workflows. It is a YAML file, and it is small on purpose. The interesting parts are the re-entrant verify loop, the fact that every step is a journaled context write, and the fact that the notify effect is journaled with a key before it fires.

fsm:
  name: incident-triage
  version: 1
  initialState: assess

  context:
    alertSeverity:     { type: enum, values: [critical, major, minor] }
    rootCause:         { type: string }
    containment:       { type: enum, values: [rollback, scale_out, restart, none] }
    containmentOutcome: { type: enum, values: [succeeded, failed] }
    verifyOutcome:     { type: enum, values: [recovered, not_recovered] }
    checks:            { type: list, of: string }   # the tool appends one entry per verify pass; the list is the loop counter
    notifyRef:         { type: string }

  agents:
    ops:
      canTransition: [assess, investigate, contain]
      guidance: full

  states:
    assess:
      description: "Assess the alert. Contribute the alertSeverity."
      requires:
        - { name: alertSeverity, from: agent }
      transitions:
        - { toState: investigate }

    investigate:
      description: "Investigate the cause. Read the logs and contribute the rootCause."
      requires:
        - { name: rootCause, from: agent }
      transitions:
        - { toState: contain }

    contain:
      description: >-
        Choose and apply the containment. The model picks the containment
        (rollback, scale out, restart); the tool applies it. The model
        chooses the value; the machine owns the effect.
      requires:
        - { name: containment, from: agent }
        - { name: containmentOutcome, from: { tool: opsTool } }
      transitions:
        - toState: verify
          guards: [{ backend: cel, expression: "context.containmentOutcome == 'succeeded'" }]
        - { toState: escalate, guards: [{ backend: cel, expression: "context.containmentOutcome == 'failed'" }] }

    verify:
      description: >-
        Verify the metric actually recovered. This is the step the model used to skip.
        It is now a required state: done is only reachable after it passes.
        Each pass appends to checks so the loop is bounded and observable.
      requires:
        - { name: verifyOutcome, from: { tool: opsTool } }
      transitions:
        - toState: notify
          guards: [{ backend: cel, expression: "context.verifyOutcome == 'recovered'" }]
        - toState: verify   # re-enter: metric not recovered yet, re-check
          guards: [{ backend: cel, expression: "context.verifyOutcome == 'not_recovered' && size(context.checks) < 3" }]
          onTransition:
            - { assign: { verifyOutcome: "" } }   # consume the trigger so the next pass re-collects a fresh outcome
        - toState: escalate
          guards: [{ backend: cel, expression: "context.verifyOutcome == 'not_recovered' && size(context.checks) >= 3" }]

    notify:
      description: >-
        Notify the on-call that the metric recovered. Journaled with a key
        so a replay does not notify twice.
      requires:
        - { name: notifyRef, from: { tool: opsTool } }
      transitions:
        - { toState: done }

    done:     { description: "Incident contained and verified recovered. Reference: {{ context.notifyRef }}.", terminal: true }
    escalate: { description: "Containment or verification failed. A human takes over with the full history.", terminal: true }

  policy:
    maxExchangesPerStage: 20
    maxExchangesPerAgentPerStage: 10
    checkpointEvery: 2

A few things to notice:

• Every step is a journaled context write. Assess writes alertSeverity. Investigate writes rootCause. Contain writes containment and, from the tool, containmentOutcome. Each write is an event in the journal, with an offset. If the process dies between contain and verify, the next process reads the journal, sees the containment that already succeeded, and enters verify instead of re-doing contain.
• The verify loop is bounded and consumes its own trigger. checks is a list in context: the tool appends one entry per verify pass, so the loop counter is data, not a variable that dies with the process. The budget is size(context.checks) < 3. The re-entry edge clears verifyOutcome in an onTransition assign, so the next pass re-collects a fresh outcome instead of routing on a stale one. When the budget is spent, the machine hands off to a human at escalate with the full history. That is the dead letter topic, and it is a declared edge, not a habit.
• The notify effect is journaled before it fires. notify writes notifyRef with a key before it pages the on-call. If the process dies after the page goes out but before the terminal state, the replay sees the journaled key and does not page a second time. One page, possibly two attempts. Same trick as Part 1.
• Recovery is replay, not restart. A workflow that dies at contain or verify does not start over at assess. The new process restores the last checkpoint, replays the events since it, and resumes exactly where the old process left off, with the same context and the same offset.

If you want the replay logic as plain code, here is the shape of it:

def replay(workflow_id, journal, snapshot):
    # Restore the last good checkpoint: (state, context, offset)
    state, context, offset = snapshot.restore(workflow_id)
    # Replay every event since the offset, deterministically, until the current offset
    for event in journal.since(workflow_id, offset):
        context = apply(context, event)      # pure: same event, same context, same result
        state = route(state, context)        # the FSM's declared edges, guarded
    # Re-hydrate the driving consumer from (state, context). It picks up at the offset.
    return DrivingConsumer(workflow_id, state, context, offset)

The apply and route steps are pure. That is the whole point of the Flink mapping: the replay is deterministic over the journal. The model’s next turn is the only non-deterministic step, and it happens after the replay, at the head of the stream, where it is contained.

---

Where the analogy breaks down

I want to be honest about the limits of this mapping, because a good analogy taken too far is worse than no analogy at all. A stream processor is the right shape for an agent runtime, but the operator is not a pure function, and some of the old assumptions do not hold.

The operator is probabilistic, so “deterministic processing” means something narrower. A Flink operator is a function: same input, same state, same output. An agent turn is not. The model can read the same alert and produce a different hypothesis, or the same hypothesis with a different containment choice. So “deterministic processing” in an agent system means the orchestration is deterministic: the journal, the state, the guards, the routing, the replay. The contributor is not. The machine is a function of its input and its state. The model is a function of its input and its temperature. You keep the model inside the deterministic boundary, and you let the boundary do the work. That is the subject of Part 3.

Checkpointing with side effects is harder when the effect is an external API call. A Flink checkpoint is a barrier: it orders the state around the in-flight data, and exactly-once follows from that ordering. An agent’s “in-flight data” is often a side effect that has already left the building: a page sent, a rollback issued, an incident closed. You cannot rewind an external API. The fix is the idempotency-key pattern from Part 1: journal the effect with a key before you fire it, and make the downstream system deduplicate. The checkpoint gives you the ordering. The key gives you the safety. You need both.

Event time and watermarks are awkward when a workflow escalates and sits waiting on a human. A Flink watermark says “no more events with a timestamp before this will arrive,” and it is what lets the job reason about event time without waiting for the wall clock. An incident that lands on escalate and sits for hours waiting on a human to act has no meaningful watermark. The “event time” of the workflow is the sequence of its own events, not the wall clock. The wall clock is irrelevant to the state machine. It is only relevant to the human, and the human’s wait is a property of the escalation record, not of the stream. So you drop the watermark and you keep the offset. The offset is what matters. The wall clock is noise.

An agent “operator” can fail semantically, not just mechanically. A Flink operator crashes, and you know it crashed. The task fails, the job restarts, the checkpoint restores. An agent can complete a turn, write a confident-sounding note, and be wrong about the world. “The rollback fixed it” in the agent’s summary is a claim, not a fact. The journal, not the model’s prose, is the thing you check. A semantic failure does not trip a circuit breaker. It does not fail a guard, unless you wrote a guard for it. That is why the deterministic shell matters: it is the only layer that can catch a wrong answer before it becomes a wrong effect.

Backpressure is a cost problem, not just a throughput problem. In a streaming system, backpressure protects the pipeline from being overwhelmed. In an agent system, backpressure protects your invoice. A retry storm across a hundred workflows is not an infrastructure problem. It is a line item. The models are expensive, the APIs are rate-limited, and the humans are slow. Backpressure is admission control, and it is cost control. Part 7 of this series is entirely about it.

The model is a provider-side dependency, and it rate-limits you. A Flink operator does not get a 429 from its input stream. A model call does, and the rate limit is not a property of your system, it is a property of the provider. A capacity plan for model calls has to account for three things the streaming world does not: the provider’s tokens-per-minute ceiling, the burst behavior of your own traffic (an alert storm is a burst, and the model does not know that), and the fallback path when the primary model is rate-limited or degraded. The fallback is not a nice-to-have. It is the same pattern as a multi-region failover: a secondary model that takes over when the primary is unavailable, with the same idempotency-key discipline so a retry under the fallback does not double an effect. The capacity plan is the document that says which model handles which state, what the fallback is, and what the per-workflow cost ceiling is when the fallback is slower or more expensive. Without it, a rate-limit event is not an incident, it is a cost event, and the cost event is the one that shows up on the invoice a week later.

None of these break the thesis. They sharpen it. The point is that a stream processor is the right shape for an agent runtime, and the shape is what fails. The operator is a new kind of operator, and it needs a few new patterns on top of the old ones. The old patterns still apply. You just have to add a few new ones, and that is what the rest of this series is about.

---

What changes at 10, 100, 10,000

The reason I frame this as a systems problem is that the failure modes change as the number of concurrent workflows grows. Here is what I have actually seen.

At 10 concurrent workflows, you can babysit. You can read every run. You can catch a lost incident in a code review. You can fix a stuck agent by restarting it and asking it where it is. The system is reliable because you are the reliability layer. The journal is a nice-to-have. The escalation record is a convenience. That is fine for a prototype and a demo. It is not an architecture.

At 100 concurrent workflows, you stop being the reliability layer. You cannot read 100 runs a day, let alone catch a subtle duplicate page. The journal, the checkpoint store, and the escalation queue stop being nice-to-have and start being load-bearing. Cost starts to matter: a retry storm across 100 workflows is a real number on your invoice, and backpressure becomes something you have to design. The question “where is this incident” stops being a question you answer by reading the logs and becomes a query you run against the journal. The offset is the answer.

At 10,000 concurrent workflows, the system is a stream processor in every sense that matters. You need admission control, rate limiting, and chaos testing. A single bad state transition fans out across thousands of runs, so a bug is not an incident, it is a population. You cannot fix it by restarting. You need replay to reconstruct a run, observability to answer “why did this happen” across a thousand of them, and a control plane to roll back a bad version of the workflow. The savepoint is how you upgrade the workflow definition without losing the 10,000 runs that are in flight. The vocabulary from the mapping table stops being a metaphor and becomes your incident runbook.

Here is the mechanic, concretely, because “the offset is the answer” is not a claim you should take on faith. Consumer lag in a data platform is the distance between the head of the topic and the offset a consumer group has processed, and it is the metric that pages the on-call when it grows past a threshold for a window. In an agent runtime, the same metric is the backlog per workflow type: events appended minus events processed. Say the incident-triage workflow appends roughly 10 events per incident (assess, investigate, contain, up to three verify passes, notify, plus the tool calls and context writes in between) and the runtime processes them at a steady 10 per minute on a good day. At 10 concurrent workflows, the gap is a number you check when something feels off: a few events deep, and it drains before anyone notices. At 100 concurrent workflows, the gap is a dashboard tile with a paging threshold: if the backlog for incident-triage workflows grows for 15 minutes and does not drain, page. The threshold is not arbitrary. It is the point at which the next incident in line inherits a wait time nobody set on purpose. At 10,000 concurrent workflows, the gap is the primary health signal for the whole fleet: it is the first number that moves before anything else, and it is the number your on-call reads first. The formula is events appended per minute - events processed per minute, and the answer it gives you changes character at each order of magnitude: a hunch, a threshold, a paging rule. The architecture does not change. The way you watch the number does.

The through-line is that the architecture you need at 10,000 is the same architecture you need at 10. The difference is that at 10, you can get away without it. That is the trap: the system works in the demo, the reliability is invisible, and you do not find out what you are missing until the scale makes it visible. Which is to say, until an 8:48 a.m. restart.

---

Operating this in production

None of the mechanisms above run themselves. Here is what actually running this looks like day to day.

Deploy a new FSM or prompt version as a savepoint-style upgrade. The version: 1 field in the YAML above is the unit you ship. A change to the verify state’s check budget, or a swap of the model that forms the root-cause hypothesis, is a production change: take a savepoint of what is in flight, ship the new version, let the in-flight runs finish under the old one while new runs start under the new one. The alternative is a fleet of incident-triage workflows that started with a 3-check verify budget and are now being re-evaluated against a 5-check budget, with no one able to say which version produced which behavior. That is not a reliability failure. It is an audit failure, and it is harder to recover from.

Monitor lag and backlog as the primary health signal. A data platform pages on consumer lag because lag is the leading indicator: by the time the downstream dashboard is stale, the lag has been growing for an hour. An agent runtime has the same shape. The backlog per workflow type (events appended minus events processed) is the leading indicator. A growing backlog in incident-triage means the model is slower, the ops tool is flakier, or contain and verify keep failing and piling into escalate. Page on the backlog trend, not on the failure count. The failure count moves after the backlog, and by then the queue is already deep.

Treat a model swap like a deploy, not a config change. Swapping the model that forms the root-cause hypothesis is a change to the operator. It needs the same discipline as any operator change: a canary on a slice of new runs, an eval gate before the swap goes to 100 percent, and a rollback path that is a savepoint, not a prayer. The model is the one part of the system that is not a pure function, and it is the one part that needs the most operational discipline.

---

The whole thing, one idea

Step back and look at what actually happened at 8:48 a.m., and what the fix was:

Failure	Systems name	Pattern that fixes it
A run dies on a process restart	State in the operator’s memory	Durable, keyed state (workflow context in a store)
The agent does not know where it is	No offset	The journal is the source of truth; the offset is where the run is
An escalated incident loses its record on restart	No durable record	The store keeps escalated runs as records, not threads; the handoff to a human survives a restart
A retry re-sends a page	Duplicate delivery	Idempotency key on every effect, journaled before it fires
You cannot say what a run did	No replay	Replay from the journal: reconstruct, debug, fork
Agents generate more work than they can handle	No backpressure	Admission control and throttling, sized to models, APIs, and humans

No new computer science. No new kind of reliability. The same patterns that keep a Flink job from losing its state on a task manager failure keep an incident-triage agent from losing its incidents on a deploy: durable state, a journal you can replay, checkpoints you can restore, and an operator that is kept inside a deterministic boundary.

I built the incident-triage workflow above because I had to. The 31 lost incidents were real, and the fix was not a better prompt. It was a journal, a checkpoint, and a replay that the model was not allowed to skip. If you are putting agents in front of anything that touches production, changes a record, or pages a human, start with the mapping table. Find the row that is missing from your architecture, and build that one first.

That is the move for this article. The next one is where the deterministic boundary gets its name: a deterministic shell around a probabilistic core, and why the LLM should never own the control flow.

Up next: Part 3 (Deterministic Shell, Probabilistic Core)