Agents in Production (Intro)

The agent did the job.

The page went out.

The rollback fired.

And it still caused a production incident.

It did exactly what it was built to do, and it did it twice.

The model was fine. The system around it was not.

We spent decades learning how to make distributed software reliable.
State, events, failure handling, replay, backpressure, idempotency, observability.
We have the vocabulary and the patterns, and they are old.
Agentic systems do not invalidate those lessons.

If anything, they make them more important, because a model is a new kind of component: probabilistic, non-replayable, and expensive. It needs the same deterministic boundaries we have always put around the things that do the work.
And it needs the operational muscle we already use to run models in production: evals before a change ships, staged rollouts, drift monitoring, and a cost budget that gets watched as closely as correctness.

we make autonomous agents reliable the same way we have always made distributed systems reliable, by putting the nondeterministic part inside a deterministic shell.

What the series argues

Three claims, each one setting up the next:

  1. An agent in production is a distributed system. It has state, failure, concurrency, retries, coordination, and observability. Those are not new problems. They have names, and the names come from distributed systems, so the fixes come from there too.
  2. A probabilistic model only works inside a deterministic shell. The model contributes a value, and the machine decides what that value means for the next step. So control flow lives in named states, guards, and declared transitions with a durable position that survives a crash. It does not live in the prompt.
  3. The shell is a production system, so someone has to operate it. It has to hold at scale, and the architecture that works at 10,000 concurrent workflows is the one you need at 10, so concurrency limits, queue depth, and cost per workflow are capacity inputs from day one. The workflow definition is a versioned artifact that gets rolled out, canaried, and rolled back like any other change, and someone is on call when a workflow gets stuck.

Each article starts from a real production failure, connects it to a solved systems problem, introduces the principle, translates it to agent architecture, shows the diagram, builds a small working example, and then gets honest about where the analogy breaks down.

How it is structured

Twelve articles in three acts.

Act I: the thesis
distributed systems, deterministic shell,
explicit state

Act II: the patterns
idempotent effects, replay,
backpressure, human review

Act III: the scale
memory, observability,
multi-agent, control plane

Act I establishes the thesis. The four articles in this section build the worldview from the ground up:

Act II proves it technically. Idempotent effects, event journaling, replay and recovery, backpressure, and durable human review. Act III shows how it scales. Memory architecture, decision observability, multi-agent coordination, and the enterprise control plane. Those are coming.

If you are putting agents in front of anything that touches a customer, changes a record, or moves money, start with Part 1. It is the mapping table that the rest of the series is built on.

Up next: Part 1 (Agents Are Distributed Systems Now)