Essays/Systems

Durable state for agents

Agents crash and machines reboot. State in a local SQLite file, every run recorded and idempotency keys let a process end without losing or doubling work.

Agents crash, sessions die mid-task, machines reboot, and rate limits cut runs off mid-sentence. In a fleet of coding agents that is an ordinary Tuesday. So the storage under everything I build must not lose data when a process ends: state lives in a local SQLite file in a predictable place, every run is recorded, and a retry cannot create a duplicate.

Why a local file and not a cloud database?

Because a file on disk does not fail when the network does. The default is SQLite in a predictable directory, one folder per tool, and it works the same on a laptop with no internet as on a machine wired into a datacenter. Cloud sync is opt-in and sits on top. It is never the only copy of the truth.

Three of these files show the pattern. One, in the coding tool my agents run in, holds more than 3,100 conversation threads going back a month. A second holds its 480 goals. A third holds agent sessions, with well over 100,000 messages between them. I can query all three with the same sqlite3 binary that is on every machine I own, and I can say from memory what each one holds. When I have to find something at 1am, a predictable file in a predictable place is worth more than a clever abstraction.

The tool that holds the agents’ terminal sessions follows the same rule. On one machine that whole session server vanished without warning. It took 53 sessions and 75 windows with it, including 18 panes running live agents, some with days of work in their scrollback. There was no crash signal and nothing in the system logs. It was recoverable only because a background auto-save had written a snapshot 26 minutes earlier. The layout was rebuilt from that file, not from anyone’s memory of what 53 sessions had held.

Tests use the same substrate, pointed somewhere else. Every tool reads an environment variable that overrides its state directory, so a test run uses a throwaway file instead of the real one. I once chased a CI failure that reproduced locally but not on the runner. The fix was to set the test environment’s home directory explicitly instead of trusting the runner’s default. A suite that shares state with anything else stops proving things.

Agents crash, and the state does not get to.

Every run is recorded, including the failures

Daemons built on these files have to survive restarts, and that is a claim I can test, not a hope. One fix reads in the commit log exactly as it should: survive SQLite busy and busy-snapshot contention instead of failing the run. Sometimes two processes reach for the same file at the same moment. The right behavior is to retry and continue, not to crash and lose what the run was in the middle of.

The scheduling daemon logs every execution, including the ones that failed. A system that remembers only its successes cannot say why a nightly job stopped firing three weeks ago. I want to answer what happened at 3am on Tuesday without asking an agent to remember. Agents do not remember, and their own account of what happened is the thing I trust least, for the reasons in Agent self-reports are not evidence.

A scheduler also has to be sure it is the only copy of itself, or two copies will take the same due job and run it twice. A pidfile guard handles that. On start the daemon looks for its own pidfile and refuses to start if another copy is alive. With SQLite underneath, a job due at 3am is neither run twice when two daemons start nor lost when the machine reboots at 2:58.

What makes a retry safe?

Idempotency keys. The clearest case I have built is a billing system, where a mistake costs money. A payment provider’s webhook can legitimately arrive twice for the same event, because networks retry and so does the provider. So there is a dedupe key on the event id, with a conflict clause that does nothing on a repeat, and the second delivery finds the row already there. Credit grants are keyed to what they grant: a monthly grant to the invoice, a top-up to the payment. Replaying the same event twice produces the same ledger, not a double credit.

Fingerprints do the same job for cheaper and more frequent cases. A loop that runs every fifteen minutes should not open a new task every fifteen minutes because the condition it watches is still true. Each task gets a stable fingerprint computed from its content, and the loop upserts against that fingerprint instead of inserting blindly. A thousand runs against an unchanged condition yield one task, updated a thousand times. This is what makes the loops in Write loops, not prompts safe to leave unattended. A loop that cannot tell the same problem from a new one floods its own task list within a day.

The contract matters as much as the keys. The design document for our workflow engine says external systems must not write rows into its database directly. Callers go through an idempotent upsert, keyed by an idempotency key and a hash of the spec, with dry-run, preflight and commit modes in front of it. Once one caller reaches around that contract, dedupe and safe retries stop being guaranteed for everyone else.

Messages between agents get what a person’s inbox gets. The mailbox that lets one agent message another in a different session is backed by tables for messages, delivery attempts and dead letters. A message that cannot be delivered does not vanish. It lands where something can retry it or a person can see why it stalled.

The layer under the model

None of this is exotic. It is plumbing more than architecture, on purpose, because the interesting part of an agent system should be what the agent decides. Models are probabilistic. The same prompt can produce a materially different result twice in a row. That is why the layer under the model cannot be probabilistic too. If the judgment varies from run to run, the state it writes to, retries against and resumes from has to behave the same way every time. That is what lets a fleet run unattended at 3am and still be trustworthy at 9am.

Back to the essays

Field Notes

A letter from Andrei Hasna on building with AI agents

Subscribe by email

By clicking “Subscribe”, I confirm that I want to get email delivered to my address and agree to Andrei Hasna’s Privacy Policy and Terms.

In each letter

  • New essays on AI agents and the systems behind them
  • What Hasna is building, and why
  • Costs, routing and results from real work
  • What did not work
  • News of the book and the podcast