Agents fail. Their sessions shouldn't.

An agent working a customer ticket builds up state as it goes. It has read the customer's message, looked up their order, called a model and started deciding what to do. In most agent frameworks that state lives in the process's memory.

Processes die. A bad deploy, an out-of-memory kill, a crash in a library, a node going away. When that happens mid-ticket, the work in memory is gone. At best the ticket starts again from scratch, repeating lookups and model calls. At worst it stalls with nobody noticing, or an action such as a refund runs twice.

This post shows a way round that, using a small support desk demo built on Laser SDK. It covers what happens when you kill an agent part way through a ticket, how the new process picks up where the old one stopped, and which parts of that the SDK gives you and which parts are your own code.

The demo

The support desk has three agents, each its own process:

• Triage reads the customer's message and works out what they want.
• Resolver looks up the order and deals with the request. A question gets an answer from the knowledge base, the order record and the customer's history. A refund request is checked against the refund policy, and if a refund is due, a human approves it.
• Billing issues approved refunds.

The agents don't call each other. They hand work on by writing messages to one durable, append-only log, and each reads the messages meant for it. Laser SDK is the Python library they use to do that. The log itself is Apache Iggy, and Laser Stack adds a key-value store (KV) and session tracking on top. It runs locally or on LaserData Cloud.

The demo has two ways to kill an agent. Each agent has a Kill button, and there is a chaos monkey that, when turned on, kills a random agent every few seconds. Either way the agent gets SIGKILL: no cleanup, no chance to save anything. A supervisor starts a new process in its place, and the ticket carries on from where it stopped. Both kinds of kill are recorded on the log, so they show in the ticket's timeline.

What makes resume work

Resuming needs the new process to answer two questions: what was I working on, and how far did I get? Three pieces answer them.

Redelivery: what was I working on? When an agent finishes with a message, it tells the log so. This is called acknowledging the message. A process killed mid-ticket never acknowledges it, so the log hands the same message to the next process that asks. Nobody has to notice the crash or resend anything.

The session: what has happened so far? Every ticket has a session, a running record of each step taken on it: the customer's message, the order lookup, the reply. Each step is called a turn. The session lives outside the process, so it survives the crash.

The checkpoint: how far did I get? A checkpoint is a bookmark in the session. The resolver takes one at a known point in its work and saves it in the key-value store under the ticket's id. After a crash, the new process loads the bookmark and asks the session for every turn recorded after it. Those turns are the work already done.

Laser SDK provides all three, plus the key-value store. The app decides where to take checkpoints and what to do when it finds one. The SDK knows nothing about tickets or refunds.

Killing the resolver mid-case

Take the duplicate scenario. Order 1042 was charged $49 twice, and the customer wants one charge back.

Triage reads the message and hands the ticket to the resolver over the log. The resolver looks up the order, takes a checkpoint and saves it to KV under the ticket id. It records the order lookup in the session as a tool call, which is two turns: the call and its result. Then it holds at its policy step.

record = await desk.get_order(body.get("order"))
cp = await session.checkpoint()
await desk.ckpts.set(t).bytes(json.dumps({"cp": cp.to_json()})).send()
lookup = await session.tool("order_lookup", {"order": body.get("order")})
await lookup.complete(json.dumps(record).encode())
# the policy hold: press Kill now
await asyncio.sleep(C.POLICY_PAUSE)
await finish(desk, t, body, record)

Press Kill on the resolver during the hold. The process dies before it acknowledges the handoff. The supervisor starts a new resolver, and the log delivers the same handoff to it.

The new resolver starts the same function. This time it finds the checkpoint:

saved = await desk.ckpts.get(t)
if saved is not None:
    cp = ls.Checkpoint.from_json(json.loads(bytes(saved).decode())["cp"])
    since = await session.turns_since(cp)
    await desk.event(t, "replay", text=f"from checkpoint, {len(since)} turn(s) since: ...")
    results = [turn for turn in since if turn.display == "tool.result"]
    if results:
        await finish(desk, t, body, json.loads(results[-1].text()))
    else:
        await finish(desk, t, body, await desk.get_order(body.get("order")))
    return

turns_since returns two turns: the order lookup call and its result. So the new resolver knows the order was already checked, and the result holds the order record. It uses that record instead of looking the order up again, and it does not rerun the policy hold. If the kill had landed before the result was recorded, it would look the order up after all. It goes straight to the decision, asks a human to approve the refund, and hands it to billing.

Nobody had to notice the crash. Nobody resent anything. The event log shows the kill, then a replay step, then the ticket carrying on.

Billing can be killed too

Redelivery means a handler can see the same message twice, so anything with a side effect has to cope with that. Billing keeps a ledger entry in KV, keyed by the handoff's idempotency key. If billing is killed part way through a refund, the new process finds the entry and finishes the refund. If the refund already went through, it finds the entry marked done and skips it. The customer is refunded once.

That check is app code on top of the SDK's keys. A later post covers it in detail.

Summing up

When an agent dies mid-ticket, a new process takes over. The ticket isn't lost, the work already done isn't repeated, and the customer isn't refunded twice. The app code needed is small: take a checkpoint, check for one when the handler starts, and keep a ledger entry for anything that moves money.

None of this is specific to support tickets. Any agent doing work in several steps can use the same pattern: record each step in the session, checkpoint at the points you would want to resume from, and check for a checkpoint before starting. Crashes then become a normal event you can test, not an incident.

That is what the Kill button and the chaos monkey are for. Run the demo, send a ticket, kill the resolver while it holds at the policy step, and watch the replay appear in the timeline. Then turn on the chaos monkey and leave it running. The code is at https://github.com/laserdata/laser-support-desk, and the README has the steps.

The next posts in this series look at the other side of the same log: how billing makes sure a refund fires once even when it is killed half way through, and how memory, analytics and evals all read the one stream the agents write to.