The Model Call You Pay For Twice

The Model Call You Pay For Twice

We kill a running agent on Restate and count the model calls its replay pays for again. Then we find what one line of code can and cannot fix.

Kill the agent in the middle of a pause

Instructor: A crash in an agent loop can cost more than a restart: it can buy the same model decision twice. We'll count exactly how.

Instructor: The loop is deliberately small: ask the model, call one tool, pause for a rate limit, ask again. Small enough to count every call.

Instructor: The kill lands during the durable pause, after the tool already ran. So the question is what the restarted process still knows.

Instructor: A clean run makes two model calls. The crashed run made three, and the tool still ran only once.

Instructor: In the log, call two repeats the first decision word for word. That extra call is the bill for the crash.

The agent loop

Instructor: Restate records only what the handler does through its context. Everything else is ordinary code that runs again on every attempt.

Instructor: One handler holds the whole loop, and every replay starts it from the top.

Instructor: The tool call and the pause both go through the context, so both are journaled.

Instructor: In naive mode, the model call is a plain HTTP request. Restate never sees it.

Instructor: In journaled mode, the same call goes through context dot run typed. Nothing else changes.

Instructor: Because the two modes differ in one line, every difference in the call counts comes from that line.

What the journal kept

Instructor: Every invocation has a journal: the record a replay reads to decide what to skip and what to run live.

Instructor: The input opens it, each durable step and timer adds an entry and its completion, and the output closes it.

Instructor: Entry one is the weather tool, and the notification after it holds the tool's result.

Instructor: Entries three and four are the pause and its completion.

Instructor: The model decision that chose the tool has no entry at all. That missing entry is the whole bug.

Instructor: Read the journal by index: the tool is there, the pause is there, and the decision behind them is not.

What the replay re-ran

Instructor: When the process comes back, Restate starts a new attempt and replays the journal against the handler.

Instructor: Replay is not a memory snapshot. The handler runs again from its first line.

Instructor: At the weather step, the journal already holds the result, so the tool is not called again. The run shows one weather call.

Instructor: The model call has nothing recorded, so it goes out to the model again.

Instructor: With a real model API, that is a second bill for a decision already made, and a slow model adds its latency to every recovery.

Journal the model call

Instructor: The fix follows from the mechanism: record the model's answer before the handler branches on it.

Instructor: Inside context dot run typed, the answer is written to the journal once it returns.

Instructor: On replay, the recorded answer comes back, so the handler takes the same branch without calling the model. The point is determinism, not caching.

Instructor: The lab measured it with a stub whose answers change from call to call: two model calls, the clean count.

Instructor: The model call now appears at entries one and seven: the decision before the tool, and the answer after the pause.

Instructor: It is the naive journal with the two missing decisions filled in.

A call cut off mid flight

Instructor: Journaling protects a call once its result is recorded. The next experiment kills the agent before that happens.

Instructor: The kill lands one second into the first model call, while the stub is still holding back its answer.

Instructor: The model did the work, but the answer never reached the journal, so the replay has nothing to return and asks again.

Instructor: Three model calls: the lost one, its replacement, and the final answer. And the journal looks exactly like a clean run's.

Instructor: So context dot run never repeats a call it recorded, but it is not exactly-once. An effect that must not happen twice also needs an idempotency key the API honors.

Instructor: In the log, the first call's answer is never delivered, because the process that asked for it is gone.

When the second answer is different

Instructor: Now let the model answer differently the second time. The cost bug becomes a correctness bug.

Instructor: Same naive agent, but the stub's answer changes from one call to the next.

Instructor: On replay, the model says answer directly. The journal says the agent called the weather tool at entry one.

Instructor: The replay cannot match its journal, so Restate stops the attempt with R T zero zero one six, a journal mismatch.

Instructor: Here the stub flipped back on the next attempt, and the run finished, after four model calls. That is luck, not recovery.

Instructor: The log shows both attempts: the mismatch, then the attempt whose answer happened to match the journal again.

When it stays different

Instructor: If the model changes its mind for good, the luck never comes. Every retry meets the same mismatch.

Instructor: After the restart, every attempt that reached the agent failed with R T zero zero one six: seventeen of them.

Instructor: Each of those attempts called the model before it reached the mismatch, so each one was paid for.

Instructor: The handler's retry policy allows twenty attempts, then pauses the invocation instead of failing it.

Instructor: That run ended paused, after eighteen model calls, and still one weather call.

Instructor: Eighteen is not a constant. Attempts made while the agent is down call no model but still count, so a longer restart delay leaves fewer paid calls: twenty down to fifteen.

Instructor: The policy comes straight from the lab's agent. On Restate server version one point seven point ten, journal mismatches were retried under it, not failed at once.

Read the mismatch

Instructor: When a replay fails, the error message tells you where the journal and the code parted ways.

Instructor: It names a journal index and the two operations that disagreed.

Instructor: Index one in the naive journal is the run of the weather tool.

Instructor: The current attempt tried to return an answer instead, because the unjournaled model call changed the branch.

Instructor: In this SDK version, the two labels print the wrong way round. It is reported upstream with a fix proposed; until that ships, trust the index, not the labels.

Instructor: The message says handler return was recorded and a run is being attempted. The journal says the opposite: entry one is the run.

Get a stuck invocation out

Instructor: A paused invocation keeps its journal, so any way out has to respect the entries already recorded.

Instructor: Resuming on new code works only if that code replays the recorded entries unchanged.

Instructor: The journaled fix adds a model step before entry one, so it cannot replay the naive journal.

Instructor: Resumed on the fixed deployment, the invocation was accepted, then failed with the same mismatch.

Instructor: Restart as new is refused while the invocation is paused. Kill it first, then restart it as new: a fresh journal, and two model calls.

Instructor: Overwriting a deployment with force is meant for local development. In production, deploy the fix as a new deployment.

Instructor: The admin API log shows each step, from the resume that failed to the restart that completed.

If the agent dies right now

Instructor: Back to the question every durable agent has to answer: if the process dies right now, what survives?

Instructor: The input survives. Every attempt starts from it.

Instructor: Everything completed through the context survives: tool results, finished timers, journaled model answers.

Instructor: Anything read outside the journal is gone, and the replay reads it again, possibly with a different value.

Instructor: A call still in flight is gone too, even a journaled one, because its result was never recorded.

Instructor: When the process comes back, unjournaled calls run and are paid for again, and a different answer ends in a journal mismatch.

Takeaways

Instructor: Every run in this module comes down to one rule about replay.

Instructor: A replay must issue the same Restate operations, in the same order, with the same inputs.

Instructor: So record anything that can differ between attempts before the handler branches on it: model answers, responses, time, randomness.

Instructor: A recorded call never runs again. A call cut off before its result is recorded can.

Instructor: And a mismatch is not a harmless retry: every attempt pays for the model again, until the invocation pauses.

Instructor: The lab reproduces every run in this deck. Count the calls yourself, and read each journal.