Capstone — A Durable Agent Runtime
This page is original to the course. It restates the exam, gives you the shape a correct answer takes, and puts a real production workflow beside it so you can compare box by box.
The exam
Design AgentRun without opening the documentation. It must:
- run for up to 72 hours and make more than 1,000 model calls;
- tolerate failing tool calls without duplicating expensive ones;
- keep working when GPUs disappear and when Workers are redeployed twice a day;
- let a user pause, resume, change instructions, and ask what it is doing;
- delegate to sub-agents;
- keep old executions alive across deployments;
- never let its history grow without bound;
- give operators exact diagnostics of what ran.
If you have done modules 1 through 10, none of these is new. The exam is whether you can put them in one picture and defend every box.
The shape that falls out
Around the box: a RetryPolicy on every Activity with bad arguments marked non-retryable; a Start-to-Close and Schedule-to-Close timeout on every Activity; a heartbeat on anything that can run longer than half a minute; Worker Versioning with executions pinned and upgraded at continue-as-new; one Task Queue per compute class; replay tests over a fixture of exported histories; and a failure-injection suite that crashes Workers, times out Activities, duplicates executions and cancels runs.
The failure column
The defense document requires one line per box. This is the table you fill in.
| Box | A Worker dies here | On retry | What history records |
|---|---|---|---|
plan() |
Replayed from history on the next Worker; nothing is lost because nothing happened outside history | Not applicable, it is a decision | Commands only |
| LLM Activity | The attempt fails at Start-to-Close or heartbeat; the Activity is retried with the same operation key | A recorded answer can be returned from the gateway ledger. If the provider charged but the gateway crashed before recording the response, reconcile the ambiguous outcome or rely on provider-side idempotency before calling again | Scheduled, Started, Completed with the result |
| Tool Activity | Heartbeat timeout fails the attempt quickly; the next attempt receives the last delivered heartbeat details | Activity code reads the details and implements safe resumption; external effects still need idempotency | Same, plus heartbeat details in the pending Activity view |
ResearchAgent child |
The child replays on its own; the parent does not notice | The child's own policy | ChildWorkflowExecution events in the parent |
| Pause wait | The condition is Workflow state; nothing polls while paused | Not applicable | WorkflowExecutionSignaled |
continue_as_new |
The new run starts atomically as the old one closes | Not applicable | WorkflowExecutionContinuedAsNew |
If you cannot fill a row from memory, that module is where your next hour goes.
The LLM row has an ambiguous commit window: the provider may finish and charge, then the gateway may crash before it stores the answer under the operation key. A retry finds an in-progress key but cannot infer the provider's outcome from that fact alone. A stable key prevents duplicate calls only if the provider honors it, the gateway controls the billable operation, or a durable provider lookup lets you reconcile the first call. The lab's fake ledger has no such crash window, so its one-bill test proves the fake's behavior, not a general exactly-once guarantee.
For the tool row, Temporal makes the last delivered heartbeat details available to the retry. It does not continue the container job automatically: your Activity must read the details, choose where to restart, and make repeated work safe.
Task Queues as compute lanes
Temporal decides where execution logically is. Kubernetes decides where processes run. They are
different layers, and the agent runtime uses both. Workers for gpu-tools run on GPU nodes and
poll only that queue. Workers for browser-tools run in a sandbox with a headless browser. The
Workflow does not know or care which host picks up a task; it only names the lane. Backlog on a
lane is the signal to scale that fleet. Appendix A1 has the details.
Nexus, in one paragraph
When the training team, the inference team and the agent team each own a Namespace, Nexus is how a Workflow in one calls an operation in another with durability, retries, cancellation and visibility intact, instead of an HTTP call that forgets everything on failure. Know that it exists. Do not build the capstone on it. Appendix A2.
The real one
The engine that rendered these slides runs its multi-leg course jobs on Temporal. The design is in
docs/design/temporal_agent_loop.md in the mini_lecture repository.
Compare it with your design:
- Stable Workflow ID per job, so the API and the reviewer always address the same execution.
- Legs as Activities with retry and heartbeat. A leg that produces one lecture and yields schedules another Activity, not another queue job.
- Checkpoint as a durable Signal wait. A reviewer may take a day; no Worker waits with them.
- Approve and revise as Signals from the API.
What it deliberately leaves outside Temporal: the job store, the agent state file, the generated artifacts, usage telemetry and the verifier database. Those are the product-facing and quality systems of record. Temporal history answers what execution happened. It does not answer whether the lecture was good. That boundary is the answer to the tenth question.
Diagnose from history alone
You will be given a broken AgentRun and you may not read the code. From the Web UI or
temporal workflow show, answer:
- What happened?
- What did Temporal think happened?
- What side effects may have occurred?
- What will happen next?
- What will happen if every Worker disappears now?
Appendix A3 maps each question to the surface that answers it.
A ninth contract question, for the product
Every box above answers what survives, what repeats and what is recorded. A product on top of the runtime needs one more answer per box: what does the user see, and from which store? The Workflow's visibility list is the operator's view, not the product's; the product reads its own projection. Appendix A4 works this through on a different scenario, an evaluation console, so the skill transfers rather than the diagram.
The ten questions
- What exactly survives when a Worker crashes?
- How does replay reconstruct Workflow state?
- Why must Workflow code be deterministic?
- Why can an Activity execute twice?
- What is the difference between Activity retries and Workflow replay?
- Signal versus Query versus Update?
- Activity versus Child Workflow?
- When should you Continue-As-New?
- How do you safely deploy changed Workflow code?
- Where does Temporal end and Kubernetes, Kafka and Postgres begin?
What the lab will test
Before building the final runtime, state which mechanism protects each requirement: crash recovery, duplicate effects, unbounded history, human input, cancellation, and code upgrades. Use the capstone evidence ledger to test your mapping.
Try it
labs/11-capstone-durable-agent-runtime/: exercise 11.1 assembles AgentRun from the module labs
with separate Task Queues; 11.2 runs 200 steps and checks history bounds and Search Attributes;
11.3 runs the failure-injection suite and deploys v2 mid-run; 11.4 is the one-page architecture
defense and the ten questions; 11.5 puts a product on top of the Workflow — appendix A4's
evaluation console, with a projection, a cursor-paginated list, an SSE stream and an approval gate.
Lab page: Open the guided lab — the exercises, every source file, and the local setup steps.