The Machine

Kill the Worker. The program continues.

Here is the experiment this module is built around. A function does step A, sleeps for thirty seconds, then does step B. Halfway through the sleep you kill -9 the process running it. You start a fresh process. Step B runs — same function, same local variables, different process. The code you wrote contains no handling for the crash.

Be precise about what "continues" means. The sleep is a durable timer that exists in the Temporal Service; it keeps counting while no Worker runs. Workflow code is not executing during the outage — nothing is. When a Worker returns, it reconstructs the Workflow by replay (module 2) and the Workflow observes the timer as already fired, so step B runs.

Temporal's name for this is Durable Execution: the ability of a Workflow Execution to maintain its state and progress even in the face of failures, crashes, or server outages. This page is about the machine behind that sentence, and the one question you must answer at the end: where was the program counter stored while the process was dead?

For the agent runtime you are building toward, the "function" is an agent loop — model call, decision, tool call, sometimes a day-long wait for a human — and every step is a place a process can die.

The program this course builds

Every module of this course works on one program, and it is worth saying what it is before the first line of code uses its name.

An agent run is a long-lived loop: decide the next step, call a model or a tool, look at what came back, decide again, and stop when the goal is met. Real runs last hours, make hundreds of model calls, wait for a person, get new instructions halfway through, hand sub-tasks to other agents, and run on machines that get redeployed underneath them. Every concern Temporal exists for shows up in that one loop, which is why we teach Temporal by building it instead of an order pipeline.

The program is called AgentRun. Its parts have fixed names that you will see in every lecture:

Module 1 builds the smallest possible stage of it: plan() decides one step, call_llm runs once, the Workflow waits thirty seconds, and it returns. That is deliberately too simple to be an agent. Each later module adds one capability and one failure you inject to prove it survives: retries and idempotency in module 4, pause and re-instruction in module 5, sub-agents and bounded history in module 6, cleanup in module 7, tests in module 8, upgrades in module 9, production lanes in module 10, and the whole design defended in module 11. By the end the same file has become a production agent runtime.

Who runs your code?

Not Temporal. This split explains most of the fault model and the scalability story, so it comes first.

The Temporal Platform consists of supervising software called the Temporal Service and application code bundled as Worker Processes. A Temporal Service consists of the Temporal Server (written in Go) and a database. Worker Processes are hosted and operated by you and execute your code, using an SDK — here, Python.

The split is strict. The Service records what happened and dispatches what should happen next; Workers poll for that work, run your Workflow and Activity code, and report back. The Service (including Temporal Cloud) doesn't execute any of your code on Temporal Service machines; it is solely responsible for orchestrating State Transitions and providing Tasks to the next available Worker.

Note the arrow's direction. Workers connect out and ask "do you have work for me?"; the Service never connects to a Worker and needs no address or open port. That is why a Worker can run behind a NAT or on a GPU box you control, against a Service somebody else hosts.

The five nouns

You need five words to describe any Temporal system. Learn them in Temporal's own terms; the precision pays off in module 2.

Temporal Service. The group of services, known as the Temporal Server, combined with Persistence and Visibility stores. Inside it: a Frontend Service (the gRPC endpoint everyone talks to), a History Service (writes each execution's ordered Event History to the database), a Matching Service (manages Task Queues and hands Tasks to polling Workers), and a Worker Service for Temporal's own system Workflows — not yours. temporal server start-dev gives you all of it in one process, Service on port 7233 and Web UI on 8233 (as of 2026-09; see upstream page).

Worker. "Worker" means one of three things, and the docs keep them apart: a Worker Program is the static code you write with the SDK; a Worker Process is the running process that polls a Task Queue, dequeues a Task, executes your code in response, and responds to the Service with the results; a Worker Entity is the individual Worker within a process that listens and polls on a single Task Queue. Every Worker has a Worker Identity, by default ${process.pid}@${os.hostname()} (as of 2026-09; see upstream page), written into Event History — your proof in the lab that a different process finished the Workflow.

Client. A Temporal Client is a library object from the SDK that your application uses to talk to the Service's Frontend: start Workflow Executions, get results, list, Query, Signal, cancel, terminate. It does not run Workflow or Activity code. In an agent product it lives in your API server; "start agent-42" is one start_workflow call.

Namespace. A Namespace is a unit of isolation within the Temporal Platform. Task Queues and Workflow Executions belong to a Namespace; Temporal guarantees a unique Workflow Id within a Namespace; retention is configured per Namespace. If no Namespace is specified, the Service uses default — what the dev server gives you.

Task Queue. A Task Queue is a lightweight, dynamically allocated queue that one or more Worker Entities poll for Tasks. It is a name, not a resource you provision.

Task Queues are routing, not Kafka

A Task Queue is created on demand when a Workflow Execution, Activity, or Nexus Operation is invoked, or when a Worker Process subscribes to start polling. No registration, no schema, no partition planning. When a named Task Queue is created, individual Task Queues for Workflows, Activities, and Nexus are created using the same name.

Workers poll via synchronous RPC, so a Worker polls only when it has spare capacity and cannot overload itself; many Workers polling the same name share the load; and Workflow and Activity Tasks persist in the queue — when a Worker Process goes down, the messages remain until a Worker recovers and can process them.

One constraint follows: any Worker can pick up any Task on a given Task Queue, so all Workers listening to a Task Queue must register all Workflows and Activities that live on that queue. A Worker polling a Task for a type it does not know fails that Task — retryable, not fatal, but it bounces until a capable Worker appears. Separate queue names split the agent fleet into lanes later (module 10); today you need one, agent-runs.

One Workflow, one Activity, one durable timer

The whole program the lab runs, shaped after the samples' hello_activity.py, is the first stage of AgentRun — the one program every module of this course grows. plan() decides one step, call_llm carries it out, a durable timer stands in for the waits every real agent has, and the run returns its answer:

import asyncio
import hashlib
from dataclasses import dataclass
from datetime import timedelta

from temporalio import activity, workflow


@dataclass
class Step:
    prompt: str
    key: str


@activity.defn
async def call_llm(prompt: str, key: str) -> str:
    return f"answer to: {prompt}"            # the real model call lives here from module 3 on


@workflow.defn
class AgentRun:
    def __init__(self) -> None:
        self.goal = ""
        self.step = 0

    def plan(self) -> Step:                  # Workflow code: a pure function of state, never a model call
        digest = hashlib.sha256(self.goal.encode()).hexdigest()[:12]
        return Step(prompt=self.goal, key=f"{workflow.info().workflow_id}:{self.step}:{digest}")

    @workflow.run
    async def run(self, goal: str, sleep_seconds: int = 30) -> str:
        self.goal = goal
        step = self.plan()
        answer = await workflow.execute_activity(
            call_llm, args=[step.prompt, step.key], start_to_close_timeout=timedelta(seconds=10)
        )
        await asyncio.sleep(sleep_seconds)   # a durable timer, not a thread sleep
        return answer

@workflow.defn marks a Workflow Definition and @workflow.run its one async entry point. plan() is Workflow code: it reads the Workflow's own state and returns a decision, and it never calls a model — module 3 is what happens when you break that rule. workflow.execute_activity(...) asks the Service to run an Activity — an ordinary Python function that may do I/O, here call_llm — and waits for the result (the mandatory start_to_close_timeout is module 4's subject; the key becomes its idempotency key there). And asyncio.sleep(30), inside a Workflow, is a Timer.

The upstream definition of a Timer is worth keeping verbatim: Timers in Temporal are persisted, meaning that even if your Worker or Temporal Service is down when the time period completes, as soon as your Worker and Temporal Service become available, the call that is awaiting the Timer in your Workflow code will resolve, causing execution to proceed. Workers consume no additional resources while waiting for a Timer to fire, so a single Worker can await millions of Timers concurrently. Treat the duration as a minimum, rounded up slightly by scheduling latency.

The Worker registers the Workflow and the Activity on the queue and polls; the starter's Client starts one execution with a Workflow ID you choose — the agent's id.

from temporalio.client import Client
from temporalio.worker import Worker

TASK_QUEUE = "agent-runs"      # deployment configuration: the Worker and the Client must agree on it,
                              # and it stays out of the Workflow module (labs/common/ holds it)

client = await Client.connect("localhost:7233")           # namespace defaults to "default"

worker = Worker(client, task_queue=TASK_QUEUE,
                workflows=[AgentRun],
                activities=[call_llm])
await worker.run()                                        # in one process

handle = await client.start_workflow(                     # in another
    AgentRun.run, "Summarize the Temporal docs", id="agent-42", task_queue=TASK_QUEUE
)
print(await handle.result())

What one execution looks like, end to end

The upstream architecture page walks through fifteen steps; here is their shape for the lab's Workflow. Commands and Events get their precise definitions in module 2.

1. The Client calls StartWorkflowExecution with the Workflow type, input and Task Queue name. The History Service writes WorkflowExecutionStarted and WorkflowTaskScheduled and asks Matching to put a Workflow Task on agent-runs.
2. Your Worker receives it and runs AgentRun.run — plan() included — until it must wait: the execute_activity. It does not run the Activity; it sends a Command back ("schedule this Activity"). The Service records ActivityTaskScheduled and queues an Activity Task.
3. A Worker — possibly the same one — polls that Task, runs call_llm, reports the result. The Service records ActivityTaskStarted and ActivityTaskCompleted and schedules a Workflow Task to wake the Workflow.
4. A Worker picks it up, restores the Workflow to where it was — answer now holds the result — and reaches asyncio.sleep(30). Another Command: "start a timer". The Service records TimerStarted. No process is now waiting on anything. This is when you kill the Worker.
5. Thirty seconds later the Service's clock fires the timer, records TimerFired, and schedules a Workflow Task. Whichever Worker polls next — restarted or on another machine — reconstructs the state, including an answer it never computed, and reaches return.
6. run returns, so the Worker sends CompleteWorkflowExecution; the Service records WorkflowExecutionCompleted; handle.result() returns an answer produced by a process that no longer exists.

The cycle — Workflow Task, Commands, Events, new Tasks — repeats as needed. The Service never called your code; it only answered polls. Between steps 4 and 5 no process held this Workflow's state and no Workflow code ran; the execution stayed open because the timer and the history were in the Service.

Workflow ID and Run ID

You chose agent-42. That is the Workflow Id: a customizable, application-level identifier for a Workflow Execution that is unique to an Open Workflow Execution within a Namespace. It is meant to be a business-process identifier — for the agent runtime, the agent id. Temporal guarantees that there can be at most one Workflow Execution with a given ID running at any point in time, so two "start agent-42" requests cannot produce two agents.

The Service assigned the second identifier. A Run Id is a globally unique, platform-level identifier for a Workflow Execution. Retries, Continue-As-New (module 6) and Resets create a chain of runs sharing a Workflow Id, each with a fresh Run Id. Do not branch on the current Run Id in Workflow code; a retry changes it, which leads to non-determinism — module 3.

Where was the program counter stored?

Where was the program counter while the Worker was dead? Nowhere. There is no persisted program counter: Temporal does not serialize the Python stack, checkpoint the heap or snapshot the process, and no memory image of your Worker ever left the machine.

What the Service persists is an Event History: an append-only, durably persisted record of every interaction this execution had with the Service — started with this input, scheduled this Activity, got this result, started this timer. A Worker reconstructs where execution logically is by re-executing the deterministic Workflow code against that record: the SDK runs your Workflow function from the top and, at each await, hands it the recorded answer instead of doing the work again, until the record runs out and the code is genuinely at a new line. Temporal stores enough history to reconstruct where execution logically is; it never stores the position itself. That reconstruction is replay, the whole of module 2.

Before you read it, write your own one-paragraph answer; comparing it against module 2 is the fastest way to find what you have not yet internalised.

What the lab will test

If the Worker dies after the Service starts the timer, draw two columns: what the Service can record alone and what waits for a Worker. In Lab 1, check both columns against the Event History and Worker identities.

Try it

Open the Lab 1 browser workspace and follow its three checkpoints:

• Start, then crash: prepare the lab, start one Worker and one Workflow, and crash the Worker as soon as its call_llm output appears. The Activity has completed and the timer is pending; record the Worker identity from history after the crash.
• Observe no-Worker progress: leave the Worker dead until the timer and sticky-queue timeout have elapsed. Refresh history and identify the Events the Service appended without application compute.
• Resume the existing run: start only the Worker, not another Workflow. Verify that a different Worker identity finishes agent-42 and that replay does not repeat the completed model call.

Then complete the Worker-versus-Service ownership table and write your own answer to “where was the program counter?” before opening the lab's collapsed answer. The optional two-Worker extension requires two terminal windows; the browser workspace intentionally runs only one process per action card.

Lab page: Open the guided lab — the exercises, every source file, and the local setup steps.