Operating Temporal in Production

AgentRun leaves the laptop

For nine modules AgentRun ran on one Task Queue, agent-runs, with one Worker you could kill -9. In production call_llm needs a rate-limited API key and execute_tool may need a GPU or a live Chromium process. The Workflow decides what happens next; it must not care which machine does it. This page is the operator's view of that split, and the boundary where Temporal stops and Kubernetes, Kafka and Postgres begin. Appendices A1 and A3 are the long form of the first four sections.

Task Queues as compute lanes

Keep the upstream definition: a Task Queue is a lightweight, dynamically allocated queue that one or more Worker Entities poll for Tasks. Task Queues don't require explicit registration; they're created on demand when a Workflow Execution or Activity is invoked, or when a Worker Process subscribes to start polling. So a name mismatch between Client and Worker is not an error: it creates two different Task Queues, the Worker receives nothing, and the Workflow Execution does not progress. Spell every name once, in a constant both sides import.

Two rules make a queue name a contract about capability. Workers pull — a Worker polls only when it has spare capacity, so a queue with no pollers is stalled, not broken. And any Worker can pick up any Task on a given Task Queue, so all Workers listening to a Task Queue must register all Workflows and Activities that live on that queue. If gpu-tools Workers must register every Activity on gpu-tools, the set of Activities on gpu-tools is exactly the set that needs a GPU.

Module 10 splits agent-runs into five lanes, each with its own Worker pool:

An Activity inherits its Workflow's Task Queue unless you say otherwise, so routing is an opt-in on every call, chosen in Workflow code:

LANE_FOR_TOOL = {"search": CPU_TOOLS, "embed": GPU_TOOLS, "open_page": BROWSER_TOOLS}

answer = await workflow.execute_activity(
    call_llm, args=[prompt, key], task_queue=PREMIUM_MODELS,
    start_to_close_timeout=timedelta(minutes=5), retry_policy=LLM_RETRY,
)
result = await workflow.execute_activity(
    execute_tool, call, task_queue=LANE_FOR_TOOL[call.tool],
    start_to_close_timeout=timedelta(minutes=30),
    heartbeat_timeout=timedelta(seconds=30), retry_policy=TOOL_RETRY,
)

LANE_FOR_TOOL is read during a Workflow Task, so the lane a step went to is replayed like any decision; moving a tool between lanes mid-run is a module 9 change. The agent-workflows pool is deliberately boring — Workflow types only — so you can run many and redeploy twice a day. The browser lane is the stateful case: open_page returns a per-process queue name, and click and extract go to it with a short schedule_to_start_timeout, so a vanished host becomes an Activity failure the Workflow handles by reopening the page (A1).

Appendix A1 is the long form of this section: lane naming, Worker sizing, and every place a Task Queue name is read.

Backlog is the scaling signal

Because Workers pull, "is this lane under-provisioned?" has a direct answer: is work waiting? For each Task Queue and Task type the Service reports (as of 2026-09; see upstream page) ApproximateBacklogCount, ApproximateBacklogAge — the age of the Task at the head of the queue — and BacklogIncreaseRate, the net Tasks per second added. Upstream says to rely on count and age for scaling decisions; the increase rate reliably says whether a backlog older than a few seconds is growing.

temporal task-queue describe --task-queue gpu-tools --task-queue-type activity

The same command lists pollers and their last poll time: under five minutes usually means every slot is full; over five minutes usually means the Workers are gone, since a Worker drops off the list after 5 minutes without a poll (as of 2026-09; see upstream page). On the SDK side the matching metric is activity_schedule_to_start_latency; rising on one lane while the others stay flat is the signature of a starved fleet.

So one autoscaling rule for all five lanes: scale on the lane's Activity backlog age, not on CPU. A gpu-tools pod at 20% CPU with a five-minute backlog age is the bottleneck; a cpu-tools pod at 90% CPU with an empty backlog is fine.

Schedule-to-Start latency and backlog age are operational signals for Worker capacity. Backlog age rises while Tasks wait; latency records their wait when a Worker eventually picks them up. A Schedule-to-Start timeout is a non-retryable application deadline: use it when the Workflow has a concrete recovery path if no Worker picks up the Task. Lab 10 sets a short timeout to demonstrate that failure boundary; it is not the normal autoscaling mechanism.

The premium-models lane adds a knob the Workflow never sees: global Task Queue rate limiting works across all Workers for a Task Queue and is frequently used to limit load on a downstream service. Cap the lane at what the model provider allows (max_task_queue_activities_per_second on the Worker, as of 2026-09; see upstream page) and call_llm simply waits in the backlog.

Search Attributes: finding one run among thousands

Event History gives you exact for one run. Visibility gives you find across all of them. It is a search index the Service updates asynchronously — eventually consistent, briefly stale — so use it to find and filter, never to read the state of one execution; for that use DescribeWorkflowExecution or the status Query.

A Search Attribute is an indexed key-value field on a Workflow Execution's Visibility record. The default ones you will query: WorkflowType, ExecutionStatus, TaskQueue, StartTime, CloseTime, HistoryLength (closed runs only), and, from module 9, TemporalWorkerDeploymentVersion and TemporalWorkflowVersioningBehavior. Custom ones you create once per Namespace; Keyword is the right default for identifiers:

temporal operator search-attribute create --name AgentStatus --type Keyword
temporal operator search-attribute create --name CurrentStep --type Int
temporal operator search-attribute create --name Owner --type Keyword

Three upstream rules. Values are stored unencrypted and bypass any Payload Codec, so no PII: Owner is an opaque account id, never an email. Values live only as long as the Workflow, and carry over a continue-as-new by default. And they are not a state store — the Workflow's variables plus the status Query are.

AgentRun sets Owner at start and upserts the rest where its state changes (names verified against temporalio 1.32.0, as of 2026-09; see upstream page):

from temporalio.common import SearchAttributeKey

AGENT_STATUS = SearchAttributeKey.for_keyword("AgentStatus")
CURRENT_STEP = SearchAttributeKey.for_int("CurrentStep")

def _publish(self, status: str) -> None:
    self.status = status                              # the Query reads this — authoritative
    workflow.upsert_search_attributes([               # the fleet view reads this — eventual
        AGENT_STATUS.value_set(status),
        CURRENT_STEP.value_set(self.step),
    ])

Call it from the pause handler too, so a paused run is findable. The upsert is a Command recorded in History, and replays like any other.

A List Filter is an SQL-like string that works identically in the Web UI filter box, temporal workflow list --query, and client.list_workflows():

WorkflowType = 'AgentRun' AND AgentStatus = 'paused'
Owner = 'acct_7f3a' AND ExecutionStatus = 'Running'
WorkflowType = 'AgentRun' AND CurrentStep > 500 AND ExecutionStatus = 'Running'
WorkflowType = 'AgentRun' AND ExecutionStatus IN ('Failed', 'TimedOut', 'Terminated')
  AND CloseTime > '2026-09-10T08:00:00Z'
WorkflowId STARTS_WITH 'agent-42'          -- the parent and its ResearchAgent children

temporal workflow count --query "WorkflowType = 'AgentRun' GROUP BY ExecutionStatus" gives the fleet at a glance; counts are approximate and GROUP BY works only there and only on ExecutionStatus (as of 2026-09; see upstream page).

Appendix A3 is the long form: the full Search Attribute catalog, List Filter syntax, and answering each operator question from history.

The operator's five questions

A run is stuck. Answer these in order; each has one surface that answers it.

Visibility answers only the fifth question, and only its "how many, which ones" half. The first four are History and Describe, per execution, strongly consistent.

Priority and Fairness on the GPU lane

gpu-tools is one lane shared by every tenant, and the enterprise tenant's 4,000-document embedding job should not make a free-tier user's single tool call wait an hour. Priority and Fairness can arbitrate which queued work reaches a shared Worker pool; your compute platform still owns GPU placement and lifecycle — which node the Worker runs on, whether a GPU is free, and what happens when it is preempted. Upstream offers the two dispatch controls inside one Task Queue (as of 2026-09; see upstream page — verify status at regeneration time):

• Priority — a priority_key in [1, 5], 1 highest, default 3. Each level is a sub-queue: all priority-1 Tasks start before the first priority-2 Task. Activities and Child Workflows inherit their Workflow's priority unless they set their own. Upstream states Priority is enabled by default in both Temporal Cloud and self-hosted Temporal.
• Fairness — a fairness_key makes a virtual queue per tenant, dispatched round-robin in proportion to fairness_weight (default 1.0; weight 2.0 is dispatched twice as often). Upstream states Fairness is available for self-hosted and Cloud; self-hosted enables it with the dynamic config matching.enableFairness, and on Cloud it is a paid, per-Namespace toggle.

Combined, the next Task is chosen by three rules in order: priority tier, strict; fairness key within the tier, weighted; FIFO within a key. For AgentRun, priority separates the interactive run someone is watching from the batch run nobody is; fairness keeps tenants from starving each other within a tier. The Python shape upstream shows:

from temporalio.common import Priority

result = await workflow.execute_activity(
    execute_tool, call, task_queue=GPU_TOOLS,
    priority=Priority(
        priority_key=1 if self.interactive else 5,   # enterprise/interactive vs batch
        fairness_key=self.owner,                      # one virtual queue per tenant
        fairness_weight=TENANT_WEIGHT[self.tier],     # e.g. enterprise 5.0, team 3.0, free 1.0
    ),
    start_to_close_timeout=timedelta(minutes=30), heartbeat_timeout=timedelta(seconds=30),
)

Operators can override a key's weight on the queue without a deploy (temporal task-queue config set --fairness-key-weight enterprise=5.0). Two limitations: both controls are enforced within a single Task Queue partition, so under imbalanced partitions a low-priority Task can go first; and the weight applies at schedule time, so it cannot throttle Tasks already in the backlog. Neither control places a job on a GPU or frees one; they order the backlog that the pool behind the lane drains. The claim carries a date, and lab 10.3 records what the dev server in front of you actually does.

Where Temporal ends

The capstone's tenth question. Temporal decides where execution logically is: which step agent-42 is on, which Activity attempt is outstanding, which lane it waits on. Three neighbours own what they own.

Kubernetes owns where processes run. The Temporal Service never executes your code; it orchestrates State Transitions and hands Tasks to the next available Worker. So one Deployment per lane — the GPU lane on the GPU node pool, the browser lane with a Chromium sidecar, premium-models with the secret mounted — each autoscaled on its lane's backlog age. When a GPU node is preempted, Kubernetes reschedules the pod; separately, Temporal times out the attempt's heartbeat and re-queues the Task on gpu-tools. Neither told the other anything. Set the Worker identity explicitly (Worker(..., identity=f"{pod}@{lane}")) so a broken history names the pod.

Kafka owns streams, not execution. A Task Queue is not a durable log consumers read at their own pace; a Task is a short-lived instruction to some Worker, and the record of what happened is History, never the queue. Outside events reach AgentRun as Signals — a Kafka consumer calling Signal-With-Start is the bridge — and events it emits leave through an Activity that produces to a topic. Do not model the agent loop as a topic with offsets.

Postgres owns your system of record. History answers what ran: this attempt started, that one completed with this payload. It does not answer whether the result was good, what the provider actually charged, or what the customer now owns. Those are rows your Activities write, keyed by the idempotency keys from module 4, and they outlive retention. The third of the five questions is answered half in History and half in Postgres.

If every Worker process dies now, History and the queued Tasks survive in the Service; Kubernetes brings the pods back; the Workers poll their lanes; replay reconstructs every run; and nothing already written to your database is written twice, because the keys are yours.

What the lab will test

When the GPU Worker pool disappears, what can an operator learn from a Task Queue, Event History, and Search Attributes? In Lab 10, find one concrete answer on each surface.

Try it

Lab 10 is in labs/10-operating-temporal-in-production/:

• 10.1 — split AgentRun across lanes: agent-workflows, cpu-tools, gpu-tools (mock), premium-models, one Worker pool per lane from separate terminals. Kill the GPU pool mid-run; read the backlog and pollers with temporal task-queue describe; bring it back and watch the backlog drain.
• 10.2 — read a live run from the UI and the CLI: create AgentStatus, CurrentStep, Owner, run a List Filter for paused runs, then answer the five questions about a run you did not start, naming the surface you used for each.
• 10.3 — set priority and fairness on the GPU lane: two tenants with different weights and an interactive run at priority 1 against a batch backlog; record the dispatch order you observe against the one the rules predict, and note the dev server version you ran it on.

Lab page: Open the guided lab — the exercises, every source file, and the local setup steps.