Determinism

Break it on purpose

Module 2 ended on a promise the code has to keep: same code plus same history gives the same sequence of Commands. This module is about the code failing to keep it. You learn the rule by violating it three ways and reading the error each time; by the end you can read that message and name the line.

Upstream's definition is the one to hold on to: a Workflow Definition is deterministic if every execution of it produces the same Commands in the same sequence given the same input. Generally speaking, this means you must take care to ensure that any time your Workflow code is executed it makes the same Workflow API calls in the same sequence, given the same input.

The contract replay depends on

When the Workflow's code replays, the Commands that are emitted are compared with the existing Event History. If a corresponding Event already exists within the Event History that matches that Command, then the Execution progresses. Using the "Execute Activity" API generates the ScheduleActivityTask Command; on re-execution that Command is compared with the Event that is in the same location within the sequence, which must be an ActivityTaskScheduled Event where the Activity name is the same as what is in the Command. If a generated Command doesn't match what it needs to in the existing Event History, then the Workflow Execution returns a non-deterministic error.

Notice what replay checks: not your intent, variable values or log lines — only that Command N matches Event N. A Workflow that could have gone either way from the same history cannot be reconstructed, because the SDK's only way to know which way it went is history, which is exactly what a non-deterministic branch refuses to consult.

Upstream names two reasons a Command might be generated out of sequence, or the wrong Command generated altogether: code changes are made to a Workflow Definition that is in use by a running Workflow Execution; or there is intrinsic non-deterministic logic, such as inline random branching. This module is the second. The first is module 9.

The coin flip

The canonical break, in Lab 3.1's shape: an agent Workflow that randomly adds a one-second jitter before calling the model. This is separate from the lab's 15-second timer after each step.

import asyncio
import random
from datetime import timedelta

from temporalio import workflow

with workflow.unsafe.imports_passed_through():
    from activities import call_llm                     # call_llm(prompt: str, key: str) -> str


@workflow.defn
class CoinFlipAgent:
    @workflow.run
    async def run(self, doc: str) -> str:
        if random.random() < 0.5:                       # BROKEN: intrinsic non-determinism
            await asyncio.sleep(1)                      # one-second jitter, TimerStarted
        key = f"{workflow.info().workflow_id}:1"
        return await workflow.execute_activity(
            call_llm, args=[doc, key], start_to_close_timeout=timedelta(minutes=2)
        )

Trace it across two Workers.

Worker 1 drew 0.31 and started the jitter timer; history records TimerStarted. On a later Workflow Task, Worker 2 replays from the top and draws 0.72. It skips the timer and tries to schedule call_llm, but the SDK finds TimerStarted at that position in history. Replay cannot continue. Lab 3.1 reproduces this direction with random.Random().random() (the unseeded instance gets past the sandbox) and reports Activity machine does not handle this event: HistoryEvent(id: 5, TimerStarted); if the original run skipped the jitter, the mirror-image error is also possible.

The upstream Python walkthrough also uses a random number gating a four-hour sleep: the first run starts a Timer; replay skips it and produces ScheduleActivityTask where history expects StartTimer. That mismatch makes Replay fail. The replay sample's README shows the opposite mismatch, produced by adding a sleep before an existing Activity: Nondeterminism("Timer machine does not handle this event: HistoryEvent(id: 8, Some(ActivityTaskScheduled))") (as of 2026-09; see upstream page). Read it as "the code asked for a timer; at event 8, history has an Activity being scheduled instead." The event id is where to look; which machine appears in the error depends on which branch the original history took.

The fix is one token: workflow.random().random() returns a random.Random seeded per Workflow Execution, so both Workers draw the same value and emit the same Command. What changed is not the shape of the branch, only where its input comes from.

The rule

Workflow = decisions. Activity = effects.

A decision is a pure function of the Workflow's inputs and history. Anything that observes the world — clock, random source, network, filesystem, environment, another thread — is an effect; effects live in Activities, whose results are recorded and returned unchanged on replay. Upstream: all operations that do not purely mutate the Workflow Execution's state should occur through a Temporal SDK API.

The recorded result is what makes the next decision reproducible. When the Workflow asks an Activity for the model's answer, that answer is written into ActivityTaskCompleted; on replay the Workflow receives the same answer and makes the same choice about it, even though calling the model again would return different tokens. Activities are not a workaround for the determinism rule; they are the mechanism that lets non-deterministic work participate in a deterministic program.

Here is how that sorts the agent runtime:

The model's answer is an effect; what you do with it is a decision. That sentence settles most arguments about where a line of agent code belongs. Carry the rule as a heuristic rather than a formal definition: a Workflow is not only a decision function, it also holds durable logical state — self.step, self.goal, self.context_summary — and coordinates messages (module 5), Timers and Child Workflows (module 6), none of which is an effect in the sense above.

What the SDK hands you instead

Upstream's Python list is blunt: no threading, no randomness, no external calls to processes, no network I/O, no global state mutation, no system date or time. For each thing you would reach for, there is a replay-safe replacement.

from temporalio import workflow

now = workflow.now()                     # time of the current Workflow Task; identical on replay
rng = workflow.random()                  # random.Random seeded per Workflow Execution
pick = rng.randint(1, 100)
run_id = workflow.uuid4()                # deterministic UUID, not uuid.uuid4()
await asyncio.sleep(3600)                # a durable Timer, not a thread sleep (module 1)
await workflow.wait_condition(lambda: self.approved)   # block until state changes, replay-safe
workflow.logger.info("step %d", step)    # suppressed during replay, so no duplicate lines

workflow.now() is history time. During replay it returns the recorded timestamp of the Workflow Task being replayed, not the wall clock — so a branch on "is it past noon?" is safe with workflow.now() and unsafe with datetime.now(). Upstream's pseudocode for the unsafe version — sleep until noon if the local clock says morning, else run the afternoon Activity — is exactly lab 3.2. Time is not the problem; unrecorded time is.

On Timers, the Python guide adds: a Workflow can sleep for months, sleeping does not tie up the process, and you can run millions of Timers off a single Worker. That is how an agent rate-limits or backs off without anyone holding a thread.

workflow.unsafe.is_replaying() exists for interceptors and metrics. Never use it in business logic: branching on replay status breaks determinism by construction.

The sandbox

The Python SDK runs Workflow code in a sandbox to help prevent non-determinism errors. It has two parts. Global state isolation: on Workflow start the file defining the Workflow is imported into a newly created sandbox, so module-level state cannot leak between executions; a known set of modules, including the standard library and Temporal itself, is passed through from outside. Restrictions: proxy objects on modules block known non-deterministic calls, at import time and run time.

With the default restrictions the sandbox raises when Workflow code calls random.random() (and the other module-level random functions), datetime.datetime.now(), time.time(), time.sleep(), uuid.uuid4(), anything in threading, socket, subprocess or urllib.request, and open() (as of 2026-09, verified against Python SDK 1.32.0's SandboxRestrictions.default; see upstream page). Most classic mistakes therefore fail fast on the first execution, with a restriction error naming the call — before there is any history to replay.

That has a consequence for the lab. The coin flip above, run as written, produces a sandbox restriction error, not a non-determinism error. To see the real replay mismatch you must get an unrecorded value past the sandbox: wrap the call in with workflow.unsafe.sandbox_unrestricted():, or construct random.Random() yourself — instantiating Random is deliberately unrestricted so seeded generators keep working, but an unseeded one is seeded from the OS. Either fairly simulates the third-party library that mutates state internally, which the sandbox docs warn it cannot catch.

Two more things you will meet. Non-standard-library modules are reloaded for every Workflow run unless passed through: wrap imports of your Activities and dataclasses in with workflow.unsafe.imports_passed_through():, as every upstream example does. And the sandbox can be switched off per block, per Workflow (@workflow.defn(sandboxed=False)) or per Worker (UnsandboxedWorkflowRunner) — removing the checks, not the requirement.

Subtler breaks

The sandbox stops the obvious ones. These get through.

Iterating a set, or a dict whose insertion order differs between runs, and issuing one Command per element: the Commands come out in a different order, and order is what replay compares. Sort first.

Module-level mutable state shared across executions in one Worker — a cache, a counter. A passed-through module carries its globals with it, and a value that depends on which executions ran earlier in this process is one the next Worker will not reproduce.

Threads, concurrent.futures, or blocking I/O inside Workflow code: even without deadlocking the event loop, they introduce timing that can reorder awaits.

Changing the Workflow code itself between original run and replay. Upstream lists what may change safely — input parameters, return values and timeouts of Activities and Child Workflows; Signal arguments; a Timer's duration, except from or to zero in Python — and what may not: reordering, adding or removing any call that produces a Command. That list is the entry ticket to module 9.

Reading the error, and forcing a replay

You do not have to crash a Worker to test replay. The SDK's Replayer runs your current Workflow code against saved histories, exactly as a fresh Worker would, and reports every mismatch.

from temporalio.client import Client, WorkflowHistory
from temporalio.worker import Replayer

client = await Client.connect("localhost:7233")
histories = client.list_workflows('WorkflowId="coinflip-1"').map_histories()

replayer = Replayer(workflows=[CoinFlipAgent])
results = await replayer.replay_workflows(histories, raise_on_replay_failure=False)
print(results)      # WorkflowReplayResults(replay_failures={...}) — empty dict means clean

Run it, read the NondeterminismError, find the event id, and open that event in the Web UI. The Command your code emitted is on one side, the Event history recorded on the other, and the line is whichever await produced the Command. Module 8 turns this loop into a CI test.

What the lab will test

For each unsafe value—randomness, wall time, network response—say whether replay can recover it from history. In Lab 3, distinguish a sandbox rejection from a Command mismatch at a specific event ID.

Try it

Go to labs/03-determinism/. Each exercise breaks one constraint, forces a replay, and has you read the error.

• 3.1 Break determinism with randomness (the coin flip). Get past the sandbox as above; force a replay by restarting the Worker mid-Activity or with the Replayer. Capture the exact error text and the event id where history and code diverged. Fix with workflow.random().
• 3.2 Break it with wall-clock time: a branch on datetime.now(). See what the sandbox does; get past it; replay; read the error; fix with workflow.now().
• 3.3 Move a network read that picks the next step into an Activity that returns the decision input. Confirm the replay is clean, then explain in one sentence why the recorded result suffices.

Lab page: Open the guided lab — the exercises, every source file, and the local setup steps.