Cancellation and Compensation

Step four fails

Your agent runtime provisions a model server in four steps: reserve a GPU, allocate a sandbox on it, start the model, register the endpoint so tool calls can reach it. The first three succeed. The fourth fails permanently: the endpoint registry rejects the name, and no retry will change that.

Temporal has already done what it can: it retried register_endpoint under its Retry Policy until the budget ran out, and it surfaces the failure to your Workflow as an ActivityError. What it cannot do is release the GPU. Nothing in the platform knows that a reserved GPU must be given back, or that stopping a model comes before releasing its sandbox. That is your rollback semantics, and it lives in your Workflow code.

Retry versus compensation

Retry repeats an operation that has not succeeded yet, until it succeeds or its budget ends. It is a policy: you configure it on the Activity and Temporal executes it with no Workflow code involved.

Compensation is a different operation that semantically undoes one that did succeed, because something later failed. release_gpu is not "reserve_gpu again"; it is a separate Activity with its own contract. Temporal retries for you; it does not compensate for you, because it has no idea what "undo" means for your resources.

Retry is configuration; compensation is architecture. The Saga pattern names a chain of steps in which every successful step registers the compensation that would undo it.

The saga as durable code

In a durable Workflow the Saga is ordinary code. After each step succeeds, append its compensation to a list held in Workflow state. If a later step fails, run the list in reverse.

from datetime import timedelta
from typing import Awaitable, Callable

from temporalio import workflow
from temporalio.common import RetryPolicy

with workflow.unsafe.imports_passed_through():
    from agent_runtime.activities import (
        allocate_sandbox, register_endpoint, release_gpu, release_sandbox,
        reserve_gpu, start_model, stop_model, unregister_endpoint,
    )

STEP = timedelta(minutes=5)
UNDO_RETRY = RetryPolicy(maximum_attempts=10)


@workflow.defn
class ProvisionModelServer:
    def __init__(self) -> None:
        self.compensations: list[Callable[[], Awaitable[None]]] = []

    def _on_failure(self, undo_activity, arg) -> None:
        async def undo() -> None:
            await workflow.execute_activity(
                undo_activity, arg,
                start_to_close_timeout=STEP, retry_policy=UNDO_RETRY,
            )
        self.compensations.append(undo)

    @workflow.run
    async def run(self, job_id: str) -> str:
        try:
            gpu = await workflow.execute_activity(
                reserve_gpu, job_id, start_to_close_timeout=STEP)
            self._on_failure(release_gpu, gpu)

            sandbox = await workflow.execute_activity(
                allocate_sandbox, gpu, start_to_close_timeout=STEP)
            self._on_failure(release_sandbox, sandbox)

            model = await workflow.execute_activity(
                start_model, sandbox, start_to_close_timeout=STEP)
            self._on_failure(stop_model, model)

            endpoint = await workflow.execute_activity(
                register_endpoint, model, start_to_close_timeout=STEP)
            self._on_failure(unregister_endpoint, endpoint)
            return endpoint
        except Exception:
            for undo in reversed(self.compensations):
                await undo()
            raise

Three things make this durable rather than merely tidy.

The list is Workflow state. If the Worker dies after stop_model ran but before release_sandbox starts, the replacement Worker replays the history: every forward step and every completed compensation comes back from recorded Activity results, the same code path re-appends the same closures, and execution resumes at the first compensation with no result in history. Lab 7.2 is this scenario.

Each compensation is an Activity with its own Retry Policy. release_gpu talks to a scheduler that may be briefly unavailable; give it a bounded retry rather than assuming rollback is cheap.

The except clause catches Exception, not BaseException, on purpose. Cancellation arrives as asyncio.CancelledError, which is a BaseException, and it deserves its own handling (below). The upstream compensation sample catches BaseException and then checks explicitly for asyncio.CancelledError and temporalio.exceptions.FailureError; either style works once you have decided what a cancel should undo.

Compensations must be idempotent too

Compensations are Activities, so everything module 4 said about at-least-once execution applies to them. release_gpu(reservation_id) may run twice: once on a Worker that died after the release went through but before the result was recorded, and once on its replacement. Releasing a reservation that is already released must be a no-op, not an error, or the rollback fails halfway and leaves the sandbox allocated.

Key each compensation on the identifier the forward step returned (reservation_id, sandbox_id, endpoint_id), never on the job alone: the keys that made the forward path safe make the reverse path safe.

Cancellation is a request

Cancellation and termination are both ways to interrupt an execution, and they are not the same thing. Upstream states the distinction precisely; keep the wording.

Canceling a Workflow provides a graceful way to stop Workflow Execution. This action resembles sending a SIGTERM to a process.

• The system records a WorkflowExecutionCancelRequested event in the Event History.
• A Workflow Task gets scheduled to process the cancelation.
• The Workflow code can handle the cancelation and execute any cleanup logic.
• The system doesn't forcefully stop the Workflow.

Terminating a Workflow forcefully stops Workflow Execution. This action resembles killing a process.

• The system records a WorkflowExecutionTerminated event in the Event History.
• The termination forcefully and immediately stops the Workflow Execution.
• The Workflow code gets no chance to handle termination.
• A Workflow Task doesn't get scheduled.

In most cases, canceling is preferable because it allows the Workflow to finish gracefully. Terminate only if the Workflow is stuck and cannot be canceled normally.

From the client, each is one call on the handle:

handle = client.get_workflow_handle("agent-run-7f3a")
await handle.cancel()      # cooperative: the Workflow gets to run cleanup
await handle.terminate()   # forceful: no Workflow code runs

In the Python SDK, cancellation is delivered as ordinary asyncio task cancellation. When the Service records the request, the SDK calls cancel() on the task running your @workflow.run method, and an asyncio.CancelledError is raised at the Workflow's next await. You can catch it, run cleanup, and re-raise so the execution closes as Cancelled. A cancellation request is a state, not an event: repeated requests are not delivered, only the first, so a Workflow that swallows the error cannot be asked again.

Cancelling the Workflow also cascades to what it is waiting on. The task awaiting an in-flight Activity is cancelled, and a cancellation request may be sent to the Activity, depending on the Activity's cancellation type.

How an Activity learns it was cancelled

An Activity does not get a signal, an exception from nowhere, or a killed process. Activities must heartbeat to receive cancellations from a Temporal Service. The request rides back on the heartbeat response. An Activity that never heartbeats runs to completion after the cancel while the Workflow has already moved on without it. Lab 7.3 shows exactly this, then adds heartbeats and watches the same Activity stop within one heartbeat interval.

Upstream lists when an Activity may receive Cancellation:

• The Activity was requested to be Cancelled. This can often cascade from Workflow Cancellation, but not always. SDKs have ways to stop Cancellation from cascading.
• The Activity was considered failed by the Server because any of the Activity timeouts have triggered (for example, the Server didn't receive a heartbeat within the Activity's Heartbeat timeout). The Cancelled Failure that the Activity receives will have message: 'TIMED_OUT'.
• The Workflow Run reached a Closed state, in which case the Cancelled Failure will have message: 'NOT_FOUND'.

An Activity may accept or ignore Cancellation: to allow Cancellation to happen, let the Cancellation Failure propagate; to ignore Cancellation, catch it and continue executing. Cancellation can only be requested a single time.

For a long-running tool in the agent, a 40-minute fine-tune or a sandboxed code run, this is the shape:

import time
from temporalio import activity
from temporalio.exceptions import CancelledError


@activity.defn
def run_long_tool(call: ToolCall) -> ToolResult:
    proc = start_tool_process(call)
    try:
        while proc.running():
            activity.heartbeat(proc.progress())  # cancellation arrives here
            time.sleep(1)
        return proc.result()
    except CancelledError:
        proc.kill()
        raise  # re-raise, or the Activity does not appear cancelled

An async def Activity receives asyncio.CancelledError instead; the pattern is the same. Set heartbeat_timeout on every long Activity, so the Service also notices when the Worker itself disappears.

Cancelling the Workflow does not terminate an external process. The request reaches the Worker, and the Worker raises inside your Activity function; nothing reaches the subprocess, the container, or the GPU job the Activity started. If the Activity heartbeats and checks, your except CancelledError is where proc.kill() happens. If it does not — or if the work was handed off to another system, as with the asynchronous completion in module 4 — the cancel closes the Workflow and the job keeps running until it finishes or something else stops it. For a GPU job that is a bill and a held reservation. So either the Activity cooperates (heartbeat plus a check, and a kill in the handler), or you build an external cancellation path — a cleanup Activity that tells the scheduler to stop the job by its id, run from the Workflow's cancel handler — and treat "the Workflow is Cancelled" as distinct from "the work has stopped".

Shielded cleanup

Cleanup after a cancel usually needs Activities of its own: stop the model, release the GPU. Workflow cancellation follows asyncio rules exactly, so cleanup that is in flight when a cancellation propagates can itself be cancelled. Upstream's pattern is a detached scope with asyncio.shield:

import asyncio
from datetime import timedelta
from temporalio import workflow


@workflow.defn
class AgentRun:
    @workflow.run
    async def run(self, goal: str) -> str:
        try:
            return await self._agent_loop(goal)
        except asyncio.CancelledError:
            workflow.logger.info("cancel requested; releasing resources")
            try:
                await asyncio.shield(
                    workflow.execute_activity(
                        release_gpu, self.gpu,
                        start_to_close_timeout=timedelta(minutes=2),
                    )
                )
            except asyncio.CancelledError:
                pass  # the shielded release still completes
            raise  # close as Cancelled, not Completed

The simplest upstream sample runs its cleanup Activity in a plain finally and it works, because the single cancellation has already been delivered and caught by then. The shield covers the case the finally does not: cleanup already running when the request lands. Lab 7.3 adds both and you read the history for ActivityTaskCancelRequested on the cleanup step.

Four ways an execution ends badly

An execution closes in one of six statuses. Two are happy (Completed, Continued-As-New). The other four are yours to tell apart in an incident:

On timeouts, upstream's guidance is that Workflows are designed to be long-running and resilient, so setting Workflow Timeouts is generally not recommended; if you need to perform an action after a specific period of time, use a Timer. The Workflow Execution Timeout defaults to infinite and covers retries and Continue-As-New; the Workflow Run Timeout defaults to the same value and covers a single run. An agent that should give up after 24 idle hours wants a timer and a deliberate cancel, not a timeout it cannot clean up after.

What the lab will test

A four-step tool sequence fails on the last step. Write the compensation order before running Lab 7; then cancel during a long Activity and compare that path with termination.

Try it

Lab 7 lives in labs/07-cancellation-and-sagas/ and has four exercises:

• 7.1 — implement the four-step saga above, make register_endpoint fail permanently, and read the reverse compensation off the Event History.
• 7.2 — kill the Worker during the second compensation; restart it and confirm the rollback finishes from where the history left it.
• 7.3 — cancel a Workflow during a long Activity, first without heartbeats (it runs to completion), then with heartbeats (it stops within one interval); then add shielded cleanup.
• 7.4 — terminate the same run instead of cancelling it: no cancel request, no finally, no rollback. The history stops at WorkflowExecutionTerminated while the tool process keeps running on the Worker and the sandbox stays allocated.

Lab page: Open the guided lab — the exercises, every source file, and the local setup steps.