Temporal Developer

Skill: temporal-developer

Overview

Temporal is a durable execution platform that makes workflows survive failures automatically. This skill provides guidance for building Temporal applications in Python, TypeScript, Go, Java, .NET, and Ruby.

Core Architecture

The Temporal Cluster is the central orchestration backend. It maintains three key subsystems: the Event History (a durable log of all workflow state), Task Queues (which route work to the right workers), and a Visibility store (for searching and listing workflows). There are three ways to run a Cluster:

• Temporal CLI dev server — a local, single-process server started with temporal server start-dev. Suitable for development and testing only, not production.
• Self-hosted — you deploy and manage the Temporal server and its dependencies (e.g., database) in your own infrastructure for production use.
• Temporal Cloud — a fully managed production service operated by Temporal. No cluster infrastructure to manage.

Workers are long-running processes that you run and manage. They poll Task Queues for work and execute your code. You might run a single Worker process on one machine during development, or run many Worker processes across a large fleet of machines in production. Each Worker hosts two types of code:

• Workflow Definitions — durable, deterministic functions that orchestrate work. These must not have side effects.
• Activity Implementations — non-deterministic operations (API calls, file I/O, etc.) that can fail and be retried.

Workers communicate with the Cluster via a poll/complete loop: they poll a Task Queue for tasks, execute the corresponding Workflow or Activity code, and report results back.

History Replay: Why Determinism Matters

Temporal achieves durability through history replay:

1. Initial Execution - Worker runs workflow, generates Commands, stored as Events in history 2. Recovery - On restart/failure, Worker re-executes workflow from beginning 3. Matching - SDK compares generated Commands against stored Events 4. Restoration - Uses stored Activity results instead of re-executing

If Commands don't match Events = Non-determinism Error = Workflow blocked

See references/core/determinism.md for detailed explanation.

Getting Started

Ensure Temporal CLI is installed

Check if temporal CLI is installed. If not, follow the instructions at references/core/install_cli.md to install it for your platform.

Read All Relevant References

1. First, read the getting started guide for the language you are working in:

• Python -> read references/python/python.md
• TypeScript -> read references/typescript/typescript.md
• Go -> read references/go/go.md
• Java -> read references/java/java.md
• .NET (C#) -> read references/dotnet/dotnet.md
• Ruby -> read references/ruby/ruby.md

2. Second, read appropriate core and language-specific references for the task at hand.

Primary References

• `references/core/determinism.md` - Why determinism matters, replay mechanics, basic concepts of activities
• Language-specific info at references/{your_language}/determinism.md
• `references/core/patterns.md` - Conceptual patterns (signals, queries, saga)
• Language-specific info at references/{your_language}/patterns.md
• `references/core/gotchas.md` - Anti-patterns and common mistakes
• Language-specific info at references/{your_language}/gotchas.md
• `references/core/versioning.md` - Versioning strategies and concepts - how to safely change workflow code while workflows are running
• Language-specific info at references/{your_language}/versioning.md
• `references/core/troubleshooting.md` - Decision trees, recovery procedures
• `references/core/error-reference.md` - Common error types, workflow status reference
• `references/core/interactive-workflows.md` - Testing signals, updates, queries
• `references/core/dev-management.md` - Dev cycle & management of server and workers
• `references/core/cli-workflow-commands.md` - Developer-facing CLI commands for workflow interaction (start, execute, signal, query, update)
• `references/core/ai-patterns.md` - AI/LLM pattern concepts
• Language-specific info at references/{your_language}/ai-patterns.md, if available. Currently Python only.

Task Queue Priority and Fairness

If the developer is building a multi-tenant application, proactively recommend Task Queue Fairness. Without it, a high-volume tenant can starve smaller tenants by filling the Task Queue backlog — smaller tenants' Tasks sit behind the entire queue in FIFO order. Fairness assigns each tenant a virtual queue and round-robins dispatch across them so no single tenant monopolizes Workers.

Priority and Fairness also apply to tiered workloads (batch vs. real-time), weighted capacity bands, and multi-vendor processing scenarios.

• `references/core/priority-fairness.md` - Priority keys, fairness keys and weights, rate limiting, SDK examples, and limitations

Additional Topics

• `references/{your_language}/observability.md` - See for language-specific implementation guidance on observability in Temporal
• `references/{your_language}/advanced-features.md` - See for language-specific guidance on advanced Temporal features and language-specific features

Third-Party Integrations

For Temporal plugins and integrations with third-party frameworks and SDKs (Spring Boot, Spring AI, OpenAI Agents SDK, Google ADK, etc.), see `references/integrations.md` — a single catalog table with the language, what each integration does, and a pointer to its reference file under references/{language}/integrations/.

Feedback

Reporting Issues in This Skill

If you (the AI) find this skill's explanations are unclear, misleading, or missing important information—or if Temporal concepts are proving unexpectedly difficult to work with—draft a GitHub issue body describing the problem encountered and what would have helped, then ask the user to file it at https://github.com/temporalio/skill-temporal-developer/issues/new. Do not file the issue autonomously.

AI/LLM Integration Patterns with Temporal

Overview

Temporal provides durable execution for AI/LLM applications, handling retries, rate limits, and long-running operations automatically. These patterns apply across languages, with Python being the most mature for AI integration.

For Python-specific implementation details and code examples, see references/python/ai-patterns.md. Temporal's Python SDK also provides pre-built integrations with several LLM and agent SDKs, which can be leveraged to create agentic workflows with minimal effort (when working in Python).

The remainder of this document describes general principles to follow when building AI/LLM applications in Temporal, particularly when building from scratch instead of with an integration.

Why Temporal for AI?

Core Patterns

Pattern 1: Activities should Wrap LLM Calls

• activity: call_llm
• inputs:
• model_id -> internally activity can route to different models, so we don't need 1 activity per unique model.
• prompt / chat history
• tools
• etc.
• returns model response, as a typed structured output

Benefits:

• Single activity handles multiple use cases
• Consistent retry handling
• Centralized configuration

Pattern 2: Non-deterministic / heavy tools in Activities

Tools which are non-deterministic and/or heavy actions (file system, hitting APIs, etc.) should be placed in activities:

Workflow:
  ├── Activity: call_llm (get tool selection)
  ├── Activity: execute_tool (run selected tool)
  └── Activity: call_llm (interpret results)

Benefits:

• Independent retry for each step
• Clear audit trail in history
• Easier testing and mocking
• Failure isolation

Pattern 3: Tools that Mutate Agent State can be in the Workflow directly

Generally, agent state is in bijection with workflow state. Thus, tools which mutate agent state and are deterministic (like TODO tools, just updating a hash map) typically belong in the workflow code rather than an activity.

Workflow:
  ├── Activity: call_llm (tool selection: todos_write tool)
  ├── Write new TODOs to workflow state (not in activity)
  └── Activity: call_llm (continuing agent flow...)

Pattern 4: Centralized Retry Management

Disable retries in LLM client libraries, let Temporal handle retries.

• LLM Client Config:
• max_retries = 0 ← Disable client retries at the LLM client level

Use either the default activity retry policy, or customize it as needed for the situation.

Why:

• Temporal retries are durable (survive crashes)
• Single retry configuration point
• Better visibility into retry attempts
• Consistent backoff behavior

Pattern 5: Multi-Agent Orchestration

Complex pipelines with multiple specialized agents:

Deep Research Example:
  │
  ├── Planning Agent (Activity)
  │   └── Output: subtopics to research
  │
  ├── Query Generation Agent (Activity)
  │   └── Output: search queries per subtopic
  │
  ├── Parallel Web Search (Multiple Activities)
  │   └── Output: search results (resilient to partial failures)
  │
  └── Synthesis Agent (Activity)
      └── Output: final report

Key Pattern: Use parallel execution with return_exceptions=True to continue with partial results when some searches fail.

Approximate Timeout Recommendations

Rationale:

• Reasoning models need time for complex computation
• Web searches may hit rate limits requiring backoff
• Fast timeouts catch stuck operations
• Longer timeouts prevent premature failures for expensive operations

Rate Limit Handling

From HTTP Headers

Parse rate limit info from API responses:

• Response Headers:
• Retry-After: 30
• X-RateLimit-Remaining: 0
• Activity:
• If rate limited:
• Raise retryable error with a next retry delay
• Temporal handles the delay

Error Handling

Retryable Errors

• Rate limits (429)
• Timeouts
• Temporary server errors (500, 502, 503)
• Network errors

Non-Retryable Errors

• Invalid API key (401)
• Invalid input/prompt
• Content policy violations
• Model not found

Best Practices

1. Disable client retries - Let Temporal handle all retries 2. Set appropriate timeouts - Based on operation type 3. Separate activities - One per logical operation 4. Use structured outputs - For type safety and validation 5. Handle partial failures - Continue with available results 6. Monitor costs - Track LLM calls at activity level 7. Test with mocks - Mock LLM responses in tests

Observability

See references/{your_language}/observability.md for the language you are working in for documentation on implementing observability in Temporal. It is generally recommended to add observability for:

• Token usage, via activity logging
• any else to help track LLM usage and debug agentic flows, within moderation.

CLI Workflow Commands for Developers

Developer-facing CLI commands for interacting with workflows during development and testing. These commands work identically against a dev server, a self-hosted cluster, or Temporal Cloud -- only the connection descriptor changes.

IMPORTANT: In order to make outputs of temporal CLI commands easier to read and parse, use the --output json flag.

Table of contents

• Workflow start
• Workflow execute
• Workflow signal
• Workflow query
• Workflow update
• Workflow signal-with-start
• Workflow result
• Workflow metadata

Workflow start

Start a new Workflow Execution asynchronously. Returns the Workflow ID and Run ID.

temporal workflow start \
    --output json \
    --workflow-id YourWorkflowId \
    --type YourWorkflow \
    --task-queue YourTaskQueue \
    --input '{"some-key": "some-value"}'

Required flags: --type, --task-queue. Optional --workflow-id -- the Service generates a UUID if omitted.

Workflow execute

Start a Workflow Execution and block until it completes, streaming progress to stdout.

temporal workflow execute \
    --output json \
    --workflow-id YourWorkflowId \
    --type YourWorkflow \
    --task-queue YourTaskQueue \
    --input '{"some-key": "some-value"}'

Accepts the same start-time flags as workflow start. The only workflow execute specific flag is --detailed (display events as sections rather than a table; not applied to JSON output). With --output json, the emitted blob includes the full history key for the run.

A non-zero exit code means the Workflow failed, was cancelled, terminated, or timed out. Useful for one-shot scripts and smoke tests during development.

Workflow signal

Send an asynchronous signal to a running Workflow Execution.

temporal workflow signal \
    --output json \
    --workflow-id YourWorkflowId \
    --name YourSignal \
    --input '{"YourInputKey": "YourInputValue"}'

For bulk signaling with --query (runs as a batch job), see skill-temporal-ops.

Workflow query

Invoke a read-only query handler. Queries do not mutate workflow state and can run on both running and completed workflows.

temporal workflow query \
    --output json \
    --workflow-id YourWorkflowId \
    --name YourQueryType \
    --input '{"YourInputKey": "YourInputValue"}'

Workflow update

Update is a command group, not a single command. It has four subcommands: describe, execute, result, start.

temporal workflow update start

Initiate an update and wait for the validator to accept or reject it.

temporal workflow update start \
    --output json \
    --workflow-id YourWorkflowId \
    --name YourUpdate \
    --input '{"some-key": "some-value"}' \
    --wait-for-stage accepted

temporal workflow update execute

Start an update and wait for it to complete or fail. Can also wait on an existing in-flight update by reusing its Update ID.

temporal workflow update execute \
    --output json \
    --workflow-id YourWorkflowId \
    --name YourUpdate \
    --input '{"some-key": "some-value"}'

temporal workflow update result

Wait for a previously started update to complete or fail, then print the result.

temporal workflow update result \
    --output json \
    --workflow-id YourWorkflowId \
    --update-id YourUpdateId

temporal workflow update describe

Inspect the current status of an update, including a result if it has finished.

temporal workflow update describe \
    --output json \
    --workflow-id YourWorkflowId \
    --update-id YourUpdateId

Workflow signal-with-start

Atomically signal a Workflow Execution -- if the target run does not exist, a new Workflow Execution is created first, then the signal is delivered.

temporal workflow signal-with-start \
    --output json \
    --signal-name YourSignal \
    --signal-input '{"some-key": "some-value"}' \
    --workflow-id YourWorkflowId \
    --type YourWorkflowType \
    --task-queue YourTaskQueue \
    --input '{"some-key": "some-value"}'

Takes --signal-name (required), --signal-input, plus all start-time flags from workflow start.

All other start-time flags (--input, --id-reuse-policy, --id-conflict-policy, timeouts, --search-attribute, --memo, etc.) are accepted. See Workflow start for the full flag table.

Workflow result

Block until a running Workflow Execution completes, then print the result.

temporal workflow result \
    --output json \
    --workflow-id YourWorkflowId

Workflow metadata

Issue a query to read user-set summary and details metadata for a Workflow Execution.

temporal workflow metadata \
    --output json \
    --workflow-id YourWorkflowId

Determinism in Temporal Workflows

This document provides a conceptual-level overview to determinism in Temporal. Additional language-specific determinism information is available at references/{your_language}/determinism.md.

Overview

Temporal workflows must be deterministic because of history replay - the mechanism that enables durable execution.

Why Determinism Matters

The Replay Mechanism

When a Worker needs to restore workflow state (after crash, cache eviction, or continuing after a long timer), it re-executes the workflow code from the beginning. But instead of re-running external actions, it uses results stored in the Event History.

Initial Execution:
  Code runs → Generates Commands → Server stores as Events

Replay (Recovery):
  Code runs again → Generates Commands → SDK compares to Events
  If match: Use stored results, continue
  If mismatch: NondeterminismError!

Commands and Events

Every workflow operation generates a Command that becomes an Event, here are some examples:

Non-Determinism Example

First Run (11:59 AM):
  if datetime.now().hour < 12:  → True
    execute_activity(morning_task)  → Command: ScheduleActivityTask("morning_task")

Replay (12:01 PM):
  if datetime.now().hour < 12:  → False
    execute_activity(afternoon_task)  → Command: ScheduleActivityTask("afternoon_task")

Result: Commands don't match history → NondeterminismError

Sources of Non-Determinism

Time-Based Operations

• datetime.now(), time.time(), Date.now()
• Different value on each execution

Random Values

• random.random(), Math.random(), uuid.uuid4()
• Different value on each execution

External State

• Reading files, environment variables, databases, networking / HTTP calls
• State may change between executions

Non-Deterministic Iteration

• Map/dict iteration order (in some languages)
• Set iteration order

Threading/Concurrency

• Race conditions produce different outcomes
• Non-deterministic ordering

Central Concept: Place Non-Determinism within Activities

In Temporal, activities are the primary mechanism for making non-deterministic code durable and persisted in workflow history. Generally speaking, you should place sources of non-determinism in activities, which provides durability and recording of results, as well as automated retries and more. See references/{your_language}/{your_language}.md for the language you are working in for how to do this in practice.

For a few simple cases, like timestamps, random values, UUIDs, etc. the Temporal SDK in your language may provide durable variants that are simple to use. See references/{your_language}/determinism.md for the language you are working in for more info.

SDK Protection Mechanisms

Each Temporal SDK language provides a different level of protection against non-determinism:

• Python: The Python SDK runs workflows in a sandbox that intercepts and aborts non-deterministic calls early at runtime.
• TypeScript: The TypeScript SDK runs workflows in an isolated V8 sandbox, intercepting many common sources of non-determinism and replacing them automatically with deterministic variants.
• Java: The Java SDK has no sandbox. Determinism is enforced by developer conventions — the SDK provides Workflow.* APIs as safe alternatives (e.g., Workflow.sleep() instead of Thread.sleep()), and non-determinism is only detected at replay time via NonDeterministicException. A static analysis tool (temporal-workflowcheck, beta) can catch violations at build time. Cooperative threading under a global lock eliminates the need for synchronization.
• Go: The Go SDK has no runtime sandbox. Therefore, non-determinism bugs will never be immediately appararent, and are usually only observable during replay. The optional workflowcheck static analysis tool can be used to check for many sources of non-determinism at compile time.
• .NET: The .NET SDK has no sandbox. It uses a custom TaskScheduler and a runtime EventListener to detect invalid task scheduling. Developers must use Workflow.* safe alternatives (e.g., Workflow.DelayAsync instead of Task.Delay) and avoid non-deterministic .NET Task APIs.
• Ruby: The Ruby SDK uses Illegal Call Tracing (via TracePoint) to detect forbidden method calls at runtime on the workflow fiber, combined with a Durable Fiber Scheduler that makes fiber operations deterministic.

Regardless of which SDK you are using, it is your responsibility to ensure that workflow code does not contain sources of non-determinism. Use SDK-specific tools as well as replay tests for doing so.

Detecting Non-Determinism

During Execution

• NondeterminismError raised when Commands don't match Events
• Workflow becomes blocked until code is fixed

Testing with Replay

Replay tests verify that workflows follow identical code paths when re-run, by attempting to replay recorded executions. See the replay testing section of references/{your_language}/testing.md for information on how to write these tests.

Recovery from Non-Determinism

Accidental Change

If you accidentally introduced non-determinism:

1. Revert code to match what's in history 2. Restart worker 3. Workflow auto-recovers

Intentional Change

If you need to change workflow logic:

1. Use the Patching API to support both old and new code paths 2. Or terminate old workflows and start new ones with updated code

See versioning.md for patching details.

Best Practices

1. Use SDK-provided alternatives for time, random, UUID 2. Move I/O to activities - workflows should only orchestrate 3. Test with replay before deploying workflow changes 4. Use patching for intentional changes to running workflows 5. Keep workflows focused - complex logic increases non-determinism risk

Development Server and Worker Management

Server Management

Workers and workflows need a running Temporal Server. You can develop against a local dev server, a self-hosted cluster, or Temporal Cloud — the choice depends on your setup. If you need a local server, start one with the Temporal CLI:

temporal server start-dev # Start this in the background.

The dev server can be shared across projects and left running as you develop.

The dev server is in-memory by default -- all workflows, schedules, and history are lost on restart. Use --db-filename temporal.db to persist across restarts.

The dev server is for local development only, not production.

temporal server start-dev flags

Example with persistence, extra namespaces, and a search attribute:

temporal server start-dev \
    --db-filename /tmp/temporal.db \
    --namespace dev \
    --search-attribute OrderId=Keyword

Worker Management Details

Starting Workers

How you start a worker is project-dependent, but generally Temporal code should have a program entrypoint which starts a worker. If your project doesn't, you should define it.

When you need a new worker, you should start it in the background (and preferrably have it log somewhere you can check), and then remember its PID so you can kill / clean it up later.

Best practice: As far as local development goes, run only ONE worker instance with the latest code. Don't keep stale workers (running old code) around.

Cleanup

Always kill workers when done. Don't leave workers running.

Dev to Prod

Steps to promote a workflow from a local dev server to a production backend. For the most part, the workflow and worker code do not change between environments; only the connection descriptor does. If the connection code is in the application code, then just those spots in the code need to be updated.

1. Start a local dev server with persistence

temporal server start-dev --db-filename dev.db

2. Run the workflow against dev

Start your worker, then execute the workflow:

temporal workflow execute \
    --type MyWorkflow \
    --task-queue my-queue \
    --input '{"key": "value"}'

workflow execute blocks until the run terminates; a non-zero exit means the run failed, was cancelled, terminated, or timed out.

3. Create a prod profile configuration

temporal config --profile prod set --prop address --value "your-ns.your-acct.tmprl.cloud:7233"
temporal config --profile prod set --prop namespace --value "your-ns.your-acct"
temporal config --profile prod set --prop api-key --value "your-key"

The profile-selecting flag is --profile <name>.

4. Smoke-test prod

temporal workflow list --profile prod --limit 1 --output json

If this returns (even an empty list), the connection descriptor is correct.

5. Run in prod

temporal workflow start \
    --profile prod \
    --type MyWorkflow \
    --task-queue my-queue \
    --input '{"key": "value"}'

workflow start is asynchronous (returns a Workflow/Run ID); use workflow execute instead if you want the CLI to block.

Common Error Types Reference

Workflow Status Reference

See Also

• Common Gotchas - Anti-patterns that cause these errors
• Troubleshooting - Decision trees for diagnosing issues

Common Temporal Gotchas

Common mistakes and anti-patterns in Temporal development. Learning from these saves significant debugging time.

This document provides a general overview of conceptual-level gotchas in Temporal. The exact form that these take and symptoms can vary by SDK language. See references/{your_language}/gotchas.md for language-specific info on common mistakes.

Non-Idempotent Activities

The Problem: Activities may execute more than once due to retries or Worker failures. If an activity calls an external service without an idempotency key, you may charge a customer twice, send duplicate emails, or create duplicate records.

Symptoms:

• Duplicate side effects (double charges, duplicate notifications)
• Data inconsistencies after retries

The Fix: Always use idempotency keys when calling external services. Use the workflow ID, activity ID, or a domain-specific identifier (like order ID) as the key.

Note: Local Activities skip the task queue for lower latency, but they're still subject to retries. The same idempotency rules apply.

Side Effects & Non-Determinism in Workflow Code

The Problem: Code in workflow functions runs on first execution AND on every replay. Any side effect (logging, notifications, metrics, etc.) will happen multiple times and non-deterministic code (IO, current time, random numbers, threading, etc.) won't replay correctly.

Symptoms:

• Non-determinism errors
• Sandbox violations, depending on SDK language
• Duplicate log entries
• Multiple notifications for the same event
• Inflated metrics

The Fix:

• Use Temporal replay-aware managed side effects for common, non-business logic cases:
• Temporal workflow logging
• Temporal date time
• Temporal UUID generation
• Temporal random number generation
• Put all other side effects in Activities

See references/core/determinism.md for more info.

Multiple Workers with Different Code

The Problem: If Worker A runs part of a workflow with code v1, then Worker B (with code v2) picks it up, replay may produce different Commands.

Symptoms:

• Non-determinism errors after deploying new code
• Errors mentioning "command mismatch" or "unexpected command"

The Fix:

• Use Worker Versioning for production deployments
• Use patching APIs
• During development: kill old workers before starting new ones
• Ensure all workers run identical code

Note: Workflows started with old code continue running after you change the code, which can then induce the above issues. During development (NOT production), you may want to terminate stale workflows (temporal workflow terminate --workflow-id <id>).

See references/core/versioning.md for more info.

Failing Activities Too Quickly

The Problem: Using aggressive activity retry policies that give up too easily.

Symptoms:

• Workflows failing on transient errors
• Unnecessary workflow failures during brief outages

The Fix: Use appropriate activity retry policies. Let Temporal handle transient failures with exponential backoff. Reserve maximum_attempts=1 for truly non-retryable operations.

Query Handler & Update Validator Mistakes

Modifying State in Queries & Update Validators

The Problem: Queries and update validators are read-only. Modifying state causes non-determinism on replay, and must strictly be avoided.

Symptoms:

• State inconsistencies after workflow replay
• Non-determinism errors

The Fix: Queries and update validators must only read state. Use Updates for operations that need to modify state AND return a result.

Blocking in Queries & Update Validators

The Problem: Queries and update validators must return immediately. They cannot await activities, child workflows, timers, or conditions.

Symptoms:

• Query / update validators timeouts
• Deadlocks

The Fix: Queries and update validators must only look at current state. Use Signals or Updates to trigger async operations.

Query vs Signal vs Update

Key rule: Query to peek, Signal to push, Update to pop.

File Organization Issues

Each SDK has specific requirements for how workflow and activity code should be organized. Mixing them incorrectly causes sandbox issues, bundling problems, or performance degradation.

See language-specific gotchas for details.

Testing Mistakes

Only Testing Happy Paths

The Problem: Not testing what happens when things go wrong.

Questions to answer:

• What happens when an Activity exhausts all retries?
• What happens when a workflow is cancelled mid-execution?
• What happens during a Worker restart?

The Fix: Test failure scenarios explicitly. Mock activities to fail, test cancellation handling, use replay testing.

Not Testing Replay Compatibility

The Problem: Changing workflow code without verifying existing workflows can still replay.

Symptoms:

• Non-determinism errors after deployment
• Stuck workflows that can't make progress

The Fix: Use replay testing against saved histories from production or staging.

Error Handling Mistakes

Swallowing Errors

The Problem: Catching errors without proper handling hides failures.

Symptoms:

• Silent failures
• Workflows completing "successfully" despite errors
• Difficult debugging

The Fix: Log errors and make deliberate decisions. Either re-raise, use a fallback, or explicitly document why ignoring is safe.

Wrong Retry Classification

The Problem: Marking transient errors as non-retryable, or permanent errors as retryable.

Symptoms:

• Workflows failing on temporary network issues (if marked non-retryable)
• Infinite retries on invalid input (if marked retryable)

The Fix:

• Retryable: Network errors, timeouts, rate limits, temporary unavailability
• Non-retryable: Invalid input, authentication failures, business rule violations, resource not found

Cancellation Handling

Not Handling Workflow Cancellation

The Problem: When a workflow is cancelled, cleanup code after the cancellation point doesn't run unless explicitly protected.

Symptoms:

• Resources not released after cancellation
• Incomplete compensation/rollback
• Leaked state

The Fix: Use language-specific cancellation scopes or try/finally blocks to ensure cleanup runs even on cancellation. See language-specific gotchas for implementation details.

Not Handling Activity Cancellation

The Problem: Activities must opt in to receive cancellation. Without proper handling, a cancelled activity continues running to completion, wasting resources.

Requirements for activity cancellation:

1. Heartbeating - Cancellation is delivered via heartbeat. Activities that don't heartbeat won't know they've been cancelled. 2. Checking for cancellation - Activity must explicitly check for cancellation or await a cancellation signal.

Symptoms:

• Cancelled activities running to completion
• Wasted compute on work that will be discarded
• Delayed workflow cancellation

The Fix: Heartbeat regularly and check for cancellation. See language-specific gotchas for implementation patterns.

CLI Gotchas for Developers

Dev Server Is In-Memory and Not for Production

The dev server loses all state on restart (use --db-filename to persist) and runs everything in a single process. See dev-management.md for the full flag table and persistence guidance. Even with persistence enabled, the dev server should NEVER be used for production deployments.

workflow update Is a Command Group, Not a Single Command

Running temporal workflow update alone will not work. Use the correct subcommand:

• temporal workflow update execute -- start an update and wait for completion.
• temporal workflow update start -- fire an update and wait for acceptance. Requires --wait-for-stage accepted.
• temporal workflow update result -- get the result of a previously started update.
• temporal workflow update describe -- check an update's current status.

--wait-for-stage Only Accepts accepted

Despite looking like an enum, the only valid value for --wait-for-stage on temporal workflow update start is accepted. Passing completed or other values will fail. The flag is required to allow a future CLI version to choose a default.

--reapply-type Only Accepts Signal or None

When resetting a workflow with temporal workflow reset, --reapply-type controls which events get reapplied after the reset point. Only Signal and None are valid values.

Payload Size Limits

The Problem: Temporal has built-in limits on payload sizes. Exceeding them causes workflows to fail.

Limits:

• Max 2MB per individual payload
• Max 4MB per gRPC message
• Max 50MB for complete workflow history (aim for < 10MB in practice)

Symptoms:

• Payload too large errors
• gRPC message size exceeded errors
• Workflow history growing unboundedly

The Fix: Store large data externally (S3/GCS) and pass references, use compression codecs, or chunk data across multiple activities. See the Large Data Handling pattern in references/core/patterns.md.

How to install Temporal CLI

macOS

Via homebrew

brew install temporal

Via tarball download

• Darwin amd64
• Darwin arm64

Extract any downloaded archive and add the temporal binary to your PATH.

Linux

Homebrew (if available), Snap, or tarball download:

brew install temporal
# or
snap install temporal

• Linux amd64
• Linux arm64

Extract any downloaded archive and add the temporal binary to your PATH.

Windows

Download the tarballs:

• Windows amd64
• Windows arm64

Extract the archive and add the temporal.exe binary to your PATH.

Docker

docker run --rm temporalio/temporal --help

tcld (Temporal Cloud CLI)

Only needed for Cloud-connected development (managing Cloud namespaces, API keys, etc.).

Homebrew:

brew install temporalio/brew/tcld

Interactive Workflows

Interactive workflows are workflows that use Temporal features such as signals or updates to pause and wait for external input. When testing and debugging these types of workflows you can send them input via the Temporal CLI.

Signals

Fire-and-forget messages to a workflow.

# Send signal to workflow
temporal workflow signal \
  --workflow-id <id> \
  --name "signal_name" \
  --input '{"key": "value"}'

Updates

Request-response style interaction (returns a value).

# Send update to workflow
temporal workflow update execute \
  --workflow-id <id> \
  --name "update_name" \
  --input '{"approved": true}'

Queries

Read-only inspection of workflow state.

# Query workflow state (read-only)
temporal workflow query \
  --workflow-id <id> \
  --name "get_status"

Typical Steps for Testing Interactive Workflows

# 1. Start worker (command is project dependent)
# 2. Start workflow (command is project dependent) This code should output the workflow ID, if not, modify it to.
temporal workflow signal --workflow-id <WORKFLOW_ID> --name "signal_name" --input '{"key": "value"}' # 3. Send it interactive events, e.g. a signal. 
# 4. Wait for workflow to complete (use Temporal CLI to check status)
# 5. Read workflow result, using the Temporal CLI
# 6. Cleanup the worker process if needed.

Temporal Workflow Patterns

Overview

Common patterns for building robust Temporal workflows. See the language-specific references for the language you are working in:

• references/{language}/{language}.md for the root level documentation for that language
• references/{language}/patterns.md for language-specific example code of the patterns in this file.

Signals

Purpose: Send data to a running workflow asynchronously (fire-and-forget).

When to Use:

• Human approval workflows
• Adding items to a workflow's queue
• Notifying workflow of external events
• Live configuration updates

Characteristics:

• Asynchronous - sender doesn't wait for response
• Can mutate workflow state
• Durable - signals are persisted in history
• Can be sent before workflow starts (signal-with-start)

Example Flow:

Client                    Workflow
  │                          │
  │──── signal(approve) ────▶│
  │                          │ (updates state)
  │                          │
  │◀──── (no response) ──────│

Note: A related but distinct pattern to signals is async activity completion. This is an advanced feature, which you may consider if the external system that would deliver the signal is unreliable and might fail to Signal, or you want the external process to Heartbeat or receive Cancellation. If this may be the case, look at language-specific advanced features for your SDK language (references/{your_language}/advanced-features.md).

Queries

Purpose: Read workflow state synchronously without modifying it.

When to Use:

• Building dashboards showing workflow progress
• Health checks and monitoring
• Debugging workflow state
• Exposing current status to external systems

Characteristics:

• Synchronous - caller waits for response
• Read-only - must not modify state
• Not recorded in history
• Executes on the worker, not persisted
• Can run even on completed workflows

Example Flow:

Client                    Workflow
  │                          │
  │──── query(status) ──────▶│
  │                          │ (reads state)
  │◀──── "processing" ───────│

Updates

Purpose: Modify workflow state and receive a response synchronously.

When to Use:

• Operations that need confirmation (add item, return count)
• Validation before accepting changes
• Replace signal+query combinations
• Request-response patterns within workflow

Characteristics:

• Synchronous - caller waits for completion
• Can mutate state AND return values
• Supports validators to reject invalid updates before they even get persisted into history
• Validators must NOT mutate workflow state or block (no activities, sleeps, or commands) — they are read-only, similar to query handlers
• Recorded in history

Example Flow:

Client                    Workflow
  │                          │
  │──── update(addItem) ────▶│
  │                          │ (validates, modifies state)
  │◀──── {count: 5} ─────────│

Child Workflows

When to Use:

• Prevent history from growing too large
• Isolate failure domains (child can fail without failing parent)
• Different retry policies for different parts

Characteristics:

• Own history (doesn't bloat parent)
• Independent lifecycle options (ParentClosePolicy)
• Can be cancelled independently
• Results returned to parent

Parent Close Policies:

• TERMINATE - Child terminated when parent closes (default)
• ABANDON - Child continues running independently
• REQUEST_CANCEL - Cancellation requested but not forced

Note: Do not need to use child workflows simply for breaking complex logic down into smaller pieces. Standard programming abstractions within a workflow can already be used for that.

Continue-as-New

Purpose: Prevent unbounded history growth by "restarting" with fresh history.

When to Use:

• Long-running workflows (entity workflows, subscriptions)
• Workflows with many iterations
• When history approaches 10,000+ events
• Periodic cleanup of accumulated state

How It Works:

Workflow (history: 10,000 events)
    │
    │ continueAsNew(currentState)
    ▼
New Workflow Execution (history: 0 events)
    │ (same workflow ID, fresh history)
    │ (receives currentState as input)

Best Practice: Check historyLength or continueAsNewSuggested periodically.

Saga Pattern

Purpose: Distributed transactions with compensation for failures.

When to Use:

• Multi-step operations that span services
• Operations requiring rollback on failure
• Financial transactions, order processing
• Booking systems with multiple reservations

How It Works:

Step 1: Reserve inventory
  └─ Compensation: Release inventory

Step 2: Charge payment
  └─ Compensation: Refund payment

Step 3: Ship order
  └─ Compensation: Cancel shipment

On failure at step 3:
  Execute: Refund payment (step 2 compensation)
  Execute: Release inventory (step 1 compensation)

Implementation Pattern:

1. Track compensation actions as you complete each step 2. On failure, execute compensations in reverse order 3. Handle compensation failures gracefully (log, alert, manual intervention)

Parallel Execution

Purpose: Run multiple independent operations concurrently.

When to Use:

• Processing multiple items that don't depend on each other
• Calling multiple APIs simultaneously
• Fan-out/fan-in patterns
• Reducing total workflow duration

Patterns:

• Promise / asyncio - Use traditional concurrency helpers (e.g. wait for all, wait for first, etc)
• Partial failure handling - Continue with successful results

Entity Workflow Pattern

Purpose: Model long-lived entities as workflows that handle events.

When to Use:

• Subscription management
• User sessions
• Shopping carts
• Any stateful entity receiving events over time

How It Works:

Entity Workflow (user-123)
    │
    ├── Receives signal: AddItem
    │   └── Updates state
    │
    ├── Receives signal: UpdateQuantity
    │   └── Updates state
    │
    ├── Receives query: GetCart
    │   └── Returns current state
    │
    └── continueAsNew when history grows

Timer Patterns

Purpose: Durable delays that survive worker restarts.

Use Cases:

• Scheduled reminders
• Timeout handling
• Delayed actions
• Polling with intervals

Characteristics:

• Timers are durable (persisted in history)
• Can be cancelled

Polling Patterns

Frequent Polling

Purpose: Frequently (once per second of faster) repeatedly check external state until condition met.

Implementation:

# Inside Activity (polling_activity):
while not condition_met:
    result = await call_external_api()
    if result.done:
        break
    activity.heartbeat("Invoking activity")
    await sleep(poll_interval)


# In workflow code:
workflow.execute_activity(
    polling_activity,
    PollingActivityInput(...),
    start_to_close_timeout=timedelta(seconds=60),
    heartbeat_timeout=timedelta(seconds=2),
)

To ensure that polling_activity is restarted in a timely manner, we make sure that it heartbeats on every iteration. Note that heartbeating only works if we set the heartbeat_timeout to a shorter value than the Activity start_to_close_timeout timeout

Advantage: Because the polling loop is inside the activity, this does not pollute the workflow history.

Infrequent Polling

Purpose: Infrequently (once per minute or slower) repeatedly poll an external service.

Implementation:

Define an Activity which fails (raises an exception) exactly when polling is not completed.

The polling loop is accomplished via activity retries, by setting the following Retry options:

• backoff_coefficient: to 1
• initial_interval: to the polling interval (e.g. 60 seconds)

This will enable the Activity to be retried exactly on the set interval.

Advantage: Individual Activity retries are not recorded in Workflow History, so this approach can poll for a very long time without affecting the history size.

Idempotency Patterns

Purpose: Ensure activities can be safely retried and replayed without causing duplicate side effects.

Why It Matters: Temporal may re-execute activities during retries (on failure) or replay (on worker restart). Without idempotency, this can cause duplicate charges, duplicate emails, duplicate database entries, etc.

Using Idempotency Keys

Pass a unique identifier to external services so they can detect and deduplicate repeated requests:

Activity: charge_payment(order_id, amount)
    │
    └── Call payment API with:
            amount: $100
            idempotency_key: "order-{order_id}"
        │
        └── Payment provider deduplicates based on key
            (second call with same key returns original result)

Good idempotency key sources:

• Workflow ID (unique per workflow execution)
• Business identifier (order ID, transaction ID)
• Workflow ID + activity name + attempt number

Check-Before-Act Pattern

Query the external system's state before making changes:

Activity: send_welcome_email(user_id)
    │
    ├── Check: Has welcome email been sent for user_id?
    │   │
    │   ├── YES: Return early (already done)
    │   │
    │   └── NO: Send email, mark as sent

Designing Idempotent Activities

1. Use unique identifiers as idempotency keys with external APIs 2. Check before acting: Query current state before making changes 3. Make operations repeatable: Ensure calling twice produces the same result 4. Record outcomes: Store transaction IDs or results for verification 5. Leverage external system features: Many APIs (Stripe, AWS, etc.) have built-in idempotency key support

Tracking State in Workflows

For complex multi-step operations, track completion status in workflow state:

Workflow State:
    payment_completed: false
    shipment_created: false

Run:
    if not payment_completed:
        charge_payment(...)
        payment_completed = true

    if not shipment_created:
        create_shipment(...)
        shipment_created = true

This ensures that on replay, already-completed steps are skipped.

Large Data Handling

Purpose: Handle data that exceeds Temporal's payload limits without polluting workflow history.

Limits (see references/core/gotchas.md for details):

• Max 2MB per individual payload
• Max 4MB per gRPC message
• Max 50MB for workflow history (aim for < 10MB)

Key Principle: Large data should never flow through workflow history. Activities read and write large data directly, passing only small references through the workflow.

Wrong Approach:

Workflow
    │
    ├── downloadFromStorage(ref) ──▶ returns large data (enters history)
    │
    ├── processData(largeData) ────▶ large data as argument (enters history AGAIN)
    │
    └── uploadToStorage(result) ───▶ large data as argument (enters history AGAIN)

This defeats the purpose—large data enters workflow history multiple times.

Correct Approach:

Workflow
    │
    └── processLargeData(inputRef) ──▶ returns outputRef (small string)
                │
                └── Activity internally:
                        download(inputRef) → process → upload → return outputRef

The workflow only handles references (small strings). The activity does all large data operations internally.

Implementation Pattern:

1. Accept a reference (URL, S3 key, database ID) as activity input 2. Download/fetch the large data inside the activity 3. Process the data inside the activity 4. Upload/store the result inside the activity 5. Return only a reference to the result

Other Strategies:

• Compression: Use a PayloadCodec to compress data automatically
• Chunking: Split large collections across multiple activities, each handling a subset

Activity Heartbeating

Purpose: Enable cancellation delivery and progress tracking for long-running activities.

Why Heartbeat:

1. Support activity cancellation - Cancellations are delivered to activities via heartbeat. Activities that don't heartbeat won't know they've been cancelled. 2. Resume progress after failure - Heartbeat details persist across retries, allowing activities to resume where they left off. 3. Detect stuck activities - If an activity stops heartbeating, Temporal can time it out and retry.

How Cancellation Works:

Workflow requests activity cancellation
    │
    ▼
Temporal Service marks activity for cancellation
    │
    ▼
Activity calls heartbeat()
    │
    ├── Not cancelled: heartbeat succeeds, continues
    │
    └── Cancelled: heartbeat raises exception
            Activity can catch this to perform cleanup

Key Point: If an activity never heartbeats, it will run to completion even if cancelled—it has no way to learn about the cancellation.

Local Activities

Purpose: Reduce latency for short, lightweight operations by skipping the task queue. ONLY use these when necessary for performance. Do NOT use these by default, as they are not durable and distributed.

When to Use:

• Short operations completing in milliseconds/seconds
• High-frequency calls where task queue overhead is significant
• Low-latency requirements where you can't afford task queue round-trip

Characteristics:

• Executes on the same worker that runs the workflow
• No task queue round-trip (lower latency)
• Still recorded in history
• Should complete quickly (default timeout is short)

Trade-offs:

• Less visibility in Temporal UI (no separate task)
• Must complete on the same worker
• Not suitable for long-running operations
• Risk with consecutive local activities: Local activity completions are only persisted when the current Workflow Task completes. Calling multiple local activities in a row (with nothing in between to yield the Workflow Task) increases the risk of losing work if the worker crashes mid-sequence. If you need a chain of operations with durable checkpoints between each step, use regular activities instead.

Choosing Between Patterns

Task Queue Priority and Fairness

Overview

Priority and Fairness control how Tasks are distributed within a Task Queue. Priority determines execution order. Fairness prevents one group of Tasks from starving others. They can be used independently or together.

Both features are in Public Preview. Priority is free. Fairness is a paid feature in Temporal Cloud.

Priority

Priority lets you control execution order within a single Task Queue by assigning a priority key (integer 1-5, lower = higher priority). Each priority level acts as a sub-queue. All priority-1 Tasks dispatch before priority-2, and so on. Tasks at the same priority level dispatch in FIFO order.

Default priority is 3. Activities inherit their parent workflow's priority unless explicitly overridden.

When to use Priority

Use Priority to differentiate execution order between types of work sharing a single Task Queue and Worker pool. For example, process payment-related Tasks before less time-sensitive inventory management Tasks, or ensure real-time Tasks run ahead of batch Tasks. You can also use it to run urgent Tasks immediately by assigning them priority 1.

CLI

temporal workflow start \
  --type ChargeCustomer \
  --task-queue my-task-queue \
  --workflow-id my-workflow-id \
  --input '{"customerId":"12345"}' \
  --priority-key 1

Go

workflowOptions := client.StartWorkflowOptions{
  ID:        "my-workflow-id",
  TaskQueue: "my-task-queue",
  Priority:  temporal.Priority{PriorityKey: 1},
}
we, err := c.ExecuteWorkflow(context.Background(), workflowOptions, MyWorkflow)

Java

WorkflowOptions options = WorkflowOptions.newBuilder()
  .setTaskQueue("my-task-queue")
  .setPriority(Priority.newBuilder().setPriorityKey(1).build())
  .build();

Python

await client.start_workflow(
  MyWorkflow.run,
  args="hello",
  id="my-workflow-id",
  task_queue="my-task-queue",
  priority=Priority(priority_key=1),
)

TypeScript

const handle = await startWorkflow(workflows.myWorkflow, {
  args: [false, 1],
  priority: { priorityKey: 1 },
});

.NET

var handle = await Client.StartWorkflowAsync(
  (MyWorkflow wf) => wf.RunAsync("hello"),
  new StartWorkflowOptions(id: "my-workflow-id", taskQueue: "my-task-queue")
  {
    Priority = new Priority(1),
  }
);

Fairness

Fairness prevents one group of Tasks from monopolizing Worker capacity. Each fairness key creates a "virtual queue" within the Task Queue. The server uses round-robin dispatch across virtual queues so no single key can block others, even with a much larger backlog.

When to use Fairness

Fairness solves the multi-tenant starvation problem. Without it, Tasks dispatch FIFO: if tenant-big enqueues 100k Tasks, tenant-small's 10 Tasks sit behind the entire backlog. With Fairness, each tenant gets its own virtual queue and Tasks are interleaved.

Common scenarios:

• Multi-tenant applications where large tenants should not block small ones.
• Tiered capacity bands where you want weighted distribution (e.g., 80% premium, 20% free) without limiting overall throughput when one band is empty.
• Batch jobs where some jobs run far more frequently than others.
• Multi-vendor processing where a few vendors generate the majority of work.

If all your Tasks can be dispatched immediately (no backlog), you don't need Fairness.

Fairness applies at Task dispatch time and considers each Task as having equal cost until dispatch. It does not account for Tasks currently being processed by Workers. So if you look at Tasks being processed by Workers, you might not see "fairness" across tenants — for example, if tenant-big already has Tasks being processed when tenant-small's Tasks are dispatched, it may still appear that tenant-big is using the most resources.

Fairness keys and weights

A fairness key is a string, typically a tenant ID or workload category. Each unique key creates a virtual queue.

A fairness weight (float, default 1.0) controls how often a key's Tasks are dispatched relative to others. A key with weight 2.0 dispatches twice as often as keys with weight 1.0.

Example with three tiers:

Tasks without a fairness key are grouped under an implicit empty-string key with weight 1.0. Adoption is incremental: unkeyed Tasks participate in round-robin alongside keyed Tasks.

Using Fairness with Priority

When combined, Priority determines which sub-queue Tasks go into (priority 1 before 2, etc.), and Fairness applies within each priority level.

SDK examples

CLI

temporal workflow start \
  --type ChargeCustomer \
  --task-queue my-task-queue \
  --workflow-id my-workflow-id \
  --input '{"customerId":"12345"}' \
  --priority-key 1 \
  --fairness-key tenant-123 \
  --fairness-weight 2.0

Go

workflowOptions := client.StartWorkflowOptions{
  ID:        "my-workflow-id",
  TaskQueue: "my-task-queue",
  Priority: temporal.Priority{
    PriorityKey:    1,
    FairnessKey:    "tenant-123",
    FairnessWeight: 2.0,
  },
}
we, err := c.ExecuteWorkflow(context.Background(), workflowOptions, MyWorkflow)

Activities:

ao := workflow.ActivityOptions{
  StartToCloseTimeout: time.Minute,
  Priority: temporal.Priority{
    PriorityKey:    1,
    FairnessKey:    "tenant-123",
    FairnessWeight: 2.0,
  },
}
ctx := workflow.WithActivityOptions(ctx, ao)
err := workflow.ExecuteActivity(ctx, MyActivity).Get(ctx, nil)

Java

WorkflowOptions options = WorkflowOptions.newBuilder()
  .setTaskQueue("my-task-queue")
  .setPriority(Priority.newBuilder()
    .setPriorityKey(1)
    .setFairnessKey("tenant-123")
    .setFairnessWeight(2.0)
    .build())
  .build();

Python

await client.start_workflow(
  MyWorkflow.run,
  args="hello",
  id="my-workflow-id",
  task_queue="my-task-queue",
  priority=Priority(priority_key=1, fairness_key="tenant-123", fairness_weight=2.0),
)

Activities:

await workflow.execute_activity(
  say_hello,
  "hi",
  priority=Priority(priority_key=1, fairness_key="tenant-123", fairness_weight=2.0),
  start_to_close_timeout=timedelta(seconds=5),
)

TypeScript

const handle = await startWorkflow(workflows.myWorkflow, {
  args: [false, 1],
  priority: { priorityKey: 1, fairnessKey: 'tenant-123', fairnessWeight: 2.0 },
});

.NET

var handle = await Client.StartWorkflowAsync(
  (MyWorkflow wf) => wf.RunAsync("hello"),
  new StartWorkflowOptions(id: "my-workflow-id", taskQueue: "my-task-queue")
  {
    Priority = new Priority(
      priorityKey: 1,
      fairnessKey: "tenant-123",
      fairnessWeight: 2.0
    )
  }
);

Child Workflows

Child workflows can set their own priority and fairness, overriding the parent.

Go:

cwo := workflow.ChildWorkflowOptions{
  WorkflowID: "child-workflow-id",
  TaskQueue:  "child-task-queue",
  Priority: temporal.Priority{
    PriorityKey:    1,
    FairnessKey:    "tenant-123",
    FairnessWeight: 2.0,
  },
}
ctx := workflow.WithChildOptions(ctx, cwo)
err := workflow.ExecuteChildWorkflow(ctx, MyChildWorkflow).Get(ctx, nil)

Java:

ChildWorkflowOptions childOptions = ChildWorkflowOptions.newBuilder()
  .setTaskQueue("child-task-queue")
  .setWorkflowId("child-workflow-id")
  .setPriority(Priority.newBuilder()
    .setPriorityKey(1)
    .setFairnessKey("tenant-123")
    .setFairnessWeight(2.0)
    .build())
  .build();
MyChildWorkflow child = Workflow.newChildWorkflowStub(MyChildWorkflow.class, childOptions);
child.run();

Python:

await workflow.execute_child_workflow(
  MyChildWorkflow.run,
  args="hello child",
  priority=Priority(priority_key=1, fairness_key="tenant-123", fairness_weight=2.0),
)

TypeScript:

const handle = await startChildWorkflow(workflows.myChildWorkflow, {
  args: [false, 1],
  priority: { priorityKey: 1, fairnessKey: 'tenant-123', fairnessWeight: 2.0 },
});

.NET:

await Workflow.ExecuteChildWorkflowAsync(
  (MyChildWorkflow wf) => wf.RunAsync("hello child"),
  new() {
    Priority = new(
      priorityKey: 1,
      fairnessKey: "tenant-123",
      fairnessWeight: 2.0
    )
  }
);

Rate limiting

Two rate-limiting controls work alongside Fairness:

• `queue-rps-limit` — overall dispatch rate for the entire Task Queue.
• `fairness-key-rps-limit-default` — per-key rate limit, scaled by weight. If the default is 10 rps and a key has weight 2.5, that key's effective limit is 25 rps.

temporal task-queue config set \
    --task-queue my-task-queue \
    --task-queue-type activity \
    --namespace my-namespace \
    --queue-rps-limit 500 \
    --queue-rps-limit-reason "overall limit" \
    --fairness-key-rps-limit-default 33.3 \
    --fairness-key-rps-limit-reason "per-key limit"

If both limits are set, the more restrictive one applies.

Fairness weight overrides

You can override the weights of up to 1000 keys through the config API. When an override is set for a key, the SDK-supplied weight is ignored. Overrides are per Task Queue and type (workflow vs. activity), so set them for both if needed.

Enabling Fairness

When you start using fairness keys, it switches your active Task Queues to fairness mode. Existing queued Tasks are processed before any new fairness-mode ones.

Temporal Cloud: automatically enabled when you start using fairness keys.

Self-hosted: set these dynamic config flags to true:

• matching.useNewMatcher
• matching.enableFairness
• matching.enableMigration (to drain existing backlogs after enabling)

Limitations

• Accuracy can degrade with a very large number of distinct fairness keys.
• Task Queue partitioning can interfere with fairness distribution. Contact Temporal Support to set a Task Queue to a single partition if needed.
• Weights apply at schedule time, not dispatch time. Changing a weight does not reorder already-backlogged Tasks.
• Fairness is not guaranteed across different Worker versions when using Worker Versioning.
• After server restarts, less-active keys may briefly dispatch new Tasks ahead of their existing backlog until ordering normalizes.

Temporal Troubleshooting Guide

Workflow Diagnosis Decision Tree

Workflow not behaving as expected?
│
├─▶ What is the workflow status?
│   │
│   ├─▶ RUNNING (but no progress)
│   │   └─▶ Go to: "Workflow Stuck" section
│   │
│   ├─▶ FAILED
│   │   └─▶ Go to: "Workflow Failed" section
│   │
│   ├─▶ TIMED_OUT
│   │   └─▶ Go to: "Timeout Issues" section
│   │
│   └─▶ COMPLETED (but wrong result)
│       └─▶ Go to: "Wrong Result" section

Workflow Stuck (RUNNING but No Progress)

Decision Tree

Workflow stuck in RUNNING?
│
├─▶ Is a worker running?
│   │
│   ├─▶ NO: Start a worker
│   │   └─▶ See references/core/dev-management.md
│   │
│   └─▶ YES: Is it on the correct task queue?
│       │
│       ├─▶ NO: Start worker with correct task queue
│       │
│       └─▶ YES: Check for non-determinism
│           │
│           ├─▶ NondeterminismError in logs?
│           │   └─▶ Go to: "Non-Determinism" section
│           │
│           ├─▶ Check history for task failures
│           │   └─▶ Run: `temporal workflow show --workflow-id <id>`
│           │       │
│           │       ├─▶ WorkflowTaskFailed event?
│           │       │   └─▶ Check error type in event details
│           │       │       └─▶ Go to relevant section in error-reference.md
│           │       │
│           │       └─▶ ActivityTaskFailed event?
│           │           └─▶ Go to: "Activity Keeps Retrying" section
│           │
│           └─▶ No errors in logs or history?
│               └─▶ Check if workflow is waiting for signal/timer

Common Causes

1. No worker running

• See references/core/dev-management.md

2. Worker on wrong task queue

• Check: Worker logs for task queue name
• Fix: Start worker with matching task queue

3. Worker has stale code

• Check: Worker startup time vs code changes
• Fix: Restart worker with updated code

4. Workflow waiting for signal

• Check: Workflow history for pending signals
• Fix: Send expected signal or check signal sender

5. Activity stuck/timing out

• Check: Activity retry attempts in history
• Fix: Investigate activity failure, increase timeout

Non-Determinism Errors

Decision Tree

NondeterminismError?
│
├─▶ Was code intentionally changed?
│   │
│   ├─▶ YES: Do you need to support in-flight workflows?
│   │   │
│   │   ├─▶ YES (production): Use patching API
│   │   │   └─▶ See: references/core/versioning.md
│   │   │
│   │   └─▶ NO (local dev/testing): Terminate or reset workflow
│   │       └─▶ `temporal workflow terminate --workflow-id <id>`
│   │       └─▶ Then start fresh with new code
│   │
│   └─▶ NO: Accidental change
│       │
│       ├─▶ Can you identify the change?
│       │   │
│       │   ├─▶ YES: Revert and restart worker. Note, this doesn't always work if workflow has progressed past the change (may induce other code paths), so may need to reset workflow.
│       │   │
│       │   └─▶ NO: Compare current code to expected history
│       │       └─▶ Check: Activity names, order, parameters

Common Causes

1. Changed call order

   # Before           # After (BREAKS)
   await activity_a   await activity_b
   await activity_b   await activity_a

2. Changed call name

   # Before                    # After (BREAKS)
   await process_order(...)    await handle_order(...)

3. Added/removed call

• Adding new activity mid-workflow
• Removing activity that was previously called

4. Using non-deterministic code

• datetime.now() in workflow (use workflow.now())
• random.random() in workflow (use workflow.random())

Recovery

Accidental Change:

1. Identify the change 2. Revert code to match history 3. Restart worker 4. Workflow automatically recovers

Intentional Change:

1. Use patching API for gradual migration 2. Or terminate old workflows, start new ones

Workflow Failed

Decision Tree

Workflow status = FAILED?
│
├─▶ Check workflow error message
│   │
│   ├─▶ Application error (your code)
│   │   └─▶ Fix the bug, start new workflow
│   │
│   ├─▶ NondeterminismError
│   │   └─▶ Go to: "Non-Determinism" section
│   │
│   └─▶ Timeout error
│       └─▶ Go to: "Timeout Issues" section

Common Causes

1. Unhandled exception in workflow

• Check error message and stack trace
• Fix bug in workflow code

2. Activity exhausted retries

• All retry attempts failed
• Check activity logs for root cause

3. Non-retryable error thrown

• Error marked as non-retryable
• Intentional failure, check business logic

Timeout Issues

Timeout Types

Diagnosis

Timeout error?
│
├─▶ Which timeout?
│   │
│   ├─▶ Workflow timeout
│   │   └─▶ Increase timeout or optimize workflow. Better yet, consider removing the workflow timeout, as it is generally discouraged unless *necessary* for your use case.
│   │
│   ├─▶ ScheduleToCloseTimeout
│   │   └─▶ Activity taking too long overall (including retries)
│   │
│   ├─▶ StartToCloseTimeout
│   │   └─▶ Single activity attempt too slow
│   │
│   └─▶ HeartbeatTimeout
│       └─▶ Activity not heartbeating frequently enough
│           └─▶ Add heartbeat() calls in long activities

Fixes

1. Increase timeout if operation legitimately takes longer 2. Add heartbeats to long-running activities 3. Optimize activity to complete faster 4. Break into smaller activities for better granularity

Activity Keeps Retrying

Decision Tree

Activity retrying repeatedly?
│
├─▶ Check activity error
│   │
│   ├─▶ Transient error (network, timeout)
│   │   └─▶ Expected behavior, will eventually succeed
│   │
│   ├─▶ Permanent error (bug, invalid input)
│   │   └─▶ Fix the bug or mark as non-retryable
│   │
│   └─▶ Resource exhausted
│       └─▶ Add backoff, check rate limits

Common Causes

1. Bug in activity code

• Fix the bug
• Consider marking certain errors as non-retryable

2. External service down

• Retries are working as intended
• Monitor service recovery

3. Invalid input

• Validate inputs before activity
• Return non-retryable error for bad input

Wrong Result (Completed but Incorrect)

Diagnosis

1. Check workflow history for unexpected activity results 2. Verify activity implementations produce correct output 3. Check for race conditions in parallel execution 4. Verify signal handling if signals are involved

Common Causes

1. Activity bug - Wrong logic in activity 2. Stale data - Activity using outdated information 3. Signal ordering - Signals processed in unexpected order 4. Parallel execution - Race condition in concurrent operations

Worker Issues

Worker Not Starting

Worker won't start?
│
├─▶ Connection error
│   └─▶ Check Temporal server is running
│       └─▶ `temporal server start-dev` (start in background, see references/core/dev-management.md)
│
├─▶ Registration error
│   └─▶ Check workflow/activity definitions are valid
│
└─▶ Other errors (imports, etc.)
    └─▶ Debug those errors as usual.

Worker Crashing

1. Out of memory - Reduce concurrent tasks, check for leaks 2. Unhandled exception - Add error handling 3. Dependency issue - Check package versions

Useful Commands

# Check Temporal server
temporal server start-dev

# List workflows
temporal workflow list

# Describe specific workflow
temporal workflow describe --workflow-id <id>

# Show workflow history
temporal workflow show --workflow-id <id>

# Terminate stuck workflow
temporal workflow terminate --workflow-id <id>

# Reset workflow to specific point
temporal workflow reset --workflow-id <id> --event-id <event-id>

Quick Reference: Status → Action

See Also

• Common Gotchas - Anti-patterns that cause these issues
• Error Reference - Quick error type lookup

Workflow Versioning Concepts

This document provides core conceptual explanations of workflow versioning in Temporal. For language-specific implementation details see references/{your_language}/versioning.md, for the language you are working in.

Overview

Workflow versioning allows safe deployment of code changes without breaking running workflows. Three approaches available:

1. Patching API - Code-level version branching 2. Workflow Type Versioning - New workflow types for incompatible changes 3. Worker Versioning - Deployment-level control with Build IDs

Why Versioning is Needed

When workers restart after deployment, they resume open workflows through history replay. If updated code produces different Commands than the original code, it causes non-determinism errors.

Original Code (recorded in history):
  await activity_a()
  await activity_b()

Updated Code (during replay):
  await activity_a()
  await activity_c()  ← Different! NondeterminismError

Approach 1: Patching API

Concept

The patching API lets you branch code based on whether a workflow was started before or after a code change.

if patched("my-change"):
    // New code path (for new and replaying new workflows)
else:
    // Old code path (for replaying old workflows)

Three-Phase Lifecycle

Phase 1: Patch In

• Add both old and new code paths
• New workflows take new path, old workflows take old path

Phase 2: Deprecate

• After all old workflows complete, remove old code
• Keep deprecation marker for history compatibility

Phase 3: Remove

• After all deprecated workflows complete
• Remove patch entirely, only new code remains

When to Use

• Adding, removing, or reordering activities/child workflows
• Changing which activity/child workflow is called
• Any change that alters the Command sequence

When NOT to Use

• Changing activity implementations (activities aren't replayed)
• Changing arguments passed to activities or child workflows
• Changing retry policies
• Changing timer durations
• Adding new signal/query/update handlers (additive changes are safe)
• Bug fixes that don't change Command sequence

Unnecessary patching adds complexity and can make workflow code unmanageable.

Approach 2: Workflow Type Versioning

Concept

Create a new workflow type (e.g., OrderWorkflowV2) instead of patching.

// Old: OrderWorkflow
// New: OrderWorkflowV2 (completely new implementation)

When to Use

• Major incompatible changes
• Complete rewrites
• When patching would be too complex
• When you want clean separation

Process

1. Create new workflow type with new name 2. Register both with worker 3. Start new workflows with new type 4. Wait for old workflows to complete 5. Remove old workflow type

Approach 3: Worker Versioning

Concept

Manage versions at deployment level using Build IDs. Multiple worker versions can run simultaneously.

Worker v1.0 (Build ID: abc123)
  └── Handles workflows started on this version

Worker v2.0 (Build ID: def456)
  └── Handles new workflows
  └── Can also handle upgraded old workflows

Key Concepts

Worker Deployment: Logical service grouping (e.g., "order-service")

Build ID: Specific code version (e.g., git commit hash)

Versioning Behaviors:

• PINNED - Workflows stay on original worker version
• AUTO_UPGRADE - Workflows can move to newer versions

When to Use PINNED

• Short-running workflows (minutes to hours)
• Consistency is critical
• Want simplest development experience
• Building new applications

When to Use AUTO_UPGRADE

• Long-running workflows (weeks or months)
• Workflows need bug fixes during execution
• Still requires patching for version transitions

Choosing an Approach

Best Practices

1. Check for open executions before removing old code 2. Use descriptive patch IDs (e.g., "add-fraud-check" not "patch-1") 3. Deploy incrementally: patch → deprecate → remove 4. Test replay compatibility before deploying changes 5. Monitor old workflow counts during migration

Finding Workflows by Version

# Find workflows with specific patch
temporal workflow list --query \
  'WorkflowType = "OrderWorkflow" AND TemporalChangeVersion = "add-fraud-check"'

# Find pre-patch workflows
temporal workflow list --query \
  'WorkflowType = "OrderWorkflow" AND TemporalChangeVersion IS NULL'

# Find workflows on specific worker version
temporal workflow list --query \
  'TemporalWorkerDeploymentVersion = "my-service:v1.0.0"'

Common Mistakes

1. Removing old code too early - Breaks replaying workflows 2. Not testing with replay - Catches issues before production 3. Patching non-Command changes - Unnecessary complexity 4. Forgetting to deprecate - Accumulates dead code

.NET SDK Advanced Features

Schedules

Create recurring workflow executions.

using Temporalio.Client.Schedules;

var scheduleId = "daily-report";
await client.CreateScheduleAsync(
    scheduleId,
    new Schedule(
        Action: ScheduleActionStartWorkflow.Create(
            (DailyReportWorkflow wf) => wf.RunAsync(),
            new(id: "daily-report", taskQueue: "reports")),
        Spec: new ScheduleSpec
        {
            Intervals = new List<ScheduleIntervalSpec>
            {
                new(Every: TimeSpan.FromDays(1)),
            },
        }));

// Manage schedules
var handle = client.GetScheduleHandle(scheduleId);
await handle.PauseAsync("Maintenance window");
await handle.UnpauseAsync();
await handle.TriggerAsync(); // Run immediately
await handle.DeleteAsync();

Async Activity Completion

For activities that complete asynchronously (e.g., human tasks, external callbacks). If you configure a HeartbeatTimeout on this activity, the external completer is responsible for sending heartbeats via the async handle. If you do NOT set a HeartbeatTimeout, no heartbeats are required.

Note: If the external system that completes the asynchronous action can reliably be trusted to do the task and Signal back with the result, and it doesn't need to Heartbeat or receive Cancellation, then consider using signals instead.

using Temporalio.Activities;
using Temporalio.Client;

[Activity]
public async Task RequestApprovalAsync(string requestId)
{
    var taskToken = ActivityExecutionContext.Current.Info.TaskToken;

    // Store task token for later completion (e.g., in database)
    await StoreTaskTokenAsync(requestId, taskToken);

    // Mark this activity as waiting for external completion
    throw new CompleteAsyncException();
}

// Later, complete the activity from another process
public async Task CompleteApprovalAsync(string requestId, bool approved)
{
    var client = await TemporalClient.ConnectAsync(new("localhost:7233"));
    // Retrieve the task token from external storage (e.g., database)
    var taskToken = await GetTaskTokenAsync(requestId);

    var handle = client.GetAsyncActivityHandle(taskToken);

    // Optional: if a HeartbeatTimeout was set, you can periodically:
    // await handle.HeartbeatAsync(progressDetails);

    if (approved)
        await handle.CompleteAsync("approved");
    else
        // You can also fail or report cancellation via the handle
        await handle.FailAsync(new ApplicationFailureException("Rejected"));
}

Worker Tuning

Configure worker performance settings.

var worker = new TemporalWorker(
    client,
    new TemporalWorkerOptions("my-task-queue")
    {
        // Workflow task concurrency
        MaxConcurrentWorkflowTasks = 100,
        // Activity task concurrency
        MaxConcurrentActivities = 100,
        // Graceful shutdown timeout
        GracefulShutdownTimeout = TimeSpan.FromSeconds(30),
    }
    .AddWorkflow<MyWorkflow>()
    .AddAllActivities(new MyActivities()));

Workflow Init Attribute

You should always put state initialization logic in the constructor of your workflow class, so that it happens before signals/updates arrive.

Normally, your constructor must have no arguments. However, if you add the [WorkflowInit] attribute, then your constructor instead receives the same workflow arguments that [WorkflowRun] receives:

[Workflow]
public class MyWorkflow
{
    private readonly string _initialValue;
    private readonly List<string> _items = new();

    [WorkflowInit]
    public MyWorkflow(string initialValue)
    {
        _initialValue = initialValue;
    }

    [WorkflowRun]
    public async Task<string> RunAsync(string initialValue)
    {
        // _initialValue and _items are already initialized
        return _initialValue;
    }
}

Constructor (with [WorkflowInit]) and [WorkflowRun] method must have the same parameters with the same types. You cannot make blocking calls (activities, sleeps, etc.) from the constructor.

Workflow Failure Exception Types

Control which exceptions cause workflow failures vs workflow task retries.

Default behavior: Only ApplicationFailureException fails a workflow. All other exceptions retry the workflow task forever (treated as bugs to fix with a code deployment).

Tip for testing: Set WorkflowFailureExceptionTypes to include Exception so any unhandled exception fails the workflow immediately rather than retrying the workflow task forever. This surfaces bugs faster.

Worker-Level Configuration

var worker = new TemporalWorker(
    client,
    new TemporalWorkerOptions("my-task-queue")
    {
        // These exception types will fail the workflow execution (not just the task)
        WorkflowFailureExceptionTypes = new[] { typeof(ArgumentException), typeof(InvalidOperationException) },
    }
    .AddWorkflow<MyWorkflow>()
    .AddAllActivities(new MyActivities()));

Dependency Injection

The .NET SDK supports dependency injection via the Temporalio.Extensions.Hosting package, which integrates with .NET's generic host.

Worker as Generic Host

using Temporalio.Extensions.Hosting;

public class Program
{
    public static async Task Main(string[] args)
    {
        var host = Host.CreateDefaultBuilder(args)
            .ConfigureServices(ctx =>
                ctx.
                    AddScoped<IOrderRepository, OrderRepository>().
                    AddHostedTemporalWorker(
                        clientTargetHost: "localhost:7233",
                        clientNamespace: "default",
                        taskQueue: "my-task-queue").
                    AddScopedActivities<MyActivities>().
                    AddWorkflow<MyWorkflow>())
            .Build();
        await host.RunAsync();
    }
}

Activity Dependency Injection

As shown in the host setup above, activities can be registered with AddScopedActivities<T>(), AddSingletonActivities<T>(), or AddTransientActivities<T>(). Activities registered this way are created via DI, allowing constructor injection:

public class MyActivities
{
    private readonly ILogger<MyActivities> _logger;
    private readonly IOrderRepository _repository;

    public MyActivities(ILogger<MyActivities> logger, IOrderRepository repository)
    {
        _logger = logger;
        _repository = repository;
    }

    [Activity]
    public async Task<Order> GetOrderAsync(string orderId)
    {
        _logger.LogInformation("Fetching order {OrderId}", orderId);
        return await _repository.GetAsync(orderId);
    }
}

Note: Dependency injection is NOT available in workflows — workflows must be self-contained for determinism.

.NET SDK Data Handling

Overview

The .NET SDK uses data converters to serialize/deserialize workflow inputs, outputs, and activity parameters.

Default Data Converter

The default converter handles:

• null
• byte[] (as binary)
• Google.Protobuf.IMessage instances
• Anything that System.Text.Json supports
• IRawValue as unconverted raw payloads

Custom Data Converter

Customize serialization by extending DefaultPayloadConverter. For example, to use camelCase property naming:

using System.Text.Json;
using Temporalio.Client;
using Temporalio.Converters;

public class CamelCasePayloadConverter : DefaultPayloadConverter
{
    public CamelCasePayloadConverter()
        : base(new JsonSerializerOptions { PropertyNamingPolicy = JsonNamingPolicy.CamelCase })
    {
    }
}

var client = await TemporalClient.ConnectAsync(new()
{
    TargetHost = "localhost:7233",
    Namespace = "my-namespace",
    DataConverter = DataConverter.Default with
    {
        PayloadConverter = new CamelCasePayloadConverter(),
    },
});

Protobuf Support

The default data converter includes built-in support for Protocol Buffer messages via Google.Protobuf.IMessage. Protobuf messages are automatically serialized using proto3 JSON.

// Any Google.Protobuf.IMessage is automatically handled
[Workflow]
public class MyWorkflow
{
    [WorkflowRun]
    public async Task<MyProtoResponse> RunAsync(MyProtoRequest request)
    {
        // Protobuf messages are serialized/deserialized automatically
        return await Workflow.ExecuteActivityAsync(
            (MyActivities a) => a.ProcessAsync(request),
            new() { StartToCloseTimeout = TimeSpan.FromMinutes(5) });
    }
}

Payload Encryption

Encrypt sensitive workflow data using a custom IPayloadCodec:

using Temporalio.Converters;
using Google.Protobuf;

public class EncryptionCodec : IPayloadCodec
{
    public Task<IReadOnlyCollection<Payload>> EncodeAsync(
        IReadOnlyCollection<Payload> payloads) =>
        Task.FromResult<IReadOnlyCollection<Payload>>(payloads.Select(p =>
            new Payload
            {
                Metadata = { ["encoding"] = "binary/encrypted" },
                Data = ByteString.CopyFrom(Encrypt(p.ToByteArray())),
            }).ToList());

    public Task<IReadOnlyCollection<Payload>> DecodeAsync(
        IReadOnlyCollection<Payload> payloads) =>
        Task.FromResult<IReadOnlyCollection<Payload>>(payloads.Select(p =>
        {
            if (p.Metadata.GetValueOrDefault("encoding") != "binary/encrypted")
                return p;
            return Payload.Parser.ParseFrom(Decrypt(p.Data.ToByteArray()));
        }).ToList());

    private byte[] Encrypt(byte[] data) => /* your encryption logic */;
    private byte[] Decrypt(byte[] data) => /* your decryption logic */;
}

// Apply encryption codec
var client = await TemporalClient.ConnectAsync(new("localhost:7233")
{
    DataConverter = DataConverter.Default with
    {
        PayloadCodec = new EncryptionCodec(),
    },
});

Search Attributes

Custom searchable fields for workflow visibility. These can be set at workflow start:

using Temporalio.Common;

var handle = await client.StartWorkflowAsync(
    (OrderWorkflow wf) => wf.RunAsync(order),
    new(id: $"order-{order.Id}", taskQueue: "orders")
    {
        TypedSearchAttributes = new SearchAttributeCollection.Builder()
            .Set(SearchAttributeKey.CreateKeyword("OrderId"), order.Id)
            .Set(SearchAttributeKey.CreateKeyword("OrderStatus"), "pending")
            .Set(SearchAttributeKey.CreateFloat("OrderTotal"), order.Total)
            .Build(),
    });

Or upserted during workflow execution:

[Workflow]
public class OrderWorkflow
{
    [WorkflowRun]
    public async Task<string> RunAsync(Order order)
    {
        // ... process order ...

        // Update search attribute
        Workflow.UpsertTypedSearchAttributes(
            SearchAttributeKey.CreateKeyword("OrderStatus").ValueSet("completed"));
        return "done";
    }
}

Querying Workflows by Search Attributes

await foreach (var wf in client.ListWorkflowsAsync(
    "OrderStatus = \"processing\" OR OrderStatus = \"pending\""))
{
    Console.WriteLine($"Workflow {wf.Id} is still processing");
}

Workflow Memo

Store arbitrary metadata with workflows (not searchable).

await client.ExecuteWorkflowAsync(
    (OrderWorkflow wf) => wf.RunAsync(order),
    new(id: $"order-{order.Id}", taskQueue: "orders")
    {
        Memo = new Dictionary<string, object>
        {
            ["customer_name"] = order.CustomerName,
            ["notes"] = "Priority customer",
        },
    });

// Read memo from workflow
[Workflow]
public class OrderWorkflow
{
    [WorkflowRun]
    public async Task<string> RunAsync(Order order)
    {
        var notes = Workflow.Memo["notes"];
        // ...
    }
}

Deterministic APIs for Values

Use these APIs within workflows for deterministic random values and UUIDs:

[Workflow]
public class MyWorkflow
{
    [WorkflowRun]
    public async Task<string> RunAsync()
    {
        // Deterministic GUID (same on replay)
        var uniqueId = Workflow.NewGuid();

        // Deterministic random (same on replay)
        var value = Workflow.Random.Next(1, 100);

        // Deterministic current time
        var now = Workflow.UtcNow;

        return uniqueId.ToString();
    }
}

Best Practices

1. Use records or classes with System.Text.Json support for input/output 2. Keep payloads small — see references/core/gotchas.md for limits 3. Encrypt sensitive data with IPayloadCodec 4. Use Workflow.NewGuid() and Workflow.Random for deterministic values 5. Use camelCase converter if interoperating with other SDKs

.NET Determinism Protection

Overview

The .NET SDK has no runtime sandbox. Determinism is enforced by developer convention and runtime task detection. Unlike the Python and TypeScript SDKs, the .NET SDK will not intercept or replace non-deterministic calls at compile time or import time. The SDK does provide a runtime EventListener that detects some invalid task scheduling, but catching all non-deterministic code requires following the rules below and testing, in particular replay tests (see references/dotnet/testing.md).

Runtime Task Detection

By default, the .NET SDK enables an EventListener that monitors task events. When workflow code accidentally starts a task on the wrong scheduler (e.g., via Task.Run), an InvalidWorkflowOperationException is thrown. This causes the workflow task to fail, which will continuously retry until the code is fixed.

// This will be detected at runtime and fail the workflow task
[Workflow]
public class BadWorkflow
{
    [WorkflowRun]
    public async Task RunAsync()
    {
        // BAD: Task.Run uses TaskScheduler.Default
        await Task.Run(() => DoSomething());
    }
}

.NET Task Determinism Rules

Many .NET Task APIs implicitly use TaskScheduler.Default, which breaks determinism. Here are the key rules:

Do NOT use:

• Task.Run — uses default scheduler. Use Workflow.RunTaskAsync.
• Task.ConfigureAwait(false) — leaves current context. Use ConfigureAwait(true) or omit.
• Task.Delay / Task.Wait / timeout-based CancellationTokenSource — uses system timers. Use Workflow.DelayAsync / Workflow.WaitConditionAsync.
• Task.WhenAny — use Workflow.WhenAnyAsync.
• Task.WhenAll — use Workflow.WhenAllAsync (technically safe currently, but wrapper is recommended).
• CancellationTokenSource.CancelAsync — use CancellationTokenSource.Cancel.
• System.Threading.Semaphore / SemaphoreSlim / Mutex — use Temporalio.Workflows.Semaphore / Mutex.

Be wary of:

• Third-party libraries that implicitly use TaskScheduler.Default
• Dataflow blocks and similar concurrency libraries with hidden default scheduler usage

Best Practices

1. *Always use `Workflow. alternatives** for Task operations in workflows 2. **Don't disable the EventListener** — it's on by default and catches mistakes at runtime 3. **Separate workflow and activity code** into different files/projects for clarity 4. **Use SortedDictionary** or sort collections before iterating — Dictionary<TKey, TValue>` iteration order is not guaranteed 5. Test with replay to catch non-determinism early 6. Review third-party library usage in workflow code for hidden default scheduler usage

.NET SDK Determinism

Overview

The .NET SDK has NO runtime sandbox (unlike Python/TypeScript). Workflows must be deterministic for replay, and determinism is enforced by developer convention and runtime task detection via an EventListener (see references/dotnet/determinism-protection.md).

Why Determinism Matters: History Replay

Temporal provides durable execution through History Replay. When a Worker restores workflow state, it re-executes workflow code from the beginning. This requires the code to be deterministic. See references/core/determinism.md for a deep explanation.

Forbidden Operations in Workflows

The following are forbidden inside workflow code but are appropriate to use in activities.

// DO NOT do these in workflows:
await Task.Run(() => { });              // Uses default scheduler
await Task.Delay(TimeSpan.FromSeconds(1)); // System timer
var now = DateTime.UtcNow;              // System clock
var r = new Random().Next();            // Non-deterministic
var id = Guid.NewGuid();               // Non-deterministic
File.ReadAllText("file.txt");           // I/O
await httpClient.GetAsync("...");       // Network I/O

Most non-determinism and side effects should be wrapped in Activities.

Safe Builtin Alternatives

Testing Replay Compatibility

Use WorkflowReplayer to verify your code changes are compatible with existing histories. See the Workflow Replay Testing section of references/dotnet/testing.md.

Best Practices

1. Always use Workflow.* APIs instead of standard .NET equivalents (see table above) 2. Never use ConfigureAwait(false) in workflows 3. Use SortedDictionary or sort before iterating collections 4. Move all I/O operations (network, filesystem, database) into activities 5. Use Workflow.Logger instead of Console.WriteLine for replay-safe logging 6. Keep workflow code focused on orchestration; delegate non-deterministic work to activities 7. Test with replay after making changes to workflow definitions

Temporal .NET SDK Reference

Overview

The Temporal .NET SDK provides a high-performance, type-safe approach to building durable workflows using C# and .NET. Workflows use attributes ([Workflow], [WorkflowRun]) and lambda expressions for type-safe invocations. Supports .NET Framework 4.6.2+ and .NET Core 3.1+ (including .NET 5+).

CRITICAL: The .NET SDK has no sandbox. Developers must be careful to avoid non-deterministic code in workflows. See the Determinism Rules section below and references/dotnet/determinism.md.

Understanding Replay

Temporal workflows are durable through history replay. For details on how this works, see references/core/determinism.md.

Quick Start

Add Dependency: Install the Temporal SDK NuGet package:

dotnet add package Temporalio

Activities.cs - Activity definitions (separate file for clarity):

using Temporalio.Activities;

public class MyActivities
{
    [Activity]
    public string Greet(string name)
    {
        return $"Hello, {name}!";
    }
}

GreetingWorkflow.workflow.cs - Workflow definition:

using Temporalio.Workflows;

[Workflow]
public class GreetingWorkflow
{
    [WorkflowRun]
    public async Task<string> RunAsync(string name)
    {
        return await Workflow.ExecuteActivityAsync(
            (MyActivities a) => a.Greet(name),
            new() { StartToCloseTimeout = TimeSpan.FromSeconds(30) });
    }
}

Worker (Program.cs) - Worker setup (registers activity and workflow, runs indefinitely and processes tasks):

using Temporalio.Client;
using Temporalio.Worker;

var client = await TemporalClient.ConnectAsync(new("localhost:7233"));

using var worker = new TemporalWorker(
    client,
    new TemporalWorkerOptions("my-task-queue")
        .AddWorkflow<GreetingWorkflow>()
        .AddAllActivities(new MyActivities()));

await worker.ExecuteAsync();

Start the dev server: Start temporal server start-dev in the background.

Start the worker: Run dotnet run in the worker project.

Starter (Program.cs) - Start a workflow execution:

using Temporalio.Client;

var client = await TemporalClient.ConnectAsync(new("localhost:7233"));

var result = await client.ExecuteWorkflowAsync(
    (GreetingWorkflow wf) => wf.RunAsync("my name"),
    new(id: $"greeting-{Guid.NewGuid()}", taskQueue: "my-task-queue"));

Console.WriteLine($"Result: {result}");

Run the workflow: Run dotnet run in the starter project. Should output: Result: Hello, my name!.

Key Concepts

Workflow Definition

• Use [Workflow] attribute on class
• Put any state initialization logic in the constructor of your workflow class to guarantee that it happens before signals/updates arrive. If your state initialization logic requires the workflow parameters, then add the [WorkflowInit] attribute and parameters to your constructor.
• Use [WorkflowRun] on the async entry point method
• Must return Task or Task<T>
• Use [WorkflowSignal], [WorkflowQuery], [WorkflowUpdate] for handlers

Activity Definition

• Use [Activity] attribute on methods
• Can be sync or async
• Instance methods support dependency injection
• Static methods are also supported

Worker Setup

• Connect client, create TemporalWorker with workflows and activities
• Use AddWorkflow<T>() and AddAllActivities(instance) or AddActivity(method)

Determinism

Workflow code must be deterministic! The .NET SDK has no sandbox. See the Determinism Rules section below and references/core/determinism.md and references/dotnet/determinism.md.

File Organization Best Practice

Keep Workflow definitions in separate files from Activity definitions. While not as critical as Python (no sandbox reloading), separation improves clarity and testability. Use the .workflow.cs extension for workflow files so the .editorconfig overrides (see below) apply only to workflow code.

MyTemporalApp/
├── Workflows/
│   └── GreetingWorkflow.workflow.cs  # Only Workflow classes
├── Activities/
│   └── TranslateActivities.cs       # Only Activity classes
├── Models/
│   └── OrderInput.cs                # Shared data models
├── Worker/
│   └── Program.cs                   # Worker setup
└── Starter/
    └── Program.cs                   # Client code to start workflows

Workflow .editorconfig

Workflow code violates some standard .NET analyzer rules. The recommended approach is to use the .workflow.cs file extension for workflow files and scope the overrides to that extension:

# Configuration specific for Temporal workflows
[*.workflow.cs]

# We use getters for queries, they cannot be properties
dotnet_diagnostic.CA1024.severity = none

# Don't force workflows to have static methods
dotnet_diagnostic.CA1822.severity = none

# Do not need ConfigureAwait for workflows
dotnet_diagnostic.CA2007.severity = none

# Do not need task scheduler for workflows
dotnet_diagnostic.CA2008.severity = none

# Workflow randomness is intentionally deterministic
dotnet_diagnostic.CA5394.severity = none

# Allow async methods to not have await in them
dotnet_diagnostic.CS1998.severity = none

# Don't force workflows to call async methods
dotnet_diagnostic.VSTHRD103.severity = none

# Don't avoid, but rather encourage things using TaskScheduler.Current in workflows
dotnet_diagnostic.VSTHRD105.severity = none

Determinism Rules

The .NET SDK has no sandbox like Python or TypeScript. Developers must avoid non-deterministic operations manually. Many standard .NET Task APIs use TaskScheduler.Default implicitly, which breaks determinism.

See references/dotnet/determinism.md for the full list of forbidden operations, safe alternatives, and best practices. See references/dotnet/determinism-protection.md for details on the runtime detection mechanism.

Common Pitfalls

1. Using `Task.Run` in workflows — Uses default scheduler, breaks determinism. Use Workflow.RunTaskAsync. 2. Using `Task.Delay` in workflows — Uses system timer. Use Workflow.DelayAsync. 3. `ConfigureAwait(false)` in workflows — Leaves the deterministic scheduler. Never use in workflows. 4. Non-`ApplicationFailureException` in workflows — Other exceptions retry the workflow task forever instead of failing the workflow. 5. Dictionary iteration in workflows — Dictionary<TKey, TValue> has no guaranteed order. Use SortedDictionary. 6. Forgetting to heartbeat — Long-running activities need ActivityExecutionContext.Current.Heartbeat() calls. 7. Using `CancellationTokenSource.CancelAsync` — Use CancellationTokenSource.Cancel instead. 8. Logging with `Console.WriteLine` in workflows — Use Workflow.Logger for replay-safe logging.

Writing Tests

See references/dotnet/testing.md for info on writing tests.

Additional Resources

Reference Files

• `references/dotnet/patterns.md` — Signals, queries, child workflows, saga pattern, etc.
• `references/dotnet/determinism.md` — Essentials of determinism in .NET
• `references/dotnet/gotchas.md` — .NET-specific mistakes and anti-patterns
• `references/dotnet/error-handling.md` — ApplicationFailureException, retry policies, non-retryable errors
• `references/dotnet/observability.md` — Logging, metrics, tracing
• `references/dotnet/testing.md` — WorkflowEnvironment, time-skipping, activity mocking
• `references/dotnet/advanced-features.md` — Schedules, worker tuning, dependency injection
• `references/dotnet/data-handling.md` — Data converters, payload encryption, etc.
• `references/dotnet/versioning.md` — Patching API, workflow type versioning, Worker Versioning
• `references/dotnet/determinism-protection.md` — Runtime task detection, .NET Task determinism rules

.NET SDK Error Handling

Overview

The .NET SDK uses ApplicationFailureException for application-specific errors and provides comprehensive retry policy configuration. Generally, the following information about errors and retryability applies across activities, child workflows and Nexus operations.

Application Failures

using Temporalio.Activities;
using Temporalio.Exceptions;

[Activity]
public async Task ValidateOrderAsync(Order order)
{
    if (!order.IsValid())
    {
        throw new ApplicationFailureException(
            "Invalid order",
            errorType: "ValidationError");
    }
}

Non-Retryable Errors

using Temporalio.Activities;
using Temporalio.Exceptions;

[Activity]
public async Task<string> ChargeCardAsync(ChargeCardInput input)
{
    if (!IsValidCard(input.CardNumber))
    {
        throw new ApplicationFailureException(
            "Permanent failure - invalid credit card",
            errorType: "PaymentError",
            nonRetryable: true); // Will not retry activity
    }
    return await ProcessPaymentAsync(input.CardNumber, input.Amount);
}

Handling Activity Errors in Workflows

using Temporalio.Workflows;
using Temporalio.Exceptions;

[Workflow]
public class MyWorkflow
{
    [WorkflowRun]
    public async Task<string> RunAsync()
    {
        try
        {
            return await Workflow.ExecuteActivityAsync(
                (MyActivities a) => a.RiskyActivityAsync(),
                new() { StartToCloseTimeout = TimeSpan.FromMinutes(5) });
        }
        catch (ActivityFailureException ex) when (!TemporalException.IsCanceledException(ex))
        {
            Workflow.Logger.LogError(ex, "Activity failed");
            throw new ApplicationFailureException(
                "Workflow failed due to activity error");
        }
    }
}

Retry Configuration

using Temporalio.Common;
using Temporalio.Workflows;

[Workflow]
public class MyWorkflow
{
    [WorkflowRun]
    public async Task<string> RunAsync()
    {
        return await Workflow.ExecuteActivityAsync(
            (MyActivities a) => a.MyActivityAsync(),
            new()
            {
                StartToCloseTimeout = TimeSpan.FromMinutes(10),
                RetryPolicy = new()
                {
                    MaximumInterval = TimeSpan.FromMinutes(1),
                    MaximumAttempts = 5,
                    NonRetryableErrorTypes = new[] { "ValidationError", "PaymentError" },
                },
            });
    }
}

Only set options such as MaximumInterval, MaximumAttempts etc. if you have a domain-specific reason to. If not, prefer to leave them at their defaults.

Timeout Configuration

[Workflow]
public class MyWorkflow
{
    [WorkflowRun]
    public async Task<string> RunAsync()
    {
        return await Workflow.ExecuteActivityAsync(
            (MyActivities a) => a.MyActivityAsync(),
            new()
            {
                StartToCloseTimeout = TimeSpan.FromMinutes(5),      // Single attempt
                ScheduleToCloseTimeout = TimeSpan.FromMinutes(30),  // Including retries
                HeartbeatTimeout = TimeSpan.FromMinutes(2),         // Between heartbeats
            });
    }
}

Workflow Failure

Critical .NET behavior: Only ApplicationFailureException will fail a workflow. All other exceptions (including standard .NET exceptions like NullReferenceException, KeyNotFoundException, etc.) will retry the workflow task indefinitely. This is by design — those are treated as bugs to be fixed with a code deployment, not reasons for the workflow to fail.

[Workflow]
public class MyWorkflow
{
    [WorkflowRun]
    public async Task<string> RunAsync()
    {
        if (someCondition)
        {
            throw new ApplicationFailureException(
                "Cannot process order",
                errorType: "BusinessError");
        }
        return "success";
    }
}

Note: Do not use nonRetryable: with ApplicationFailureException inside a workflow (as opposed to an activity).

Best Practices

1. Use specific error types for different failure modes 2. Mark permanent failures as non-retryable in activities 3. Configure appropriate retry policies 4. Log errors before re-raising 5. Use ActivityFailureException to catch activity failures in workflows 6. Design code to be idempotent for safe retries (see more at references/core/patterns.md) 7. Only throw ApplicationFailureException from workflows to fail them — other exceptions will retry the workflow task

.NET Gotchas

.NET-specific mistakes and anti-patterns. See also Common Gotchas for language-agnostic concepts.

.NET Task Determinism

The biggest .NET gotcha. Many Task APIs implicitly use TaskScheduler.Default, which breaks determinism. The SDK detects some of these at runtime via an EventListener, but not all.

Task.Run

// BAD: Uses TaskScheduler.Default
await Task.Run(() => DoSomething());

// GOOD: Uses current (deterministic) scheduler
await Workflow.RunTaskAsync(() => DoSomething());

Task.Delay / Thread.Sleep

// BAD: Uses system timer
await Task.Delay(TimeSpan.FromMinutes(5));

// GOOD: Creates durable timer in event history
await Workflow.DelayAsync(TimeSpan.FromMinutes(5));

ConfigureAwait(false)

// BAD: Leaves the deterministic context
var result = await SomeCallAsync().ConfigureAwait(false);

// GOOD: Stays on deterministic scheduler (or just omit ConfigureAwait)
var result = await SomeCallAsync().ConfigureAwait(true);
var result = await SomeCallAsync(); // Also fine

Task.WhenAll / Task.WhenAny

// BAD: Potential non-determinism
await Task.WhenAll(task1, task2);
await Task.WhenAny(task1, task2);

// GOOD: Deterministic wrappers
await Workflow.WhenAllAsync(task1, task2);
await Workflow.WhenAnyAsync(task1, task2);

Threading Primitives

// BAD: System threading primitives
var mutex = new System.Threading.Mutex();
var semaphore = new SemaphoreSlim(1);

// GOOD: Temporal workflow-safe alternatives
var mutex = new Temporalio.Workflows.Mutex();
var semaphore = new Temporalio.Workflows.Semaphore(1);

See references/dotnet/determinism-protection.md for the complete list.

Wrong Retry Classification

Example: Transient network errors should be retried. Authentication errors should not be. See references/dotnet/error-handling.md to understand how to classify errors.

Heartbeating

Forgetting to Heartbeat Long Activities

// BAD: No heartbeat, can't detect stuck activities
[Activity]
public async Task ProcessLargeFileAsync(string path)
{
    foreach (var chunk in ReadChunks(path))
        await ProcessAsync(chunk); // Takes hours, no heartbeat

// GOOD: Regular heartbeats with progress
[Activity]
public async Task ProcessLargeFileAsync(string path)
{
    var chunks = ReadChunks(path);
    for (var i = 0; i < chunks.Count; i++)
    {
        ActivityExecutionContext.Current.Heartbeat($"Processing chunk {i}");
        await ProcessAsync(chunks[i]);
    }
}

Heartbeat Timeout Too Short

// BAD: Heartbeat timeout shorter than processing time
await Workflow.ExecuteActivityAsync(
    (MyActivities a) => a.ProcessChunkAsync(),
    new()
    {
        StartToCloseTimeout = TimeSpan.FromMinutes(30),
        HeartbeatTimeout = TimeSpan.FromSeconds(10), // Too short!
    });

// GOOD: Heartbeat timeout allows for processing variance
await Workflow.ExecuteActivityAsync(
    (MyActivities a) => a.ProcessChunkAsync(),
    new()
    {
        StartToCloseTimeout = TimeSpan.FromMinutes(30),
        HeartbeatTimeout = TimeSpan.FromMinutes(2),
    });

Set heartbeat timeout as high as acceptable for your use case — each heartbeat counts as an action.

Cancellation

Not Handling Workflow Cancellation

// BAD: Cleanup doesn't run on cancellation
[Workflow]
public class BadWorkflow
{
    [WorkflowRun]
    public async Task RunAsync()
    {
        await Workflow.ExecuteActivityAsync(
            (MyActivities a) => a.AcquireResourceAsync(),
            new() { StartToCloseTimeout = TimeSpan.FromMinutes(5) });
        await Workflow.ExecuteActivityAsync(
            (MyActivities a) => a.DoWorkAsync(),
            new() { StartToCloseTimeout = TimeSpan.FromMinutes(5) });
        await Workflow.ExecuteActivityAsync(
            (MyActivities a) => a.ReleaseResourceAsync(), // Never runs if cancelled!
            new() { StartToCloseTimeout = TimeSpan.FromMinutes(5) });
    }
}

// GOOD: Use try/finally for cleanup
[Workflow]
public class GoodWorkflow
{
    [WorkflowRun]
    public async Task RunAsync()
    {
        await Workflow.ExecuteActivityAsync(
            (MyActivities a) => a.AcquireResourceAsync(),
            new() { StartToCloseTimeout = TimeSpan.FromMinutes(5) });
        try
        {
            await Workflow.ExecuteActivityAsync(
                (MyActivities a) => a.DoWorkAsync(),
                new() { StartToCloseTimeout = TimeSpan.FromMinutes(5) });
        }
        finally
        {
            await Workflow.ExecuteActivityAsync(
                (MyActivities a) => a.ReleaseResourceAsync(),
                new()
                {
                    StartToCloseTimeout = TimeSpan.FromMinutes(5),
                    CancellationToken = CancellationToken.None,
                });
        }
    }
}

Not Handling Activity Cancellation

Activities must opt in to receive cancellation. This requires:

1. Heartbeating — Cancellation is delivered via heartbeat 2. Checking the cancellation token — Token is triggered when heartbeat detects cancellation

// BAD: Activity ignores cancellation
[Activity]
public async Task LongActivityAsync()
{
    await DoExpensiveWorkAsync(); // Runs to completion even if cancelled
}

// GOOD: Heartbeat, check cancellation, and handle cleanup
[Activity]
public async Task LongActivityAsync()
{
    try
    {
        foreach (var item in items)
        {
            ActivityExecutionContext.Current.Heartbeat();
            ActivityExecutionContext.Current.CancellationToken.ThrowIfCancellationRequested();
            await ProcessAsync(item);
        }
    }
    catch (OperationCanceledException)
    {
        await CleanupAsync();
        throw;
    }
}

Testing

Not Testing Failures

It is important to make sure workflows work as expected under failure paths in addition to happy paths. Please see references/dotnet/testing.md for more info.

Not Testing Replay

Replay tests help you test that you do not have hidden sources of non-determinism bugs in your workflow code. Please see references/dotnet/testing.md for more info.

Timers and Sleep

Using Task.Delay

// BAD: Task.Delay uses system timer, not deterministic during replay
[Workflow]
public class BadWorkflow
{
    [WorkflowRun]
    public async Task RunAsync()
    {
        await Task.Delay(TimeSpan.FromMinutes(1)); // SDK will detect and fail the task
    }
}

// GOOD: Use Workflow.DelayAsync for deterministic timers
[Workflow]
public class GoodWorkflow
{
    [WorkflowRun]
    public async Task RunAsync()
    {
        await Workflow.DelayAsync(TimeSpan.FromMinutes(1)); // Deterministic
    }
}

Why this matters: Task.Delay uses the system clock, which differs between original execution and replay. Workflow.DelayAsync creates a durable timer in the event history, ensuring consistent behavior during replay.

Dictionary Iteration Order

// BAD: Dictionary iteration order is not guaranteed
var dict = new Dictionary<string, int> { ["b"] = 2, ["a"] = 1 };
foreach (var kvp in dict) // Order may differ between executions!
    await ProcessAsync(kvp.Key, kvp.Value);

// GOOD: Use SortedDictionary or sort before iterating
var dict = new SortedDictionary<string, int> { ["b"] = 2, ["a"] = 1 };
foreach (var kvp in dict) // Always iterates in key order
    await ProcessAsync(kvp.Key, kvp.Value);

.NET SDK Observability

Overview

The .NET SDK provides observability through logging, metrics, and tracing using standard .NET patterns.

Logging

Workflow Logging (Replay-Safe)

Use Workflow.Logger for replay-safe logging that avoids duplicate messages:

[Workflow]
public class MyWorkflow
{
    [WorkflowRun]
    public async Task<string> RunAsync(string name)
    {
        Workflow.Logger.LogInformation("Workflow started for {Name}", name);

        var result = await Workflow.ExecuteActivityAsync(
            (MyActivities a) => a.MyActivityAsync(),
            new() { StartToCloseTimeout = TimeSpan.FromMinutes(5) });

        Workflow.Logger.LogInformation("Activity completed with {Result}", result);
        return result;
    }
}

The workflow logger automatically:

• Suppresses duplicate logs during replay
• Includes workflow context (workflow ID, run ID, etc.)

Activity Logging

Use ActivityExecutionContext.Current.Logger for context-aware activity logging:

[Activity]
public async Task<string> ProcessOrderAsync(string orderId)
{
    var logger = ActivityExecutionContext.Current.Logger;
    logger.LogInformation("Processing order {OrderId}", orderId);

    // Perform work...

    logger.LogInformation("Order processed successfully");
    return "completed";
}

Customizing Logger Configuration

using Microsoft.Extensions.Logging;

var client = await TemporalClient.ConnectAsync(new("localhost:7233")
{
    LoggerFactory = LoggerFactory.Create(builder =>
        builder
            .AddSimpleConsole(options => options.TimestampFormat = "[HH:mm:ss] ")
            .SetMinimumLevel(LogLevel.Information)),
});

Metrics

Enabling SDK Metrics

Metrics are configured on TemporalRuntime. Create the runtime globally before any client/worker and set a Prometheus endpoint or custom metric meter.

using Temporalio.Client;
using Temporalio.Runtime;

// Create runtime with Prometheus endpoint
var runtime = new TemporalRuntime(new()
{
    Telemetry = new() { Metrics = new() { Prometheus = new("0.0.0.0:9000") } },
});

// Use this runtime for all clients
var client = await TemporalClient.ConnectAsync(
    new("localhost:7233") { Runtime = runtime });

Alternatively, use Temporalio.Extensions.DiagnosticSource to bridge metrics to a .NET System.Diagnostics.Metrics.Meter for integration with OpenTelemetry or other .NET metrics pipelines.

Key SDK Metrics

• temporal_request — Client requests to server
• temporal_workflow_task_execution_latency — Workflow task processing time
• temporal_activity_execution_latency — Activity execution time
• temporal_workflow_task_replay_latency — Replay duration

Search Attributes (Visibility)

See the Search Attributes section of references/dotnet/data-handling.md

Best Practices

1. Use Workflow.Logger in workflows, ActivityExecutionContext.Current.Logger in activities 2. Don't use Console.WriteLine in workflows — it will produce duplicate output on replay 3. Configure metrics for production monitoring 4. Use Search Attributes for business-level visibility