Temporal Cloud

Temporal Cloud Skill

Help users diagnose and resolve Temporal Cloud connectivity, authentication, and configuration issues using tcld and temporal CLI.

Core Philosophy

Cloud issues are frustrating because they sit at the intersection of configuration, networking, authentication, and Temporal-specific code. Most problems fall into predictable patterns. This skill provides systematic diagnosis to quickly identify root causes and prescribe fixes.

References:

• See references/cloud-troubleshooting-reference.md for full CLI command reference and error codes
• See references/common-scenarios.md for step-by-step setup walkthroughs
• Environment configuration docs - SDK setup for connecting to Cloud
• HA namespace connectivity - multi-region endpoint and DNS setup
• Dev Success troubleshooting guide - companion connection troubleshooting guide

Out of scope: Worker performance tuning, scaling, metrics interpretation, SDK-specific config, deployment patterns. Those topics are covered by separate worker-focused skills.

Issue Classification

The Process

Step 1: Identify the Category

Ask the user:

• What's the exact error message? (copy-paste if possible)
• What are you trying to do? (tcld command, starting workers, running workflows)
• What changed recently? (new certs, new namespace, new region)

Step 2: Gather Context

For SDK/client snippet reviews:

• Which auth method are you using: API key or mTLS?
• Which SDK and version are you using?
• What exact HostPort / address are you using?
• What exact Namespace are you using?
• Is this SDK code, temporal CLI, or tcld?

For tcld issues:

• Can you run tcld account get?
• Multiple Temporal accounts?

For connection issues:

• What's your exact address / HostPort?
• Using mTLS or API keys?
• Which SDK and version are you using?
• Any firewall/proxy between you and Cloud?

For ambiguous runtime errors:

• Where exactly do you see the error: workflow start, signal/update, polling, querying, logs?
• Is this happening before work starts, while polling, or while workflow code is already running?
• Are pollers present on the relevant task queue?
• Did this start after a traffic spike, deploy, or config change?

For certificate issues:

• When were certs generated?
• What CA was used?
• Is CA uploaded to namespace?

For worker issues:

• Are workers running? How many?
• What does temporal task-queue describe show?
• Any errors in worker logs?

Step 3: Apply Decision Tree

Use the appropriate decision tree based on category (see below).

Step 4: Provide Fix

Give specific commands to resolve the issue, with verification steps.

Always include a confidence score for the proposed diagnosis or fix:

• Confidence: 9-10/10 when the symptom, operation, and confirming signals line up cleanly
• Confidence: 6-8/10 when the evidence is good but one plausible alternative remains
• Confidence: 1-5/10 when the issue is still ambiguous and the "fix" is really the next discriminating check

If the problem is ambiguous, say so explicitly and keep the recommendation scoped to the next check rather than presenting a speculative root cause as settled.

Decision Trees

tcld Login Issues

Symptom: tcld login not working
│
├─ Can `tcld account get` run?
│  ├─ Yes → Login is valid; continue with account verification
│  └─ No → Run `tcld login`
│
├─ Token refresh failed?
│  └─ tcld logout && tcld login
│
├─ Wrong organization/account?
│  ├─ tcld account get
│  └─ Verify the expected namespace appears in `tcld namespace list`
│
└─ "unauthorized" or auth errors?
   └─ tcld logout && tcld login

Connection Failures

Docs: Environment configuration - SDK connection options

Endpoint check before network debugging:

Exception: Namespaces using Flexible Auth (pre-release) cannot use Namespace Endpoints yet.

Symptom: Can't connect to Temporal Cloud
│
├─ Check: Using Namespace Endpoint?
│  ├─ Using regional endpoint (`*.api.temporal.io`) without HA need?
│  │  └─ Switch to Namespace Endpoint (`<ns>.<acct>.tmprl.cloud:7233`)
│  ├─ Using old/stale endpoint format?
│  │  └─ Switch to Namespace Endpoint
│  └─ Endpoint looks correct → Continue
│
├─ Check: DNS resolution
│  └─ nslookup <host-from-address>
│     ├─ Fails → DNS issue (check network, VPN)
│     └─ Succeeds → Continue
│
├─ Check: Port connectivity
│  └─ nc -zv <host-from-address> 7233
│     ├─ Fails → Firewall blocking port 7233
│     └─ Succeeds → Continue
│
├─ Check: TLS handshake
│  └─ openssl s_client -connect <address>
│     ├─ Fails → Certificate issue (see mTLS tree)
│     └─ Succeeds → Continue
│
└─ Check: Temporal CLI test
   └─ temporal workflow list --limit 1 --address ...
      ├─ PERMISSION_DENIED → Check namespace name format
      ├─ UNAUTHENTICATED → Certificate not accepted
      └─ Works → Connection OK, issue elsewhere

Ambiguous Runtime Errors

Do not assume these are pure connectivity failures. Classify them by operation first.

Use this decision sequence:

Symptom: ambiguous runtime error
│
├─ Check: Which operation returned the error?
│  ├─ start / signal / update / query request
│  ├─ poll loop / worker logs
│  └─ UI / metrics only
│
├─ Check: Is work reaching a task queue?
│  ├─ No pollers listed
│  │  └─ Treat as worker connectivity / config until proven otherwise
│  ├─ Pollers listed, backlog growing
│  │  └─ Worker capacity / tuning issue (out of scope for this skill)
│  └─ Pollers listed, no backlog issue
│     └─ Continue
│
├─ For `context deadline exceeded`
│  ├─ Happens before any work starts
│  │  └─ Check endpoint format, auth, proxy, DNS, firewall
│  ├─ Happens on workflow start with large payloads
│  │  └─ Consider payload size / client timeout path
│  └─ Happens during local execution / queries
│     └─ Consider blocked execution path, local activity, or client-side timeout
│
└─ For `workflow is busy`
   ├─ Identify exact API / operation
   ├─ Check whether user is conflating this with a generic workflow failure
   └─ Explain that the error class is operation-specific before prescribing fixes

If the operation and surrounding signals still do not make the error interpretable, label it as ambiguous and gather more context before prescribing a fix.

When responding, attach a confidence score from 1-10 to the proposed diagnosis or next step. Ambiguous cases should carry a low-confidence score and a narrow next check rather than a broad claimed fix.

SDK Snippet Review

When the user pastes SDK config, validate the config itself before suggesting lower-level networking checks.

Review in this order: 1. Auth method: API key vs mTLS 2. Address / HostPort: should be Namespace Endpoint (<ns>.<acct>.tmprl.cloud:7233) for most cases 3. Namespace: full Cloud namespace format (<namespace>.<account-id>) 4. TLS config: empty tls.Config{} is normal for API key auth; client cert/key required for mTLS 5. Environment config: prefer TEMPORAL_ADDRESS, TEMPORAL_NAMESPACE, TEMPORAL_API_KEY, TEMPORAL_TLS_CLIENT_CERT_PATH, TEMPORAL_TLS_CLIENT_KEY_PATH

Common snippet diagnoses:

• Using regional endpoint (*.api.temporal.io) when Namespace Endpoint would work → simplify to <ns>.<acct>.tmprl.cloud:7233
• Empty HostPort + Cloud namespace/auth → missing explicit Cloud endpoint
• API key without TLS enabled by the SDK/runtime → check SDK-specific Cloud guidance
• Old endpoint format (stale docs/examples) → update to current Namespace Endpoint

If the snippet is wrong, fix that first. Do not lead with DNS/TLS debugging until the endpoint and namespace are plausible.

Certificate Errors

Symptom: x509 certificate errors
│
├─ "certificate signed by unknown authority"
│  ├─ Is CA uploaded to namespace?
│  │  └─ tcld namespace accepted-client-ca list --namespace <ns>
│  │     ├─ CA not listed → Add it:
│  │     │  tcld namespace accepted-client-ca add \
│  │     │    --namespace <ns> --ca-certificate-file ca.pem
│  │     └─ CA listed → Cert not signed by that CA
│  │        └─ Verify: openssl verify -CAfile ca.pem client.pem
│  │
│  └─ Self-signed cert without CA?
│     └─ Must use CA-signed certs for Cloud
│
├─ "certificate has expired"
│  ├─ Check expiry: openssl x509 -enddate -noout -in cert.pem
│  └─ Generate new cert:
│     tcld generate-certificates end-entity-certificate \
│       --organization <org> --validity-period 365d \
│       --ca-certificate-file ca.pem --ca-key-file ca.key \
│       --certificate-file client.pem --key-file client.key
│
├─ "private key does not match"
│  └─ Wrong key file - verify match:
│     openssl x509 -modulus -noout -in cert.pem | md5
│     openssl rsa -modulus -noout -in key.pem | md5
│
└─ "bad certificate" from server
   └─ Server rejected cert - CA not in accepted list
      └─ tcld namespace accepted-client-ca list --namespace <ns>

Namespace Issues

Symptom: namespace not found or access denied
│
├─ Check: Namespace name format
│  ├─ Format: <namespace-name>.<account-id>
│  ├─ Example: my-namespace.a1b2c3
│  └─ Wrong format → Use full namespace name
│
├─ Check: Namespace exists
│  └─ tcld namespace list
│     └─ Not listed → Wrong account, or namespace not created
│
├─ Check: Address format
│  ├─ Namespace Endpoint (recommended): <namespace>.<account>.tmprl.cloud:7233
│  ├─ Regional Endpoint (HA only): <region>.<cloud_provider>.api.temporal.io:7233
│  └─ Using wrong or stale endpoint? → switch to Namespace Endpoint
│
└─ Check: User permissions
   └─ tcld user list
      └─ Verify user has access to namespace

Worker Issues

This skill diagnoses Cloud connectivity issues for workers. Worker performance tuning, scaling, and deployment patterns are out of scope.

Docs: Environment configuration - SDK connection setup

Symptom: Workers not picking up tasks
│
├─ Check: Are workers running?
│  └─ Verify worker process is up, check logs for errors
│
├─ Check: Task queue status
│  └─ temporal task-queue describe --task-queue <queue>
│     ├─ No pollers listed → Workers not connected (Cloud issue)
│     │  ├─ Check task queue name matches
│     │  ├─ Check namespace in worker config
│     │  ├─ Verify SDK connection options (see env config docs)
│     │  └─ Check for connection errors in logs
│     │
│     │  Note: if only metrics or dashboards say "no pollers", verify with CLI or another direct surface before concluding workers are absent.
│     │
│     ├─ Pollers listed but backlog growing
│     │  └─ NOT a Cloud issue (worker scaling/tuning problem)
│     │
│     └─ Pollers listed, no backlog
│        └─ Workers healthy, issue is elsewhere
│
├─ Check: DNS caching (common in K8s)
│  └─ Stale DNS can cause workers to connect to wrong endpoint
│     └─ Restart workers to refresh DNS
│
└─ Check: Rate limiting
   └─ RESOURCE_EXHAUSTED in logs?
      └─ See Rate Limiting section

Scope clarification:

HA Namespace Connectivity

Docs: HA namespace connectivity

HA (multi-region) namespaces use a hierarchical DNS structure:

• Namespace endpoint: <ns>.<acct>.tmprl.cloud (CNAME to active region)
• Regional endpoint: <region>.region.tmprl.cloud
• During failover, CNAME switches regions (15s TTL, ~30s convergence)

Symptom: HA namespace connectivity or failover issues
│
├─ Check: DNS resolution
│  └─ nslookup <namespace>.tmprl.cloud
│     ├─ Should return CNAME → <region>.region.tmprl.cloud
│     └─ Then resolve to IP address
│
├─ Symptom: Clients not failing over
│  ├─ Check: DNS caching
│  │  ├─ TTL is 15s - clients should converge within 30s
│  │  ├─ Some DNS resolvers cache longer
│  │  └─ Fix: Restart workers to refresh DNS
│  │
│  └─ Check: SDK connection caching
│     └─ Some SDKs cache connections - may need restart
│
├─ Symptom: PrivateLink not working after failover
│  ├─ Check: Regional DNS override configured?
│  │  └─ Need Route 53 private hosted zone for region.tmprl.cloud
│  │     mapping regional endpoints to VPC endpoint IPs
│  │
│  └─ Check: Inter-region connectivity
│     └─ Workers need Transit Gateway or VPC Peering to reach
│        VPC endpoints in both regions
│
├─ Symptom: GCP Private Service Connect not working
│  └─ NOT SUPPORTED: Private connectivity not yet offered for
│     GCP Multi-region Namespaces
│
└─ Symptom: sa-east-1 region issues
   └─ NOT SUPPORTED: sa-east-1 not available for Multi-region namespaces (because there are no other regions on the continent)

Worker placement for HA:

• Option A: Run workers in both regions continuously
• Option B: Single region + Transit Gateway/VPC Peering for failover access

Private Connectivity Issues

Symptom: PrivateLink or Private Service Connect not working
│
├─ AWS PrivateLink
│  ├─ Check: VPC endpoint status
│  │  └─ AWS Console → VPC → Endpoints → Status = available
│  │
│  ├─ Check: Security groups
│  │  └─ Allow outbound to endpoint on port 7233
│  │
│  ├─ Check: DNS resolution
│  │  └─ Should resolve to private IP (10.x or 172.x)
│  │
│  └─ Check: Connectivity rules
│     ├─ tcld connectivity-rule list --namespace <ns>
│     └─ If needed, attach rules with:
│        tcld namespace set-connectivity-rules --namespace <ns> --connectivity-rule-ids <id>
│
├─ GCP Private Service Connect
│  ├─ Check: PSC endpoint status
│  │  └─ GCP Console → Network Services → Private Service Connect
│  │
│  ├─ Check: Firewall rules
│  │  └─ Allow egress to PSC endpoint
│  │
│  └─ Check: DNS configuration
│     └─ Cloud DNS zone for tmprl.cloud pointing to PSC
│
└─ General
   └─ Verify rule details:
      tcld connectivity-rule get --connectivity-rule-id <id>

Rate Limiting

Symptom: RESOURCE_EXHAUSTED errors
│
├─ Check: Which operation is limited?
│  └─ Error message indicates operation type
│
├─ "namespace write ops" exceeded
│  ├─ Too many workflow starts
│  ├─ Too many signals/updates
│  └─ Fix: Add backoff, batch operations
│
├─ "namespace read ops" exceeded
│  ├─ Too many list/query operations
│  └─ Fix: Add caching, reduce polling frequency
│
└─ Poll operations rate limited
   ├─ Too many pollers across workers
   └─ Fix: Reduce MaxConcurrentPollers settings

Common Scenarios

See references/common-scenarios.md for step-by-step walkthroughs:

• API key connectivity setup
• New namespace setup (mTLS)
• Certificate rotation
• Switching from mTLS to API keys

Common Pitfalls

Tips

• Always get the exact error message - Copy-paste, don't paraphrase
• Check the simple things first - DNS, port connectivity, cert expiry
• Use Namespace Endpoint by default - <ns>.<acct>.tmprl.cloud:7233 works for both mTLS and API key auth
• Use openssl for cert issues - It gives clearer error messages than SDKs
• tcld account get is your friend - Shows the current account context
• Namespace names are case-sensitive - Match exactly
• API keys are easier than mTLS - Consider for simpler setups
• Point users to env config docs - Environment configuration covers all SDK connection options

Temporal Cloud Skill

A skill to help troubleshoot Temporal Cloud connectivity, authentication, and configuration issues.

[!WARNING]

This Skill is currently in Public Preview, and will continue to evolve and improve.

We would love to hear your feedback - positive or negative - over in the Community Slack, in the #topic-ai channel

Installation

Via npx skills - supports all major coding agents

1. npx skills add temporalio/skills-temporal-cloud 2. Follow prompts

Via manually cloning the skill repo:

1. mkdir -p ~/.claude/skills && git clone https://github.com/temporalio/skills-temporal-cloud ~/.claude/skills/temporal-cloud

Appropriately adjust the installation directory based on your coding agent.

Cloud Troubleshooting Reference

Technical reference for debugging Temporal Cloud connectivity, authentication, and configuration issues.

Table of Contents

• CLI Tools Overview — tcld vs temporal comparison, related resources
• Connectivity Diagnostic Ladder — 7-layer bottom-up debugging model
• tcld Commands — auth, namespace, certs, private connectivity, account management
• temporal CLI Commands — connection testing, task queues, workflows, endpoint selection
• OpenSSL Commands — certificate inspection, TLS connection testing
• Environment Variables — SDK connection env vars
• Client and Version Considerations — tcld vs temporal CLI vs SDK, version matters for diagnosis
• Error Codes Reference — connection errors, ambiguous errors, TLS, auth, rate limiting
• Cloud Endpoint Patterns — standard, regional, private connectivity endpoints
• Quick Diagnostic Commands — full connectivity check scripts (mTLS + API key)

CLI Tools Overview

Two main CLIs for Temporal Cloud:

Related resources:

• Worker best practices - deployment, tuning, and operational guidance
• Dev Success: Troubleshooting Connection Issues - companion step-by-step connection troubleshooting guide

Connectivity Diagnostic Ladder

Important: Run diagnostics from the same machine, pod, or container where the problem occurs. Network reachability, DNS resolution, and TLS behavior can differ between your laptop and the production environment.

Connection issues have layers. If a lower layer fails, everything above it will fail too. Debug bottom-up:

Start at step 1. Move up only after the current step succeeds.

tcld Commands

Authentication

# Standard login (opens browser)
tcld login

# Clear stale tokens and re-auth
tcld logout && tcld login

# Check current account context
tcld account get

Namespace Management

# List namespaces
tcld namespace list

# Get namespace details
tcld namespace get --namespace <namespace>

# List accepted client CA certificates
tcld namespace accepted-client-ca list --namespace <namespace>

# Add CA certificate
tcld namespace accepted-client-ca add --namespace <namespace> --ca-certificate-file ca.pem

# Remove CA certificate (by fingerprint)
tcld namespace accepted-client-ca remove --namespace <namespace> --fp <fingerprint>

Certificate Generation

# Generate new CA
tcld generate-certificates certificate-authority-certificate \
  --organization temporal \
  --validity-period 365d \
  --ca-certificate-file output_dir/ca.pem \
  --ca-key-file output_dir/ca.key

# Generate leaf certificate from CA
tcld generate-certificates end-entity-certificate \
  --organization temporal \
  --validity-period 365d \
  --ca-certificate-file output_dir/ca.pem \
  --ca-key-file output_dir/ca.key \
  --certificate-file output_dir/client.pem \
  --key-file output_dir/client.key

# Generate another leaf certificate
tcld generate-certificates end-entity-certificate \
  --organization temporal \
  --validity-period 365d \
  --ca-certificate-file ca.pem \
  --ca-key-file ca.key \
  --certificate-file output_dir/client.pem \
  --key-file output_dir/client.key

Private Connectivity

# List connectivity rules for a namespace
tcld connectivity-rule list --namespace <namespace>

# Get rule details
tcld connectivity-rule get --connectivity-rule-id <rule-id>

# Create AWS PrivateLink rule
tcld connectivity-rule create \
  --connectivity-type private \
  --connection-id "vpce-abcde" \
  --region "aws-us-east-1"

# Create GCP Private Service Connect rule
tcld connectivity-rule create \
  --connectivity-type private \
  --connection-id "1234567890" \
  --region "gcp-us-central1" \
  --gcp-project-id "my-project-123"

# Attach connectivity rules to a namespace
tcld namespace set-connectivity-rules \
  --namespace "my-namespace.abc123" \
  --connectivity-rule-ids "rule-id-1" \
  --connectivity-rule-ids "rule-id-2"

Account & User Management

# Get account info
tcld account get

# List users
tcld user list

# List service accounts
tcld service-account list

# Create service account
tcld service-account create --name <name> --description "<description>"

# Create API key for service account
tcld apikey create --service-account-id <id> --description "<description>"

temporal CLI Commands

Connection Testing

# Basic connectivity test
temporal workflow list --limit 1 \
  --address <namespace>.tmprl.cloud:7233 \
  --namespace <namespace> \
  --tls-cert-path client.pem \
  --tls-key-path client.key

# With API key (same Namespace Endpoint, different auth)
temporal workflow list --limit 1 \
  --address <namespace>.tmprl.cloud:7233 \
  --namespace <namespace> \
  --api-key <key>

# Cluster health check
temporal operator cluster health \
  --address <namespace>.tmprl.cloud:7233 \
  --namespace <namespace> \
  --tls-cert-path client.pem \
  --tls-key-path client.key

Task Queue Diagnostics

# Describe task queue (shows pollers, backlog)
temporal task-queue describe \
  --task-queue <queue-name> \
  --address <address> \
  --namespace <namespace> \
  --tls-cert-path client.pem \
  --tls-key-path client.key

# Get task queue types
temporal task-queue get-build-ids \
  --task-queue <queue-name> \
  --address <address> \
  --namespace <namespace>

Workflow Operations

# List workflows
temporal workflow list \
  --address <address> \
  --namespace <namespace>

# Describe specific workflow
temporal workflow describe \
  --workflow-id <id> \
  --address <address> \
  --namespace <namespace>

# Show workflow history
temporal workflow show \
  --workflow-id <id> \
  --address <address> \
  --namespace <namespace>

Endpoint Selection

Namespace Endpoints work for both mTLS and API key auth. Use Regional Endpoints only for advanced multi-region HA routing. See namespace access docs.

OpenSSL Commands

Certificate Inspection

# Check certificate expiry
openssl x509 -enddate -noout -in cert.pem

# View full certificate details
openssl x509 -text -noout -in cert.pem

# Check certificate chain
openssl verify -CAfile ca.pem cert.pem

# View certificate subject/issuer
openssl x509 -subject -issuer -noout -in cert.pem

# Check if cert matches private key
openssl x509 -modulus -noout -in cert.pem | md5
openssl rsa -modulus -noout -in key.pem | md5
# (outputs should match)

TLS Connection Testing

# Test TLS handshake
openssl s_client -connect <namespace>.tmprl.cloud:7233 \
  -cert client.pem -key client.key -CAfile ca.pem

# Test with SNI (required for Temporal Cloud)
openssl s_client -connect <namespace>.tmprl.cloud:7233 \
  -servername <namespace>.tmprl.cloud \
  -cert client.pem -key client.key

# Show server certificate
openssl s_client -connect <namespace>.tmprl.cloud:7233 \
  -showcerts </dev/null 2>/dev/null | openssl x509 -text -noout

Environment Variables

SDK Connection

All auth methods use the same Namespace Endpoint.

Client and Version Considerations

When diagnosing connection or auth issues, first determine what the user is running:

1. Which client? — tcld, temporal CLI, or an SDK (Go, Java, TypeScript, Python, .NET, Ruby) 2. Which version? (e.g., tcld v0.x.x, temporal v1.x.x, go.temporal.io/sdk v1.x.x) 3. Which runtime/language version? (Go 1.22, Java 17, Node 20, Python 3.12, etc. — SDK only)

Each client has different connection and auth paths. tcld uses browser-based OAuth to the control plane. temporal CLI uses mTLS or API keys to namespace endpoints. SDKs use mTLS or API keys but with SDK-specific TLS and connection handling.

Why SDK version matters:

SDK repositories and development docs:

• Go: github.com/temporalio/sdk-go — development docs
• Java: github.com/temporalio/sdk-java — development docs
• TypeScript: github.com/temporalio/sdk-typescript — development docs
• Python: github.com/temporalio/sdk-python — development docs
• .NET: github.com/temporalio/sdk-dotnet — development docs
• Ruby: github.com/temporalio/sdk-ruby — development docs

Error Codes Reference

Connection Errors

Ambiguous Error Notes

Some high-frequency error strings are too ambiguous to classify from text alone. Use surrounding context before deciding whether the problem is Cloud connectivity, worker capacity, or application behavior.

context deadline exceeded

Do not treat this as a pure network error by default.

Common possibilities:

• wrong endpoint or stale endpoint format
• proxy, DNS, or firewall problem
• client-side timeout before work is accepted
• oversized payload on workflow start
• blocked or slow local execution path

Best first question:

• where exactly did this occur: workflow start, signal/update, query, poll loop, or general logs?

context deadline exceeded with PrivateLink

When using AWS PrivateLink or GCP Private Service Connect, context deadline exceeded usually means one of these issues in order of likelihood:

1. PrivateLink not enabled on the namespace

Verify in the Cloud UI: https://cloud.temporal.io/namespaces/<namespace> → Connect → look for PrivateLink. If not enabled, submit a support ticket including your AWS Account ID and Region.

2. Network connectivity to the VPC endpoint

Test from within your application environment (e.g., a Kubernetes pod):

# Install if needed: apk add netcat-openbsd
nc -v -w2 <vpc-endpoint-dns>:7233

A successful result shows succeeded!. If it times out, the issue is likely VPC security groups or network ACLs not permitting TCP on port 7233.

3. TLS handshake failure

Test the full TLS handshake with client certificates:

openssl s_client \
  -connect <vpc-endpoint-dns>:7233 \
  -cert client.pem -key client.key \
  -servername <namespace>.tmprl.cloud

A successful handshake shows SSL handshake has read NNNN bytes with Verification: OK. If it shows read 0 bytes, check VPC endpoint security group inbound rules for port 7233.

4. Missing TLS server name override in SDK configuration

When connecting through a VPC endpoint, the TLS server name must be overridden to match the namespace endpoint. Without this, the TLS handshake will fail even if network connectivity succeeds.

temporal CLI:

temporal workflow count \
  --address <vpc-endpoint-dns>:7233 \
  --tls-cert-path ./client.pem \
  --tls-key-path ./client.key \
  --tls-server-name <namespace>.tmprl.cloud \
  --namespace <namespace>

Java SDK (PrivateLink with mTLS):

SslContext sslContext = SimpleSslContextBuilder
    .forPKCS8(clientCertInputStream, clientKeyInputStream).build();

WorkflowServiceStubs service =
    WorkflowServiceStubs.newServiceStubs(
        WorkflowServiceStubsOptions.newBuilder()
            .setSslContext(sslContext)
            .setTarget("<vpc-endpoint-dns>:7233")
            .setChannelInitializer(
                c -> c.overrideAuthority("<namespace>.tmprl.cloud"))
            .build());

Go SDK:

c, err := client.Dial(client.Options{
    HostPort:  "<vpc-endpoint-dns>:7233",
    Namespace: "<namespace>",
    ConnectionOptions: client.ConnectionOptions{
        TLS: &tls.Config{
            Certificates: []tls.Certificate{cert},
            ServerName:   "<namespace>.tmprl.cloud",
        },
    },
})

Python SDK:

client = await Client.connect(
    "<vpc-endpoint-dns>:7233",
    namespace="<namespace>",
    tls=TLSConfig(
        client_cert=cert_bytes,
        client_private_key=key_bytes,
        domain="<namespace>.tmprl.cloud",
    ),
)

TypeScript SDK:

const connection = await NativeConnection.connect({
  address: "<vpc-endpoint-dns>:7233",
  tls: {
    serverNameOverride: "<namespace>.tmprl.cloud",
    clientCertPair: {
      crt: fs.readFileSync(clientCertPath),
      key: fs.readFileSync(clientKeyPath),
    },
  },
});

Note: Temporal recommends using private DNS instead of manual server name overrides when possible. With private DNS configured, the namespace endpoint (<namespace>.tmprl.cloud:7233) resolves directly to the VPC endpoint, eliminating the need for TLS server name overrides.

workflow is busy / RESOURCE_EXHAUSTED: Workflow is busy

This is often interpreted too broadly. It commonly indicates contention or temporary execution delay on a specific operation, not necessarily a hard workflow failure.

Common possibilities:

• operation-level contention or throttling (e.g., concurrent signals/updates to the same workflow)
• too many pending activities, child workflows, or timers on a single workflow execution
• user misunderstanding of what the error class refers to

Best first questions:

• which operation returned the error?
• were pollers present?
• was backlog or schedule-to-start latency increasing at the same time?

no pollers

Treat this as a worker reachability or configuration clue, not a complete diagnosis.

Common possibilities:

• workers are not connected to the expected namespace or task queue
• workers are up but misconfigured
• metrics or dashboards are delayed or misleading

Best first checks:

• verify pollers from a direct task-queue surface
• compare with worker logs
• separate "no pollers" from "pollers present but overloaded"

TLS/Certificate Errors

Authentication Errors

Rate Limiting

Cloud Endpoint Patterns

Standard Endpoints

Namespace Endpoint (recommended, all auth): <namespace>.<account>.tmprl.cloud:7233
Regional Endpoint (advanced HA only):       <region>.<cloud_provider>.api.temporal.io:7233
Web UI:                                     https://cloud.temporal.io/namespaces/<namespace>

Namespace Endpoints work for both mTLS and API key auth. All namespaces can use them (exception: Flexible Auth pre-release namespaces).

Regional Endpoints

Pattern: <region>.<cloud_provider>.api.temporal.io:7233

Example: us-east-1.aws.api.temporal.io:7233

Only recommended for advanced multi-region HA routing. For the full list of supported regions and providers, see Cloud regions. See also namespace access docs.

Note: Namespace name includes account ID suffix. Full format: <namespace-name>.<account-id>

Private Connectivity Endpoints

When using PrivateLink/Private Service Connect:

# AWS PrivateLink
Address: <namespace>.<account>.tmprl.cloud:7233
(resolves to VPC endpoint)

# GCP Private Service Connect
Address: <namespace>.<account>.tmprl.cloud:7233
(resolves to PSC endpoint)

Quick Diagnostic Commands

Full Connectivity Check

#!/bin/bash
# --- mTLS variant ---
# NOTE: Run this script from the machine/pod experiencing the issue.
NS="your-namespace.account-id"
CERT="client.pem"
KEY="client.key"
HOST="$NS.tmprl.cloud"
ADDRESS="$HOST:7233"

echo "=== DNS Resolution ==="
nslookup "$HOST"

echo "=== Port Connectivity ==="
nc -zv "$HOST" 7233

echo "=== TLS Handshake ==="
openssl s_client -connect "$ADDRESS" \
  -servername "$HOST" \
  -cert $CERT -key $KEY </dev/null 2>&1 | head -20

echo "=== Temporal CLI Test ==="
temporal workflow list --limit 1 \
  --address "$ADDRESS" \
  --namespace $NS \
  --tls-cert-path $CERT \
  --tls-key-path $KEY

#!/bin/bash
# --- API key variant (same Namespace Endpoint, different auth) ---
# NOTE: Run this script from the machine/pod experiencing the issue.
NS="your-namespace.account-id"
HOST="$NS.tmprl.cloud"
ADDRESS="$HOST:7233"
API_KEY="${TEMPORAL_API_KEY}"

echo "=== DNS Resolution ==="
nslookup "$HOST"

echo "=== Port Connectivity ==="
nc -zv "$HOST" 7233

echo "=== TLS Handshake ==="
openssl s_client -connect "$ADDRESS" \
  -servername "$HOST" </dev/null 2>&1 | head -20

echo "=== Temporal CLI Test ==="
temporal workflow list --limit 1 \
  --address "$ADDRESS" \
  --namespace $NS \
  --api-key "$API_KEY"

Certificate Expiry Check

#!/bin/bash
for cert in *.pem; do
  echo "$cert:"
  openssl x509 -enddate -noout -in "$cert" 2>/dev/null || echo "  (not a certificate)"
done

Common Scenarios

Step-by-step walkthroughs for common Temporal Cloud setup and configuration tasks.

API Key Connectivity

User is connecting to Temporal Cloud with an API key.

Checklist: 1. Use the full namespace name (<name>.<account-id>) 2. Use the Namespace Endpoint (<ns>.<acct>.tmprl.cloud:7233) - works for all auth methods 3. Keep credentials in environment variables where possible 4. Verify with temporal workflow list

export TEMPORAL_NAMESPACE="my-ns.abc123"
export TEMPORAL_ADDRESS="my-ns.abc123.tmprl.cloud:7233"
export TEMPORAL_API_KEY="<api-key>"

temporal workflow list --limit 1 \
  --address "$TEMPORAL_ADDRESS" \
  --namespace "$TEMPORAL_NAMESPACE" \
  --api-key "$TEMPORAL_API_KEY"

New Namespace Setup

User is setting up connection to a new namespace for the first time.

Docs: Environment configuration - full SDK setup guide

Checklist: 1. Get namespace name (full format with account ID) 2. Generate or obtain certificates 3. Upload CA to namespace 4. Test connection with temporal CLI 5. Configure workers using environment configuration docs

# 1. Verify namespace exists and get full name
tcld namespace list

# 2. Generate certs (if needed)
tcld generate-certificates certificate-authority-certificate \
  --organization mycompany \
  --validity-period 365d \
  --ca-certificate-file certs/ca.pem \
  --ca-key-file certs/ca.key
tcld generate-certificates end-entity-certificate \
  --organization mycompany \
  --validity-period 365d \
  --ca-certificate-file certs/ca.pem \
  --ca-key-file certs/ca.key \
  --certificate-file certs/client.pem \
  --key-file certs/client.key

# 3. Upload CA
tcld namespace accepted-client-ca add \
  --namespace my-ns.abc123 \
  --ca-certificate-file certs/ca.pem

# 4. Test connection
temporal workflow list --limit 1 \
  --address my-ns.abc123.tmprl.cloud:7233 \
  --namespace my-ns.abc123 \
  --tls-cert-path certs/client.pem \
  --tls-key-path certs/client.key

Certificate Rotation

Rotate certs before expiry. If rotating CA, add new CA first, deploy new certs, then remove old CA.

# Check expiry
openssl x509 -enddate -noout -in client.pem

# Generate new leaf cert (same CA)
tcld generate-certificates end-entity-certificate \
  --organization mycompany \
  --validity-period 365d \
  --ca-certificate-file ca.pem \
  --ca-key-file ca.key \
  --certificate-file new-certs/client.pem \
  --key-file new-certs/client.key

# If rotating CA too:
# 1. Generate new CA
tcld generate-certificates certificate-authority-certificate \
  --organization mycompany \
  --validity-period 365d \
  --ca-certificate-file new-ca/ca.pem \
  --ca-key-file new-ca/ca.key

# 2. Add new CA to namespace (keep old one temporarily)
tcld namespace accepted-client-ca add \
  --namespace my-ns.abc123 \
  --ca-certificate-file new-ca/ca.pem

# 3. Deploy new certs to workers

# 4. Remove old CA
tcld namespace accepted-client-ca list --namespace my-ns.abc123
tcld namespace accepted-client-ca remove \
  --namespace my-ns.abc123 \
  --fp <old-ca-fingerprint>

Switching from mTLS to API Keys

User wants to use API keys instead of certificates.

# 1. Create service account
tcld service-account create \
  --name worker-sa \
  --description "Service account for workers"

# 2. Create API key
tcld apikey create \
  --service-account-id <sa-id> \
  --description "Worker API key"

# 3. Test with temporal CLI (same Namespace Endpoint, just swap auth)
temporal workflow list --limit 1 \
  --address my-ns.abc123.tmprl.cloud:7233 \
  --namespace my-ns.abc123 \
  --api-key <api-key>

# 4. Update worker config to use API key instead of certs