
Restate
- 9 installs
- 5 repo stars
- Updated June 1, 2026
- schpet/toolbox
Helps with ai & agent building tasks.
About
restate is a Claude Code skill for ai & agent building. It helps solo builders move faster with AI-assisted development.
- restate
- AI & Agent Building
- AI-coding skill
Restate by the numbers
- 9 all-time installs (skills.sh)
- Ranked #12,152 of 16,546 AI & Agent Building skills by installs in the Skillselion catalog
- Data as of Aug 5, 2026 (Skillselion catalog sync)
npx skills add https://github.com/schpet/toolbox --skill restateAdd your badge
Show developers this skill is listed on Skillselion. Paste this into your README.
| Installs | 9 |
|---|---|
| repo stars | ★ 5 |
| Last updated | June 1, 2026 |
| Repository | schpet/toolbox ↗ |
What it does
Helps with ai & agent building tasks.
Files
Restate Durable Execution Framework
Restate is a durable execution framework that makes applications resilient to failures. Use this skill when building:
- Durable workflows with automatic retries
- Services with persisted state (Virtual Objects)
- Microservice orchestration with transactional guarantees
- Event processing with exactly-once semantics
- Long-running tasks that survive crashes
When to Use Restate
Use Restate when:
- Building workflows that must complete despite failures
- Need automatic retry and recovery without manual retry logic
- Building stateful services (shopping carts, user sessions, payment processing)
- Orchestrating multiple services with saga/compensation patterns
- Processing events with exactly-once delivery guarantees
- Scheduling durable timers and cron jobs
Core Concepts
Service Types
Restate supports three service types:
1. Services - Stateless handlers with durable execution
- Use for: microservice orchestration, sagas, idempotent requests
2. Virtual Objects - Stateful handlers with K/V state isolated per key
- Use for: entities (shopping cart), state machines, actors, stateful event processing
- Only one handler runs at a time per object key (consistency guarantee)
3. Workflows - Special Virtual Objects where run handler executes exactly once
- Use for: order processing, human-in-the-loop, long-running provisioning
See Services Concepts for detailed comparison.
Durable Building Blocks
Restate provides these building blocks through the SDK context:
- Journaled actions (
ctx.run()) - Persist results of side effects - State (
ctx.get/set/clear) - K/V state for Virtual Objects - Timers (
ctx.sleep()) - Durable sleep that survives restarts - Service calls (
ctx.serviceClient()) - RPC with automatic retries - Awakeables - Wait for external events/signals
See Durable Building Blocks.
TypeScript SDK Quick Reference
Installation
npm install @restatedev/restate-sdkBasic Service
import * as restate from "@restatedev/restate-sdk";
const myService = restate.service({
name: "MyService",
handlers: {
greet: async (ctx: restate.Context, name: string) => {
return "Hello, " + name + "!";
},
},
});
restate.endpoint().bind(myService).listen(9080);Virtual Object (Stateful)
const counter = restate.object({
name: "Counter",
handlers: {
add: async (ctx: restate.ObjectContext, value: number) => {
const current = (await ctx.get<number>("count")) ?? 0;
ctx.set("count", current + value);
return current + value;
},
get: restate.handlers.object.shared(
async (ctx: restate.ObjectSharedContext) => {
return (await ctx.get<number>("count")) ?? 0;
}
),
},
});Workflow
const paymentWorkflow = restate.workflow({
name: "PaymentWorkflow",
handlers: {
run: async (ctx: restate.WorkflowContext, payment: Payment) => {
// Step 1: Reserve funds
const reservation = await ctx.run("reserve", () =>
reserveFunds(payment)
);
// Step 2: Wait for approval (awakeable)
const approved = await ctx.promise<boolean>("approval");
if (!approved) {
await ctx.run("cancel", () => cancelReservation(reservation));
return { status: "cancelled" };
}
// Step 3: Complete payment
await ctx.run("complete", () => completePayment(reservation));
return { status: "completed" };
},
approve: async (ctx: restate.WorkflowSharedContext) => {
ctx.promise<boolean>("approval").resolve(true);
},
reject: async (ctx: restate.WorkflowSharedContext) => {
ctx.promise<boolean>("approval").resolve(false);
},
},
});Key SDK Patterns
// Journaled action - result persisted, replayed on retry
const result = await ctx.run("action-name", async () => {
return await callExternalApi();
});
// Durable timer - survives restarts
await ctx.sleep(60_000); // 60 seconds
// Call another service
const client = ctx.serviceClient(OtherService);
const response = await client.handler(input);
// Async call (fire and forget)
ctx.serviceSendClient(OtherService).handler(input);
// Delayed call
ctx.serviceSendClient(OtherService, { delay: 60_000 }).handler(input);
// Awakeable - wait for external signal
const { id, promise } = ctx.awakeable<string>();
// Give `id` to external system, then:
const result = await promise;
// Random (deterministic)
const value = ctx.rand.random();
const uuid = ctx.rand.uuidv4();Running Locally
1. Start Restate server:
npx @restatedev/restate-server2. Run your service:
npx ts-node src/app.ts3. Register service with Restate:
npx @restatedev/restate deployments register http://localhost:90804. Invoke handlers via HTTP:
# Service handler
curl localhost:8080/MyService/greet -H 'content-type: application/json' -d '"World"'
# Virtual Object handler (with key)
curl localhost:8080/Counter/user123/add -H 'content-type: application/json' -d '5'
# Start workflow
curl localhost:8080/PaymentWorkflow/order-456/run -H 'content-type: application/json' -d '{"amount": 100}'Documentation References
Concepts
- Services - Service types and use cases
- Invocations - How invocations work
- Durable Execution - Execution guarantees
- Durable Building Blocks - SDK primitives
TypeScript SDK
- Overview - SDK setup and service definitions
- State - K/V state management
- Journaling Results - Side effects and
ctx.run() - Durable Timers - Sleep and scheduling
- Service Communication - Calling other services
- Awakeables - External events
- Workflows - Workflow implementation
- Error Handling - Error patterns
- Serving - Running services
- Testing - Testing strategies
- Clients - Client SDK
Guides
- Error Handling - Comprehensive error handling
- Sagas - Saga pattern with compensations
- Cron Jobs - Scheduled tasks
- Parallelizing Work - Fan-out patterns
- Databases - Database integration
- Lambda Deployment - AWS Lambda deployment
Use Cases
- Workflows
- Async Tasks
- Event Processing
- Microservice Orchestration
Operations
- HTTP Invocation
- Service Registration
- Versioning
- Architecture
Important Guidelines
1. Side effects must be wrapped in `ctx.run()` - External calls, random values, timestamps must go through the context to be journaled and replayed correctly.
2. State access only in Virtual Objects/Workflows - Plain Services don't have state access.
3. Handlers must be deterministic - Same inputs should produce same outputs. Use ctx.rand for randomness.
4. One handler per Virtual Object key at a time - Restate ensures consistency by queuing concurrent requests to the same key.
5. Workflows `run` handler executes exactly once - Use other handlers to query/signal the workflow.
6. Register services after code changes - Run restate deployments register to update handler definitions.
--- Generated from Restate documentation. Run `scripts/sync-docs.sh` to update.
License
The content in the references/ directory is derived from the Restate documentation and TypeScript SDK.
Restate Architecture
Restate is designed to be extremely simple to get started with by delivering all the functionality in a single binary with minimal upfront configuration needs. In particular, when starting out by running Restate on a single node, you don't need to understand its internal architecture in a great level of detail.
As you begin to plan for more complex deployment scenarios, you will benefit from having a deeper understanding of the various components and how they fit together to support scalable and resilient clusters. The goal of this section is to introduce the terminology we use throughout the server documentation and inform the choices involved in configuring Restate clusters.
Overview
Restate is implemented with a three-layered architecture: a control plane, distributed log, and processors.
Control plane
This component stores all metadata about deployments (what services exist behind which endpoint addresses / URLs / ARNs) and is responsible for assigning the leaders to log partitions and processors.
Distributed Log a.k.a Bifrost
Restate uses a distributed log, called Bifrost, to durably record all events in the system before acting on them, similar to the function of a write-ahead log (WAL) in a database system. The design of Bifrost is based on the idea of a virtual consensus, as described in the Delos paper. The consensus algorithm powering Bifrost is based on Flexible Paxos.
To support scale-out operations, Restate splits the services key-space across multiple partitions, backed by logs. Currently, we map one partition to one log, though this relationship may change in the future. Thus you will see references to both partition ids and log ids, depending on context, so be aware that these are distinct concepts even if they might be the same value.
Each Bifrost log is a chain of append-only segments. The control plane takes care of sealing prior segments and extending this chain. The individual segments are backed by loglets - currently, Bifrost ships with a local loglet suitable for single-node deployments, and a replicated loglet which supports clustered deployments.
Processors
The Processors (also called Partition Processors) receive invocations, process events from the durable log, and interact with the services/functions containing the application/workflow logic. The Processors encapsulate the state machines for durable execution and invocation life-cycles.
The Processors maintain the “state of the world” derived from log events. This state includes ongoing invocations, the journal of each invocation, durable promises, and persisted key-value state. Partition state is stored in RocksDB and can be periodically snapshotted to an object store (this is required in multi-node clusters).
Nodes and roles
You'll see many mentions of the terms server and node throughout this documentation. Generally, we use the term "server" to refer to a running instance of the restate-server binary. This binary can host multiple functions. When you start a single-node Restate server, for example when doing some local development or testing, you are hosting all the essential features in a single process. These include accepting incoming requests, durably recording events, processing work (delegating invocations to services, handling key-value operations), as well as maintaining metadata used internally by the system.
At its simplest, running a cluster is not that different - multiple nodes cooperate to share the responsibilities we mentioned earlier. This is accomplished by having multiple copies of the server process running on separate machines, although it is possible to create test clusters on a single machine. Nodes are therefore distinct instances of the Restate server within a cluster.
Restate clusters are designed to scale out in support of large deployments. As you add more machines, it becomes wasteful to replicate all the functionality across all the machines in a cluster, since not all features need to scale out at the same rate. Roles control which features run on any given node, enabling specialization within the cluster.
Here is an overview of the different roles that can run on a node:
- Metadata server: the source of truth for cluster-wide information
- Ingress: the entry point for external requests
- Log server: responsible for durably persisting the log
- Worker: houses the partition processors
- Admin: giving access to the admin API and running the cluster controller
Metadata store
The Restate metadata store is part of the control plane and is the internal source of truth for node membership and responsibilities. It is essential to the correctness of the overall system: In a cluster this service enables distributed consensus about other components' configuration. All nodes in a Restate cluster must be able to access the metadata store, though not all members of the cluster need to be part of hosting it. Restate includes a built-in Raft-based metadata store which is hosted on all nodes running the metadata-server role.
The metadata store is designed to support relatively low volumes of read and write operations (at least compared to other parts of Restate), with the highest level of integrity and availability.
[//]: # (For some weird reason this shifts to the right if I use two hashtags) <h2>Ingress</h2>
External requests enter the Restate cluster via the HTTP ingress component, which runs on nodes assigned the http-ingress role. Compared to other roles, the HTTP ingress role does not involve long-lived state and it can move around relatively freely, since it only handles ongoing client connections.
The fine-grained http-ingress role is a new addition in Restate 1.2. For backwards-compatibility, nodes running the worker role will continue to run the ingress function in version 1.2.
Log servers
Log server nodes running the log-server role are responsible for durably persisting the log. If the log is the equivalent of a WAL, then partition stores are the materializations that enable efficient reads of the events (invocation journals, key-value data) that have been recorded. Depending on the configured log replication requirements, Restate will replicate log records to multiple log servers to persist a given log, and this will change over time to support maintenance and resizing of the cluster.
Workers
Nodes assigned the worker role run the partition processors, which are the Restate components responsible for maintaining the partition store. Partition processors can operate in either leader or follower mode. Only a single leader for a given partition can be active at a time, and this is the sole processor that handles invocations to deployed services. Followers keep up with the log without taking action, and are ready to take over in the event that the partition's leader becomes unavailable. The overall number of processors per partition is configurable via the partition replication configuration option.
Partition processors replicate their state by following and applying the log for their partition. If a processor needs to stop, for example for scheduled maintenance, it will typically catch up on the records it missed by reading them from the cluster's log servers once it comes back online. Occasionally, a worker node might lose a disk - or you might need to grow your cluster by adding fresh nodes to it. In these cases, it's far more efficient to obtain a snapshot of the partition state from a recent point in time than to replay all the missing log events. Restate clusters can be configured to use an external object store as the snapshot repository, allowing partition processors to skip ahead in the log. This also enables us to trim logs which might otherwise grow unboundedly.
Admin
Nodes running the admin role expose the admin REST API, which can be used to manage the cluster (e.g. registering services, canceling invocations, etc.) and to obtain information about available services and running invocations. The admin role also exposes the SQL endpoint to query the cluster status which gives advanced information about the cluster. One of the admin nodes runs the cluster controller which is responsible for configuring the partition processors and selecting the leader. In case the admin node running the cluster controller fails, another admin node will take over and start a new cluster controller.
Other reading material
- Distributed Restate - a first look
- Every System is a Log: blog post on the idea at the basis of Restate
- Virtual Consensus in Delos
- An introduction to Virtual Consensus in Delos - Jack Vanlightly
Durable Building Blocks
Distributed systems are inherently complex and failures are inevitable. Almost any application is a distributed system, since they are composed of different components that communicate over the network (e.g. services, databases, queues, etc). With every component, the number of possible failure scenarios increases: network partitions, hardware failures, timeouts, race conditions etc. Building reliable applications is a challenging task.
Restate lets you write distributed applications that are resilient to failures. It does this by providing a distributed, durable version of common building blocks.
For these building blocks, Restate handles failure recovery, idempotency, state, and consistency. This way, you can implement otherwise tricky patterns in a few lines of code without worrying about these concerns.
You implement your business logic in handlers that have access to these building blocks via the Restate SDK, that is loaded as a dependency.
Let's have a look at a handler that processes food orders:
!!steps Durable functions
Handlers take part in durable execution, meaning that Restate keeps track of their progress and recovers them to the previously reached state in case of failures.
```ts !!tabs TypeScript order_processor.ts CODE_LOAD::ts/src/concepts/food_ordering.ts?1
CODE_LOAD::java/src/main/java/concepts/buildingblocks/OrderWorkflow.java?1
CODE_LOAD::go/concepts/foodordering.go?1
CODE_LOAD::python/src/concepts/food_ordering.py?1
## !!steps Durable RPCs and queues
Handlers can call other handlers in a resilient way, with or without waiting for the response.
When a failure happens, Restate handles retries and recovers partial progress.
CODE_LOAD::ts/src/concepts/food_ordering.ts?2
CODE_LOAD::java/src/main/java/concepts/buildingblocks/OrderWorkflow.java?2
CODE_LOAD::go/concepts/foodordering.go?2
CODE_LOAD::python/src/concepts/food_ordering.py?2
## !!steps Durable promises and timers
Register promises in Restate to make them resilient to failures (e.g. webhooks, timers).
Restate lets the handler suspend while awaiting the promise, and invokes it again when the result is available.
A great match for function-as-a-service platforms.
CODE_LOAD::ts/src/concepts/food_ordering.ts?3
CODE_LOAD::java/src/main/java/concepts/buildingblocks/OrderWorkflow.java?3
CODE_LOAD::go/concepts/foodordering.go?3
CODE_LOAD::python/src/concepts/food_ordering.py?3
## !!steps Consistent K/V state
Persist application state in Restate with a simple concurrency model and no extra setup. Restate makes sure state remains consistent amid failures.
CODE_LOAD::ts/src/concepts/food_ordering.ts?4
CODE_LOAD::java/src/main/java/concepts/buildingblocks/OrderWorkflow.java?4
CODE_LOAD::go/concepts/foodordering.go?4
CODE_LOAD::python/src/concepts/food_ordering.py?4
## !!steps Journaling actions
Store the result of an action in Restate. The result gets replayed in case of failures and the action is not executed again.
CODE_LOAD::ts/src/concepts/food_ordering.ts?5
CODE_LOAD::java/src/main/java/concepts/buildingblocks/OrderWorkflow.java?5
CODE_LOAD::go/concepts/foodordering.go?5
CODE_LOAD::python/src/concepts/food_ordering.py?5
Durable Execution
Restate provides resilience for applications via its Durable Execution mechanism.
A Durable Execution engine tracks code execution to enable recovery of partial progress in case of failures.
Restate implements Durable Execution by keeping track of the progress of execution in a central, persisted log that can be replayed in case of failures. Restate uses a combination of a server and SDK libraries to provide durable execution.
The SDKs are responsible for tracking the progress of the execution and sending it to the runtime.
The Restate server is responsible for storing the progress in a durable log and triggering retries in case of failures. When the Restate server triggers a retry, it sends the progress log to the SDK, which replays the log to continue the execution from where it left off.
In case of a failure (e.g. timeout, infrastructure crash, network glitch), Restate will retry the execution by invoking the handler again and sending over the latest version of the journal. The handler then starts executing again and whenever it encounters an action on the Restate context, it will skip execution and will inject the response it finds in the journal. This way the handler can recover up to the point where it crashed.
Invocations
An invocation is a request to execute a handler that is part of a Restate service.
There are three ways to invoke a handler:
Send a request to the Restate Server (port <code>8080</code>), with the handler name in the path, and the payload body. <br/> <a href="/invoke/http">Learn more</a> </p> ), },
title: 'Programmatically', iconPath: '/img/code-icon.svg', description: ( <p> Use the SDK to send requests within Restate handlers. Or use generated HTTP clients anywhere else. <br/> <a href="/invoke/clients">Learn more</a> </p> ), },
title: 'Kafka events', iconPath: '/img/kafka-icon.svg', description: ( <p> Restate subscribes to a Kafka topic, and invokes a handler should for each message that arrives. <br/> <a href="/invoke/kafka">Learn more</a> </p> ), }, ]
All invocations are proxied through the Restate Server, which registers the request, routes the request to the correct handler, and drives the execution of the handler.
Invocations get a unique identifier. This identifier is used to track the progress of the invocation, and lets you correlate logs and metrics.
Invocation types
!!steps
Request-response invocations allow you to call another handler and wait for the response.
```typescript !!tabs TypeScript CODE_LOAD::ts/src/concepts/invocations/rpc.ts#rpc_call
CODE_LOAD::ts/src/concepts/invocations/ingress/rpc.ts#rpc_call_node
CODE_LOAD::java/src/main/java/concepts/invocations/RpcCalls.java#rpc
CODE_LOAD::java/src/main/java/concepts/invocations/RpcCalls.java#rpc_java
CODE_LOAD::go/concepts/invocations/call/call.go#rpc_call
CODE_LOAD::python/src/concepts/invocations/rpc.py#rpc_call
curl localhost:8080/GreeterService/greet --json '"Hi"'
## !!steps
**One-way invocations** allow you to trigger an asynchronous action.
This returns an invocation ID with which you can retrieve the result of the invocation later, if desired.
CODE_LOAD::ts/src/concepts/invocations/one_way.ts#one_way_call
CODE_LOAD::ts/src/concepts/invocations/ingress/one_way.ts#one_way_call_node
CODE_LOAD::java/src/main/java/concepts/invocations/OneWayCalls.java#one_way_call
CODE_LOAD::java/src/main/java/concepts/invocations/OneWayCalls.java#one_way_call_java
CODE_LOAD::go/concepts/invocations/send/send.go#one_way_call
CODE_LOAD::python/src/concepts/invocations/one_way.py#one_way_call
curl localhost:8080/GreeterService/greet/send --json '"Hi"'
## !!steps
**Delayed invocations** allow you to schedule an invocation for a later point in time.
CODE_LOAD::ts/src/concepts/invocations/delayed.ts#delayed_call
CODE_LOAD::ts/src/concepts/invocations/ingress/delayed.ts#delayed_call_node
CODE_LOAD::java/src/main/java/concepts/invocations/DelayedCalls.java#delayed_call
CODE_LOAD::java/src/main/java/concepts/invocations/DelayedCalls.java#delayed_call_java
CODE_LOAD::go/concepts/invocations/delayed_send/delayed_send.go#delayed_call
CODE_LOAD::python/src/concepts/invocations/delayed.py#delayed_call
curl localhost:8080/GreeterService/greet/send?delay=10s --json '"Hi"'
Learn more about [HTTP invocations](/invoke/http), [SDK clients](/invoke/clients), [Kafka events](/invoke/kafka).
## Idempotent invocations
You can add an idempotency key to your request header to make the invocation idempotent.
Restate will then deduplicate requests with the same idempotency key, and will only execute the handler once.
Duplicate requests will get the same response as the first request, or will latch on to the first invocation if it's still running.
curl localhost:8080/GreeterService/greet \
!focus
-H 'idempotency-key: ad5472esg4dsg525dssdfa5loi' \ --json '"Hi"'
## Inspecting invocations
Restate proxies and manages inbound as well as service-to-service invocations.
This makes it a great source of observability data for your application.
You can inspect invocations via the UI:
Or via the CLI:
restate services list
NAME REVISION FLAVOR DEPLOYMENT TYPE DEPLOYMENT ID 🌎 CartObject 1 ⬅️ 🚶🚶🚶 HTTP 2 dp_11pXug0mWsff2NOoRBZbOcV 🌎 CheckoutService 1 HTTP 2 dp_11pXug0mWsff2NOoRBZbOcV 🌎 TicketObject 1 ⬅️ 🚶🚶🚶 HTTP 2 dp_11pXug0mWsff2NOoRBZbOcV
restate services describe CartObject
📜 Service Information: ――――――――――――――――――――――― Name: CartObject Service type: VirtualObject Revision: 1 Public: true Deployment ID: dp_11pXug0mWsff2NOoRBZbOcV Deployment Type: HTTP 2 Protocol Style: Streaming Endpoint: http://localhost:9080/ Created at: 2024-04-23T12:32:16.691000000Z
🔌 Handlers: ―――――――――――― HANDLER INPUT TYPE OUTPUT TYPE addTicket one of "empty or value of value with content-type "application/json" content-type /" checkout one of "empty or value of value with content-type "application/json" content-type /" expireTicket one of "empty or value of value with content-type "application/json" content-type /"
restate invocations list
❯ [2024-04-23 14:41:59.365 +02:00] inv_1fmRNvSNVxNp5PTqHI4HLJ17HpxzhB3MEV Target: CartObject/Mary/addTicket Status: backing-off (18 seconds and 284 ms. Retried 9 time(s). Next retry in in 8 seconds and 220 ms)) Deployment: dp_11pXug0mWsff2NOoRBZbOcV [required] Error: [2024-04-23 14:42:13.706 +02:00] [500] Failing Caused by: UNKNOWN
restate invocations describe inv_1fmRNvSNVxNp5PTqHI4HLJ17HpxzhB3MEV
📜 Invocation Information: ―――――――――――――――――――――――――― Created at: 2024-04-23 14:41:59.365 +02:00 (a minute ago) Target: CartObject/Mary/addTicket Status: backing-off (1 minute, 23 seconds and 937 ms. Retried 14 time(s). Next retry in in 991 ms)) Deployment: dp_11pXug0mWsff2NOoRBZbOcV [required] Error: [2024-04-23 14:43:13.248 +02:00] [500] Failing Caused by: UNKNOWN Modified at: 2024-04-23 14:41:59.388 +02:00
💡 This invocation is bound to run on deployment 'dp_11pXug0mWsff2NOoRBZbOcV'. To guarantee safety and correctness, invocations that made progress on a deployment cannot move to newer deployments automatically.
🚂 Invocation Progress: ――――――――――――――――――――――― [Ingress] └──(this)─> CartObject/Mary/addTicket ▸ ├──── ☑️ #1 Call TicketObject/seat2B/reserve inv_19maBIcE9uRD1CrHgpGXZ7FcXPsz4bzkbL └────>> backing-off
Restate also [exposes traces](/operate/monitoring/tracing) via OpenTelemetry, which can be sent to your observability platform of choice (e.g. Jaeger).
<div>
</div>
## Cancelling and killing invocations
You can cancel and kill invocations via the Restate UI, the CLI, or with the SDK clients.
- For cancellations, Restate will gracefully stop the handler by executing all compensation actions.
- For kills, Restate will immediately stop the handler without executing any compensation actions.
If necessary, you can register compensating actions in your handlers to ensure that the system remains consistent amid cancellations ([sagas guide](/guides/sagas)).
restate invocations cancel --kill inv_1fmRNvSNVxNp5PTqHI4HLJ17HpxzhB3MEV
❯ [2024-04-23 14:41:59.365 +02:00] inv_1fmRNvSNVxNp5PTqHI4HLJ17HpxzhB3MEV Target: CartObject/Mary/addTicket Status: backing-off (25 minutes, 29 seconds and 200 ms. Retried 141 time(s). Next retry in in 12 seconds and 94 ms)) Deployment: dp_11pXug0mWsff2NOoRBZbOcV [required] Error: [2024-04-23 15:07:27.860 +02:00] [500] Failing Caused by: UNKNOWN
✔ Are you sure you want to kill this invocation · yes
✅ Request was sent successfully
Services
This is what a Restate application looks like from a helicopter view:
<center>
</center>
1. Services: Restate services contain functions/handlers which process incoming requests and execute business logic. Services run like regular code in your infrastructure, for example a NodeJS/Java app in a Docker container or a Python function on AWS Lambda. Services embed the Restate SDK as a dependency, and their handlers use it to persist the progress they make. Services can be written in any language for which there is an SDK available: TypeScript, Java, Kotlin, Go, Python, and Rust. 2. [Restate Server](/concepts/services#restate-server): The server sits in front of your services, similar to a reverse proxy or message broker. It proxies incoming requests to the corresponding services and drives their execution till the end. 3. [Invocation](/concepts/invocations): An invocation is a request to execute a handler.
There are three types of services in Restate:
[//]: # (This is an html table because markdown tables don't support setting the column width and for some reason the workflow column was very large) <table><thead><tr><th>Services (plain)</th><th>Virtual objects</th><th>Workflows</th></tr></thead><tbody><tr><td width={"30%"Set of handlers durably executed</td><td width={"30%"Set of handlers durably executed</td><td width={"30%"The workflow <code>run</code> handler is durably executed a single time.</td></tr><tr><td>No associated K/V store</td><td>Handlers share K/V state; isolated per virtual object</td><td>K/V state isolated per workflow execution. Can only be set by the <code>run</code> handler.</td></tr><tr><td>No concurrency limits; unlimited scale-out on platforms like AWS Lambda.</td><td> <ul><li>To guard state consistency, only one handler with write access to the state can run at a time per virtual object: queue per key.</li><li>Handlers marked as shared don't have write access to state and can run concurrently to the exclusive handlers.</li></ul></td><td>The run handler can run only a single time per workflow ID. Other handlers can run concurrently to interact with the workflow.</td></tr><tr><td>Example use cases: <ul><li>Microservice orchestration</li><li>Sagas and distributed transactions</li><li>Exactly-once webhook callback processing</li><li>Idempotent requests</li><li>Parallelization, chaining API calls, and complex routing</li><li>Async task scheduling</li></ul></td><td>Example use cases: <ul><li>Stateful handlers and entities: e.g. shopping cart</li><li>Atomic, durable state machines: e.g. payment processing</li><li>Stateful agents, actors, and digital twins: e.g. AI chat sessions</li><li>Locking mechanisms: database writes</li><li>Stateful Kafka event processing: e.g. enrichment, joins</li></ul></td><td>Example use cases: <ul><li>Payments, order processing and logistics</li><li>Human-in-the-loop workflow: e.g. signup, email approval</li><li>Long-running tasks with failures/timeouts: e.g. infrastructure provisioning</li><li>Flexibility and dynamic routing: e.g. workflow interpreters</li></ul></td></tr></tbody></table>
Services
Services expose a collection of handlers:
!!steps
Restate makes sure that handlers run to completion, even in the presence of failures. Restate persists the results of actions and recovers them after failures.
```ts !!tabs TypeScript https://github.com/restatedev/examples/blob/main/typescript/basics/src/0_durable_execution.ts CODE_LOAD::ts/src/concepts/services/subscription_service.ts?1
CODE_LOAD::java/src/main/java/concepts/services/SubscriptionService.java?1
CODE_LOAD::kotlin/src/main/kotlin/concepts/services/SubscriptionService.kt?1
CODE_LOAD::go/concepts/services/subscriptionservice.go?1
CODE_LOAD::python/src/concepts/services/subscription_service.py?1
CODE_LOAD::rust/src/concepts/services.rs?1
## !!steps
The handlers of services are independent and can be invoked concurrently.
CODE_LOAD::ts/src/concepts/services/subscription_service.ts?2
CODE_LOAD::java/src/main/java/concepts/services/SubscriptionService.java?2
CODE_LOAD::kotlin/src/main/kotlin/concepts/services/SubscriptionService.kt?2
CODE_LOAD::go/concepts/services/subscriptionservice.go?2
CODE_LOAD::python/src/concepts/services/subscription_service.py?2
CODE_LOAD::rust/src/concepts/services.rs?2
## !!steps
Handlers use regular code and control flow, no custom DSLs.
CODE_LOAD::ts/src/concepts/services/subscription_service.ts?3
CODE_LOAD::java/src/main/java/concepts/services/SubscriptionService.java?3
CODE_LOAD::kotlin/src/main/kotlin/concepts/services/SubscriptionService.kt?3
CODE_LOAD::go/concepts/services/subscriptionservice.go?3
CODE_LOAD::python/src/concepts/services/subscription_service.py?3
CODE_LOAD::rust/src/concepts/services.rs?3
## !!steps
Handlers are exposed over HTTP. When the Restate Server receives a request, it sets up an HTTP2 connection with the service and streams events back and forth over this connection.
Alternatively, you can create a serverless function, like an AWS Lambda handler.
CODE_LOAD::ts/src/concepts/services/subscription_service.ts?4
CODE_LOAD::java/src/main/java/concepts/services/SubscriptionService.java?4
CODE_LOAD::kotlin/src/main/kotlin/concepts/services/SubscriptionService.kt?4
CODE_LOAD::go/concepts/services/subscriptionservice.go?4
CODE_LOAD::python/src/concepts/services/subscription_service.py?4
CODE_LOAD::rust/src/concepts/services.rs?4
## Virtual objects
Virtual objects expose a set of handlers with access to K/V state stored in Restate.
## !!steps
A virtual object is **uniquely identified and accessed by its key**.
CODE_LOAD::ts/src/concepts/services/virtual_objects.ts?1
CODE_LOAD::java/src/main/java/concepts/services/Greeter.java?1
CODE_LOAD::kotlin/src/main/kotlin/concepts/services/GreeterObject.kt?1
CODE_LOAD::go/concepts/virtualobjects/main.go?1
CODE_LOAD::python/src/concepts/services/virtual_objects.py?1
CODE_LOAD::rust/src/concepts/vo.rs?1
## !!steps
Each virtual object has access to its own **isolated K/V state**, stored in Restate.
The handlers of a virtual object can read and write to the state of the object.
Restate delivers the state together with the request to the virtual object, so virtual objects have their state locally accessible without requiring any database connection or lookup.
State is exclusive, and atomically committed with the handler execution.
CODE_LOAD::ts/src/concepts/services/virtual_objects.ts?2
CODE_LOAD::java/src/main/java/concepts/services/Greeter.java?2
CODE_LOAD::kotlin/src/main/kotlin/concepts/services/GreeterObject.kt?2
CODE_LOAD::go/concepts/virtualobjects/main.go?2
CODE_LOAD::python/src/concepts/services/virtual_objects.py?2
CODE_LOAD::rust/src/concepts/vo.rs?2
## !!steps
When a handler is invoked, it can **read and write to the state** of the virtual object.
To ensure consistent writes to the state, Restate provides **concurrency guarantees**: at most one handler can execute at a time for a given virtual object.
CODE_LOAD::ts/src/concepts/services/virtual_objects.ts?3
CODE_LOAD::java/src/main/java/concepts/services/Greeter.java?3
CODE_LOAD::kotlin/src/main/kotlin/concepts/services/GreeterObject.kt?3
CODE_LOAD::go/concepts/virtualobjects/main.go?3
CODE_LOAD::python/src/concepts/services/virtual_objects.py?3
CODE_LOAD::rust/src/concepts/vo.rs?3
## !!steps
If you want to **allow concurrent reads** to the state, you can mark a handler as a **shared handler**.
This allows the handler to run concurrently with other handlers, but it cannot write to the state.
CODE_LOAD::ts/src/concepts/services/virtual_objects.ts?4
CODE_LOAD::java/src/main/java/concepts/services/Greeter.java?4
CODE_LOAD::kotlin/src/main/kotlin/concepts/services/GreeterObject.kt?4
CODE_LOAD::go/concepts/virtualobjects/main.go?4
CODE_LOAD::python/src/concepts/services/virtual_objects.py?4
CODE_LOAD::rust/src/concepts/vo.rs?4
## Workflows
A workflow is a special type of Virtual Object that can be used to implement a set of steps that need to be executed durably.
Workflows have additional capabilities such as signaling, querying, additional invocation options, and a longer retention time in the CLI.
## !!steps
A workflow has a **run handler** that implements the **workflow logic**.
The `run` handler runs exactly once per workflow ID (object).
CODE_LOAD::ts/src/concepts/services/signup_workflow.ts?1
CODE_LOAD::java/src/main/java/concepts/services/SignupWorkflow.java?1
CODE_LOAD::kotlin/src/main/kotlin/concepts/services/SignupWorkflow.kt?1
CODE_LOAD::go/concepts/workflow/workflows.go?1
CODE_LOAD::python/src/concepts/services/signup_workflow.py?1
CODE_LOAD::rust/src/concepts/workflows.rs?1
## !!steps
The run handler executes a set of **durable steps/activities**. These can either be:
- Inline activities: run blocks, sleep, mutating K/V state,...
- Calls to other handlers implementing the activities.
CODE_LOAD::ts/src/concepts/services/signup_workflow.ts?2
CODE_LOAD::java/src/main/java/concepts/services/SignupWorkflow.java?2
CODE_LOAD::kotlin/src/main/kotlin/concepts/services/SignupWorkflow.kt?2
CODE_LOAD::go/concepts/workflow/workflows.go?2
CODE_LOAD::python/src/concepts/services/signup_workflow.py?2
CODE_LOAD::rust/src/concepts/workflows.rs?2
## !!steps
You can define other handlers in the same workflow that can run concurrently to the `run` handler and:
- **Query** the workflow (get information out of it) by getting K/V state or awaiting promises that are resolved by the workflow.
- **Signal** the workflow (send information to it) by resolving promises that the workflow waits on.
For example, the click handler signals the workflow that the email link was clicked.
CODE_LOAD::ts/src/concepts/services/signup_workflow.ts?3
CODE_LOAD::java/src/main/java/concepts/services/SignupWorkflow.java?3
CODE_LOAD::kotlin/src/main/kotlin/concepts/services/SignupWorkflow.kt?3
CODE_LOAD::go/concepts/workflow/workflows.go?3
CODE_LOAD::python/src/concepts/services/signup_workflow.py?3
CODE_LOAD::rust/src/concepts/workflows.rs?3
## Restate Server
The Restate Server sits like reverse-proxy or message broker in front of your services and proxies invocations to them.
The Restate Server is written in Rust, to be self-contained and resource-efficient.
It has an event-driven foundation to suit low-latency requirements.
<div className={"text-center"
</div>
The Restate Server runs as a single binary with zero dependencies. It runs with low operational overhead on any platform, also locally.
You can run the Restate Server in a highly-available configuration, with multiple instances behind a load balancer.
Restate is also available as a [fully managed cloud service](https://restate.dev/cloud/), if all you want is to use it and let us operate it.
[Contact our team for more information.](https://restate.dev/get-restate-cloud/)
Learn more about the Restate Server:
- [Deploying Restate Servers and clusters](/deploy/overview)
- [Restate's architecture](/references/architecture)
- [Deep-dive architecture blog post with benchmark results](https://restate.dev/blog/the-anatomy-of-a-durable-execution-stack-from-first-principles/).
- [Durable Execution](/concepts/durable_execution): the main feature the Restate Server implements and how it works
Cron Jobs
This guide shows how to use the Restate to schedule cron jobs.
A cron job is a scheduled task that runs periodically at a specified time or interval. It is often used for background tasks like cleanup or sending notifications.
Restate has no built-in functionality for cron jobs. But Restate's durable building blocks make it easy to implement a service that does this for us, and uses the guarantees Restate gives to make sure tasks get executed reliably.
Restate has many features that make it a good fit for implementing cron jobs:
- Durable timers: Schedule tasks to run at a specific time in the future. Restate ensures execution.
- Task resiliency: Restate ensures that tasks are retried until they succeed.
- Task control: Cancel and inspect running jobs.
- K/V state: We store the details of the cron jobs in Restate, so we can retrieve them later and query them from the outside.
- FaaS support: Run your services on FaaS infrastructure, like AWS Lambda. Restate will scale your scheduler and tasks to zero while they sleep.
- Scalability: Restate can handle many cron jobs running in parallel, and can scale horizontally to handle more load.
- Observability: See the execution history of the cron jobs, and their status in the Restate UI.
Example
The example implements a cron service that you can copy over to your own project.
Usage: 1. Send requests to CronJobInitiator.create() to start new jobs with standard cron expressions:
"cronExpression": "0 0 * * *", # E.g. run every day at midnight
"service": "TaskService", # Schedule any Restate handler
"method": "executeTask",
"key": "taskId", # Optional, Virtual Object key
"payload": "Hello midnight!"
2. Each job gets a unique ID and runs as a CronJob virtual object 3. Jobs automatically reschedule themselves after each execution
```ts !!tabs TypeScript https://github.com/restatedev/examples/blob/main/typescript/patterns-use-cases/src/cron/cron_service.ts // collapse_prequel CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/typescript/patterns-use-cases/src/cron/cron_service.ts
// collapse_prequel CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/java/patterns-use-cases/src/main/java/my/example/cron/Cron.java
// collapse_prequel CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/go/patterns-use-cases/src/cron/cron.go
This pattern is implementable with any of our SDKs. We are still working on translating all patterns to all SDK languages.
If you need help with a specific language, please reach out to us via [Discord](https://discord.com/invite/skW3AZ6uGd) or [Slack](https://join.slack.com/t/restatecommunity/shared_invite/zt-2v9gl005c-WBpr167o5XJZI1l7HWKImA).
## Adapt to your use case
Note that this implementation is fully resilient, but you might need to make some adjustments to make this fit your use case:
- Take into account time zones.
- Adjust how you want to handle tasks that fail until the next task gets scheduled. With the current implementation, you would have concurrent executions of the same cron job (one retrying and the other starting up).
If you want to cancel the failing task when a new one needs to start, you can do the following: at the beginning of the `execute` call, retrieve the `next_execution_id` from the job state and check if it is completed by [attaching to it](/develop/ts/service-communication#re-attach-to-an-invocation) with [a timeout set to 0](/develop/ts/journaling-results#combineable-promise-combinators). If it is not completed, [cancel it](/develop/ts/service-communication#cancel-an-invocation) and start the new iteration.
## Running the example
restate example typescript-patterns-use-cases && cd typescript-patterns-use-cases
restate example java-patterns-use-cases && cd java-patterns-use-cases
restate example go-patterns-use-cases && cd go-patterns-use-cases
restate-server
npx tsx watch ./src/cron/task_service.ts
./gradlew -PmainClass=my.example.cron.TaskService run
go run ./src/cron
restate deployments register localhost:9080
For example, run `executeTask` every minute:
curl localhost:8080/CronJobInitiator/create --json '{ "cronExpression": " *", "service": "TaskService", "method": "executeTask", "payload": "Hello new minute!" }'
curl localhost:8080/CronJobInitiator/create --json '{ "cronExpression": " *", "service": "TaskService", "method": "executeTask", "payload": "Hello new minute!" }'
curl localhost:8080/CronJobInitiator/Create --json '{ "cronExpression": " *", "service": "TaskService", "method": "executeTask", "payload": "Hello new minute!" }'
For example, or run `executeTask` at midnight:
curl localhost:8080/CronJobInitiator/create --json '{ "cronExpression": "0 0 *", "service": "TaskService", "method": "executeTask", "payload": "Hello midnight!" }'
curl localhost:8080/CronJobInitiator/create --json '{ "cronExpression": "0 0 *", "service": "TaskService", "method": "executeTask", "payload": "Hello midnight!" }'
curl localhost:8080/CronJobInitiator/Create --json '{ "cronExpression": "0 0 *", "service": "TaskService", "method": "executeTask", "payload": "Hello midnight!" }'
You can also use the cron service to execute handlers on Virtual Objects, by specifying the Virtual Object key in the request.
You will get back a response with the job ID.
Using the job ID, you can then get information about the job:
curl localhost:8080/CronJob/myJobId/getInfo
curl localhost:8080/CronJob/myJobId/getInfo
curl localhost:8080/CronJob/myJobId/GetInfo
Or cancel the job later:
curl localhost:8080/CronJob/myJobId/cancel
curl localhost:8080/CronJob/myJobId/cancel
curl localhost:8080/CronJob/myJobId/Cancel
In the UI, you can see how the tasks are scheduled, and how the state of the cron jobs is stored in Restate.
You can kill and restart any of the services or the Restate Server, and the scheduled tasks will still be there.
Databases and Restate
Restate is not only a system for resilience and durability in communication and orchestration, it can also store data like a database system, to enable writing long-lived stateful logic with strong out-of-the-box resilience.
To avoid confusion: In this page, we refer to the type of state that outlives a single workflow or durable handler execution, and that uses Restate’s Virtual Objects. This is a feature not available in typical workflow systems or durable execution frameworks.
This guide discusses thoughts on when to use Restate for that type of state, when to use a database, and how to integrate Restate and databases.
What state to store in Restate, what to store in databases
State stored directly in Restate (Virtual Objects) has a set of advantages:
- Ultra robust: No coordination between systems is needed. Instead, state access/updates directly participate in the durable execution logic (go through the same consensus log), making the state always aligned with the business logic: no lost updates, duplications of state changes, stale views of state, etc.
- Simple correctness model: The single-writer-per-key and linearizable consistency model means developers don’t need to worry about locks, race conditions, dirty reads, change visibility, zombie processes, etc.
- Efficient: Computation and access patterns to state align naturally, you cannot get into situations where multiple function executions try to access/update the same database entry and contend on locks or interfere with each other's transaction.
No write amplification through separate durability in an external system (instead state is simply committed with the durable execution journal data).
- Serverless: when Restate invokes a handler, it attaches the relevant state to the request. On serverless, for example AWS lambda, this means your state is locally available when the function executes, no access delays, synchronization, wait times, etc.
- No additional dependency - the K/V store is integrated in the core of Restate. You don't need to do anything extra to start using state when developing locally, or migrating the Restate app to cloud/production.
State stored directly in Restate (Virtual Objects) has some limitations:
- K/V interface with single-key transactions (though a single key can store a document with flexible structures)
- SQL is supported only for analytics / introspection, not for updates/transactions.
- Modifiable only from the Virtual Object, not from other services. Any modification needs to be sent as a request to the Virtual Object.
- While state can be exported in a standard way (psql client), it is managed by a less standard system, compared to some databases.
When to use which?
Use a database:
- When you need complex access patterns, full SQL, text search, time-series analysis, etc.
- For core business data, like your user database, products database, history of transactions, etc. that you want to access from other services as well
Use Restate for any state requiring tight integration with function logic, resilience, and correctness:
- State that is part of transactional state machines (payment status, activation status, ...)
- Session state (shopping cart, ongoing orders, LLM context, AI agent context, ...)
- State that is part of distributed orchestration and coordination (e.g., versions/life-cycle of shared resources, transaction IDs, reference counts, distributed locks/semaphores/leases, ...)
- Agents / Digital-twins (agent/twin memory and context)
- State in event processing pipelines (aggregates, joins)
- State of control planes (desired cluster status, resource references, ...)
- Consistent state/metadata overlay over eventual consistent infra
How to interact with Databases from Restate
This set of examples shows various patterns to access databases from Restate handlers.
```ts https://github.com/restatedev/examples/blob/main/typescript/patterns-use-cases/src/database/main.ts CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/typescript/patterns-use-cases/src/database/main.ts
### Two-phase commit
CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/typescript/patterns-use-cases/src/database/2phasecommit.ts
## Running the examples
This is purely optional, the example code and comments document the behavior well.
Running the example can be interesting, though, if you want to play with specific failure scenarios, like pausing/killing processes at specific points and observe the behavior.
[Check out the readme of the example on how to run the example.](https://github.com/restatedev/examples/blob/main/typescript/patterns-use-cases/README.md#database-interaction-patterns)Durable webhooks
Restate handlers can be used as durable processors of webhook events.
What does this give you?
- Restate persists all incoming events, and ensures that they are processed exactly once, across failures and restarts. Restate guarantees your handler runs till completion.
- Let Restate deduplicate events on an idempotency key. If the sender of the event retries, Restate will not process the event again.
- Use any of Restate's **durable SDK constructs** when processing the events: durable calls/messaging to other services, durable timers, scheduling tasks, K/V state, concurrency guarantees etc.
- Any handler can be a durable webhook endpoint. No need to do anything special or extra!
Just point your webhook endpoint to your handler: restate:8080/MyService/myHandler.
Example
This example processes webhook callbacks from a payment provider.
The payment provider notifies us about payment success or failure of invoices by sending webhook events to our handler. The handler then routes the event to the correct processor via a one-way message.
```ts !!tabs TypeScript https://github.com/restatedev/examples/blob/main/typescript/patterns-use-cases/src/webhookcallbacks/webhook_callback_router.ts // collapse_prequel CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/typescript/patterns-use-cases/src/webhookcallbacks/webhook_callback_router.ts
// collapse_prequel CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/go/patterns-use-cases/src/webhookcallbacks/callbackrouter.go
This pattern is implementable with any of our SDKs. We are still working on translating all patterns to all SDK languages.
If you need help with a specific language, please reach out to us via [Discord](https://discord.com/invite/skW3AZ6uGd) or [Slack](https://join.slack.com/t/restatecommunity/shared_invite/zt-2v9gl005c-WBpr167o5XJZI1l7HWKImA).
Error Handling
Restate handles retries for failed invocations. By default, Restate infinitely retries all errors with an exponential backoff strategy.
This guide helps you fine-tune the retry behavior for your use cases.
Infrastructure errors (transient) vs. application errors (terminal)
In Restate, we distinguish between two types of errors: transient errors and terminal errors.
- Transient errors are temporary and can be retried. They are typically caused by infrastructure issues (network problems, service overload, API unavailability,...).
- Terminal errors are permanent and should not be retried. They are typically caused by application logic (invalid input, business rule violation, ...).
Handling transient errors via retries
Restate assumes by default that all errors are transient errors and therefore retryable. If you do not want an error to be retried, you need to specifically label it as a terminal error (see below).
Restate lets you configure the retry strategy at different levels: at the Restate-level (global) and at the run-block-level.
At the Restate-Level (Global)
This defines the default retry policy that will be used for all invocations, unless overridden at the service-, or run-block-level.
You can set the global retry policy in the Restate Server configuration. By default, Restate will use an exponential backoff retry policy:
```toml restate.toml [worker.invoker.retry-policy] type = "exponential" # retry strategy; required initial-interval = "50ms" # time between the first and second retry; required factor = 2.0 # factor used to calculate the next retry interval; required max-interval = "10s" # max time between retries; default: unset (=interval keeps increasing)
You can tune this policy to your needs. Note that all durations should follow the [humantime format](https://docs.rs/humantime/latest/humantime/fn.parse_duration.html).
You can also use a **fixed-delay retry policy**:[worker.invoker.retry-policy] type = "fixed-delay" # retry strategy; required interval = "50ms" # time between retries; required max-attempts = "10" # max number of attempts before terminal error; default: unset (=infinite)
If you set a maximum number of attempts, then the handler will throw a terminal error once the retries are exhausted.
Then run the Restate Server with:
restate-server --config-file restate.toml
Or set it [via environment variables](/operate/configuration/server#environment-variables), for example:RESTATE_WORKER__INVOKER__RETRY_POLICY__TYPE=fixed-delay \ RESTATE_WORKER__INVOKER__RETRY_POLICY__INTERVAL=100ms \ restate-server
### At the Run-Block-Level
Handlers use run blocks to execute non-deterministic actions, often involving other systems and services (API call, DB write, ...).
These run blocks are especially prone to transient failures, and you might want to configure a specific retry policy for them.
Most Restate SDKs allow this:
CODE_LOAD::ts/src/guides/retries.ts?1
CODE_LOAD::python/src/guides/retries.py?1
CODE_LOAD::java/src/main/java/guides/RetryRunService.java?1
CODE_LOAD::kotlin/src/main/kotlin/guides/RetryRunService.kt?1
CODE_LOAD::go/guides/retries.go?1
CODE_LOAD::rust/src/guides/retries.rs
Note that these retries are coordinated and initiated by the Restate Server.
So the handler goes through the regular retry cycle of suspension and re-invocation.
If you set a maximum number of attempts, then the run block will fail with a TerminalException once the retries are exhausted.
Service-level retry policies are planned and will come soon.
## Application errors (terminal)
By default, Restate infinitely retries all errors.
In some cases, you might not want to retry an error (e.g. because of business logic, because the issue is not transient, ...).
For these cases you can throw a terminal error. Terminal errors are permanent and are not retried by Restate.
You can throw a terminal error as follows:
CODE_LOAD::ts/src/develop/error_handling.ts
CODE_LOAD::python/src/develop/error_handling.py
CODE_LOAD::java/src/main/java/develop/ErrorHandling.java
CODE_LOAD::kotlin/src/main/kotlin/develop/ErrorHandling.kt
CODE_LOAD::go/develop/errorhandling.go
CODE_LOAD::rust/src/guides/retries.rs#terminal_error
You can throw terminal errors from any place in your handler, including run blocks.
Unless catched, terminal errors stop the execution and are propagated back to the caller.
If the caller is another Restate service, the terminal error will propagate across RPCs, and will get thrown at the line where the RPC was made.
If this is not caught, it will propagate further up the call stack until it reaches the original caller.
You can catch terminal errors just like any other error, and build control flow around this.
For example, the catch block can run undo actions for the actions you did earlier in your handler, to bring it to a consistent state before rethrowing the terminal error.
For example, to catch a terminal error of a run block:
CODE_LOAD::ts/src/guides/retries.ts#catch
CODE_LOAD::python/src/guides/retries.py#catch
CODE_LOAD::java/src/main/java/guides/RetryRunService.java#catch
CODE_LOAD::kotlin/src/main/kotlin/guides/RetryRunService.kt#catch
CODE_LOAD::go/guides/retries.go#catch
CODE_LOAD::rust/src/guides/retries.rs#catch
When you throw a terminal error, you might need to undo the actions you did earlier in your handler to make sure that your system remains in a consistent state.
Have a look at our [sagas guide](/guides/sagas) to learn more.
## Cancellations are Terminal Errors
You can cancel invocations via the [CLI](/operate/invocation#cancelling-invocations), UI and programmatically.
When you cancel an invocation, it throws a terminal error in the handler processing the invocation the next time it awaits a Promise or Future of a Restate Context action (e.g. run block, RPC, sleep,...; `RestatePromise` in TypeScript, `DurableFuture` in Java).
Unless caught, This terminal error will propagate up the call stack until it reaches the original caller.
Here again, the handler needs to have [compensation logic](/guides/sagas) in place to make sure the system remains in a consistent state, when you cancel an invocation.
## Timeouts between Restate and the service
There are two types of timeouts describing the behavior between Restate and the service.
### Inactivity timeout
When the Restate Server does not receive a next journal entry from a running handler within the inactivity timeout, it will ask the handler to suspend.
This timer guards against stalled service/handler invocations. Once it expires, Restate triggers a graceful termination by asking the service invocation to suspend (which preserves intermediate progress).
By default, the inactivity timeout is set to one minute.
You can increase the inactivity timeout if you have long-running `ctx.run` blocks, that lead to long pauses between journal entries. Otherwise, this timeout might kill the ongoing execution.
### Abort timeout
This timer guards against stalled service/handler invocations that are supposed to terminate.
The abort timeout is started after the 'inactivity timeout' has expired and the service/handler invocation has been asked to gracefully terminate.
Once the timer expires, it will abort the service/handler invocation.
By default, the abort timeout is set to one minute.
This timer potentially interrupts user code.
If the user code needs longer to gracefully terminate, then this value needs to be set accordingly.
If you have long-running `ctx.run` blocks, you need to increase both timeouts to prevent the handler from terminating prematurely.
### Configuring the timeouts
You can set the inactivity timeout via the UI, the CLI or the [Restate Server configuration](/operate/configuration/server).
Via the CLI:
restate services config edit
Then you can adapt the configuration file and save it for the new settings to take effect.
Via the Restate Server Configuration:
[worker.invoker] inactivity-timeout = "1m" abort-timeout = "1m"
restate-server --config-file restate.toml
Both timeouts follow the [humantime](https://docs.rs/humantime/latest/humantime/) format.
Or set it [via environment variables](/operate/configuration/server#environment-variables), for example:RESTATE_WORKER__INVOKER__INACTIVITY_TIMEOUT=5m \ RESTATE_WORKER__INVOKER__ABORT_TIMEOUT=5m \ restate-server
## Common patterns
These are some common patterns for handling errors in Restate:
### Sagas
Have a look at the [sagas guide](/guides/sagas) to learn how to revert your system back to a consistent state after a terminal error.
Keep track of compensating actions throughout your business logic and apply them in the catch block after a terminal error.
### Dead-letter queue
A [dead-letter queue (DLQ)](https://aws.amazon.com/what-is/dead-letter-queue/) is a queue where you can send messages that could not be processed due to errors.
You can implement this in Restate by wrapping your handler in a try-catch block. In the catch block you can forward the failed invocation to a DLQ Kafka topic or a catch-all handler which for example reports them or backs them up.
<details>
<summary>Catching failed invocations before handler execution starts</summary>
Some errors might happen before the handler code gets invoked/starts running (e.g. service does not exist, request decoding errors in SDK HTTP server, ...).
By default, Restate fails these requests with `400`.
Handle these as follows:
- In case the caller waited for the response of the failed call, the caller can handle the propagation to the DLQ.
- If the caller did not wait for the response (one-way send), you would lose these messages.
- Decoding errors can be caught by doing the decoding inside the handler.
The called handler then takes raw input and does the decoding and validation itself.
In this case, it would be included in the try-catch block which would do the dispatching:
CODE_LOAD::ts/src/guides/retries.ts#raw
CODE_LOAD::python/src/guides/retries.py#raw
CODE_LOAD::java/src/main/java/guides/RetryRunService.java#raw
CODE_LOAD::kotlin/src/main/kotlin/guides/RetryRunService.kt#raw
CODE_LOAD::go/guides/retries.go#raw
CODE_LOAD::rust/src/guides/retries.rs#raw
The other errors mainly occur due to misconfiguration of your setup (e.g. wrong service name, wrong handler name, forgot service registration...).
You cannot handle those.
</details>
### Timeouts for context actions
You can set timeouts for context actions like calls, awakeables, etc. to bound the time they take:
CODE_LOAD::ts/src/guides/retries.ts#timeout
CODE_LOAD::python/src/guides/retries.py#timeout
CODE_LOAD::java/src/main/java/guides/RetryRunService.java#timeout
CODE_LOAD::kotlin/src/main/kotlin/guides/RetryRunService.kt#timeout
CODE_LOAD::go/guides/retries.go#timeout
Deploying Restate TypeScript services on AWS Lambda
This tutorial shows how to deploy a greeter service written with the Restate TypeScript SDK on AWS Lambda via AWS console.
- The prerequisites for running Restate TS services
- An AWS account with permissions for Lambda.
<a href={"/get_started/quickstart"Get the Greeter service template</a></span>
<a href={"/develop/ts/serving#creating-a-lambda-handler"Convert the endpoint to a Lambda handler</a></span>
Now, we need to create a zip file that includes the service code and the required dependencies to run it. To build the code and make the zip file, do
npm run bundleGo to the Lambda UI in the AWS console. Click on Create function. Fill in the name of your function. You can leave the settings to the default.
<details> <summary>View</summary>

</details>
Click Create function.
You should now see a function overview with your new function in it.
<details> <summary>View</summary>

</details>
The next step is uploading the zip file with our function code. Open the Code tab in the section below the function overview. Click on Upload from and select your zip file. You should now see the uploaded code in the browser editor.
By default, Lambda assumes that your handler can be found under index.handler. So this means that you should have the Restate Lambda handler assigned to export const handler in the file src/app.ts, as shown in the code. This handler will then be included in index.js after creating the zip, and be used by AWS Lambda as the entry point of the Lambda function. To change that you can scroll down to Runtime settings and change the handler reference.
Finally, let's publish a new version of our Lambda function. Go to the tab Versions and click Publish new version and then Publish.
Our Lambda function should now be working!
Run the Restate Server via one of the options listed in the docs.
<details>
<summary>Running Restate in a Docker container</summary>
If you run Restate in a Docker container, then make sure it can using your local AWS creds (defined in ~/.aws):
docker run -e AWS_PROFILE -v ~/.aws/:/root/.aws --name restate_dev --rm -p 8080:8080 -p 9070:9070 -p 9071:9071 --add-host=host.docker.internal:host-gateway docker.restate.dev/restatedev/restate:VAR::RESTATE_VERSION</details>
Connect to the Restate Server (e.g. via an SSH session if it is running on EC2) and execute the registration command:
```shell !!tabs CLI restate deployments register arn:aws:lambda:eu-central-1:000000000000:function:my-greeter:1
curl localhost:9070/deployments --json '{"arn": "arn:aws:lambda:eu-central-1:000000000000:function:my-greeter:1" }'
Make sure you replace the Lambda function ARN with the one you deployed, including it's version tag (here `1`).
When executing this command, you should see the discovered services printed out!
curl localhost:8080/Greeter/greet --json '"Hi"'
The Greeter service should say hi back.
Here are some next steps for you to try:
- Add a new method to the greeter function and redeploy the Lambda function with the new methods enabled.
- Create and deploy a new Lambda function that calls the greeter function.
Parallelizing work
This guide shows how to use the Restate to execute a list of tasks in parallel and then gather their result, also known as fan-out, fan-in.
How does Restate help?
- Restate lets you schedule the tasks asynchronously and guarantees that all tasks will run, with retries and recovery on failures.
- Restate turns Promises/Futures into durable, distributed constructs that are persisted in Restate and can be recovered and awaited on another process.
- You can deploy the subtask executors on serverless infrastructure, like AWS Lambda, to let them scale automatically. The main task, that is idle while waiting on the subtasks, gets suspended until it can make progress.
Fan out: You can fan out tasks with Restate by creating a handler that processes a single subtask, and then scheduling it repeatedly from another handler.
Fan in: You can fan in the results of the subtasks by using Restate's Promise Combinators to wait for all promises to resolve.
Example
The example implements a worker service: 1. It splits a task into subtasks. 2. It schedules all the subtasks. Each subtask results in a promise that gets added to a list. 3. The result is gathered by waiting for all promises to resolve.
You can run this on FaaS infrastructure, like AWS Lambda, and it will scale automatically. The run handler will then suspend while it waits for all subtasks to finish. Restate will then resume the handler when all subtasks are done.
```ts !!tabs TypeScript https://github.com/restatedev/examples/blob/main/typescript/patterns-use-cases/src/parallelizework/fan_out_worker.ts // collapse_prequel CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/typescript/patterns-use-cases/src/parallelizework/fan_out_worker.ts
// collapse_prequel CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/java/patterns-use-cases/src/main/java/my/example/parallelizework/FanOutWorker.java
// collapse_prequel CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/kotlin/patterns-use-cases/src/main/kotlin/my/example/parallelizework/FanOutWorker.kt
// collapse_prequel CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/python/patterns-use-cases/parallelizework/app.py
// collapse_prequel CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/go/patterns-use-cases/src/parallelizework/fanoutworker.go
In this example, we parallelize RPC calls, but this can also be used to parallelize `ctx.run` actions.
This pattern is implementable with any of our SDKs. We are still working on translating all patterns to all SDK languages.
If you need help with a specific language, please reach out to us via [Discord](https://discord.com/invite/skW3AZ6uGd) or [Slack](https://join.slack.com/t/restatecommunity/shared_invite/zt-2v9gl005c-WBpr167o5XJZI1l7HWKImA).
## Running the example
restate example typescript-patterns-use-cases && cd typescript-patterns-use-cases
restate example java-patterns-use-cases && cd java-patterns-use-cases
restate example kotlin-patterns-use-cases && cd kotlin-patterns-use-cases
restate example python-patterns-use-cases && cd python-patterns-use-cases
restate example go-patterns-use-cases && cd go-patterns-use-cases
restate-server
npx tsx watch ./src/parallelizework/fan_out_worker.ts
./gradlew -PmainClass=my.example.parallelizework.FanOutWorker run
./gradlew -PmainClass=my.example.parallelizework.FanOutWorkerKt run
python parallelizework/app.py
go run ./src/parallelizework
restate deployments register localhost:9080
curl localhost:8080/worker/run \ --json '{"description": "get out of bed,shower,make coffee,have breakfast"}'
curl localhost:8080/FanOutWorker/run \ --json '{"description": "get out of bed,shower,make coffee,have breakfast"}'
curl localhost:8080/FanOutWorker/run \ --json '{"description": "get out of bed,shower,make coffee,have breakfast"}'
curl localhost:8080/FanOutWorker/run \ --json '{"description": "get out of bed,shower,make coffee,have breakfast"}'
curl localhost:8080/FanOutWorker/Run \ --json '{"description": "get out of bed,shower,make coffee,have breakfast"}'
See how all tasks get spawned in parallel, finish at different times, and then get aggregated.
!mark gold
[restate] [worker/runSubtask][inv_17jBqoqRG0TN3msVqHEpZn2aQMOX5kSKrf][2025-01-17T08:51:44.993Z] INFO: Started executing subtask: get out of bed
!mark green
[restate] [worker/runSubtask][inv_1f8R1NuF0LF27EdQ0R6s7PR8hld245OM8h][2025-01-17T08:51:44.995Z] INFO: Started executing subtask: shower
!mark blue
[restate] [worker/runSubtask][inv_101oPhGwxQqZ0sQebkQnpGyV9Rp3oj9CSJ][2025-01-17T08:51:44.997Z] INFO: Started executing subtask: make coffee
!mark red
[restate] [worker/runSubtask][inv_1eKDShaxMCEB6DXasrR5OtRXJEvA2je33X][2025-01-17T08:51:44.998Z] INFO: Started executing subtask: have breakfast
!mark gold
[restate] [worker/runSubtask][inv_17jBqoqRG0TN3msVqHEpZn2aQMOX5kSKrf][2025-01-17T08:51:47.003Z] INFO: Execution subtask finished: get out of bed
!mark blue
[restate] [worker/runSubtask][inv_101oPhGwxQqZ0sQebkQnpGyV9Rp3oj9CSJ][2025-01-17T08:51:48.007Z] INFO: Execution subtask finished: make coffee
!mark green
[restate] [worker/runSubtask][inv_1f8R1NuF0LF27EdQ0R6s7PR8hld245OM8h][2025-01-17T08:51:48.999Z] INFO: Execution subtask finished: shower
!mark red
[restate] [worker/runSubtask][inv_1eKDShaxMCEB6DXasrR5OtRXJEvA2je33X][2025-01-17T08:51:49.001Z] INFO: Execution subtask finished: have breakfast [restate] [worker/run][inv_18QHSeAYfvim1oNXRl9I5105veQcTW3BEl][2025-01-17T08:51:49.007Z] INFO: Aggregated result: get out of bed: DONE,shower: DONE,make coffee: DONE,have breakfast: DONE
2025-01-17 10:00:58 INFO [FanOutWorker/run][inv_1jNoSMJtWluo4Ir43OUyDAxD9weMAQ4OeR] dev.restate.sdk.core.InvocationStateMachine - Start invocation 2025-01-17 10:00:58 INFO [FanOutWorker/runSubtask][inv_1kdpBvVXdqyo3saU6KThul6Jgkfot6LcRP] dev.restate.sdk.core.InvocationStateMachine - Start invocation
!focus
!mark gold
2025-01-17 10:00:58 INFO [FanOutWorker/runSubtask][inv_1kdpBvVXdqyo3saU6KThul6Jgkfot6LcRP] my.example.parallelizework.utils.Utils - Started executing subtask: get out of bed 2025-01-17 10:00:58 INFO [FanOutWorker/runSubtask][inv_162MCD5ertQ65pdG0uDIYRMgLBFYZkNPnb] dev.restate.sdk.core.InvocationStateMachine - Start invocation
!focus
!mark green
2025-01-17 10:00:58 INFO [FanOutWorker/runSubtask][inv_162MCD5ertQ65pdG0uDIYRMgLBFYZkNPnb] my.example.parallelizework.utils.Utils - Started executing subtask: shower 2025-01-17 10:00:58 INFO [FanOutWorker/runSubtask][inv_10bPiFTjBUXX35qtzOTPNr0vfgoYYVehpf] dev.restate.sdk.core.InvocationStateMachine - Start invocation
!focus
!mark blue
2025-01-17 10:00:58 INFO [FanOutWorker/runSubtask][inv_10bPiFTjBUXX35qtzOTPNr0vfgoYYVehpf] my.example.parallelizework.utils.Utils - Started executing subtask: make coffee 2025-01-17 10:00:58 INFO [FanOutWorker/runSubtask][inv_1115lzidXq7M7CtLZn0aEyUvMC4zkXYLWF] dev.restate.sdk.core.InvocationStateMachine - Start invocation
!focus
!mark red
2025-01-17 10:00:58 INFO [FanOutWorker/runSubtask][inv_1115lzidXq7M7CtLZn0aEyUvMC4zkXYLWF] my.example.parallelizework.utils.Utils - Started executing subtask: have breakfast
!focus
!mark green
2025-01-17 10:00:59 INFO [FanOutWorker/runSubtask][inv_162MCD5ertQ65pdG0uDIYRMgLBFYZkNPnb] my.example.parallelizework.utils.Utils - Execution subtask finished: shower 2025-01-17 10:00:59 INFO [FanOutWorker/runSubtask][inv_162MCD5ertQ65pdG0uDIYRMgLBFYZkNPnb] dev.restate.sdk.core.InvocationStateMachine - End invocation
!focus
!mark gold
2025-01-17 10:01:00 INFO [FanOutWorker/runSubtask][inv_1kdpBvVXdqyo3saU6KThul6Jgkfot6LcRP] my.example.parallelizework.utils.Utils - Execution subtask finished: get out of bed 2025-01-17 10:01:00 INFO [FanOutWorker/runSubtask][inv_1kdpBvVXdqyo3saU6KThul6Jgkfot6LcRP] dev.restate.sdk.core.InvocationStateMachine - End invocation
!focus
!mark blue
2025-01-17 10:01:04 INFO [FanOutWorker/runSubtask][inv_10bPiFTjBUXX35qtzOTPNr0vfgoYYVehpf] my.example.parallelizework.utils.Utils - Execution subtask finished: make coffee 2025-01-17 10:01:04 INFO [FanOutWorker/runSubtask][inv_10bPiFTjBUXX35qtzOTPNr0vfgoYYVehpf] dev.restate.sdk.core.InvocationStateMachine - End invocation
!focus
!mark red
2025-01-17 10:01:05 INFO [FanOutWorker/runSubtask][inv_1115lzidXq7M7CtLZn0aEyUvMC4zkXYLWF] my.example.parallelizework.utils.Utils - Execution subtask finished: have breakfast 2025-01-17 10:01:05 INFO [FanOutWorker/runSubtask][inv_1115lzidXq7M7CtLZn0aEyUvMC4zkXYLWF] dev.restate.sdk.core.InvocationStateMachine - End invocation 2025-01-17 10:01:05 INFO [FanOutWorker/run][inv_1jNoSMJtWluo4Ir43OUyDAxD9weMAQ4OeR] my.example.parallelizework.utils.Utils - Aggregated result: get out of bed: DONE, shower: DONE, make coffee: DONE, have breakfast: DONE 2025-01-17 10:01:05 INFO [FanOutWorker/run][inv_1jNoSMJtWluo4Ir43OUyDAxD9weMAQ4OeR] dev.restate.sdk.core.InvocationStateMachine - End invocation
2025-03-06 12:20:18 INFO [FanOutWorker/run][inv_1fGEUyfogPKK5cbSCSWzpCkcDpkQIKSzMB] dev.restate.sdk.core.InvocationStateMachine - Start invocation 2025-03-06 12:20:18 INFO [FanOutWorker/runSubtask][inv_146fBfVLISKb2sCWqesf6uMReXdRKvqmv7] dev.restate.sdk.core.InvocationStateMachine - Start invocation
!mark gold
2025-03-06 12:20:18 INFO [FanOutWorker/runSubtask][inv_146fBfVLISKb2sCWqesf6uMReXdRKvqmv7] FanOutWorker - Started executing subtask: get out of bed 2025-03-06 12:20:18 INFO [FanOutWorker/runSubtask][inv_18T9WW6paOhm6eciCeBt5iqHXRY4h2NRvP] dev.restate.sdk.core.InvocationStateMachine - Start invocation
!mark green
2025-03-06 12:20:18 INFO [FanOutWorker/runSubtask][inv_18T9WW6paOhm6eciCeBt5iqHXRY4h2NRvP] FanOutWorker - Started executing subtask: shower 2025-03-06 12:20:18 INFO [FanOutWorker/runSubtask][inv_10kE3b5UcL8L64ghFpHQjeeAEowKNis4dH] dev.restate.sdk.core.InvocationStateMachine - Start invocation
!mark blue
2025-03-06 12:20:18 INFO [FanOutWorker/runSubtask][inv_10kE3b5UcL8L64ghFpHQjeeAEowKNis4dH] FanOutWorker - Started executing subtask: make coffee 2025-03-06 12:20:18 INFO [FanOutWorker/runSubtask][inv_1fCFmQ9ulbxL2MBwYCDRgV8Was8PDBedW1] dev.restate.sdk.core.InvocationStateMachine - Start invocation
!mark red
2025-03-06 12:20:18 INFO [FanOutWorker/runSubtask][inv_1fCFmQ9ulbxL2MBwYCDRgV8Was8PDBedW1] FanOutWorker - Started executing subtask: have breakfast
!mark blue
2025-03-06 12:20:21 INFO [FanOutWorker/runSubtask][inv_10kE3b5UcL8L64ghFpHQjeeAEowKNis4dH] FanOutWorker - Execution subtask finished: make coffee 2025-03-06 12:20:21 INFO [FanOutWorker/runSubtask][inv_10kE3b5UcL8L64ghFpHQjeeAEowKNis4dH] dev.restate.sdk.core.InvocationStateMachine - End invocation
!mark gold
2025-03-06 12:20:24 INFO [FanOutWorker/runSubtask][inv_146fBfVLISKb2sCWqesf6uMReXdRKvqmv7] FanOutWorker - Execution subtask finished: get out of bed 2025-03-06 12:20:24 INFO [FanOutWorker/runSubtask][inv_146fBfVLISKb2sCWqesf6uMReXdRKvqmv7] dev.restate.sdk.core.InvocationStateMachine - End invocation
!mark red
2025-03-06 12:20:25 INFO [FanOutWorker/runSubtask][inv_1fCFmQ9ulbxL2MBwYCDRgV8Was8PDBedW1] FanOutWorker - Execution subtask finished: have breakfast 2025-03-06 12:20:25 INFO [FanOutWorker/runSubtask][inv_1fCFmQ9ulbxL2MBwYCDRgV8Was8PDBedW1] dev.restate.sdk.core.InvocationStateMachine - End invocation
!mark green
2025-03-06 12:20:27 INFO [FanOutWorker/runSubtask][inv_18T9WW6paOhm6eciCeBt5iqHXRY4h2NRvP] FanOutWorker - Execution subtask finished: shower 2025-03-06 12:20:27 INFO [FanOutWorker/runSubtask][inv_18T9WW6paOhm6eciCeBt5iqHXRY4h2NRvP] dev.restate.sdk.core.InvocationStateMachine - End invocation 2025-03-06 12:20:27 INFO [FanOutWorker/run][inv_1fGEUyfogPKK5cbSCSWzpCkcDpkQIKSzMB] FanOutWorker - Aggregated result: get out of bed: DONE, shower: DONE, make coffee: DONE, have breakfast: DONE 2025-03-06 12:20:27 INFO [FanOutWorker/run][inv_1fGEUyfogPKK5cbSCSWzpCkcDpkQIKSzMB] dev.restate.sdk.core.InvocationStateMachine - End invocation
!mark gold
[2025-01-17 12:00:05,183] [12245] [INFO] - Started executing subtask: get out of bed
!mark green
[2025-01-17 12:00:05,184] [12247] [INFO] - Started executing subtask: shower
!mark blue
[2025-01-17 12:00:05,184] [12245] [INFO] - Started executing subtask: make coffee
!mark red
[2025-01-17 12:00:05,185] [12245] [INFO] - Started executing subtask: have breakfast
!mark blue
[2025-01-17 12:00:05,188] [12245] [INFO] - Execution subtask finished: make coffee
!mark gold
[2025-01-17 12:00:08,193] [12245] [INFO] - Execution subtask finished: get out of bed
!mark green
[2025-01-17 12:00:10,194] [12247] [INFO] - Execution subtask finished: shower
!mark red
[2025-01-17 12:00:15,196] [12245] [INFO] - Execution subtask finished: have breakfast [2025-01-17 12:00:15,198] [12245] [INFO] - Aggregated result: get out of bed: DONE,shower: DONE,make coffee: DONE,have breakfast: DONE
2025/01/16 16:41:22 INFO Handling invocation method=FanOutWorker/Run invocationID=inv_1lkcVTBmCorR3fSPhE0pNTiO8XFXoV34C5 2025/01/16 16:41:22 INFO Handling invocation method=FanOutWorker/RunSubtask invocationID=inv_1jpZWOrDK45b2ZwWCapl68GgXzNfoOh0BP
!focus
!mark gold
2025/01/16 16:41:22 Started executing subtask: get out of bed 2025/01/16 16:41:22 INFO Handling invocation method=FanOutWorker/RunSubtask invocationID=inv_10eVGnmjP1ET4PgI3z82rvXcVlCnSOep3P
!focus
!mark green
2025/01/16 16:41:22 Started executing subtask: shower 2025/01/16 16:41:22 INFO Handling invocation method=FanOutWorker/RunSubtask invocationID=inv_1i3RduoDMNnb4ideAtWaWCLRDaIO62eghP
!focus
!mark blue
2025/01/16 16:41:22 Started executing subtask: make coffee 2025/01/16 16:41:22 INFO Handling invocation method=FanOutWorker/RunSubtask invocationID=inv_142WnXnWDxfy6k4JanZ7DQVqAL6zmktuxP
!focus
!mark red
2025/01/16 16:41:22 Started executing subtask: have breakfast
!focus
!mark gold
2025/01/16 16:41:24 Execution subtask finished: get out of bed 2025/01/16 16:41:24 INFO Invocation completed successfully method=FanOutWorker/RunSubtask invocationID=inv_1jpZWOrDK45b2ZwWCapl68GgXzNfoOh0BP
!focus
!mark green
2025/01/16 16:41:25 Execution subtask finished: shower 2025/01/16 16:41:25 INFO Invocation completed successfully method=FanOutWorker/RunSubtask invocationID=inv_10eVGnmjP1ET4PgI3z82rvXcVlCnSOep3P
!focus
!mark red
2025/01/16 16:41:25 Execution subtask finished: have breakfast 2025/01/16 16:41:25 INFO Invocation completed successfully method=FanOutWorker/RunSubtask invocationID=inv_142WnXnWDxfy6k4JanZ7DQVqAL6zmktuxP
!focus
!mark blue
2025/01/16 16:41:26 Execution subtask finished: make coffee 2025/01/16 16:41:26 INFO Invocation completed successfully method=FanOutWorker/RunSubtask invocationID=inv_1i3RduoDMNnb4ideAtWaWCLRDaIO62eghP 2025/01/16 16:41:26 Aggregated result: get out of bed: DONE,shower: DONE,have breakfast: DONE,make coffee: DONE 2025/01/16 16:41:26 INFO Invocation completed successfully method=FanOutWorker/Run invocationID=inv_1lkcVTBmCorR3fSPhE0pNTiO8XFXoV34C5
## Related resources
- Promise combinator docs: [TS](/develop/ts/journaling-results#combineable-promise-combinators) /
[Java/Kotlin](/develop/java/journaling-results#durable-future-combinators) /
[Python](/develop/python/journaling-results#waiting-multiple-futures) /
[Go](/develop/go/journaling-results#selectors) /
[Rust](https://docs.rs/restate-sdk/latest/restate_sdk/macro.select.html)
- [Async Tasks use case page](/use-cases/async-tasks)
- Concurrent async tasks with [Java](https://github.com/restatedev/examples/blob/main/java/patterns-use-cases/README.md#concurrent-async-tasks)Sagas
When building distributed systems, it is crucial to ensure that the system remains consistent even in the presence of failures. One way to achieve this is by using the Saga pattern.
A Saga is a design pattern for handling transactions that span multiple services. It breaks the process into a sequence of local operations, each with a corresponding compensating action.
If a failure occurs partway through, these compensations are triggered to undo completed steps, ensuring your system stays consistent even when things go wrong.
How does Restate help?
Restate makes it easy to implement resilient sagas in your code:
- Durable Execution: Restate guarantees that your code runs to completion. If a transient failure occurs, Restate automatically retries from the point of failure and ensures that all compensations run.
- Resilience built-in: No need to manually track state or retry logic. Restate handles all persistence and compensation orchestration for you.
- Code-first approach: Define sagas using regular code, no DSLs. Track compensations in a list, and execute them on non-transient failures.
Example
Here is a typical travel booking workflow, where you book a flight, then rent a car, and finally book a hotel. If any step fails for a non-transient reason (e.g. driver license not accepted, hotel full), we want to roll back the previous steps to keep the system consistent.
Restate lets us implement this purely in code without any DSLs or extra infrastructure.
- Wrap your business logic in a try-block, and throw a terminal error for cases where you want to compensate and finish.
- For each step you do in your try-block, add a compensation to a list.
- In the catch block, in case of a terminal error, you run the compensations in reverse order, and rethrow the error.
Note that for Golang we use defer to run the compensations at the end.
```ts !!tabs TypeScript https://github.com/restatedev/examples/blob/main/typescript/patterns-use-cases/src/sagas/booking_workflow.ts // collapse_prequel CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/typescript/patterns-use-cases/src/sagas/booking_workflow.ts
// collapse_prequel CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/java/patterns-use-cases/src/main/java/my/example/sagas/BookingWorkflow.java
// collapse_prequel CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/kotlin/patterns-use-cases/src/main/kotlin/my/example/sagas/BookingWorkflow.kt
// collapse_prequel CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/python/patterns-use-cases/sagas/app.py
// collapse_prequel CODE_LOAD::https://raw.githubusercontent.com/restatedev/examples/refs/heads/main/go/patterns-use-cases/src/sagas/bookingworkflow.go
This pattern is implementable with any of our SDKs. We are still working on translating all patterns to all SDK languages.
If you need help with a specific language, please reach out to us via [Discord](https://discord.com/invite/skW3AZ6uGd) or [Slack](https://join.slack.com/t/restatecommunity/shared_invite/zt-2v9gl005c-WBpr167o5XJZI1l7HWKImA).
## When to use Sagas
Restate automatically retries all transient failures, like network hiccups or temporary service outages. But not all failures are temporary.
For these failures, sagas are essential:
1. **Business logic requirements**:
- Some failures are not transient but a business decision (e.g. “Hotel is full” or “Driver license not accepted”), retrying won't help.
- In this case, you can throw a terminal error to stop the execution and trigger the compensations.
2. **User/system-initiated cancellations**:
- If a user [cancels](/operate/invocation#cancelling-invocations) a long-running invocation (say via UI or CLI), this triggers a terminal error.
- Restate will not retry.
- Again, a saga can kick in to undo previous successful operations so the system doesn't end up in an inconsistent state (e.g., booking a hotel but not a car).
## Running the example
restate example typescript-patterns-use-cases && cd typescript-patterns-use-cases
restate example java-patterns-use-cases && cd java-patterns-use-cases
restate example kotlin-patterns-use-cases && cd kotlin-patterns-use-cases
restate example python-patterns-use-cases && cd python-patterns-use-cases
restate example go-patterns-use-cases && cd go-patterns-use-cases
restate-server
npx tsx watch ./src/sagas/booking_workflow.ts
./gradlew -PmainClass=my.example.sagas.BookingWorkflow run
./gradlew -PmainClass=my.example.sagas.BookingWorkflowKt run
python sagas/app.py
go run ./src/sagas
restate deployments register localhost:9080
curl localhost:8080/BookingWorkflow/run --json '{ "flight": { "flightId": "12345", "passengerName": "John Doe" }, "car": { "pickupLocation": "Airport", "rentalDate": "2024-12-16" }, "hotel": { "arrivalDate": "2024-12-16", "departureDate": "2024-12-20"
}'
curl localhost:8080/BookingWorkflow/run --json '{ "flight": { "flightId": "12345", "passengerName": "John Doe" }, "car": { "pickupLocation": "Airport", "rentalDate": "2024-12-16" }, "hotel": { "arrivalDate": "2024-12-16", "departureDate": "2024-12-20"
}'
curl localhost:8080/BookingWorkflow/run --json '{ "flight": { "flightId": "12345", "passengerName": "John Doe" }, "car": { "pickupLocation": "Airport", "rentalDate": "2024-12-16" }, "hotel": { "arrivalDate": "2024-12-16", "departureDate": "2024-12-20"
}'
curl localhost:8080/BookingWorkflow/run --json '{ "flight": { "flightId": "12345", "passengerName": "John Doe" }, "car": { "pickupLocation": "Airport", "rentalDate": "2024-12-16" }, "hotel": { "arrivalDate": "2024-12-16", "departureDate": "2024-12-20"
}'
curl localhost:8080/BookingWorkflow/Run --json '{ "flight": { "flightId": "12345", "passengerName": "John Doe" }, "car": { "pickupLocation": "Airport", "rentalDate": "2024-12-16" }, "hotel": { "arrivalDate": "2024-12-16", "departureDate": "2024-12-20"
}'
See in the Restate UI (`localhost:9070`) how all steps were executed, and how the compensations were triggered because the hotel was full.
## Advanced: Idempotency and compensations
Since sagas in Restate are implemented in user code, compensations are flexible and powerful, as long as they're idempotent: you can reset service state, call other services to undo prior actions, use `ctx.run` to delete rows or reverse database operations.
The example above uses the customer ID to guarantee idempotency, so that on retries it will not create duplicate bookings or rentals.
The example assumes that the API provider deduplicates the requests based on this ID.
Based on the API you are using, generating the idempotency key and registering the compensation can be done in different ways:
1. **Two-phase APIs**: First you _reserve_, then _confirm_ or _cancel_. Register the compensation after reservation, when you have the resource ID.
Reservations that are not confirmed, get automatically cancelled by the API after a timeout.
CODE_LOAD::ts/src/guides/sagas/booking_workflow.ts#twostep
CODE_LOAD::python/src/guides/sagas/app.py#twostep
CODE_LOAD::java/src/main/java/guides/sagas/BookingWorkflow.java#twostep
CODE_LOAD::kotlin/src/main/kotlin/guides/sagas/BookingWorkflow.kt#twostep
CODE_LOAD::go/guides/sagas/bookingworkflow.go#twostep
2. **One-shot APIs with idempotency key**: First, you generate an idempotency key and persist it in Restate. Then, you register the compensation (e.g. `refund`), and finally do the action (e.g. `charge`).
We need to register the compensation before doing the action, because there is a chance that the action succeeded but that we never got the confirmation.
CODE_LOAD::ts/src/guides/sagas/booking_workflow.ts#idempotency
CODE_LOAD::python/src/guides/sagas/app.py#idempotency
CODE_LOAD::java/src/main/java/guides/sagas/BookingWorkflow.java#idempotency
CODE_LOAD::kotlin/src/main/kotlin/guides/sagas/BookingWorkflow.kt#idempotency
CODE_LOAD::go/guides/sagas/bookingworkflow.go#idempotency
## Related resources
- [Error Handling guide](/guides/error-handling)
- Terminal errors: [Java](/develop/java/error-handling) / [TS](/develop/ts/error-handling) / [Go](/develop/go/error-handling) / [Python](/develop/python/error-handling) / [Rust](https://docs.rs/restate-sdk/latest/restate_sdk/errors/index.html)
- [Cancellation of invocations](/operate/invocation#cancelling-invocations)
- [Blog post: Graceful cancellations: How to keep your application and workflow state consistent 💪](https://restate.dev/blog/graceful-cancellations-how-to-keep-your-application-and-workflow-state-consistent/)
- [Microservice orchestration use case page](/use-cases/microservice-orchestration)
Clients
The Restate SDK client library lets you invoke Restate handlers from anywhere in your application. Use this only in non-Restate services without access to the Restate Context.
The UI helps you with invoking your services programmatically. Open the UI at port 9070, register your service, click on the service, open the playground, and copy over the code snippet to invoke your service in your preferred language.
Always invoke handlers via the context, if you have access to it. Restate then attaches information about the invocation to the parent invocation.
Have a look at the documentation of the SDKs:
HTTP
You can invoke handlers over HTTP with or without waiting for a response, and with or without an idempotency key.
Make sure to first register the handler you want to invoke.
The UI helps you with invoking your services. Open the UI at port 9070, register your service, click on the service, open the playground, and invoke your handlers from there.
Request-response calls over HTTP
You can invoke services over HTTP 1.1 or higher. Request/response bodies should be encoded as JSON.
Invoking Services
Invoke myHandler of myService as follows:
```shell !result curl localhost:8080/MyService/myHandler --json '{"name": "Mary", "age": 25}'
### Invoking Virtual Objects
Invoke `myHandler` of `myVirtualObject` for `myKey` as follows:
curl localhost:8080/MyVirtualObject/myObjectKey/myHandler --json '{"name": "Mary", "age": 25}'
### Invoke Workflows
Call the `run` handler of the `MyWorkflow` as follows:
curl localhost:8080/MyWorkflow/myWorkflowId/run --json '{"name": "Mary", "age": 25}'
Follow the same pattern for calling the other handlers of the workflow.
Note that all invocations go first via the Restate Server. The server then forwards the request to the appropriate service.
Therefore, `localhost:8080` refers to ingress port of the Restate Server, not the service instance.
## Sending a message over HTTP
If you do not want to wait for the response, you can also send a message by adding `/send` to the URL path:
curl localhost:8080/MyService/myHandler/send --json '{"name": "Mary", "age": 25}'
{"invocationId":"inv_1aiqX0vFEFNH1Umgre58JiCLgHfTtztYK5","status":"Accepted"}
The response contains the [Invocation ID](/operate/invocation#invocation-identifier).
You can use this identifier [to cancel](/operate/invocation#cancelling-invocations) or [kill the invocation](/operate/invocation#killing-invocations).
## Sending a delayed message over HTTP
You can **delay the message** by adding a delay request parameter in ISO8601 notation or using [humantime format](https://docs.rs/humantime/latest/humantime/):
curl localhost:8080/MyService/myHandler/send?delay=10s --json '{"name": "Mary", "age": 25}'
curl localhost:8080/MyService/myHandler/send?delay=PT10S --json '{"name": "Mary", "age": 25}'
You cannot yet use this feature for workflows.
Workflows can only be scheduled with a delay from within another Restate handler ([TS](/develop/ts/service-communication#delayed-calls)/[Java/Kotlin](/develop/java/service-communication#delayed-calls)).
## Invoke a handler idempotently
You can send requests to Restate providing an idempotency key, through the [`Idempotency-Key` header](https://datatracker.ietf.org/doc/draft-ietf-httpapi-idempotency-key-header/):
curl localhost:8080/MyService/myHandler \
!focus
-H 'idempotency-key: ad5472esg4dsg525dssdfa5loi' \ --json '{"name": "Mary", "age": 25}'
After the invocation completes, Restate persists the response for a retention period of one day (24 hours).
If you re-invoke the service with the same idempotency key within 24 hours, Restate sends back the same response and doesn't re-execute the request to the service.
With Restate and an idempotency key, you can make any service call idempotent, without any extra code or setup.
This is a very powerful feature to ensure that your system stays consistent and doesn't perform the same operation multiple times.
<details'The retention time is in humantime format. </details>
Retrieve result of invocations and workflows
Restate allows you to retrieve the result of workflows and invocations with an idempotency key. There are two options:
- To attach to an invocation or workflow and wait for it to finish, use
/attach. - To peek at the output of an invocation or workflow, use
/output. This will return: {"message":"not ready"}for ongoing workflows- The result for finished workflows
{"message":"not found"}for non-existing workflows
You can attach to a service/object invocation only if the invocation used an idempotency key:
# Via invocation ID
curl localhost:8080/restate/invocation/myInvocationId/attach
curl localhost:8080/restate/invocation/myInvocationId/output
# For Services, via idempotency key
curl localhost:8080/restate/invocation/MyService/myHandler/myIdempotencyKey/attach
curl localhost:8080/restate/invocation/MyService/myHandler/myIdempotencyKey/output
# For Virtual Objects, via idempotency key
curl localhost:8080/restate/invocation/myObject/myKey/myHandler/myIdempotencyKey/attach
curl localhost:8080/restate/invocation/myObject/myKey/myHandler/myIdempotencyKey/output
# For Workflows, with the Workflow ID
curl localhost:8080/restate/workflow/MyWorkflow/myWorkflowId/attach
curl localhost:8080/restate/workflow/MyWorkflow/myWorkflowId/outputOpenAPI support
Restate exposes for every service an OpenAPI 3.1 definition, to get it:
curl localhost:9070/services/MyService/openapi > MyService_openapi.jsonYou can use this definition with any OpenAPI 3.1 compliant tool to generate clients for your service, such as openapi-generator.
Depending on the SDKs, the rich input/output JSON schemas are included as well. At the moment, rich schemas are supported for:
- TypeScript SDK with Zod schemas
- Python SDK with Pydantic models
- Java and Kotlin SDK
Kafka
You can invoke handlers via Kafka events, by doing the following:
Make sure to first register the handler you want to invoke.
You can invoke any handler via Kafka events. The event payload will be (de)serialized as JSON.
- When invoking Virtual Object or Workflow handlers via Kafka, the key of the Kafka record will be used to determine the Virtual Object/Workflow key.
The key needs to be a valid UTF-8 string. The events are delivered to the subscribed handler in the order in which they arrived on the topic partition.
- When invoking Virtual Object or Workflow _shared_ handlers via Kafka, the key of the Kafka record will be used to determine the Virtual Object/Workflow key.
The key needs to be a valid UTF-8 string. The events are delivered to the subscribed handler in parallel without ordering guarantees.
- When invoking Service handlers over Kafka, events are delivered in parallel without ordering guarantees.
Since you can invoke any handler via Kafka events, a single handler can be invoked both by RPC and via Kafka.
Define the Kafka cluster that Restate needs to connect to in the Restate configuration file:
```toml restate.toml [[ingress.kafka-clusters]] name = "my-cluster" brokers = ["PLAINTEXT://broker:9092"]
And make sure the Restate Server uses it via `restate-server --config-file restate.toml`.
Check the [configuration docs](/operate/configuration/server) for more details.
<details]</details>
<a href={"/operate/registration"Register the service</a> you want to invoke.</span>
Let Restate forward events from the Kafka topic to the event handler by creating a subscription using the Admin API:
curl localhost:9070/subscriptions --json '{
"source": "kafka://my-cluster/my-topic",
"sink": "service://MyService/handle",
"options": {"auto.offset.reset": "earliest"}
}'Once you've created a subscription, Restate immediately starts consuming events from Kafka. The handler will be invoked for each event received from Kafka.
The options field is optional and accepts any configuration parameter from librdkafka configuration.
Have a look at the invocation docs for more commands to manage subscriptions.
<details> <summary>Kafka connection configuration</summary>
You can pass arbitrary Kafka cluster options in the restate.toml, and those options will be applied for all the subscriptions to that cluster, for example:
```toml restate.toml [[ingress.kafka-clusters]] name = "my-cluster" brokers = ["PLAINTEXT://broker:9092"] "sasl.username" = "me" "sasl.password" = "pass"
For the full list of options, check [librdkafka configuration](https://github.com/confluentinc/librdkafka/blob/master/CONFIGURATION.md).
</details>
<details>
<summary>Multiple Kafka clusters support</summary>
You can configure multiple kafka clusters in the `restate.toml` file:
[[ingress.kafka-clusters]] name = "my-cluster-1" brokers = ["PLAINTEXT://localhost:9092"]
[[ingress.kafka-clusters]] name = "my-cluster-2" brokers = ["PLAINTEXT://localhost:9093"]
And then, when creating the subscriptions, you refer to the specific cluster by `name`:
Subscription to my-cluster-1
curl localhost:9070/subscriptions --json '{ "source": "kafka://my-cluster-1/topic-1", "sink": "service://MyService/handleCluster1" }'
Subscription to my-cluster-2
curl localhost:9070/subscriptions --json '{ "source": "kafka://my-cluster-2/topic-2", "sink": "service://MyService/handleCluster2" }'
</details>
<details>
<summary>Raw event support</summary>
By default handlers will deserialize the event payload as JSON.
By using serdes `restate.serde.binary` you can override this behaviour. Check [Typescript SDK > Serialization](../develop/ts/serialization) for more details.
</details>
<details>
<summary>Event metadata</summary>
Each event carries within the `CODE_LOAD::ts/src/develop/kafka.ts#headers` map the following entries:
* `restate.subscription.id`: The subscription identifier, as shown by the [Admin API](/category/admin-api).
* `kafka.offset`: The record offset.
* `kafka.partition`: The record partition.
* `kafka.timestamp`: The record timestamp.
</details>
<details>
<summary>Raw event support</summary>
By default handlers will deserialize the event payload as JSON.
By declaring the handler input parameter as `byte[]` and annotating it `@Raw` the JSON deserialization will be skipped, and the event payload will be passed as is.
</details>
<details>
<summary>Event metadata</summary>
Each event carries within the `CODE_LOAD::java/src/main/java/develop/MyKafkaVirtualObject.java#headers` map the following entries:
* `restate.subscription.id`: The subscription identifier, as shown by the [Admin API](/category/admin-api).
* `kafka.offset`: The record offset.
* `kafka.partition`: The record partition.
* `kafka.timestamp`: The record timestamp.
</details>
<details>
<summary>Event metadata</summary>
Each event carries within the `CODE_LOAD::go/develop/kafka.go#headers` map the following entries:
* `restate.subscription.id`: The subscription identifier, as shown by the [Admin API](/category/admin-api).
* `kafka.offset`: The record offset.
* `kafka.partition`: The record partition.
* `kafka.timestamp`: The record timestamp.
</details>
Tooling
Restate offers the following tools:
- CLI: A command-line interface to interact with Restate services, deployments, and invocations.
You can find useful commands throughout this entire section of the documentation.
- `restatectl`: A command-line utility to control running Restate clusters.
- UI: A graphical interface to manage, debug, and configure your services.
Operating Restate and Restate Services
Restate Service Configuration
Services have some configuration settings:
- Retention time of idempotency keys
- Whether they are private or not
- Inactivity and abort timeout
These can be set via: 1. **UI**: click on your registered deployment and adapt the settings 2. CLI: execute the command, adapt the config file and save it:
restate services config edit3. **Admin API**: You can find an example of an Admin API request, in the docs on private services