Ai Infrastructure Replicate

Replicate SDK Patterns

Quick Guide: Use the replicate npm package to run open-source ML models on serverless GPUs. Use replicate.run() for synchronous execution that returns output directly, replicate.stream() for SSE-based streaming, or replicate.predictions.create() for async background jobs with webhook notifications. Models are referenced as owner/model (uses latest version) or owner/model:version (pinned). File outputs are FileOutput objects implementing ReadableStream. Cold starts are expected for infrequently-used models -- use deployments with min_instances to keep models warm.

---

<critical_requirements>

CRITICAL: Before Using This Skill

All code must follow project conventions in CLAUDE.md (kebab-case, named exports, import ordering, import type, named constants)

(You MUST never hardcode API tokens -- always use environment variables via `process.env.REPLICATE_API_TOKEN`)

(You MUST handle `FileOutput` objects for models that return files -- do not assume outputs are plain strings or URLs)

(You MUST validate webhooks using `validateWebhook()` from the `replicate` package -- never trust unverified webhook payloads)

(You MUST account for cold starts when running infrequently-used models -- use deployments for latency-sensitive applications)

(You MUST specify model versions (`owner/model:version`) in production to ensure reproducible results -- unversioned references use the latest, which can change)

</critical_requirements>

---

Auto-detection: Replicate, replicate, replicate.run, replicate.stream, replicate.predictions, replicate.deployments, replicate.trainings, replicate.models, FileOutput, validateWebhook, REPLICATE_API_TOKEN, serverless GPU, cold start, webhook_events_filter

When to use:

• Running open-source ML models (Llama, Stable Diffusion, Whisper, etc.) without managing GPU infrastructure
• Generating images, transcribing audio, running LLMs, or any ML inference via API
• Streaming LLM output in real-time with server-sent events
• Processing predictions asynchronously with webhook notifications
• Fine-tuning models with custom training data
• Running models on dedicated hardware with custom scaling via deployments

Key patterns covered:

• Client initialization and configuration (auth, user agent, file encoding)
• Running predictions (replicate.run(), replicate.predictions.create(), replicate.wait())
• Streaming output (replicate.stream() with SSE events)
• Model versioning (owner/model vs owner/model:version)
• File input/output handling (FileOutput, file uploads, Buffer inputs)
• Webhooks (setup, event filtering, signature validation)
• Deployments (custom hardware, scaling, keeping models warm)
• Training / fine-tuning

When NOT to use:

• You need a unified multi-provider LLM SDK (OpenAI, Anthropic, Google) -- use a provider-agnostic SDK
• You want to run models locally -- Replicate is a cloud-only serverless platform
• You need sub-second latency guarantees without deployments -- cold starts can take minutes

---

Examples Index

• Core: Setup, Predictions & Files -- Client init, run(), predictions.create(), wait(), file I/O, error handling
• Streaming & Webhooks -- stream(), SSE events, webhook setup, signature validation
• Deployments & Training -- Custom hardware, scaling, fine-tuning, model management
• Quick API Reference -- Method signatures, constructor options, error types, model reference format

---

<philosophy>

Philosophy

Replicate provides serverless GPU infrastructure for running open-source ML models. You send inputs, Replicate allocates GPU hardware, runs the model, and returns outputs. No Docker, no CUDA drivers, no GPU provisioning.

Core principles:

1. Serverless execution -- Models run on-demand on Replicate's infrastructure. You pay only for compute time. Cold starts are a trade-off for not maintaining always-on GPUs. 2. Model marketplace -- Thousands of community and official models available at replicate.com/explore. Run any public model with just its identifier. 3. Version pinning for reproducibility -- Models are versioned with SHA-256 hashes. Pin to a version in production (owner/model:abc123...) to guarantee identical behavior across deploys. 4. Three execution modes -- replicate.run() for synchronous wait, replicate.stream() for real-time SSE output, replicate.predictions.create() for fire-and-forget with webhooks. 5. File-first I/O -- Many models accept and produce files (images, audio, video). The SDK handles file uploads automatically and returns FileOutput objects for file outputs.

</philosophy>

---

<patterns>

Core Patterns

Pattern 1: Client Setup

Initialize the Replicate client. It auto-reads REPLICATE_API_TOKEN from the environment.

// lib/replicate.ts -- basic setup
import Replicate from "replicate";

const replicate = new Replicate();

export { replicate };

// lib/replicate.ts -- explicit auth + custom user agent
import Replicate from "replicate";

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN, // Auto-reads from env if omitted
  userAgent: "my-app/1.0.0",
});

export { replicate };

Why good: Minimal setup, env var auto-detected, explicit auth optional but useful for clarity

// BAD: Hardcoded token
const replicate = new Replicate({
  auth: "r8_abc123...",
});

Why bad: Hardcoded API token is a security risk, will leak in version control

See: examples/core.md for full constructor options, error handling patterns

---

Pattern 2: Running Predictions

Use replicate.run() for synchronous execution. Returns the model output directly.

// Run an image generation model
const [output] = await replicate.run("black-forest-labs/flux-schnell", {
  input: {
    prompt: "a serene mountain landscape at sunset",
  },
});

// output is a FileOutput object for image models
console.log(output.url()); // URL of generated image

// Run an LLM -- output is a string for text models
const output = await replicate.run("meta/meta-llama-3-70b-instruct", {
  input: {
    prompt: "Explain TypeScript generics in 3 sentences.",
    max_tokens: 512,
  },
});

console.log(output); // Text response

Why good: Simple API, returns output directly, destructuring works for array outputs (images)

// BAD: Not pinning version in production
const output = await replicate.run("community-user/experimental-model", {
  input: { prompt: "hello" },
});

Why bad: Community models without version pinning can change behavior unexpectedly when authors push updates

See: examples/core.md for version pinning, predictions.create() + wait(), and progress callbacks

---

Pattern 3: Streaming

Use replicate.stream() for real-time SSE output from language models.

const stream = replicate.stream("meta/meta-llama-3-70b-instruct", {
  input: {
    prompt: "Write a short poem about TypeScript.",
    max_tokens: 512,
  },
});

for await (const event of stream) {
  if (event.event === "output") {
    process.stdout.write(event.data);
  }
}

Why good: Progressive output for better UX, event-based with typed event and data fields

// BAD: Using replicate.run() for user-facing LLM output
const output = await replicate.run("meta/meta-llama-3-70b-instruct", {
  input: { prompt: "Write a long essay..." },
});
// User waits for entire generation to complete before seeing anything

Why bad: No progressive feedback, user sees a blank screen for seconds

See: examples/streaming-webhooks.md for event types, error handling, cancellation

---

Pattern 4: Model Versioning

Models are referenced as owner/model (latest version) or owner/model:sha256hash (pinned version).

// Development: use latest version for convenience
const output = await replicate.run("stability-ai/sdxl", {
  input: { prompt: "a cat" },
});

// Production: pin to a specific version for reproducibility
const VERSION_HASH =
  "39ed52f2a78e934b3ba6e2a89f5b1c712de7dfea535525255b1aa35c5565e08b";
const output = await replicate.run(`stability-ai/sdxl:${VERSION_HASH}`, {
  input: { prompt: "a cat" },
});

Why good: Pinned version guarantees identical behavior, hash is immutable

See: examples/core.md for listing model versions, getting version details

---

Pattern 5: File Handling

Models that output files return FileOutput objects implementing ReadableStream.

import { writeFile } from "node:fs/promises";

const [output] = await replicate.run("black-forest-labs/flux-schnell", {
  input: { prompt: "a sunset over mountains" },
});

// FileOutput has .url() and .blob() methods
console.log(output.url()); // Underlying URL

// Save to disk
const blob = await output.blob();
const buffer = Buffer.from(await blob.arrayBuffer());
await writeFile("./output.png", buffer);

// File inputs: pass URLs, Buffers, or ReadStreams
import { readFile } from "node:fs/promises";

const imageBuffer = await readFile("./input.png");

const output = await replicate.run("some-user/image-model", {
  input: {
    image: imageBuffer, // Auto-uploaded (max 100 MiB)
  },
});

Why good: FileOutput is a ReadableStream, works with Node.js stream APIs, .url() for the underlying URL

// BAD: Treating file output as a plain URL string
const [output] = await replicate.run("black-forest-labs/flux-schnell", {
  input: { prompt: "hello" },
});
const url = output; // WRONG: output is a FileOutput object, not a string

Why bad: FileOutput is an object, not a string -- use .url() to get the URL

See: examples/core.md for file uploads, large file handling, encoding strategies

---

Pattern 6: Async Predictions with Webhooks

Use replicate.predictions.create() for background jobs with webhook notifications.

const prediction = await replicate.predictions.create({
  model: "owner/model", // OR version: "sha256hash" for pinned version
  input: { prompt: "a painting of a cat" },
  webhook: "https://my.app/webhooks/replicate",
  webhook_events_filter: ["completed"],
});

console.log(prediction.id); // Use to track status
console.log(prediction.status); // "starting"

// Webhook signature validation (CRITICAL for security)
import { validateWebhook } from "replicate";

async function handleWebhook(request: Request): Promise<Response> {
  const secret = process.env.REPLICATE_WEBHOOK_SIGNING_SECRET;
  const isValid = await validateWebhook(request, secret);

  if (!isValid) {
    return new Response("Invalid signature", { status: 401 });
  }

  const prediction = await request.json();
  // Process prediction.output safely
  return new Response("OK", { status: 200 });
}

Why good: Decoupled processing, secure signature validation, filtered events reduce noise

See: examples/streaming-webhooks.md for webhook event types, polling alternative

---

Pattern 7: Deployments

Deployments give you a private, fixed endpoint with custom hardware and scaling.

// Create a prediction on a deployment (no cold start if min_instances > 0)
const prediction = await replicate.deployments.predictions.create(
  "my-org/my-deployment",
  {
    input: { prompt: "hello world" },
  },
);

const result = await replicate.wait(prediction);
console.log(result.output);

Why good: Predictable latency with min_instances, private endpoint, custom hardware selection

See: examples/deployments-training.md for creating/managing deployments, training API

---

Pattern 8: Error Handling

Catch API errors with status codes. The SDK auto-retries on 429 and 5xx errors (5 retries by default with exponential backoff).

try {
  const output = await replicate.run("owner/model", {
    input: { prompt: "hello" },
  });
} catch (error) {
  if (error instanceof Error) {
    console.error(`Replicate error: ${error.message}`);

    // Check for specific HTTP status codes in the error
    if ("status" in error) {
      const status = (error as { status: number }).status;
      if (status === 401) {
        throw new Error("Invalid API token. Check REPLICATE_API_TOKEN.");
      }
      if (status === 422) {
        console.error("Invalid input parameters");
      }
      if (status === 429) {
        console.error(
          "Rate limited -- SDK auto-retries (5 attempts) exhausted",
        );
      }
    }
  }
  throw error;
}

Why good: Checks error type, handles specific status codes, re-throws unexpected errors

See: examples/core.md for full error handling example with status code handling

</patterns>

---

<performance>

Performance Optimization

Cold Start Mitigation

Frequent model with varying load   -> Use deployments with min_instances >= 1
One-off batch jobs                  -> Use predictions.create() with webhooks (no waiting)
Popular public models               -> Usually warm, replicate.run() is fine
Custom/niche models                 -> Expect 30s-5min cold start on first run

Key Optimization Patterns

• Use deployments for latency-sensitive applications -- set min_instances: 1 to eliminate cold starts
• Use webhooks instead of polling for async jobs -- reduces API calls and latency
• Batch file inputs as URLs instead of uploading buffers -- avoids 100 MiB upload limit and is faster
• Pin model versions in production -- avoids unexpected behavior changes and enables caching
• Use `replicate.stream()` for LLMs -- progressive output feels faster than waiting for full completion
• Cancel unneeded predictions with replicate.predictions.cancel() -- stops billing immediately

</performance>

---

<decision_framework>

Decision Framework

Which Execution Method to Use

Is this a user-facing LLM response?
+-- YES -> Use replicate.stream() for real-time SSE output
+-- NO -> Do you need the result immediately?
    +-- YES -> Use replicate.run() (blocks until complete)
    +-- NO -> Use replicate.predictions.create() + webhook
        +-- Need to poll instead? -> Use replicate.wait(prediction)

Model Reference Format

Are you in development/prototyping?
+-- YES -> Use owner/model (latest version, convenient)
+-- NO -> Are you in production?
    +-- YES -> Use owner/model:version_hash (pinned, reproducible)
    +-- Does the model change frequently?
        +-- YES -> Pin version, test updates explicitly
        +-- NO -> Either format works, prefer pinned

Deployments vs Direct API

Do you need consistent low latency?
+-- YES -> Create a deployment with min_instances >= 1
+-- NO -> Do you need custom hardware (A100, H100)?
    +-- YES -> Create a deployment with specific hardware
    +-- NO -> Use replicate.run() / replicate.stream() directly
        (Replicate auto-allocates hardware)

When to Use This SDK vs Other AI SDKs

Are you running open-source models on serverless GPUs?
+-- YES -> Use Replicate SDK
+-- NO -> Are you calling proprietary APIs (OpenAI, Anthropic)?
    +-- YES -> Not this skill's scope -- use provider-specific SDKs
    +-- NO -> Do you need to switch between multiple providers?
        +-- YES -> Not this skill's scope -- use a unified provider SDK
        +-- NO -> Do you want to self-host models?
            +-- YES -> Not this skill's scope -- consider Cog or vLLM
            +-- NO -> Replicate SDK is appropriate

</decision_framework>

---

<red_flags>

RED FLAGS

High Priority Issues:

• Hardcoding REPLICATE_API_TOKEN in source code (security breach risk)
• Treating FileOutput as a string (it is a ReadableStream object -- use .url() or .blob())
• Not validating webhook signatures with validateWebhook() (allows forged webhook payloads)
• Using replicate.run() for long-running models in request handlers (blocks the response, can timeout)

Medium Priority Issues:

• Not pinning model versions in production (owner/model uses latest, which can change without notice)
• Relying solely on default retry behavior for production (5 retries with exponential backoff may be too aggressive for some use cases)
• Uploading large files as Buffer instead of hosting them at a URL (100 MiB limit on uploads)
• Ignoring cold start latency for infrequently-used models (first request can take minutes)

Common Mistakes:

• Confusing replicate.run() (returns output directly) with replicate.predictions.create() (returns a prediction object with status/id)
• Destructuring image output incorrectly: const output = await replicate.run(...) instead of const [output] = await replicate.run(...) (image models return arrays)
• Using replicate.stream() with models that do not support streaming (only language models with SSE support)
• Forgetting that replicate.predictions.create() accepts either a version hash or a model string (owner/model) -- use version for pinned reproducibility, model for latest-version convenience
• Not consuming the async iterator from replicate.stream() (events are lost)

Gotchas & Edge Cases:

• Prediction inputs and outputs are automatically deleted after one hour -- persist outputs via webhooks or download immediately
• The SDK auto-retries on 429 (rate limit) and 5xx errors -- 5 retries by default with exponential backoff. GET requests retry on 429 and 5xx; non-GET requests retry only on 429
• replicate.stream() returns ServerSentEvent objects with .event ("output", "error", "done") and .data (string) properties
• File uploads are limited to 100 MiB -- for larger files, host them at a URL and pass the URL as input
• Browser usage is not supported -- the SDK requires a server-side environment (Node.js 18+, Bun, Deno, Cloudflare Workers)
• webhook_events_filter accepts ["start", "output", "logs", "completed"] -- use ["completed"] unless you need intermediate status updates
• The Prefer: wait header enables sync mode on the HTTP API (up to 60s), but replicate.run() already handles this automatically
• Community models may disappear or change without warning -- pin versions and maintain fallbacks for critical workflows
• replicate.wait() polls the API until the prediction completes -- use webhooks for production to avoid polling overhead
• FileOutput.url() returns the underlying URL, but these URLs are temporary -- download or persist the file before it expires

</red_flags>

---

<critical_reminders>

CRITICAL REMINDERS

All code must follow project conventions in CLAUDE.md (kebab-case, named exports, import ordering, import type, named constants)

(You MUST never hardcode API tokens -- always use environment variables via `process.env.REPLICATE_API_TOKEN`)

(You MUST handle `FileOutput` objects for models that return files -- do not assume outputs are plain strings or URLs)

(You MUST validate webhooks using `validateWebhook()` from the `replicate` package -- never trust unverified webhook payloads)

(You MUST account for cold starts when running infrequently-used models -- use deployments for latency-sensitive applications)

(You MUST specify model versions (`owner/model:version`) in production to ensure reproducible results -- unversioned references use the latest, which can change)

Failure to follow these rules will produce insecure, unreliable, or unpredictable AI integrations.

</critical_reminders>

Replicate SDK -- Setup, Predictions & File Handling Examples

Client initialization, production config, running predictions, file I/O, model versioning, and error handling. See SKILL.md for core patterns.

Related examples:

• streaming-webhooks.md -- Streaming output, SSE events, webhooks
• deployments-training.md -- Deployments, training, model management

---

Basic Client Setup

// lib/replicate.ts
import Replicate from "replicate";

// Reads REPLICATE_API_TOKEN from env automatically
const replicate = new Replicate();

export { replicate };

---

Production Configuration

// lib/replicate.ts
import Replicate from "replicate";

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN,
  userAgent: "my-app/1.0.0",
});

export { replicate };

Constructor Options

new Replicate({
  auth: process.env.REPLICATE_API_TOKEN, // Auto-reads from env if omitted
  baseUrl: "https://api.replicate.com/v1", // Override for proxies
  userAgent: "my-app/1.0.0", // Custom user agent string
  fetch: globalThis.fetch, // Custom fetch implementation
  fileEncodingStrategy: "default", // "default" | "upload" | "data-uri"
  useFileOutput: true, // Return FileOutput objects (default: true)
});

---

Running Predictions with replicate.run()

Image Generation

import Replicate from "replicate";

const replicate = new Replicate();

// Image models return arrays of FileOutput objects
const [output] = await replicate.run("black-forest-labs/flux-schnell", {
  input: {
    prompt: "a serene mountain landscape at sunset, photorealistic",
    aspect_ratio: "16:9",
    num_outputs: 1,
  },
});

console.log(output.url()); // Temporary URL to generated image

Text Generation (LLM)

const output = await replicate.run("meta/meta-llama-3-70b-instruct", {
  input: {
    prompt: "Explain the difference between TCP and UDP in 3 sentences.",
    max_tokens: 256,
    temperature: 0.7,
  },
});

// LLM output is a string (or array of strings for some models)
console.log(output);

Audio Transcription

const output = await replicate.run("openai/whisper", {
  input: {
    audio: "https://example.com/audio-file.mp3",
    model: "large-v3",
    language: "en",
  },
});

// Output contains transcription text and segments
console.log(output.text);

---

Version Pinning

// Development: latest version (convenient but unpredictable)
const output = await replicate.run("stability-ai/sdxl", {
  input: { prompt: "a cat" },
});

// Production: pin to a specific version hash
const SDXL_VERSION =
  "39ed52f2a78e934b3ba6e2a89f5b1c712de7dfea535525255b1aa35c5565e08b";
const output = await replicate.run(`stability-ai/sdxl:${SDXL_VERSION}`, {
  input: { prompt: "a cat" },
});

Listing Model Versions

// Get all versions of a model
const versions = await replicate.models.versions.list("stability-ai", "sdxl");
for (const version of versions.results) {
  console.log(version.id, version.created_at);
}

// Get a specific version
const version = await replicate.models.versions.get(
  "stability-ai",
  "sdxl",
  "39ed52f2a78e934b3ba6e2a89f5b1c712de7dfea535525255b1aa35c5565e08b",
);
console.log(version.openapi_schema); // Model's input/output schema

---

Async Predictions with predictions.create()

// Create a prediction (returns immediately, does not wait for completion)
// Use `version` for pinned reproducibility, or `model` for latest-version convenience
const prediction = await replicate.predictions.create({
  model: "owner/model", // OR version: "sha256hash" for pinned version
  input: {
    prompt: "a painting of a cat in the style of Van Gogh",
  },
  webhook: "https://my.app/webhooks/replicate",
  webhook_events_filter: ["completed"],
});

console.log(prediction.id); // "abc123xyz"
console.log(prediction.status); // "starting"

// Option 1: Poll until complete
const result = await replicate.wait(prediction);
console.log(result.output);
console.log(result.status); // "succeeded" | "failed" | "canceled"

// Option 2: Check status manually
const updated = await replicate.predictions.get(prediction.id);
console.log(updated.status);

// Option 3: Cancel if no longer needed
await replicate.predictions.cancel(prediction.id);

---

Progress Callbacks

import type { Prediction } from "replicate";

function onProgress(prediction: Prediction): void {
  console.log(`Status: ${prediction.status}`);
  if (prediction.logs) {
    console.log(prediction.logs);
  }
}

const output = await replicate.run(
  "black-forest-labs/flux-schnell",
  { input: { prompt: "a sunset" } },
  onProgress,
);

---

File Input Handling

URL Input (Recommended for Large Files)

// Pass a URL -- no upload needed, no size limit
const output = await replicate.run("owner/image-upscaler", {
  input: {
    image: "https://example.com/photo.jpg",
    scale: 4,
  },
});

Buffer Input (Auto-Uploaded, Max 100 MiB)

import { readFile } from "node:fs/promises";

const imageBuffer = await readFile("./input.png");

const output = await replicate.run("owner/image-upscaler", {
  input: {
    image: imageBuffer, // Automatically uploaded to Replicate
    scale: 4,
  },
});

Explicit File Upload via Files API

// Upload a file and get a persistent reference
const file = await replicate.files.create(
  await readFile("./training-data.zip"),
  { filename: "training-data.zip" },
);

console.log(file.id); // File ID for later reference
console.log(file.urls); // Temporary download URLs

---

File Output Handling

import { writeFile } from "node:fs/promises";

const [output] = await replicate.run("black-forest-labs/flux-schnell", {
  input: { prompt: "a beautiful garden" },
});

// FileOutput implements ReadableStream
// Method 1: Get the URL
const url = output.url();
console.log(url); // Temporary URL (expires -- download promptly)

// Method 2: Get as Blob, then save
const blob = await output.blob();
const buffer = Buffer.from(await blob.arrayBuffer());
await writeFile("./output.png", buffer);

// Method 3: Write directly using Node.js stream utilities
await writeFile("./output.png", output);

---

Error Handling

import Replicate from "replicate";

const replicate = new Replicate();

async function safePrediction(prompt: string): Promise<string | null> {
  try {
    const output = await replicate.run("meta/meta-llama-3-70b-instruct", {
      input: { prompt, max_tokens: 512 },
    });

    return String(output);
  } catch (error) {
    if (!(error instanceof Error)) {
      throw error;
    }

    console.error(`Replicate error: ${error.message}`);

    // Check for HTTP status codes
    if ("status" in error) {
      const status = (error as { status: number }).status;

      switch (status) {
        case 401:
          throw new Error("Invalid API token. Check REPLICATE_API_TOKEN.");
        case 404:
          console.error("Model not found. Check the model identifier.");
          return null;
        case 422:
          console.error("Invalid input parameters for this model.");
          return null;
        case 429:
          console.error("Rate limited. All retries exhausted.");
          return null;
      }

      // 5xx errors -- SDK auto-retried, all attempts failed
      if (status >= 500) {
        console.error("Server error after all retries.");
        return null;
      }
    }

    throw error; // Re-throw unexpected errors
  }
}

const result = await safePrediction("Hello!");
if (result) {
  console.log(result);
}

---

Listing Predictions

// List recent predictions (paginated)
const page = await replicate.predictions.list();

for (const prediction of page.results) {
  console.log(prediction.id, prediction.status, prediction.model);
}

// Get next page
if (page.next) {
  const nextPage = await replicate.predictions.list({ cursor: page.next });
}

---

_For streaming and webhooks, see streaming-webhooks.md. For deployments and training, see deployments-training.md. For API reference tables, see reference.md._

Replicate SDK -- Deployments & Training Examples

Deployment management, custom hardware, scaling configuration, fine-tuning models, and model CRUD operations. See SKILL.md for core patterns.

Related examples:

• core.md -- Client setup, predictions, file handling
• streaming-webhooks.md -- Streaming output, webhooks

---

Deployments

Why Use Deployments

Deployments give you a private, fixed API endpoint with control over:

• Hardware -- Choose specific GPU types (e.g., A40, A100)
• Scaling -- Set min_instances to keep models warm (eliminates cold starts)
• Versioning -- Pin to a model version independently of the public model page

Creating a Deployment

const deployment = await replicate.deployments.create({
  name: "my-llama-deployment",
  model: "meta/meta-llama-3-70b-instruct",
  version: "abc123...",
  hardware: "gpu-a100-large",
  min_instances: 1, // Always warm -- no cold starts
  max_instances: 5, // Auto-scale up to 5 instances
});

console.log(deployment.name);
console.log(deployment.current_release);

Running Predictions on a Deployment

// Use the deployment name instead of model identifier
const prediction = await replicate.deployments.predictions.create(
  "my-org/my-llama-deployment",
  {
    input: {
      prompt: "Summarize this article...",
      max_tokens: 1024,
    },
  },
);

// Wait for result
const result = await replicate.wait(prediction);
console.log(result.output);

Streaming from a Deployment

const stream = replicate.stream("my-org/my-llama-deployment", {
  input: { prompt: "Hello world" },
});

for await (const event of stream) {
  if (event.event === "output") {
    process.stdout.write(event.data);
  }
}

Managing Deployments

// List all deployments
const deployments = await replicate.deployments.list();
for (const d of deployments.results) {
  console.log(d.name, d.current_release?.model);
}

// Get a specific deployment
const deployment = await replicate.deployments.get(
  "my-org/my-llama-deployment",
);

// Update deployment (change hardware, version, or scaling)
await replicate.deployments.update("my-org/my-llama-deployment", {
  version: "new-version-hash",
  hardware: "gpu-a100-large",
  min_instances: 2,
  max_instances: 10,
});

// Delete a deployment
await replicate.deployments.delete("my-org/my-llama-deployment");

Available Hardware

const hardware = await replicate.hardware.list();
for (const hw of hardware) {
  console.log(hw.name, hw.sku);
}

See reference.md for the full hardware SKU table.

---

Training (Fine-Tuning)

Starting a Training Job

const training = await replicate.trainings.create(
  "owner",
  "model-name",
  "version-hash",
  {
    input: {
      train_data: "https://example.com/my-training-data.zip",
      num_train_epochs: 4,
      learning_rate: 0.0001,
    },
    destination: "my-org/my-fine-tuned-model",
    webhook: "https://my.app/webhooks/training",
    webhook_events_filter: ["completed"],
  },
);

console.log(training.id);
console.log(training.status); // "starting"

Monitoring Training Progress

// Poll for training status
const training = await replicate.trainings.get(training.id);
console.log(training.status); // "starting" | "processing" | "succeeded" | "failed"
console.log(training.logs); // Training logs

// Wait for completion
const result = await replicate.wait(training);

if (result.status === "succeeded") {
  console.log("Training complete!");
  console.log("New model version:", result.output?.version);
} else {
  console.error("Training failed:", result.error);
}

Listing and Canceling Trainings

// List all trainings
const trainings = await replicate.trainings.list();
for (const t of trainings.results) {
  console.log(t.id, t.status, t.model);
}

// Cancel a running training
await replicate.trainings.cancel(training.id);

---

Model Management

Getting Model Info

const model = await replicate.models.get("stability-ai", "sdxl");

console.log(model.owner);
console.log(model.name);
console.log(model.description);
console.log(model.visibility); // "public" | "private"
console.log(model.latest_version?.id);

Creating a Model

// Create a new model (for training destinations or custom models)
const model = await replicate.models.create("my-org", "my-custom-model", {
  description: "A fine-tuned image generation model",
  visibility: "private",
  hardware: "gpu-a40-large",
});

Listing Models

// List your models
const models = await replicate.models.list();
for (const m of models.results) {
  console.log(m.owner, m.name, m.run_count);
}

// Search public models
const results = await replicate.models.search("text to image");
for (const m of results.results) {
  console.log(m.owner, m.name, m.description);
}

Model Versions

// List all versions
const versions = await replicate.models.versions.list("stability-ai", "sdxl");
for (const v of versions.results) {
  console.log(v.id, v.created_at);
}

// Get a specific version (includes input/output schema)
const version = await replicate.models.versions.get(
  "stability-ai",
  "sdxl",
  "39ed52f2a78e934b3ba6e2a89f5b1c712de7dfea535525255b1aa35c5565e08b",
);

// Access input/output schema from OpenAPI spec
console.log(version.openapi_schema);

---

Collections

// List model collections
const collections = await replicate.collections.list();
for (const c of collections.results) {
  console.log(c.slug, c.name, c.description);
}

// Get a specific collection
const collection = await replicate.collections.get("text-to-image");
for (const model of collection.models) {
  console.log(model.owner, model.name);
}

---

Files API

import { readFile } from "node:fs/promises";

// Upload a file
const file = await replicate.files.create(await readFile("./data.zip"), {
  filename: "data.zip",
});
console.log(file.id);

// List files
const files = await replicate.files.list();
for (const f of files.results) {
  console.log(f.id, f.name, f.size);
}

// Get file info
const fileInfo = await replicate.files.get(file.id);

// Delete a file
await replicate.files.delete(file.id);

---

_For client setup and predictions, see core.md. For streaming and webhooks, see streaming-webhooks.md. For API reference tables, see reference.md._

Replicate SDK -- Streaming & Webhooks Examples

Streaming output with SSE events, webhook setup, event filtering, and signature validation. See SKILL.md for core patterns.

Related examples:

• core.md -- Client setup, predictions, file handling
• deployments-training.md -- Deployments, training, model management

---

Basic Streaming with replicate.stream()

import Replicate from "replicate";

const replicate = new Replicate();

const stream = replicate.stream("meta/meta-llama-3-70b-instruct", {
  input: {
    prompt: "Explain async/await in TypeScript.",
    max_tokens: 512,
  },
});

for await (const event of stream) {
  if (event.event === "output") {
    process.stdout.write(event.data);
  }
}
console.log(); // newline

---

Handling All SSE Event Types

const stream = replicate.stream("meta/meta-llama-3-70b-instruct", {
  input: { prompt: "Tell me a story." },
});

const chunks: string[] = [];

for await (const event of stream) {
  switch (event.event) {
    case "output":
      // Progressive text output
      chunks.push(event.data);
      process.stdout.write(event.data);
      break;

    case "error":
      // Prediction error (event.data is JSON with details)
      console.error("Stream error:", event.data);
      break;

    case "done":
      // Prediction complete (event.data may contain reason)
      console.log("\nStream complete");
      break;
  }
}

const fullOutput = chunks.join("");
console.log("Total length:", fullOutput.length);

SSE Event Types Reference

---

Streaming with Error Handling

try {
  const stream = replicate.stream("meta/meta-llama-3-70b-instruct", {
    input: { prompt: "Hello" },
  });

  for await (const event of stream) {
    if (event.event === "output") {
      process.stdout.write(event.data);
    }
    if (event.event === "error") {
      throw new Error(`Prediction error: ${event.data}`);
    }
  }
} catch (error) {
  if (error instanceof Error) {
    console.error("Stream failed:", error.message);
  }
  throw error;
}

---

Streaming on Deployments

// Stream from a deployment (consistent latency if min_instances > 0)
const stream = replicate.stream("my-org/my-deployment", {
  input: { prompt: "Summarize this document..." },
});

for await (const event of stream) {
  if (event.event === "output") {
    process.stdout.write(event.data);
  }
}

---

Webhook Setup

Creating a Prediction with Webhooks

const prediction = await replicate.predictions.create({
  version: "27b93a2413e7f36cd83da926f3656280b2931564ff050bf9575f1fdf9bcd7478",
  input: { prompt: "a painting of a sunset" },
  webhook: "https://my.app/webhooks/replicate",
  webhook_events_filter: ["completed"],
});

console.log(prediction.id); // Track via ID

Webhook Events Filter Options

// Receive all events (verbose, useful for debugging)
const prediction = await replicate.predictions.create({
  version: "abc123...",
  input: { prompt: "hello" },
  webhook: "https://my.app/webhooks/replicate",
  webhook_events_filter: ["start", "output", "logs", "completed"],
});

// Receive only the final result (recommended for production)
const prediction = await replicate.predictions.create({
  version: "abc123...",
  input: { prompt: "hello" },
  webhook: "https://my.app/webhooks/replicate",
  webhook_events_filter: ["completed"],
});

---

Webhook Signature Validation

Using the Built-in Validator

import { validateWebhook } from "replicate";

async function handleWebhook(request: Request): Promise<Response> {
  const secret = process.env.REPLICATE_WEBHOOK_SIGNING_SECRET;

  if (!secret) {
    throw new Error("REPLICATE_WEBHOOK_SIGNING_SECRET not set");
  }

  const isValid = await validateWebhook(request, secret);

  if (!isValid) {
    return new Response("Invalid webhook signature", { status: 401 });
  }

  const prediction = await request.json();

  // Safe to process -- signature verified
  console.log("Prediction status:", prediction.status);
  console.log("Prediction output:", prediction.output);

  if (prediction.status === "succeeded") {
    // Handle successful prediction
    await processOutput(prediction.output);
  } else if (prediction.status === "failed") {
    console.error("Prediction failed:", prediction.error);
  }

  return new Response("OK", { status: 200 });
}

Manual Signature Validation

import { validateWebhook } from "replicate";

// When you have raw request data instead of a Request object
const isValid = await validateWebhook({
  id: request.headers["webhook-id"],
  timestamp: request.headers["webhook-timestamp"],
  signature: request.headers["webhook-signature"],
  body: rawBody, // string | ArrayBuffer | ReadableStream
  secret: process.env.REPLICATE_WEBHOOK_SIGNING_SECRET,
});

---

Webhook Payload Structure

// The webhook POST body is the same as the prediction object
interface WebhookPayload {
  id: string;
  model: string;
  version: string;
  status: "starting" | "processing" | "succeeded" | "failed" | "canceled";
  input: Record<string, unknown>;
  output: unknown; // Model-specific output
  error: string | null;
  logs: string;
  created_at: string;
  started_at: string | null;
  completed_at: string | null;
  urls: {
    get: string;
    cancel: string;
    stream?: string;
  };
}

---

Polling as Alternative to Webhooks

// For environments where webhooks are not feasible
const prediction = await replicate.predictions.create({
  version: "abc123...",
  input: { prompt: "hello" },
});

// replicate.wait() polls until the prediction completes
const result = await replicate.wait(prediction);

if (result.status === "succeeded") {
  console.log(result.output);
} else if (result.status === "failed") {
  console.error("Failed:", result.error);
} else if (result.status === "canceled") {
  console.log("Prediction was canceled");
}

---

_For client setup and file handling, see core.md. For deployments and training, see deployments-training.md. For API reference tables, see reference.md._

Replicate SDK Quick Reference

Constructor options, API methods, error types, model reference format, and webhook events. See SKILL.md for core concepts and examples/ for code examples.

---

Package Installation

npm install replicate

---

Client Configuration

import Replicate from "replicate";

const replicate = new Replicate({
  auth: process.env.REPLICATE_API_TOKEN, // Auto-reads from env if omitted
  baseUrl: "https://api.replicate.com/v1", // Override for proxies
  userAgent: "my-app/1.0.0", // Custom user agent string
  fetch: globalThis.fetch, // Custom fetch implementation (Node.js 18+)
  fileEncodingStrategy: "default", // "default" | "upload" | "data-uri"
  useFileOutput: true, // Return FileOutput objects for file outputs (default: true)
});

Environment Variables

---

Model Reference Format

owner/model                          -- Uses latest version (convenient, not reproducible)
owner/model:sha256hash               -- Pinned to a specific version (production use)

Examples

---

API Methods Reference

Core Methods

// Run a model synchronously (waits for completion)
const output = await replicate.run(
  "owner/model",               // or "owner/model:version"
  {
    input: { prompt: "..." },  // Model-specific input parameters
  },
  onProgress?,                 // Optional: (prediction: Prediction) => void
);

// Stream model output as SSE events
const stream = replicate.stream(
  "owner/model",               // or deployment name
  {
    input: { prompt: "..." },  // Model-specific input parameters
  },
);
// Returns AsyncGenerator<ServerSentEvent>

// Wait for a prediction to complete (polling)
const result = await replicate.wait(prediction);

Predictions

// Create (async, returns immediately)
// Specify EITHER model OR version (not both)
const prediction = await replicate.predictions.create({
  model: "owner/name",                      // Use for latest version
  // OR: version: "sha256hash",             // Use for pinned reproducibility
  input: { prompt: "..." },                 // Required: model inputs
  webhook?: "https://...",                   // Optional: webhook URL
  webhook_events_filter?: ["completed"],     // Optional: event filter
});

// Get prediction status
const prediction = await replicate.predictions.get(prediction.id);

// List predictions (paginated)
const page = await replicate.predictions.list();

// Cancel a running prediction
await replicate.predictions.cancel(prediction.id);

Models

// Get model info
const model = await replicate.models.get("owner", "name");

// List models (paginated)
const models = await replicate.models.list();

// Search models
const results = await replicate.models.search("query");

// Create a model
await replicate.models.create("owner", "name", {
  description?: string,
  visibility: "public" | "private",
  hardware?: string,
});

// List model versions
const versions = await replicate.models.versions.list("owner", "name");

// Get a specific version
const version = await replicate.models.versions.get("owner", "name", "version_id");

Deployments

// Create a deployment
await replicate.deployments.create({
  name: string,
  model: string,
  version: string,
  hardware: string,
  min_instances?: number,
  max_instances?: number,
});

// Run prediction on deployment
await replicate.deployments.predictions.create("owner/deployment", {
  input: { ... },
  webhook?: string,
});

// Get deployment info
await replicate.deployments.get("owner/deployment");

// List deployments
await replicate.deployments.list();

// Update deployment
await replicate.deployments.update("owner/deployment", { ... });

// Delete deployment
await replicate.deployments.delete("owner/deployment");

Trainings

// Start training
await replicate.trainings.create("owner", "model", "version", {
  input: { train_data: "https://...", ... },
  destination: "owner/new-model",
  webhook?: string,
  webhook_events_filter?: string[],
});

// Get training status
await replicate.trainings.get(training.id);

// List trainings
await replicate.trainings.list();

// Cancel training
await replicate.trainings.cancel(training.id);

Files

// Upload file
await replicate.files.create(buffer, { filename: "data.zip" });

// Get file info
await replicate.files.get(file.id);

// List files
await replicate.files.list();

// Delete file
await replicate.files.delete(file.id);

Hardware

const hardware = await replicate.hardware.list();
// Returns: Array<{ name: string, sku: string }>

Collections

await replicate.collections.get("collection-slug");
await replicate.collections.list();

---

Prediction Status Values

---

SSE Event Types (Streaming)

---

Webhook Events

Webhook payload is the same as the prediction object (JSON POST to your URL).

---

Webhook Signature Validation

import { validateWebhook } from "replicate";

// With a Request object
const isValid = await validateWebhook(request, secret);

// With raw data
const isValid = await validateWebhook({
  id: headers["webhook-id"],
  timestamp: headers["webhook-timestamp"],
  signature: headers["webhook-signature"],
  body: rawBody,
  secret: process.env.REPLICATE_WEBHOOK_SIGNING_SECRET,
});

---

FileOutput Object

// Returned by replicate.run() for models that output files
interface FileOutput extends ReadableStream {
  url(): URL; // Get the underlying URL object (temporary -- download promptly)
  blob(): Promise<Blob>; // Get as Blob
  toString(): string; // String representation
}

---

Error Types

The SDK throws errors with HTTP status codes for API failures:

The SDK automatically retries on 429 and 5xx errors (5 retries by default with exponential backoff). GET requests retry on 429 and 5xx; non-GET requests retry only on 429.

---

Hardware SKUs

---

Platform Support