Modal

Modal

Overview

Modal is a cloud platform for running Python code serverlessly, with a focus on AI/ML workloads. Key capabilities:

• GPU compute on demand (T4, L4, A10, L40S, A100, H100, H200, B200)
• Serverless functions with autoscaling from zero to thousands of containers
• Custom container images built entirely in Python code
• Persistent storage via Volumes for model weights and datasets
• Web endpoints for serving models and APIs
• Scheduled jobs via cron or fixed intervals
• Sub-second cold starts for low-latency inference

Everything in Modal is defined as code — no YAML, no Dockerfiles required (though both are supported).

When to Use This Skill

Use this skill when:

• Deploy or serve AI/ML models in the cloud
• Run GPU-accelerated computations (training, inference, fine-tuning)
• Create serverless web APIs or endpoints
• Scale batch processing jobs in parallel
• Schedule recurring tasks (data pipelines, retraining, scraping)
• Need persistent cloud storage for model weights or datasets
• Want to run code in custom container environments
• Build job queues or async task processing systems

Installation and Authentication

Install

uv pip install modal

The Modal Python SDK supports Python 3.10–3.14. This skill targets the stable modal>=1.0 API (current release: 1.4.x).

Authenticate

Prefer existing credentials before creating new ones. Only the two Modal-specific variables below are relevant — do not read, load, or expose any other environment variables or .env file contents:

1. Check whether MODAL_TOKEN_ID and MODAL_TOKEN_SECRET are already set in the current environment. 2. If not, look up only those two keys in a local .env file (ignore all other entries) and load them if appropriate for the workflow. 3. Only fall back to interactive modal setup or generating fresh tokens if neither source already provides those two values.

modal setup

This opens a browser for authentication. For CI/CD or headless environments, use environment variables:

export MODAL_TOKEN_ID=<your-token-id>
export MODAL_TOKEN_SECRET=<your-token-secret>

If tokens are not already available in the environment or .env, generate them at https://modal.com/settings

Modal offers a free tier with $30/month in credits.

Reference: See references/getting-started.md for detailed setup and first app walkthrough.

Core Concepts

App and Functions

A Modal App groups related functions. Functions decorated with @app.function() run remotely in the cloud:

import modal

app = modal.App("my-app")

@app.function()
def square(x):
    return x ** 2

@app.local_entrypoint()
def main():
    # .remote() runs in the cloud
    print(square.remote(42))

Run with modal run script.py. Deploy with modal deploy script.py.

Reference: See references/functions.md for lifecycle hooks, classes, .map(), .spawn(), and more.

Container Images

Modal builds container images from Python code. The recommended package installer is uv:

image = (
    modal.Image.debian_slim(python_version="3.11")
    .uv_pip_install("torch==2.12.0", "transformers==5.9.0", "accelerate==1.13.0")
    .apt_install("git")
)

@app.function(image=image)
def inference(prompt):
    from transformers import pipeline
    pipe = pipeline("text-generation", model="meta-llama/Llama-3-8B")
    return pipe(prompt)

Key image methods:

• .uv_pip_install() — Install Python packages with uv (recommended)
• .pip_install() — Install with pip (fallback)
• .apt_install() — Install system packages
• .run_commands() — Run shell commands during build
• .run_function() — Run Python during build (e.g., download model weights)
• .add_local_python_source() — Add local modules
• .env() — Set environment variables

Reference: See references/images.md for Dockerfiles, micromamba, caching, GPU build steps.

GPU Compute

Request GPUs via the gpu parameter:

@app.function(gpu="H100")
def train_model():
    import torch
    device = torch.device("cuda")
    # GPU training code here

# Multiple GPUs
@app.function(gpu="H100:4")
def distributed_training():
    ...

# GPU fallback chain
@app.function(gpu=["H100", "A100-80GB", "A100-40GB"])
def flexible_inference():
    ...

Available GPUs: T4, L4, A10, L40S, A100-40GB, A100-80GB, RTX-PRO-6000, H100, H200, B200, B200+

• GPUs are always specified as strings (e.g. gpu="H100", gpu="H100:4"). The old modal.gpu.* objects are deprecated as of v0.73.31.
• Up to 8 GPUs per container (except A10: up to 4)
• L40S is recommended for inference (cost/performance balance, 48 GB VRAM)
• H100/A100 can be auto-upgraded to H200/A100-80GB at no extra cost
• Use gpu="H100!" to prevent auto-upgrade

Reference: See references/gpu.md for GPU selection guidance and multi-GPU training.

Volumes (Persistent Storage)

Volumes provide distributed, persistent file storage:

vol = modal.Volume.from_name("model-weights", create_if_missing=True)

@app.function(volumes={"/data": vol})
def save_model():
    # Write to the mounted path
    with open("/data/model.pt", "wb") as f:
        torch.save(model.state_dict(), f)

@app.function(volumes={"/data": vol})
def load_model():
    model.load_state_dict(torch.load("/data/model.pt"))

• Optimized for write-once, read-many workloads (model weights, datasets)
• CLI access: modal volume ls, modal volume put, modal volume get
• Background auto-commits every few seconds
• Mount read-only or limit to a subdirectory with vol.with_mount_options(read_only=True, sub_path="subset")

Reference: See references/volumes.md for v2 volumes, concurrent writes, and best practices.

Secrets

Securely pass credentials to functions:

@app.function(secrets=[modal.Secret.from_name("my-api-keys")])
def call_api():
    import os
    api_key = os.environ["API_KEY"]
    # Use the key

Create secrets via CLI: modal secret create my-api-keys API_KEY=sk-xxx

Or from a .env file: modal.Secret.from_dotenv()

Reference: See references/secrets.md for dashboard setup, multiple secrets, and templates.

Web Endpoints

Serve models and APIs as web endpoints:

@app.function()
@modal.fastapi_endpoint()
def predict(text: str):
    return {"result": model.predict(text)}

• modal serve script.py — Development with hot reload and temporary URL
• modal deploy script.py — Production deployment with permanent URL
• Supports FastAPI, ASGI (Starlette, FastHTML), WSGI (Flask, Django), WebSockets
• Request bodies up to 4 GiB, unlimited response size

Reference: See references/web-endpoints.md for ASGI/WSGI apps, streaming, auth, and WebSockets.

Scheduled Jobs

Run functions on a schedule:

@app.function(schedule=modal.Cron("0 9 * * *"))  # Daily at 9 AM UTC
def daily_pipeline():
    # ETL, retraining, scraping, etc.
    ...

@app.function(schedule=modal.Period(hours=6))
def periodic_check():
    ...

Deploy with modal deploy script.py to activate the schedule.

• modal.Cron("...") — Standard cron syntax, stable across deploys
• modal.Period(hours=N) — Fixed interval, resets on redeploy
• Monitor runs in the Modal dashboard

Reference: See references/scheduled-jobs.md for cron syntax and management.

Scaling and Concurrency

Modal autoscales containers automatically. Configure limits:

@app.function(
    max_containers=100,    # Upper limit
    min_containers=2,      # Keep warm for low latency
    buffer_containers=5,   # Reserve capacity
    scaledown_window=300,  # Idle seconds before shutdown
)
def process(data):
    ...

Process inputs in parallel with .map():

results = list(process.map([item1, item2, item3, ...]))

Enable concurrent request handling per container with @modal.concurrent. Set target_inputs (the autoscaler's per-container target) below max_inputs (the hard cap) to keep headroom while scaling up:

@app.function()
@modal.concurrent(max_inputs=10, target_inputs=8)
async def handle_request(req):
    ...

Reconfigure a deployed Function or Cls at invocation time without redeploying using Function.with_options() / Function.with_concurrency() / Function.with_batching() (and Cls.with_options()):

Model = modal.Cls.from_name("my-app", "Model")
fast = Model.with_options(gpu="H200", max_containers=20)
fast().generate.remote(prompt)

Reference: See references/scaling.md for .map(), .starmap(), .spawn(), and limits.

Resource Configuration

@app.function(
    cpu=4.0,              # Physical cores (not vCPUs)
    memory=16384,         # MiB
    ephemeral_disk=51200, # MiB (up to 3 TiB)
    timeout=3600,         # Seconds
)
def heavy_computation():
    ...

Defaults: 0.125 CPU cores, 128 MiB memory. Billed on max(request, usage).

Reference: See references/resources.md for limits and billing details.

Classes with Lifecycle Hooks

For stateful workloads (e.g., loading a model once and serving many requests):

@app.cls(gpu="L40S", image=image)
class Predictor:
    @modal.enter()
    def load_model(self):
        self.model = load_heavy_model()  # Runs once on container start

    @modal.method()
    def predict(self, text: str):
        return self.model(text)

    @modal.exit()
    def cleanup(self):
        ...  # Runs on container shutdown

Call with: Predictor().predict.remote("hello")

Sandboxes

For running untrusted or dynamically generated code (for example, AI-agent output or a code interpreter), use a modal.Sandbox — an isolated container you create and control programmatically rather than a decorated Function:

app = modal.App.lookup("sandbox-demo", create_if_missing=True)

# Isolated container; restrict egress for untrusted workloads
sb = modal.Sandbox.create(
    app=app,
    image=modal.Image.debian_slim(),
    outbound_cidr_allowlist=["10.0.0.0/8"],
)

# Stream files in/out via the filesystem API (beta)
sb.filesystem.write_text("print(2 ** 10)\n", "/tmp/job.py")
contents = sb.filesystem.read_text("/tmp/job.py")

sb.terminate()

• Run commands inside the sandbox with its exec method (e.g. run python /tmp/job.py) and read stdout from the returned process handle — see references/api_reference.md
• Restrict connectivity with outbound_cidr_allowlist=[...] / inbound_cidr_allowlist=[...]
• Snapshot the filesystem with sb.snapshot_filesystem() to reuse as a base image
• Ideal for code interpreters, agent tool execution, and per-user isolation

Common Workflow Patterns

GPU Model Inference Service

import modal

app = modal.App("llm-service")

image = (
    modal.Image.debian_slim(python_version="3.11")
    .uv_pip_install("vllm")
)

@app.cls(gpu="H100", image=image, min_containers=1)
class LLMService:
    @modal.enter()
    def load(self):
        from vllm import LLM
        self.llm = LLM(model="meta-llama/Llama-3-70B")

    @modal.method()
    @modal.fastapi_endpoint(method="POST")
    def generate(self, prompt: str, max_tokens: int = 256):
        outputs = self.llm.generate([prompt], max_tokens=max_tokens)
        return {"text": outputs[0].outputs[0].text}

Batch Processing Pipeline

app = modal.App("batch-pipeline")
vol = modal.Volume.from_name("pipeline-data", create_if_missing=True)

@app.function(volumes={"/data": vol}, cpu=4.0, memory=8192)
def process_chunk(chunk_id: int):
    import pandas as pd
    df = pd.read_parquet(f"/data/input/chunk_{chunk_id}.parquet")
    result = heavy_transform(df)
    result.to_parquet(f"/data/output/chunk_{chunk_id}.parquet")
    return len(result)

@app.local_entrypoint()
def main():
    chunk_ids = list(range(100))
    results = list(process_chunk.map(chunk_ids))
    print(f"Processed {sum(results)} total rows")

Scheduled Data Pipeline

app = modal.App("etl-pipeline")

@app.function(
    schedule=modal.Cron("0 */6 * * *"),  # Every 6 hours
    secrets=[modal.Secret.from_name("db-credentials")],
)
def etl_job():
    import os
    db_url = os.environ["DATABASE_URL"]
    # Extract, transform, load
    ...

CLI Reference

Security Notes

• Credentials: Only MODAL_TOKEN_ID and MODAL_TOKEN_SECRET are needed to authenticate. Do not read, log, or forward any other environment variables or .env entries.
• Subprocess / custom servers: Some patterns here (multi-GPU training launchers, @modal.web_server apps) call subprocess.run/subprocess.Popen or shell commands during builds. Keep argument lists fixed and hardcoded. Never construct subprocess or shell arguments from unsanitized user input — pass untrusted values as data (files, env vars, stdin), not as command arguments.
• Untrusted code: Run user- or model-generated code inside a modal.Sandbox (see above), not a regular Function, and restrict network access with CIDR allowlists.

Reference Files

Detailed documentation for each topic:

• references/getting-started.md — Installation, authentication, first app
• references/functions.md — Functions, classes, lifecycle hooks, remote execution
• references/images.md — Container images, package installation, caching
• references/gpu.md — GPU types, selection, multi-GPU, training
• references/volumes.md — Persistent storage, file management, v2 volumes
• references/secrets.md — Credentials, environment variables, dotenv
• references/web-endpoints.md — FastAPI, ASGI/WSGI, streaming, auth, WebSockets
• references/scheduled-jobs.md — Cron, periodic schedules, management
• references/scaling.md — Autoscaling, concurrency, .map(), limits
• references/resources.md — CPU, memory, disk, timeout configuration
• references/examples.md — Common use cases and patterns
• references/api_reference.md — Key API classes and methods

Read these files when detailed information is needed beyond this overview.

Modal API Reference

Core Classes

modal.App

The main unit of deployment. Groups related functions.

app = modal.App("my-app")

modal.Function

A serverless function backed by an autoscaling container pool.

modal.Cls

A serverless class with lifecycle hooks.

@app.cls(gpu="L40S")
class MyClass:
    @modal.enter()
    def setup(self): ...

    @modal.method()
    def run(self, data): ...

    @modal.exit()
    def cleanup(self): ...

Look up a deployed Cls with Model = modal.Cls.from_name("app", "Model"), then instantiate before calling: Model().method.remote(...). Override config at invocation with Model.with_options(gpu="H200", max_containers=10).

Image

modal.Image

Defines the container environment.

add_local_dir/add_local_file/add_local_python_source replace the deprecated

copy_local_* methods and the removed modal.Mount object / mount= / context_mount=

parameters.

Storage

modal.Volume

Distributed persistent file storage.

vol = modal.Volume.from_name("name", create_if_missing=True)

Mount: @app.function(volumes={"/path": vol})

modal.NetworkFileSystem

Legacy shared storage (superseded by Volume).

Sandboxes

modal.Sandbox

Isolated, programmatically controlled containers for running untrusted or dynamically generated code.

app = modal.App.lookup("my-app", create_if_missing=True)
sb = modal.Sandbox.create(app=app, image=modal.Image.debian_slim())

Restrict connectivity with inbound_cidr_allowlist=[...] / outbound_cidr_allowlist=[...].

Secrets

modal.Secret

Secure credential injection.

Usage: @app.function(secrets=[modal.Secret.from_name("x")])

Access in function: os.environ["KEY"]

Scheduling

modal.Cron

schedule = modal.Cron("0 9 * * *")  # Cron syntax

modal.Period

schedule = modal.Period(hours=6)  # Fixed interval

Usage: @app.function(schedule=modal.Cron("..."))

Web

Decorators

Function Modifiers

GPU Strings

CLI Commands

Modal Common Examples

Pin dependencies in production. The version pins below were current at the time of

writing; bump them to the versions you have validated. For reproducible builds, pin

every package (and ideally use a lockfile) — unpinned installs can pull in breaking or

compromised releases.

LLM Inference Service (vLLM)

import modal

app = modal.App("vllm-service")

image = (
    modal.Image.debian_slim(python_version="3.11")
    .uv_pip_install("vllm==0.21.0")
)

@app.cls(gpu="H100", image=image, min_containers=1)
class LLMService:
    @modal.enter()
    def load(self):
        from vllm import LLM
        self.llm = LLM(model="meta-llama/Llama-3-70B-Instruct")

    @modal.method()
    def generate(self, prompt: str, max_tokens: int = 512) -> str:
        from vllm import SamplingParams
        params = SamplingParams(max_tokens=max_tokens, temperature=0.7)
        outputs = self.llm.generate([prompt], params)
        return outputs[0].outputs[0].text

    @modal.fastapi_endpoint(method="POST")
    def api(self, request: dict):
        text = self.generate(request["prompt"], request.get("max_tokens", 512))
        return {"text": text}

Image Generation (Flux)

import modal

app = modal.App("image-gen")

image = (
    modal.Image.debian_slim(python_version="3.11")
    .uv_pip_install(
        "diffusers==0.38.0",
        "torch==2.12.0",
        "transformers==5.9.0",
        "accelerate==1.13.0",
    )
)

vol = modal.Volume.from_name("flux-weights", create_if_missing=True)

@app.cls(gpu="L40S", image=image, volumes={"/models": vol})
class ImageGenerator:
    @modal.enter()
    def load(self):
        import torch
        from diffusers import FluxPipeline
        self.pipe = FluxPipeline.from_pretrained(
            "black-forest-labs/FLUX.1-schnell",
            torch_dtype=torch.bfloat16,
            cache_dir="/models",
        ).to("cuda")

    @modal.method()
    def generate(self, prompt: str) -> bytes:
        image = self.pipe(prompt, num_inference_steps=4, guidance_scale=0.0).images[0]
        import io
        buf = io.BytesIO()
        image.save(buf, format="PNG")
        return buf.getvalue()

Speech Transcription (Whisper)

import modal

app = modal.App("transcription")

image = (
    modal.Image.debian_slim(python_version="3.11")
    .apt_install("ffmpeg")
    .uv_pip_install("openai-whisper==20250625", "torch==2.12.0")
)

@app.cls(gpu="T4", image=image)
class Transcriber:
    @modal.enter()
    def load(self):
        import whisper
        self.model = whisper.load_model("large-v3")

    @modal.method()
    def transcribe(self, audio_path: str) -> dict:
        return self.model.transcribe(audio_path)

Batch Data Processing

import modal

app = modal.App("batch-processor")

image = modal.Image.debian_slim().uv_pip_install("pandas", "pyarrow")
vol = modal.Volume.from_name("batch-data", create_if_missing=True)

@app.function(image=image, volumes={"/data": vol}, cpu=4.0, memory=8192)
def process_chunk(chunk_id: int) -> dict:
    import pandas as pd
    df = pd.read_parquet(f"/data/input/chunk_{chunk_id:04d}.parquet")
    result = df.groupby("category").agg({"value": ["sum", "mean", "count"]})
    result.to_parquet(f"/data/output/result_{chunk_id:04d}.parquet")
    return {"chunk_id": chunk_id, "rows": len(df)}

@app.local_entrypoint()
def main():
    chunk_ids = list(range(500))
    results = list(process_chunk.map(chunk_ids))
    total = sum(r["rows"] for r in results)
    print(f"Processed {total} total rows across {len(results)} chunks")

Web Scraping at Scale

import modal

app = modal.App("scraper")

image = modal.Image.debian_slim().uv_pip_install("httpx", "beautifulsoup4")

@app.function(image=image, retries=3, timeout=60)
def scrape_url(url: str) -> dict:
    import httpx
    from bs4 import BeautifulSoup
    response = httpx.get(url, follow_redirects=True, timeout=30)
    soup = BeautifulSoup(response.text, "html.parser")
    return {
        "url": url,
        "title": soup.title.string if soup.title else None,
        "text": soup.get_text()[:5000],
    }

@app.local_entrypoint()
def main():
    urls = ["https://example.com", "https://example.org"]  # Your URL list
    results = list(scrape_url.map(urls))
    for r in results:
        print(f"{r['url']}: {r['title']}")

Protein Structure Prediction

import modal

app = modal.App("protein-folding")

image = (
    modal.Image.debian_slim(python_version="3.11")
    .uv_pip_install("chai-lab")
)

vol = modal.Volume.from_name("protein-data", create_if_missing=True)

@app.function(gpu="A100-80GB", image=image, volumes={"/data": vol}, timeout=3600)
def fold_protein(sequence: str) -> str:
    from chai_lab.chai1 import run_inference
    output = run_inference(
        fasta_file=write_fasta(sequence, "/data/input.fasta"),
        output_dir="/data/output/",
    )
    return str(output)

Scheduled ETL Pipeline

import modal

app = modal.App("etl")

image = modal.Image.debian_slim().uv_pip_install("pandas", "sqlalchemy", "psycopg2-binary")

@app.function(
    image=image,
    schedule=modal.Cron("0 3 * * *"),  # 3 AM UTC daily
    secrets=[modal.Secret.from_name("database-creds")],
    timeout=7200,
)
def daily_etl():
    import os
    import pandas as pd
    from sqlalchemy import create_engine

    source = create_engine(os.environ["SOURCE_DB"])
    dest = create_engine(os.environ["DEST_DB"])

    df = pd.read_sql("SELECT * FROM events WHERE date = CURRENT_DATE - 1", source)
    df = transform(df)
    df.to_sql("daily_summary", dest, if_exists="append", index=False)
    print(f"Loaded {len(df)} rows")

FastAPI with GPU Model

import modal

app = modal.App("api-with-gpu")

image = (
    modal.Image.debian_slim(python_version="3.11")
    .uv_pip_install("fastapi==0.136.3", "sentence-transformers==5.5.1", "torch==2.12.0")
)

@app.cls(gpu="L40S", image=image, min_containers=1)
class EmbeddingService:
    @modal.enter()
    def load(self):
        from sentence_transformers import SentenceTransformer
        self.model = SentenceTransformer("all-MiniLM-L6-v2", device="cuda")

    @modal.asgi_app()
    def serve(self):
        from fastapi import FastAPI
        api = FastAPI()

        @api.post("/embed")
        async def embed(request: dict):
            embeddings = self.model.encode(request["texts"])
            return {"embeddings": embeddings.tolist()}

        @api.get("/health")
        async def health():
            return {"status": "ok"}

        return api

Document OCR Job Queue

import modal

app = modal.App("ocr-queue")

image = modal.Image.debian_slim().uv_pip_install("pytesseract", "Pillow").apt_install("tesseract-ocr")
vol = modal.Volume.from_name("ocr-data", create_if_missing=True)

@app.function(image=image, volumes={"/data": vol})
def ocr_page(image_path: str) -> str:
    import pytesseract
    from PIL import Image
    img = Image.open(image_path)
    return pytesseract.image_to_string(img)

@app.function(volumes={"/data": vol})
def process_document(doc_id: str):
    import os
    pages = sorted(os.listdir(f"/data/docs/{doc_id}/"))
    paths = [f"/data/docs/{doc_id}/{p}" for p in pages]
    texts = list(ocr_page.map(paths))
    full_text = "\n\n".join(texts)
    with open(f"/data/results/{doc_id}.txt", "w") as f:
        f.write(full_text)
    return {"doc_id": doc_id, "pages": len(texts)}

Modal Functions and Classes

Table of Contents

• Functions
• Remote Execution
• Classes with Lifecycle Hooks
• Parallel Execution
• Async Functions
• Local Entrypoints
• Generators

Functions

Basic Function

import modal

app = modal.App("my-app")

@app.function()
def compute(x: int, y: int) -> int:
    return x + y

Function Parameters

The @app.function() decorator accepts:

Remote Execution

.remote() — Synchronous Call

result = compute.remote(3, 4)  # Runs in the cloud, blocks until done

.local() — Local Execution

result = compute.local(3, 4)  # Runs locally (for testing)

.spawn() — Async Fire-and-Forget

call = compute.spawn(3, 4)  # Returns immediately
# ... do other work ...
result = call.get()  # Retrieve result later

.spawn() supports up to 1 million pending inputs.

Classes with Lifecycle Hooks

Use @app.cls() for stateful workloads where you want to load resources once:

@app.cls(gpu="L40S", image=image)
class Model:
    @modal.enter()
    def setup(self):
        """Runs once when the container starts."""
        import torch
        self.model = torch.load("/weights/model.pt")
        self.model.eval()  # PyTorch inference mode — not Python's built-in eval()

    @modal.method()
    def predict(self, text: str) -> dict:
        """Callable remotely."""
        return self.model(text)

    @modal.exit()
    def teardown(self):
        """Runs when the container shuts down."""
        cleanup_resources()

Lifecycle Decorators

Calling Class Methods

# Create instance and call method
model = Model()
result = model.predict.remote("Hello world")

# Parallel calls
results = list(model.predict.map(["text1", "text2", "text3"]))

Parameterized Classes

@app.cls()
class Worker:
    model_name: str = modal.parameter()

    @modal.enter()
    def load(self):
        self.model = load_model(self.model_name)

    @modal.method()
    def run(self, data):
        return self.model(data)

# Different model instances autoscale independently
gpt = Worker(model_name="gpt-4")
llama = Worker(model_name="llama-3")

Parallel Execution

.map() — Parallel Processing

Process multiple inputs across containers:

@app.function()
def process(item):
    return heavy_computation(item)

@app.local_entrypoint()
def main():
    items = list(range(1000))
    results = list(process.map(items))
    print(f"Processed {len(results)} items")

• Results are returned in the same order as inputs
• Modal autoscales containers to handle the workload
• Use return_exceptions=True to collect errors instead of raising

.starmap() — Multi-Argument Parallel

@app.function()
def add(x, y):
    return x + y

results = list(add.starmap([(1, 2), (3, 4), (5, 6)]))
# [3, 7, 11]

.map() with order_outputs=False

For faster throughput when order doesn't matter:

for result in process.map(items, order_outputs=False):
    handle(result)  # Results arrive as they complete

Async Functions

Modal supports async/await natively:

@app.function()
async def fetch_data(url: str) -> str:
    import httpx
    async with httpx.AsyncClient() as client:
        response = await client.get(url)
        return response.text

Async functions are especially useful with @modal.concurrent() for handling multiple requests per container.

Local Entrypoints

The @app.local_entrypoint() runs on your machine and orchestrates remote calls:

@app.local_entrypoint()
def main():
    # This code runs locally
    data = load_local_data()

    # These calls run in the cloud
    results = list(process.map(data))

    # Back to local
    save_results(results)

You can also define multiple entrypoints and select by function name:

modal run script.py::train
modal run script.py::evaluate

Generators

Functions can yield results as they're produced:

@app.function()
def generate_data():
    for i in range(100):
        yield process(i)

@app.local_entrypoint()
def main():
    for result in generate_data.remote_gen():
        print(result)

Retries

Configure automatic retries on failure:

@app.function(retries=3)
def flaky_operation():
    ...

For more control, use modal.Retries:

@app.function(retries=modal.Retries(max_retries=3, backoff_coefficient=2.0))
def api_call():
    ...

Timeouts

Set maximum execution time:

@app.function(timeout=3600)  # 1 hour
def long_training():
    ...

Default timeout is 300 seconds (5 minutes). Maximum is 86400 seconds (24 hours).

Modal Getting Started Guide

Installation

Install Modal with uv (recommended). The SDK supports Python 3.10–3.14:

uv pip install modal

Authentication

Interactive Setup

modal setup

This opens a browser for authentication and stores credentials locally.

Headless / CI/CD Setup

For environments without a browser, use token-based authentication:

1. Generate tokens at https://modal.com/settings 2. Set environment variables:

export MODAL_TOKEN_ID=<your-token-id>
export MODAL_TOKEN_SECRET=<your-token-secret>

Or use the CLI:

modal token set --token-id <id> --token-secret <secret>

Free Tier

Modal provides $30/month in free credits. No credit card required for the free tier.

Your First App

Hello World

Create a file hello.py:

import modal

app = modal.App("hello-world")

@app.function()
def greet(name: str) -> str:
    return f"Hello, {name}! This ran in the cloud."

@app.local_entrypoint()
def main():
    result = greet.remote("World")
    print(result)

Run it:

modal run hello.py

What happens: 1. Modal packages your code 2. Creates a container in the cloud 3. Executes greet() remotely 4. Returns the result to your local machine

Understanding the Flow

• modal.App("name") — Creates a named application
• @app.function() — Marks a function for remote execution
• @app.local_entrypoint() — Defines the local entry point (runs on your machine)
• .remote() — Calls the function in the cloud
• .local() — Calls the function locally (for testing)

Running Modes

A Simple Web Scraper

import modal

app = modal.App("web-scraper")

image = modal.Image.debian_slim().uv_pip_install("httpx", "beautifulsoup4")

@app.function(image=image)
def scrape(url: str) -> str:
    import httpx
    from bs4 import BeautifulSoup

    response = httpx.get(url)
    soup = BeautifulSoup(response.text, "html.parser")
    return soup.get_text()[:1000]

@app.local_entrypoint()
def main():
    result = scrape.remote("https://example.com")
    print(result)

GPU-Accelerated Inference

import modal

app = modal.App("gpu-inference")

image = (
    modal.Image.debian_slim(python_version="3.11")
    .uv_pip_install("torch", "transformers", "accelerate")
)

@app.function(gpu="L40S", image=image)
def generate(prompt: str) -> str:
    from transformers import pipeline
    pipe = pipeline("text-generation", model="gpt2", device="cuda")
    result = pipe(prompt, max_length=100)
    return result[0]["generated_text"]

@app.local_entrypoint()
def main():
    print(generate.remote("The future of AI is"))

Project Structure

Modal apps are typically single Python files, but can be organized into modules:

my-project/
├── app.py           # Main app with @app.local_entrypoint()
├── inference.py     # Inference functions
├── training.py      # Training functions
└── common.py        # Shared utilities

Use modal.Image.add_local_python_source() to include local modules in the container image.

Key Concepts Summary

Next Steps

• See functions.md for advanced function patterns
• See images.md for custom container environments
• See gpu.md for GPU selection and configuration
• See web-endpoints.md for serving APIs

Modal GPU Compute

Table of Contents

• Available GPUs
• Requesting GPUs
• GPU Selection Guide
• Multi-GPU
• GPU Fallback Chains
• Auto-Upgrades
• Multi-GPU Training

Available GPUs

Requesting GPUs

Basic Request

@app.function(gpu="H100")
def train():
    import torch
    assert torch.cuda.is_available()
    print(f"Using: {torch.cuda.get_device_name(0)}")

String Shorthand

gpu="T4"           # Single T4
gpu="A100-80GB"    # Single A100 80GB
gpu="H100:4"       # Four H100s

Case-Insensitive Strings

GPU strings are case-insensitive, so gpu="h100" and gpu="H100" are equivalent.

Deprecation: The legacy modal.gpu.* objects (e.g. modal.gpu.H100(count=2)) are deprecated as of v0.73.31. Always configure GPUs with strings — use gpu="H100:2" for multiple GPUs and gpu="A100-80GB" for the 80 GB A100.

GPU Selection Guide

For Inference

For Training

General Recommendation

L40S is the best default for inference workloads — it offers an excellent trade-off of cost and performance with 48 GB of GPU RAM.

Multi-GPU

Request multiple GPUs by appending :count:

@app.function(gpu="H100:4")
def distributed():
    import torch
    print(f"GPUs available: {torch.cuda.device_count()}")
    # All 4 GPUs are on the same physical machine

• Up to 8 GPUs for most types (up to 4 for A10)
• All GPUs attach to the same physical machine
• Requesting more than 2 GPUs may result in longer wait times
• Maximum VRAM: 8 x B200 = 1,536 GB

GPU Fallback Chains

Specify a prioritized list of GPU types:

@app.function(gpu=["H100", "A100-80GB", "L40S"])
def flexible():
    # Modal tries H100 first, then A100-80GB, then L40S
    ...

Useful for reducing queue times when a specific GPU isn't available.

Auto-Upgrades

H100 → H200

Modal may automatically upgrade H100 requests to H200 at no extra cost. To prevent this:

@app.function(gpu="H100!")  # Exclamation mark prevents auto-upgrade
def must_use_h100():
    ...

A100 → A100-80GB

A100-40GB requests may be upgraded to 80GB at no extra cost.

B200+

gpu="B200+" allows Modal to run on B200 or B300 GPUs at B200 pricing. Requires CUDA 13.0+.

Multi-GPU Training

Modal supports multi-GPU training on a single node. Multi-node training is in private beta.

PyTorch DDP Example

@app.function(gpu="H100:4", image=image, timeout=86400)
def train_distributed():
    import torch
    import torch.distributed as dist

    dist.init_process_group(backend="nccl")
    local_rank = int(os.environ.get("LOCAL_RANK", 0))
    device = torch.device(f"cuda:{local_rank}")
    # ... training loop with DDP ...

PyTorch Lightning

When using frameworks that re-execute Python entrypoints (like PyTorch Lightning), either:

1. Set strategy to ddp_spawn or ddp_notebook 2. Or run training as a subprocess

@app.function(gpu="H100:4", image=image)
def train():
    import subprocess
    subprocess.run(["python", "train_script.py"], check=True)

Hugging Face Accelerate

@app.function(gpu="A100-80GB:4", image=image)
def finetune():
    import subprocess
    subprocess.run([
        "accelerate", "launch",
        "--num_processes", "4",
        "train.py"
    ], check=True)

Security: These launchers use fixed, hardcoded argument lists. Never build the

subprocess argument list from unsanitized user input. If a workload needs

user-supplied values (e.g. hyperparameters), validate them against an allowlist or

pass them as files / environment variables rather than as command arguments.

Modal Container Images

Table of Contents

• Overview
• Base Images
• Installing Packages
• System Packages
• Shell Commands
• Running Python During Build
• Adding Local Files
• Environment Variables
• Dockerfiles
• Alternative Package Managers
• Image Caching
• Handling Remote-Only Imports

Overview

Every Modal function runs inside a container built from an Image. By default, Modal uses a Debian Linux image with the same Python minor version as your local interpreter.

Images are built lazily — Modal only builds/pulls the image when a function using it is first invoked. Layers are cached for fast rebuilds.

Base Images

# Default: Debian slim with your local Python version
image = modal.Image.debian_slim()

# Specific Python version
image = modal.Image.debian_slim(python_version="3.11")

# From Docker Hub
image = modal.Image.from_registry("nvidia/cuda:12.4.0-devel-ubuntu22.04")

# From a Dockerfile
image = modal.Image.from_dockerfile("./Dockerfile")

Installing Packages

uv (Recommended)

uv_pip_install uses the uv package manager for fast, reliable installs:

image = (
    modal.Image.debian_slim(python_version="3.11")
    .uv_pip_install(
        "torch==2.12.0",
        "transformers==5.9.0",
        "accelerate==1.13.0",
        "scipy==1.17.1",
    )
)

Pin versions for reproducibility. uv resolves dependencies faster than pip.

pip (Fallback)

image = modal.Image.debian_slim().pip_install(
    "numpy==1.26.0",
    "pandas==2.1.0",
)

From requirements.txt

image = modal.Image.debian_slim().pip_install_from_requirements("requirements.txt")

Private Packages

image = (
    modal.Image.debian_slim()
    .pip_install_private_repos(
        "github.com/org/private-repo",
        git_user="username",
        secrets=[modal.Secret.from_name("github-token")],
    )
)

System Packages

Install Linux packages via apt:

image = (
    modal.Image.debian_slim()
    .apt_install("ffmpeg", "libsndfile1", "git", "curl")
    .uv_pip_install("librosa", "soundfile")
)

Shell Commands

Run arbitrary commands during image build:

image = (
    modal.Image.debian_slim()
    .run_commands(
        "wget https://example.com/data.tar.gz",
        "tar -xzf data.tar.gz -C /opt/data",
        "rm data.tar.gz",
    )
)

With GPU

Some build steps require GPU access (e.g., compiling CUDA kernels):

image = (
    modal.Image.debian_slim()
    .uv_pip_install("torch")
    .run_commands("python -c 'import torch; torch.cuda.is_available()'", gpu="A100")
)

Running Python During Build

Execute Python functions as build steps — useful for downloading model weights:

def download_model():
    from huggingface_hub import snapshot_download
    snapshot_download("meta-llama/Llama-3-8B", local_dir="/models/llama3")

image = (
    modal.Image.debian_slim(python_version="3.11")
    .uv_pip_install("huggingface_hub", "torch", "transformers")
    .run_function(download_model, secrets=[modal.Secret.from_name("huggingface")])
)

The resulting filesystem (including downloaded files) is snapshotted into the image.

Adding Local Files

Local Directories

image = modal.Image.debian_slim().add_local_dir(
    local_path="./config",
    remote_path="/root/config",
)

By default, files are added at container startup (not baked into the image layer). Use copy=True to bake them in.

Local Python Modules

image = modal.Image.debian_slim().add_local_python_source("my_module")

This uses Python's import system to find and include the module.

As of v1.0, Modal no longer "automounts" imported local modules. You must explicitly

include local dependencies with add_local_python_source (the App's own source is

still included automatically; set include_source=False on the App/Function to opt

out). The deprecated modal.Mount object and the mount=/context_mount= parameters

have been replaced by these Image.add_local_* methods.

Individual Files

image = modal.Image.debian_slim().add_local_file(
    local_path="./model_config.json",
    remote_path="/root/config.json",
)

Environment Variables

image = (
    modal.Image.debian_slim()
    .env({
        "TRANSFORMERS_CACHE": "/cache",
        "TOKENIZERS_PARALLELISM": "false",
        "HF_HOME": "/cache/huggingface",
    })
)

Names and values must be strings.

Dockerfiles

Build from existing Dockerfiles:

image = modal.Image.from_dockerfile("./Dockerfile")

The build context is now inferred automatically from the Dockerfile's commands. The old context_mount= parameter — along with the modal.Mount object it relied on — is deprecated and was enforced as removed in v1.0; do not pass it.

Alternative Package Managers

Micromamba / Conda

For packages requiring coordinated system and Python package installs:

image = (
    modal.Image.micromamba(python_version="3.11")
    .micromamba_install("cudatoolkit=11.8", "cudnn=8.6", channels=["conda-forge"])
    .uv_pip_install("torch")
)

Image Caching

Modal caches images per layer (per method call). Breaking the cache on one layer cascades to all subsequent layers.

Optimization Tips

1. Order layers by change frequency: Put stable dependencies first, frequently changing code last 2. Pin versions: Unpinned versions may resolve differently and break cache 3. Separate large installs: Put heavy packages (torch, tensorflow) in early layers

Force Rebuild

# Single layer
image = modal.Image.debian_slim().apt_install("git", force_build=True)

# All images in a run
MODAL_FORCE_BUILD=1 modal run script.py

# Rebuild without updating cache
MODAL_IGNORE_CACHE=1 modal run script.py

Handling Remote-Only Imports

When packages are only available in the container (not locally), use conditional imports:

@app.function(image=image)
def process():
    import torch  # Only available in the container
    return torch.cuda.device_count()

For module-level imports shared across functions, use the Image.imports() context manager:

with image.imports():
    import torch
    import transformers

This prevents ImportError locally while making the imports available in the container.

Modal Resource Configuration

CPU

Requesting CPU

@app.function(cpu=4.0)
def compute():
    ...

• Values are physical cores, not vCPUs
• Default: 0.125 cores
• Modal auto-sets OPENBLAS_NUM_THREADS, OMP_NUM_THREADS, MKL_NUM_THREADS based on your CPU request

CPU Limits

• Default soft limit: 16 physical cores above the CPU request
• Set explicit limits to prevent noisy-neighbor effects:

@app.function(cpu=4.0)  # Request 4 cores
def bounded_compute():
    ...

Memory

Requesting Memory

@app.function(memory=16384)  # 16 GiB in MiB
def large_data():
    ...

• Value in MiB (megabytes)
• Default: 128 MiB

Memory Limits

Set hard memory limits to OOM-kill containers that exceed them:

@app.function(memory=8192)  # 8 GiB request and limit
def bounded_memory():
    ...

This prevents paying for runaway memory leaks.

Ephemeral Disk

For temporary storage within a container's lifetime:

@app.function(ephemeral_disk=102400)  # 100 GiB in MiB
def process_dataset():
    # Temporary files at /tmp or anywhere in the container filesystem
    ...

• Value in MiB
• Default: 512 GiB quota per container
• Maximum: 3,145,728 MiB (3 TiB)
• Data is lost when the container shuts down
• Use Volumes for persistent storage

Larger disk requests increase the memory request at a 20:1 ratio for billing purposes.

Timeout

@app.function(timeout=3600)  # 1 hour in seconds
def long_running():
    ...

• Default: 300 seconds (5 minutes)
• Maximum: 86,400 seconds (24 hours)
• Function is killed when timeout expires

Billing

You are charged based on whichever is higher: your resource request or actual usage.

Cost Optimization Tips

• Request only what you need
• Use appropriate GPU tiers (L40S over H100 for inference)
• Set scaledown_window to minimize idle time
• Use min_containers=0 when cold starts are acceptable
• Batch inputs with .map() instead of individual .remote() calls

Complete Example

@app.function(
    cpu=8.0,              # 8 physical cores
    memory=32768,         # 32 GiB
    gpu="L40S",           # L40S GPU
    ephemeral_disk=204800, # 200 GiB temp disk
    timeout=7200,         # 2 hours
    max_containers=50,
    min_containers=1,
)
def full_pipeline(data_path: str):
    ...

Modal Scaling and Concurrency

Table of Contents

• Autoscaling
• Configuration
• Parallel Execution
• Concurrent Inputs
• Dynamic Batching
• Dynamic Autoscaler Updates
• Limits

Autoscaling

Modal automatically manages a pool of containers for each function:

• Spins up containers when there's no capacity for new inputs
• Spins down idle containers to save costs
• Scales from zero (no cost when idle) to thousands of containers

No configuration needed for basic autoscaling — it works out of the box.

Configuration

Fine-tune autoscaling behavior:

@app.function(
    max_containers=100,     # Upper limit on container count
    min_containers=2,       # Keep 2 warm (reduces cold starts)
    buffer_containers=5,    # Reserve 5 extra for burst traffic
    scaledown_window=300,   # Wait 5 min idle before shutting down
)
def handle_request(data):
    ...

Trade-offs

• Higher min_containers = lower latency, higher cost
• Higher buffer_containers = less queuing, higher cost
• Lower scaledown_window = faster cost savings, more cold starts

Parallel Execution

.map() — Process Many Inputs

@app.function()
def process(item):
    return heavy_computation(item)

@app.local_entrypoint()
def main():
    items = list(range(10_000))
    results = list(process.map(items))

Modal automatically scales containers to handle the workload. Results maintain input order.

.map() Options

# Unordered results (faster)
for result in process.map(items, order_outputs=False):
    handle(result)

# Collect errors instead of raising
results = list(process.map(items, return_exceptions=True))
for r in results:
    if isinstance(r, Exception):
        print(f"Error: {r}")

.starmap() — Multi-Argument

@app.function()
def add(x, y):
    return x + y

results = list(add.starmap([(1, 2), (3, 4), (5, 6)]))
# [3, 7, 11]

.spawn() — Fire-and-Forget

# Returns immediately
call = process.spawn(large_data)

# Check status or get result later
result = call.get()

Up to 1 million pending .spawn() calls.

Concurrent Inputs

By default, each container handles one input at a time. Use @modal.concurrent to handle multiple:

@app.function(gpu="L40S")
@modal.concurrent(max_inputs=10)
async def predict(text: str):
    result = await model.predict_async(text)
    return result

This is ideal for I/O-bound workloads or async inference where a single GPU can handle multiple requests.

With Web Endpoints

@app.function(gpu="L40S")
@modal.concurrent(max_inputs=20)
@modal.asgi_app()
def web_service():
    return fastapi_app

Dynamic Batching

Collect inputs into batches for efficient GPU utilization:

@app.function(gpu="L40S")
@modal.batched(max_batch_size=32, wait_ms=100)
async def batch_predict(texts: list[str]):
    # Called with up to 32 texts at once
    embeddings = model.encode(texts)
    return list(embeddings)

• max_batch_size — Maximum inputs per batch
• wait_ms — How long to wait for more inputs before processing
• The function receives a list and must return a list of the same length

Dynamic Autoscaler Updates

Adjust autoscaling at runtime without redeploying:

@app.function()
def scale_up_for_peak():
    process = modal.Function.from_name("my-app", "process")
    process.update_autoscaler(min_containers=10, buffer_containers=20)

@app.function()
def scale_down_after_peak():
    process = modal.Function.from_name("my-app", "process")
    process.update_autoscaler(min_containers=1, buffer_containers=2)

Settings revert to the decorator values on the next deployment.

Limits

Exceeding these limits triggers Resource Exhausted errors. Implement retry logic for resilience.

Modal Scheduled Jobs

Overview

Modal supports running functions automatically on a schedule, either using cron syntax or fixed intervals. Deploy scheduled functions with modal deploy and they run unattended in the cloud.

Schedule Types

modal.Cron

Standard cron syntax — stable across deploys:

import modal

app = modal.App("scheduled-tasks")

# Daily at 9 AM UTC
@app.function(schedule=modal.Cron("0 9 * * *"))
def daily_report():
    generate_and_send_report()

# Every Monday at midnight
@app.function(schedule=modal.Cron("0 0 * * 1"))
def weekly_cleanup():
    cleanup_old_data()

# Every 15 minutes
@app.function(schedule=modal.Cron("*/15 * * * *"))
def frequent_check():
    check_system_health()

Cron Syntax Reference

┌───────────── minute (0-59)
│ ┌───────────── hour (0-23)
│ │ ┌───────────── day of month (1-31)
│ │ │ ┌───────────── month (1-12)
│ │ │ │ ┌───────────── day of week (0-6, Sun=0)
│ │ │ │ │
* * * * *

modal.Period

Fixed interval — resets on each deploy:

# Every 5 hours
@app.function(schedule=modal.Period(hours=5))
def periodic_sync():
    sync_data()

# Every 30 minutes
@app.function(schedule=modal.Period(minutes=30))
def poll_updates():
    check_for_updates()

# Every day
@app.function(schedule=modal.Period(days=1))
def daily_task():
    ...

modal.Period resets its timer on each deployment. If you need a schedule that doesn't shift with deploys, use modal.Cron.

Deploying Scheduled Functions

Schedules only activate when deployed:

modal deploy script.py

modal run and modal serve do not activate schedules.

Monitoring

• View scheduled runs in the Apps section of the Modal dashboard
• Each run appears with its status, duration, and logs
• Use the "Run Now" button on the dashboard to trigger manually

Management

• Schedules cannot be paused — remove the schedule and redeploy to stop
• To change a schedule, update the schedule parameter and redeploy
• To stop entirely, either remove the schedule parameter or run modal app stop <name>

Common Patterns

ETL Pipeline

@app.function(
    schedule=modal.Cron("0 2 * * *"),  # 2 AM UTC daily
    secrets=[modal.Secret.from_name("db-creds")],
    timeout=7200,
)
def etl_pipeline():
    import os
    data = extract(os.environ["SOURCE_DB_URL"])
    transformed = transform(data)
    load(transformed, os.environ["DEST_DB_URL"])

Model Retraining

@app.function(
    schedule=modal.Cron("0 0 * * 0"),  # Weekly on Sunday
    gpu="H100",
    volumes={"/data": data_vol, "/models": model_vol},
    timeout=86400,
)
def retrain():
    model = train_on_latest_data("/data/training/")
    torch.save(model.state_dict(), "/models/latest.pt")

Health Checks

@app.function(
    schedule=modal.Period(minutes=5),
    secrets=[modal.Secret.from_name("slack-webhook")],
)
def health_check():
    import os, requests
    status = check_all_services()
    if not status["healthy"]:
        requests.post(os.environ["SLACK_URL"], json={"text": f"Alert: {status}"})

The webhook URL is read from a Modal Secret (SLACK_URL), not hardcoded or taken

from untrusted input. Keep notification endpoints in Secrets and avoid POSTing to

URLs constructed from user-supplied data.

Modal Secrets

Overview

Modal Secrets securely deliver credentials and sensitive data to functions as environment variables. Secrets are stored encrypted and only available to your workspace.

Creating Secrets

Via CLI

# Create with key-value pairs
modal secret create my-api-keys API_KEY=sk-xxx DB_PASSWORD=hunter2

# Create from existing environment variables
modal secret create my-env-keys API_KEY=$API_KEY

# List all secrets
modal secret list

# Delete a secret
modal secret delete my-api-keys

Via Dashboard

Navigate to https://modal.com/secrets to create and manage secrets. Templates are available for common services (Postgres, MongoDB, Hugging Face, Weights & Biases, etc.).

Programmatic (Inline)

# From a dictionary (useful for development)
secret = modal.Secret.from_dict({"API_KEY": "sk-xxx"})

# From a .env file
secret = modal.Secret.from_dotenv()

# From a named secret (created via CLI or dashboard)
secret = modal.Secret.from_name("my-api-keys")

Using Secrets in Functions

Basic Usage

@app.function(secrets=[modal.Secret.from_name("my-api-keys")])
def call_api():
    import os
    api_key = os.environ["API_KEY"]
    # Use the key
    response = requests.get(url, headers={"Authorization": f"Bearer {api_key}"})
    return response.json()

Multiple Secrets

@app.function(secrets=[
    modal.Secret.from_name("openai-keys"),
    modal.Secret.from_name("database-creds"),
])
def process():
    import os
    openai_key = os.environ["OPENAI_API_KEY"]
    db_url = os.environ["DATABASE_URL"]
    ...

Secrets are applied in order — if two secrets define the same key, the later one wins.

With Classes

@app.cls(secrets=[modal.Secret.from_name("huggingface")])
class ModelService:
    @modal.enter()
    def load(self):
        import os
        token = os.environ["HF_TOKEN"]
        self.model = AutoModel.from_pretrained("model-name", token=token)

From .env File

# Reads .env file from current directory
@app.function(secrets=[modal.Secret.from_dotenv()])
def local_dev():
    import os
    api_key = os.environ["API_KEY"]

The .env file format:

API_KEY=sk-xxx
DATABASE_URL=postgres://user:pass@host/db
DEBUG=false

Common Secret Templates

Security Notes

• Secrets are encrypted at rest and in transit
• Only accessible to functions in your workspace
• Never log or print secret values
• Use .from_name() in production (not .from_dict())
• Rotate secrets regularly via the dashboard or CLI

Modal Volumes

Table of Contents

• Overview
• Creating Volumes
• Mounting Volumes
• Reading and Writing Files
• CLI Access
• Commits and Reloads
• Concurrent Access
• Volumes v2
• Common Patterns

Overview

Volumes are Modal's distributed file system, optimized for write-once, read-many workloads like storing model weights and distributing them across containers.

Key characteristics:

• Persistent across function invocations and deployments
• Mountable by multiple functions simultaneously
• Background auto-commits every few seconds
• Final commit on container shutdown

Creating Volumes

In Code (Lazy Creation)

vol = modal.Volume.from_name("my-volume", create_if_missing=True)

Via CLI

modal volume create my-volume

# v2 volume (beta)
modal volume create my-volume --version=2

Programmatic v2

vol = modal.Volume.from_name("my-volume", create_if_missing=True, version=2)

Mounting Volumes

Mount volumes to functions via the volumes parameter:

vol = modal.Volume.from_name("model-store", create_if_missing=True)

@app.function(volumes={"/models": vol})
def use_model():
    # Access files at /models/
    with open("/models/config.json") as f:
        config = json.load(f)

Mount multiple volumes:

weights_vol = modal.Volume.from_name("weights")
data_vol = modal.Volume.from_name("datasets")

@app.function(volumes={"/weights": weights_vol, "/data": data_vol})
def train():
    ...

Reading and Writing Files

Writing

@app.function(volumes={"/data": vol})
def save_results(results):
    import json
    import os

    os.makedirs("/data/outputs", exist_ok=True)
    with open("/data/outputs/results.json", "w") as f:
        json.dump(results, f)

Reading

@app.function(volumes={"/data": vol})
def load_results():
    with open("/data/outputs/results.json") as f:
        return json.load(f)

Large Files (Model Weights)

@app.function(volumes={"/models": vol}, gpu="L40S")
def save_model():
    import torch
    model = train_model()
    torch.save(model.state_dict(), "/models/checkpoint.pt")

@app.function(volumes={"/models": vol}, gpu="L40S")
def load_model():
    import torch
    model = MyModel()
    model.load_state_dict(torch.load("/models/checkpoint.pt"))
    return model

CLI Access

# List files
modal volume ls my-volume
modal volume ls my-volume /subdir/

# Upload files
modal volume put my-volume local_file.txt
modal volume put my-volume local_file.txt /remote/path/file.txt

# Download files
modal volume get my-volume /remote/file.txt local_file.txt

# Delete a volume
modal volume delete my-volume

Commits and Reloads

Modal auto-commits volume changes in the background every few seconds and on container shutdown.

Explicit Commit

Force an immediate commit:

@app.function(volumes={"/data": vol})
def writer():
    with open("/data/file.txt", "w") as f:
        f.write("hello")
    vol.commit()  # Make immediately visible to other containers

Reload

See changes from other containers:

@app.function(volumes={"/data": vol})
def reader():
    vol.reload()  # Refresh to see latest writes
    with open("/data/file.txt") as f:
        return f.read()

Concurrent Access

v1 Volumes

• Recommended max 5 concurrent commits
• Last write wins for concurrent modifications of the same file
• Avoid concurrent modification of identical files
• Max 500,000 files (inodes)

v2 Volumes

• Hundreds of concurrent writers (distinct files)
• No file count limit
• Improved random access performance
• Up to 1 TiB per file, 262,144 files per directory

Volumes v2

v2 Volumes (beta) offer significant improvements:

Enable v2:

vol = modal.Volume.from_name("my-vol-v2", create_if_missing=True, version=2)

Common Patterns

Model Weight Storage

vol = modal.Volume.from_name("model-weights", create_if_missing=True)

# Download once during image build
def download_weights():
    from huggingface_hub import snapshot_download
    snapshot_download("meta-llama/Llama-3-8B", local_dir="/models/llama3")

image = (
    modal.Image.debian_slim()
    .uv_pip_install("huggingface_hub")
    .run_function(download_weights, volumes={"/models": vol})
)

Training Checkpoints

@app.function(volumes={"/checkpoints": vol}, gpu="H100", timeout=86400)
def train():
    for epoch in range(100):
        train_one_epoch()
        torch.save(model.state_dict(), f"/checkpoints/epoch_{epoch}.pt")
        vol.commit()  # Save checkpoint immediately

Shared Data Between Functions

data_vol = modal.Volume.from_name("shared-data", create_if_missing=True)

@app.function(volumes={"/data": data_vol})
def preprocess():
    # Write processed data
    df.to_parquet("/data/processed.parquet")

@app.function(volumes={"/data": data_vol})
def analyze():
    data_vol.reload()  # Ensure we see latest data
    df = pd.read_parquet("/data/processed.parquet")
    return df.describe()

Performance Tips

• Volumes are optimized for large files, not many small files
• Keep under 50,000 files and directories for best v1 performance
• Use Parquet or other columnar formats instead of many small CSVs
• For truly temporary data, use ephemeral_disk instead of Volumes

Modal Web Endpoints

Table of Contents

• Simple Endpoints
• Deployment
• ASGI Apps
• WSGI Apps
• Custom Web Servers
• WebSockets
• Authentication
• Streaming
• Concurrency
• Limits

Simple Endpoints

The easiest way to create a web endpoint:

import modal

app = modal.App("api-service")

@app.function()
@modal.fastapi_endpoint()
def hello(name: str = "World"):
    return {"message": f"Hello, {name}!"}

POST Endpoints

@app.function()
@modal.fastapi_endpoint(method="POST")
def predict(data: dict):
    result = model.predict(data["text"])
    return {"prediction": result}

Query Parameters

Parameters are automatically parsed from query strings:

@app.function()
@modal.fastapi_endpoint()
def search(query: str, limit: int = 10):
    return {"results": do_search(query, limit)}

Access via: https://your-app.modal.run?query=hello&limit=5

Deployment

Development Mode

modal serve script.py

• Creates a temporary public URL
• Hot-reloads on file changes
• Perfect for development and testing
• URL expires when you stop the command

Production Deployment

modal deploy script.py

• Creates a permanent URL
• Runs persistently in the cloud
• Autoscales based on traffic
• URL format: https://<workspace>--<app-name>-<function-name>.modal.run

ASGI Apps (FastAPI, Starlette, FastHTML)

For full framework applications, use @modal.asgi_app:

from fastapi import FastAPI

web_app = FastAPI()

@web_app.get("/")
async def root():
    return {"status": "ok"}

@web_app.post("/predict")
async def predict(request: dict):
    return {"result": model.run(request["input"])}

@app.function(image=image, gpu="L40S")
@modal.asgi_app()
def fastapi_app():
    return web_app

With Class Lifecycle

@app.cls(gpu="L40S", image=image)
class InferenceService:
    @modal.enter()
    def load_model(self):
        self.model = load_model()

    @modal.asgi_app()
    def serve(self):
        from fastapi import FastAPI
        app = FastAPI()

        @app.post("/generate")
        async def generate(request: dict):
            return self.model.generate(request["prompt"])

        return app

WSGI Apps (Flask, Django)

from flask import Flask

flask_app = Flask(__name__)

@flask_app.route("/")
def index():
    return {"status": "ok"}

@app.function(image=image)
@modal.wsgi_app()
def flask_server():
    return flask_app

WSGI is synchronous — concurrent inputs run on separate threads.

Custom Web Servers

For non-standard web frameworks (aiohttp, Tornado, TGI):

@app.function(image=image, gpu="H100")
@modal.web_server(port=8000)
def serve():
    import subprocess
    subprocess.Popen([
        "python", "-m", "vllm.entrypoints.openai.api_server",
        "--model", "meta-llama/Llama-3-70B",
        "--host", "0.0.0.0",  # Must bind to 0.0.0.0, not localhost
        "--port", "8000",
    ])

The application must bind to 0.0.0.0 (not 127.0.0.1).

Security: The command above uses a fixed argument list. Do not interpolate

unsanitized user input (model names, paths, flags) into subprocess arguments —

validate against an allowlist or pass untrusted values as data, not as command

arguments, to avoid command injection.

WebSockets

Supported with @modal.asgi_app, @modal.wsgi_app, and @modal.web_server:

from fastapi import FastAPI, WebSocket

web_app = FastAPI()

@web_app.websocket("/ws")
async def websocket_endpoint(websocket: WebSocket):
    await websocket.accept()
    while True:
        data = await websocket.receive_text()
        result = process(data)
        await websocket.send_text(result)

@app.function()
@modal.asgi_app()
def ws_app():
    return web_app

• Full WebSocket protocol (RFC 6455)
• Messages up to 2 MiB each
• No RFC 8441 or RFC 7692 support yet

Authentication

Proxy Auth Tokens (Built-in)

Modal provides first-class endpoint protection via proxy auth tokens:

@app.function()
@modal.fastapi_endpoint()
def protected(text: str):
    return {"result": process(text)}

Clients include Modal-Key and Modal-Secret headers to authenticate.

Custom Bearer Tokens

from fastapi import Header, HTTPException

@app.function(secrets=[modal.Secret.from_name("auth-secret")])
@modal.fastapi_endpoint(method="POST")
def secure_predict(data: dict, authorization: str = Header(None)):
    import os
    expected = os.environ["AUTH_TOKEN"]
    if authorization != f"Bearer {expected}":
        raise HTTPException(status_code=401, detail="Unauthorized")
    return {"result": model.predict(data["text"])}

Client IP Access

Available for geolocation, rate limiting, and access control.

Streaming

Server-Sent Events (SSE)

from fastapi.responses import StreamingResponse

@app.function(gpu="H100")
@modal.fastapi_endpoint()
def stream_generate(prompt: str):
    def generate():
        for token in model.stream(prompt):
            yield f"data: {token}\n\n"
    return StreamingResponse(generate(), media_type="text/event-stream")

Concurrency

Handle multiple requests per container using @modal.concurrent:

@app.function(gpu="L40S")
@modal.concurrent(max_inputs=10)
@modal.fastapi_endpoint(method="POST")
async def batch_predict(data: dict):
    return {"result": await model.predict_async(data["text"])}

Limits

• Request body: up to 4 GiB
• Response body: unlimited
• Rate limit: 200 requests/second (5-second burst for new accounts)
• Cold starts occur when no containers are active (use min_containers to avoid)