Baseten

Baseten Skill

Product summary

Baseten is a training and inference platform for deploying AI models at scale. Use Truss, an open-source framework, to package models into deployable containers with a config.yaml file specifying GPU, dependencies, and runtime behavior. Push models with truss push to get an autoscaling API endpoint. For hosted models without deployment, use Model APIs (OpenAI-compatible endpoints for LLMs like DeepSeek, Qwen, GLM). Orchestrate multi-step workflows with Chains, each step running on independent hardware. Key files: config.yaml (model configuration), model/model.py (custom inference logic), .trussrc (local CLI config). Primary docs: https://docs.baseten.co

When to use

• Deploying models: Package open-source LLMs, embeddings, or custom models with Truss and push to production.
• Running inference: Call deployed models via REST API or use Model APIs for instant access to hosted models.
• Building pipelines: Orchestrate multi-model workflows (RAG, image generation, transcription) with Chains.
• Iterating rapidly: Use truss push --watch for live reload during development, then promote to production.
• Scaling automatically: Configure autoscaling to handle traffic spikes and scale to zero when idle.
• Training and fine-tuning: Run training jobs on H100/H200 GPUs and deploy checkpoints directly to production.
• Async processing: Queue long-running inference tasks with webhooks for batch processing.

Quick reference

Essential CLI commands

config.yaml essentials

model_name: my-model
description: Model description
resources:
  accelerator: L4  # GPU type (L4, A100, H100, etc.)
  cpu: "4"
  memory: 16Gi
requirements:
  - torch
  - transformers
secrets:
  hf_token: null  # Placeholder; set in Baseten UI
environment_variables:
  HF_HOME: /app/hf_cache
runtime:
  predict_concurrency: 5  # Parallel requests per replica

Inference endpoints

Model class structure

class Model:
    def __init__(self, **kwargs):
        # Read config, secrets, data_dir
        pass
    
    def load(self):
        # Download weights, initialize GPU (runs once at startup)
        pass
    
    def predict(self, model_input):
        # Run inference (called per request)
        return output

Decision guidance

Workflow

Deploy a model

1. Create a Truss project: Run truss init or create a directory with config.yaml and optional model/model.py. 2. Configure resources: Set accelerator, cpu, memory, and requirements in config.yaml. 3. Add model code (optional): Write model/model.py with __init__, load, and predict methods if config alone isn't enough. 4. Authenticate: Run truss auth login and set BASETEN_API_KEY environment variable. 5. Push to development: Run truss push --watch to create a development deployment with live reload. 6. Test and iterate: Edit files, save, and watch logs update in seconds. Use truss watch to re-attach if disconnected. 7. Deploy to production: Run truss push to create an immutable production deployment. 8. Promote to environment: Use the Baseten UI or API to promote the deployment to the production environment for a stable endpoint.

Call a deployed model

1. Get model ID: Find it in the Baseten dashboard or from truss push output. 2. Set API key: Export BASETEN_API_KEY environment variable. 3. Make a request: Use the /production/predict endpoint with Bearer token authentication. 4. Handle streaming (optional): Set stream: true in the request body to receive tokens incrementally.

Run async inference

1. Set up a webhook: Deploy an HTTPS endpoint to receive results. 2. Submit async request: POST to /production/async_predict with model_input and webhook_endpoint. 3. Receive request ID: Store it for status polling. 4. Check status (optional): Poll /async_request/{id} to track progress. 5. Receive results: Baseten POSTs the output to your webhook when complete.

Configure autoscaling

1. Set autoscaling parameters in config.yaml or via the Baseten UI:

• min_replica: Minimum running instances (0 = scale to zero).
• max_replica: Maximum instances (cost ceiling).
• concurrency_target: Requests per replica before scaling.
• target_utilization_percentage: Headroom before scaling (default 70%).
• scale_down_delay: Wait time before removing idle replicas (default 900s).

2. Monitor metrics: Check the Metrics tab in the dashboard for replica count, queue depth, and latency. 3. Adjust based on traffic: Increase max_replica for higher peak load; increase min_replica to keep replicas warm.

Common gotchas

• API key exposure: Never commit BASETEN_API_KEY to version control. Use environment variables or secrets management.
• Model load timeout: The load() method has a 30-minute timeout. If it exceeds this, the deployment fails. Optimize weight downloads with BDN (Baseten Delivery Network) instead of bundling large files.
• Hot reload limitations: --watch-hot-reload doesn't re-run __init__() or load(). If you add new instance state, do a full reload with truss push --watch.
• Bundled data size: Keep data/ directory under ~1 GB. Larger bundles slow down every cold start. Use BDN for weights over 1 GB.
• Async webhook loss: If webhook delivery fails after retries, outputs are lost. Save results in postprocess() to cloud storage as a backup.
• Memory pressure: OOMKilled errors mean the container ran out of memory. Reduce batch sizes, payload sizes, or move to a larger instance type.
• Development deployment limits: Single replica, no gRPC, no TRT-LLM builds. Use truss push (not --watch) for production features.
• Concurrency vs. async: predict_concurrency limits sync requests per replica. Async requests queue separately but share capacity; sync takes priority.
• Cold start delays: Models with long load times may expire async requests if max_time_in_queue_seconds is too short. Set it to account for startup time.
• Config changes requiring redeploy: GPU type, Python version, system packages, and data/ directory changes require a full truss push, not live patching.

Verification checklist

Before submitting a deployment:

• [ ] config.yaml has valid YAML syntax and required fields (model_name, resources.accelerator).
• [ ] model/model.py (if present) has __init__, load, and predict methods.
• [ ] load() completes within 30 minutes (test locally with truss container run).
• [ ] predict() returns JSON-serializable output (dict, list, str, or Pydantic model).
• [ ] All secrets are defined in config.yaml with null values; actual values are set in Baseten UI.
• [ ] requirements.txt or requirements list all Python dependencies.
• [ ] Test the deployment with a sample request: curl or Python requests to /production/predict.
• [ ] Autoscaling settings match your traffic pattern (min/max replicas, concurrency target).
• [ ] Logs show no errors: check the Logs tab in the Baseten dashboard.
• [ ] For async workflows, webhook endpoint is HTTPS and responds with 2xx status.

Resources

• Comprehensive navigation: https://docs.baseten.co/llms.txt
• Quickstart: https://docs.baseten.co/quickstart (run inference in under 2 minutes)
• Deploy your first model: https://docs.baseten.co/development/model/build-your-first-model (config-only deployment)
• Truss configuration reference: https://docs.baseten.co/reference/truss-configuration (all config.yaml options)
• Inference API reference: https://docs.baseten.co/reference/inference-api/overview (predict, async_predict, status endpoints)
• Autoscaling guide: https://docs.baseten.co/deployment/autoscaling/overview (replica scaling, concurrency, cold starts)

---

For additional documentation and navigation, see: https://docs.baseten.co/llms.txt