Databricks Model Serving

Model Serving Endpoints

FIRST: Use the parent databricks-core skill for CLI basics, authentication, and profile selection.

Model Serving provides managed endpoints for serving LLMs, custom ML models, and external models as scalable REST APIs. Endpoints are identified by name (unique per workspace).

Endpoint Types

Endpoint Structure

Serving Endpoint (top-level, identified by NAME)
  ├── Config
  │     ├── Served Entities (model references + scaling config)
  │     └── Traffic Config (routing percentages across entities)
  ├── AI Gateway (rate limits, usage tracking)
  └── State (READY / NOT_READY, config_update status)

• Served Entities: Each entity references a model (from Unity Catalog or MLflow) with scaling parameters. Get the entity name from served_entities[].name in the get output — needed for build-logs and logs commands.
• Traffic Config: Routes requests across served entities by percentage (for A/B testing, canary deployments).
• State: Endpoints transition NOT_READY → READY after creation or config update. Poll via get to check state.ready.

CLI Discovery — ALWAYS Do This First

Do NOT guess command syntax. Discover available commands and their usage dynamically:

# List all serving-endpoints subcommands
databricks serving-endpoints -h

# Get detailed usage for any subcommand (flags, args, JSON fields)
databricks serving-endpoints <subcommand> -h

Run databricks serving-endpoints -h before constructing any command. Run databricks serving-endpoints <subcommand> -h to discover exact flags, positional arguments, and JSON spec fields for that subcommand.

Create an Endpoint

Do NOT list endpoints before creating.

databricks serving-endpoints create <ENDPOINT_NAME> \
  --json '{
    "served_entities": [{
      "entity_name": "<MODEL_CATALOG_PATH>",
      "entity_version": "<VERSION>",
      "min_provisioned_throughput": 0,
      "max_provisioned_throughput": 0,
      "workload_size": "Small",
      "scale_to_zero_enabled": true
    }],
    "traffic_config": {
      "routes": [{
        "served_entity_name": "<ENTITY_NAME>",
        "traffic_percentage": 100
      }]
    }
  }' --profile <PROFILE>

• Discover available Foundation Models: see Foundation Model API endpoints below for the runtime-list snippet and default-picking rules. You can also check the system.ai catalog in Unity Catalog, or run databricks serving-endpoints list --profile <PROFILE> to see what's deployed in the workspace. Use databricks serving-endpoints get-open-api <ENDPOINT_NAME> --profile <PROFILE> to inspect a specific endpoint's API schema.
• Long-running operation; the CLI waits for completion by default. Use --no-wait to return immediately, then poll:

  databricks serving-endpoints get <ENDPOINT_NAME> --profile <PROFILE>
  # Check: state.ready == "READY"

• For provisioned throughput or custom model endpoints, run databricks serving-endpoints create -h to discover the required JSON fields for your endpoint type.

MLflow Deployments client (Python alternative)

mlflow.deployments.get_deploy_client("databricks").create_endpoint(name=..., config={...}) takes the same JSON shape as the CLI. Two gotchas:

• `tags=` is a top-level kwarg, NOT a field inside config. Same [{key, value}] shape as serving-endpoints patch --add-tags.
• `traffic_config.routes[].served_model_name` = `"<model>-<version>"` (e.g. "turbine_failure-3"). The API auto-derives this from the entity, but you reference the exact string in traffic_config — get the format wrong and the route silently doesn't match.

Zero-downtime version swap

To roll an endpoint to a new model version: repoint the alias and call update_endpoint with the new served_entities + matching traffic_config. Missing either half is the common bug — alias-only doesn't update the endpoint; update_endpoint-only leaves the alias pointing at the old version.

from mlflow.tracking import MlflowClient
from mlflow.deployments import get_deploy_client

registry = MlflowClient(registry_uri="databricks-uc")
deploy   = get_deploy_client("databricks")

registry.set_registered_model_alias(FULL_NAME, "prod", new_version)
deploy.update_endpoint(endpoint=ENDPOINT_NAME, config={
    "served_entities": [{"entity_name": FULL_NAME, "entity_version": new_version,
                         "workload_size": "Small", "scale_to_zero_enabled": True}],
    "traffic_config": {"routes": [
        {"served_model_name": f"{NAME}-{new_version}", "traffic_percentage": 100}
    ]},
})

The CLI equivalent is databricks serving-endpoints update-config <NAME> --json '...'. Either way, poll both state.ready and state.config_update afterward — see Endpoint Readiness below.

Endpoint Readiness

After create or update-config, the endpoint provisions compute and loads the model. Do not query the endpoint until it is ready. Two state fields matter and they mean different things:

• state.ready — READY once the endpoint has any working config. Stays READY during a version swap.
• state.config_update — NOT_UPDATING once the current config update finishes; IN_PROGRESS during a version swap.

A loop watching only state.ready will say "ready" mid version-swap while the old version is still serving. Poll both:

databricks serving-endpoints get <ENDPOINT_NAME> --profile <PROFILE> \
  | jq '{ready: .state.ready, config_update: .state.config_update}'
# Fully ready when ready == "READY" AND config_update == "NOT_UPDATING"

Provisioning may take several minutes. Provisioned throughput endpoints take the longest (GPU allocation). Queries to endpoints that are not yet READY return 404 or 503.

Query an Endpoint

Chat / agent endpoints use the messages array:

databricks serving-endpoints query <ENDPOINT_NAME> \
  --json '{"messages": [{"role": "user", "content": "Hello"}]}' --profile <PROFILE>

Classical-ML endpoints use dataframe_records (one record per row):

databricks serving-endpoints query <ENDPOINT_NAME> \
  --json '{"dataframe_records": [{"vibration": 0.42, "rpm": 18.3, "temp_c": 71.2}]}'

• Use --stream for streaming responses on chat endpoints.
• For embeddings or other custom schemas: use get-open-api <ENDPOINT_NAME> first to discover the request/response shape.

Get Endpoint Schema (OpenAPI)

Returns the OpenAPI 3.1 JSON schema describing what each served model accepts and returns. Use this to understand an endpoint's input/output format before querying it.

databricks serving-endpoints get-open-api <ENDPOINT_NAME> --profile <PROFILE>

The schema shows paths per served model (e.g., /served-models/<model-name>/invocations) with full request/response definitions including parameter types, enums, and nullable fields.

Other Commands

Run databricks serving-endpoints <subcommand> -h for usage details.

What's Next

Integrate with a Databricks App

After creating a serving endpoint, wire it into a Databricks App.

Step 1 — Check if the `serving` plugin is available in the AppKit template:

databricks apps manifest --profile <PROFILE>

If the output includes a serving plugin, scaffold with:

databricks apps init --name <APP_NAME> \
  --features serving \
  --set "serving.serving-endpoint.name=<ENDPOINT_NAME>" \
  --run none --profile <PROFILE>

Step 2 — If no `serving` plugin, add the endpoint resource manually to an existing app's databricks.yml:

resources:
  apps:
    my_app:
      resources:
        - name: my-model-endpoint
          serving_endpoint:
            name: <ENDPOINT_NAME>
            permission: CAN_QUERY

And inject the endpoint name as an environment variable in app.yaml:

env:
  - name: SERVING_ENDPOINT
    valueFrom: serving-endpoint

Then wire the endpoint into your app via the serving() plugin or a custom route in onPluginsReady. For the full app integration pattern, use the `databricks-apps` skill and read the Model Serving Guide.

Develop & deploy new models

This skill is ops-focused (manage existing endpoints). For the dev-side flow — training, MLflow tracking, UC registration, custom PyFunc authoring, and hand-rolled ResponsesAgent code — see [databricks-ml-training](../databricks-ml-training/SKILL.md) (experimental).

Foundation Model API endpoints

Pay-per-token, pre-provisioned in every workspace. New models land regularly and a static skill list goes stale fast — always list at runtime instead of hard-coding names. Filter by the databricks- name prefix AND by the served entity being in system.ai.* (other endpoints like databricks-app-template-serving share the prefix but aren't FM API endpoints).

# FM API endpoints in this workspace, grouped by task (chat / embeddings / etc.)
databricks serving-endpoints list \
  | jq -r '.[]
      | select(.name | startswith("databricks-"))
      | select((.config.served_entities[0].entity_name // "") | startswith("system.ai."))
      | "\(.task)\t\(.name)"' \
  | sort

Defaults when the user doesn't specify: pick the highest-numbered Claude Sonnet for agents, the highest-numbered -codex-max for code, databricks-gte-large-en for embeddings — resolve actual names from the live list above.

Off-platform streaming

For apps deployed outside Databricks Apps (Vercel, AWS, standalone Node.js) hitting Databricks AI Gateway with Vercel AI SDK v6, see references/off-platform-streaming.md. For AppKit-based apps, use the databricks-apps skill's built-in serving plugin instead.

Troubleshooting

Off-Platform Streaming with AI SDK v6

These patterns are for apps deployed outside Databricks Apps (e.g., Vercel, AWS, standalone Node.js servers) using direct AI SDK v6 integration with Databricks AI Gateway. For AppKit-based apps, use the `databricks-apps` skill's built-in serving plugin instead.

AI SDK v6 Streaming Pattern

Use this pattern for streaming AI chat with Databricks AI Gateway and Vercel AI SDK v6 in off-platform apps.

Dependencies: ai@6, @ai-sdk/react@3, @ai-sdk/openai, @databricks/sdk-experimental

Auth helper — works for both local dev (CLI profile) and deployed apps (service principal token):

import { Config } from "@databricks/sdk-experimental";

async function getDatabricksToken() {
  if (process.env.DATABRICKS_TOKEN) {
    return process.env.DATABRICKS_TOKEN;
  }
  const config = new Config({
    profile: process.env.DATABRICKS_CONFIG_PROFILE || "DEFAULT",
  });
  await config.ensureResolved();
  const headers = new Headers();
  await config.authenticate(headers);
  const authHeader = headers.get("Authorization");
  if (!authHeader) {
    throw new Error(
      "Failed to get Databricks token. Check your CLI profile or set DATABRICKS_TOKEN.",
    );
  }
  return authHeader.replace("Bearer ", "");
}

Server route (POST /api/chat):

import { createOpenAI } from "@ai-sdk/openai";
import { streamText, type UIMessage } from "ai";

app.post("/api/chat", async (req, res) => {
  const { messages } = req.body;

  // AI SDK v6 client sends UIMessage objects with a parts array.
  // Convert to CoreMessage format for streamText().
  const coreMessages = (messages as UIMessage[]).map((m) => ({
    role: m.role as "user" | "assistant" | "system",
    content:
      m.parts
        ?.filter((p) => p.type === "text" && p.text)
        .map((p) => p.text)
        .join("") ??
      m.content ??
      "",
  }));

  try {
    const token = await getDatabricksToken();
    const endpoint = process.env.DATABRICKS_ENDPOINT || "<ENDPOINT_NAME>";

    // AI Gateway URL uses /mlflow/v1 path, NOT /openai/v1
    // URL varies by cloud: .cloud.databricks.com (AWS), .azuredatabricks.net (Azure), .gcp.databricks.com (GCP)
    const databricks = createOpenAI({
      baseURL: `https://${process.env.DATABRICKS_WORKSPACE_ID}.ai-gateway.cloud.databricks.com/mlflow/v1`,
      apiKey: token,
    });

    const result = streamText({
      model: databricks.chat(endpoint),
      messages: coreMessages,
      maxOutputTokens: 1000,
    });

    result.pipeTextStreamToResponse(res);
  } catch (err) {
    const message = (err as Error).message;
    console.error(`[chat] Streaming request failed:`, message);
    res.status(502).json({ error: "Chat request failed", detail: message });
  }
});

Environment variables:

• DATABRICKS_WORKSPACE_ID — for explicit setup: databricks api get /api/2.1/unity-catalog/current-metastore-assignment --profile <PROFILE> → workspace_id field
• DATABRICKS_ENDPOINT — model endpoint name (e.g. databricks-meta-llama-3-3-70b-instruct). Run databricks serving-endpoints list --profile <PROFILE> to see available models.

Streaming Client Pattern (AI SDK v6)

import { useChat } from "@ai-sdk/react";
import { TextStreamChatTransport } from "ai";
import { useState } from "react";

export function ChatPage() {
  const [input, setInput] = useState("");

  const { messages, sendMessage, status } = useChat({
    transport: new TextStreamChatTransport({ api: "/api/chat" }),
  });

  return (
    <div className="flex flex-col h-full">
      <div className="flex-1 overflow-y-auto space-y-4 p-4">
        {messages.map((m) => (
          <div key={m.id} className={m.role === "user" ? "text-right" : ""}>
            <span className="text-sm font-medium">
              {m.role === "user" ? "You" : "Assistant"}
            </span>
            {m.parts.map((part, i) =>
              part.type === "text" ? (
                <p key={`${m.id}-${i}`} className="whitespace-pre-wrap">
                  {part.text}
                </p>
              ) : null,
            )}
          </div>
        ))}
        {status === "submitted" && <div className="p-4">Loading...</div>}
      </div>
      <form
        onSubmit={(e) => {
          e.preventDefault();
          if (input.trim()) {
            void sendMessage({ text: input });
            setInput("");
          }
        }}
        className="border-t p-4 flex gap-2"
      >
        <input
          value={input}
          onChange={(e) => setInput(e.target.value)}
          placeholder="Ask a question..."
          className="flex-1 border rounded px-3 py-2"
          disabled={status !== "ready"}
        />
        <button type="submit" disabled={status !== "ready"}>
          {status === "submitted" || status === "streaming"
            ? "Sending..."
            : "Send"}
        </button>
      </form>
    </div>
  );
}

Key differences from AI SDK v5: use sendMessage({ text }) (NOT append), render m.parts array (NOT m.content), and status states are ready, submitted, streaming.

Embeddings Pattern

Generate text embeddings using a Databricks AI Gateway endpoint.

import { WorkspaceClient } from "@databricks/sdk-experimental";

const workspaceClient = new WorkspaceClient({
  host: process.env.DATABRICKS_HOST,
});

export async function generateEmbedding(text: string): Promise<number[]> {
  const endpoint =
    process.env.DATABRICKS_EMBEDDING_ENDPOINT || "databricks-gte-large-en";
  const result = await workspaceClient.servingEndpoints.query({
    name: endpoint,
    input: text,
  });
  return result.data![0].embedding!;
}

Common embedding endpoints: databricks-gte-large-en (1024d), databricks-bge-large-en (1024d). Set DATABRICKS_EMBEDDING_ENDPOINT in .env and app.yaml.

For vector similarity search with these embeddings, see the `databricks-lakebase` skill.

Troubleshooting