Databricks Apps

Databricks Apps Development

FIRST: Use the parent databricks-core skill for CLI basics, authentication, and profile selection.

For data UI design (required for any data-displaying app): if the app shows ANY data — a dashboard, KPI/overview page, report, chart, table, query results, OR a conversational / chat / Genie natural-language assistant — you MUST use the databricks-app-design skill (alongside this one) to decide layout, charts, KPIs, semantic color, required states, and AI-result trust, and map them to AppKit components. This includes chat/Genie apps, not just dashboards — if in doubt, use it.

Build apps that deploy to Databricks Apps platform.

Required Reading by Phase

Generic Guidelines

• App name: ≤26 characters, lowercase letters/numbers/hyphens only (no underscores). dev- prefix adds 4 chars, max 30 total.
• Validation: databricks apps validate --profile <PROFILE> before deploying.
• Smoke tests (AppKit only): ALWAYS update tests/smoke.spec.ts selectors BEFORE running validation. Default template checks for "Minimal Databricks App" heading and "hello world" text — these WILL fail in your custom app. See testing guide.
• Smoke test selectors: use only Playwright locator APIs — getByRole, getByText, getByPlaceholder, getByLabel. getByLabelText does not exist in Playwright (it is a React Testing Library method) and throws TypeError at runtime. See testing guide or npx playwright codegen.
• Smoke test data: keep result sets under the 1 MB analytics-event payload cap. Queries returning thousands of rows cause INVALID_REQUEST: Event exceeds max size of 1048576 bytes and net::ERR_ABORTED, leaving every asserted UI element absent. Use LIMIT or an aggregated query (e.g. COUNT(*) GROUP BY status) — never raw row dumps.
• AppKit version: never override the @databricks/appkit or @databricks/appkit-ui version in package.json — databricks apps init sets the correct version. Do not run npm install @databricks/appkit@<version> unless explicitly asked by the user. If you need a different version, re-scaffold with databricks apps init --version <version>.
• Authentication: covered by parent databricks-core skill.
• AppKit API surface: before writing code that calls AppKit APIs (createApp, plugin shapes, useAnalyticsQuery, etc.), run npx @databricks/appkit docs <section> and use the actual signature. Training data has stale shapes; a single invented signature fails tsc --noEmit during validate. The docs ship with the installed AppKit and are the authoritative source.
• TypeScript casts: never use as unknown as <T> double-assertions — appkit lint enforces no-double-type-assertion and one violation fails the entire validate step. Instead: narrow with Zod (z.infer<typeof schema>), use a runtime type guard, or write a typed mapper function. If a query result needs reshaping, type the row schema via queryKey types rather than casting.

Project Structure (after databricks apps init --features analytics)

• client/src/App.tsx — main React component (start here)
• config/queries/*.sql — SQL query files (queryKey = filename without .sql)
• server/server.ts — backend entry (onPluginsReady + Express routes)
• tests/smoke.spec.ts — smoke test (⚠️ MUST UPDATE selectors for your app)
• client/src/appKitTypes.d.ts — auto-generated types (npm run typegen)

Project Structure (after databricks apps init --features lakebase)

• server/server.ts — backend with Lakebase pool + Express routes
• client/src/App.tsx — React frontend
• app.yaml — manifest with database resource declaration
• package.json — includes @databricks/lakebase dependency
• Note: No `config/queries/` — Lakebase apps use appkit.lakebase.query() in Express routes, not SQL files

Data Discovery

Before writing any SQL, use the parent databricks-core skill for data exploration — search information_schema by keyword, then batch discover-schema for the tables you need. Do NOT skip this step.

State Storage Guidance (evaluate BEFORE the Decision Gate):

If the user's app description involves storing or persisting data — forms, CRUD operations, user submissions, orders, todos, or other user-generated content — the app likely needs a Lakebase database.

1. Ask the user whether the app needs persistent storage (Lakebase) before scaffolding. Do not silently add Lakebase. 2. If confirmed, use the `databricks-lakebase` skill to create a Lakebase project and obtain the branch and database resource names. 3. Scaffold with --features lakebase and pass --set lakebase.postgres.branch=<BRANCH_NAME> --set lakebase.postgres.database=<DATABASE_NAME>. 4. If the app also reads from Unity Catalog tables, proceed to the Data Access Decision Gate below to determine whether to add --features analytics or use Lakebase synced tables.

Do NOT add Lakebase to analytics, dashboard, or visualization apps unless the user explicitly requests persistent write-back storage. Read-only data display, filters, and preferences do not require a database.

Development Workflow (FOLLOW THIS ORDER)

Data Access Decision Gate (REQUIRED before scaffolding):

If the app reads from Unity Catalog / lakehouse tables, you MUST show the comparison below to the user and ask them to choose. Do not skip this. Do not choose for them.

After showing the table, add a brief recommendation. Default to recommending Analytics (B) for most read-only apps — dashboards, charts, filtered queries, browsing, and aggregations. Recommend Lakebase synced tables (A) only when the app needs sub-second latency for full-text search, typeahead/autocomplete, real-time lookups by ID, or operational data serving. Note: "search" or "filter" in a prompt usually means SQL WHERE clauses (Analytics), not full-text search (Lakebase). Always let the user make the final call.

After the user chooses:

• (A) Lakebase synced tables → scaffold with --features lakebase. See Lakebase Guide for full workflow.
• (B) Analytics → scaffold with --features analytics.
• Both → scaffold with --features analytics,lakebase if the app needs both patterns.
• If the app does NOT read Unity Catalog data (pure CRUD, Genie, Model Serving), skip this gate and scaffold with the appropriate --features flag.

Analytics apps (--features analytics):

1. Create SQL files in config/queries/ 2. Run npm run typegen — verify all queries show ✓ 3. Read client/src/appKitTypes.d.ts to see generated types 4. THEN write App.tsx using the generated types 5. Update tests/smoke.spec.ts selectors 6. Run databricks apps validate --profile <PROFILE>

DO NOT write UI code before running typegen — types won't exist and you'll waste time on compilation errors.

Lakebase apps (--features lakebase): No SQL files or typegen. See Lakebase Guide for the onPluginsReady pattern: initialize schema at startup, register Express routes in server/server.ts, then build the React frontend.

When to Use What

After completing the decision gate above, use this routing table:

• Read analytics data → display in chart/table: Use visualization components with queryKey prop
• Read analytics data → custom display (KPIs, cards): Use useAnalyticsQuery hook
• Read analytics data → need computation before display: Still use useAnalyticsQuery, transform client-side
• Read lakehouse data at low latency (lookups, search, catalogs): Use Lakebase synced tables — see Lakebase Guide
• Read/write persistent data (users, orders, CRUD state): Use Lakebase via Express routes in onPluginsReady — see Lakebase Guide
• Natural language query interface over tables (Genie): Use genie() plugin — see Genie Guide
• Call ML model endpoint: Use serving() plugin — see Model Serving Guide
• Trigger or monitor a Lakeflow Job from the app: Use the jobs() plugin — see Jobs Guide
• ⚠️ NEVER add custom endpoints to run SELECT queries against the warehouse — always use SQL files in config/queries/
• ⚠️ NEVER use `useAnalyticsQuery` for Lakebase data — it queries the SQL warehouse only

Frameworks

AppKit (Recommended)

TypeScript/React framework with type-safe SQL queries and built-in components.

Official Documentation — the source of truth for all API details:

npx @databricks/appkit docs                              # ← ALWAYS start here to see available pages
npx @databricks/appkit docs <query>                      # view a section by name or doc path
npx @databricks/appkit docs --full                       # full index with all API entries
npx @databricks/appkit docs "appkit-ui API reference"    # example: section by name
npx @databricks/appkit docs ./docs/plugins/analytics.md  # example: specific doc file

DO NOT guess doc paths. Run without args first, pick from the index. The <query> argument accepts both section names (from the index) and file paths. Docs are the authority on component props, hook signatures, and server APIs — skill files only cover anti-patterns and gotchas.

App Manifest and Scaffolding

Agent workflow for scaffolding: get the manifest first, then build the init command.

1. Get the manifest (JSON schema describing plugins and their resources):

   databricks apps manifest --profile <PROFILE>
   # See plugins available in a specific AppKit version:
   databricks apps manifest --version <VERSION> --profile <PROFILE>
   # Custom template:
   databricks apps manifest --template <GIT_URL> --profile <PROFILE>

The output defines:

• Plugins: each has a key (plugin ID for --features), plus requiredByTemplate, and resources.
• requiredByTemplate: If true, that plugin is mandatory for this template — do not add it to --features (it is included automatically); you must still supply all of its required resources via --set. If false or absent, the plugin is optional — add it to --features only when the user's prompt indicates they want that capability (e.g. analytics/SQL), and then supply its required resources via --set.
• Resources: Each plugin has resources.required and resources.optional (arrays). Each item has resourceKey and fields (object: field name → description/env). Use --set <plugin>.<resourceKey>.<field>=<value> for each required resource field of every plugin you include.

2. Scaffold (DO NOT use npx; use the CLI only):

   databricks apps init --name <NAME> --features <plugin1>,<plugin2> \
     --set <plugin1>.<resourceKey>.<field>=<value> \
     --set <plugin2>.<resourceKey>.<field>=<value> \
     --description "<DESC>" --run none --profile <PROFILE>
   # --run none: skip auto-run after scaffolding (review code first)
   # With custom template:
   databricks apps init --template <GIT_URL> --name <NAME> --features ... --set ... --profile <PROFILE>

Optionally use --version <VERSION> to target a specific AppKit version.

• Required: --name, --profile. Name: ≤26 chars, lowercase letters/numbers/hyphens only. Use --features only for optional plugins the user wants (plugins with requiredByTemplate: false or absent); mandatory plugins must not be listed in --features.
• Resources: Pass --set for every required resource (each field in resources.required) for (1) all plugins with requiredByTemplate: true, and (2) any optional plugins you added to --features. Add --set for resources.optional only when the user requests them.
• Discovery: Use the parent databricks-core skill to resolve IDs (e.g. warehouse: databricks warehouses list --profile <PROFILE> or databricks experimental aitools tools get-default-warehouse --profile <PROFILE>).

DO NOT guess plugin names, resource keys, or property names — always derive them from databricks apps manifest output. Example: if the manifest shows plugin analytics with a required resource resourceKey: "sql-warehouse" and fields: { "id": ... }, include --set analytics.sql-warehouse.id=<ID>.

Scaffolding Rules Protocol — databricks apps manifest may emit scaffolding.rules at the template level (top-level scaffolding.rules) and on individual plugins (plugins[].scaffolding.rules). Each block has must / should / never arrays of short directive strings. Consume them as follows:

1. Gather — for every plugin in your final --features list AND every plugin with requiredByTemplate: true, read plugins[].scaffolding.rules. Union those with the top-level template scaffolding.rules into one working set, tagged by source (template vs <plugin>). 2. Precedence — manifest rules override the directives baked into this skill. Where the manifest is silent on a topic, this skill's content is the floor. 3. Phase ordering — rules whose text begins with Before init MUST be executed before databricks apps init. Rules beginning with After init MUST be executed after init completes (e.g. migrations, typegen, connectivity checks). Rules without a phase prefix apply throughout the scaffold/develop loop. 4. Conflict detection — if a plugin must rule contradicts a template never rule on the same target (or vice versa), STOP and ask the user which to follow before proceeding. Do not silently pick one. Treat must vs never on the same action as a conflict; should is advisory and does not block. 5. Reporting — before running databricks apps init, surface the merged working set to the user grouped by phase (Before init / After init / Always) and by severity (must / should / never), so the active guardrails are explicit.

READ [AppKit Overview](references/appkit/overview.md) for project structure, workflow, and pre-implementation checklist.

Genie Agent Workflow — when the user wants a Genie-powered app, do not start by asking for a Genie Space ID. Instead:

1. Ask which Unity Catalog tables the app should query (fully qualified: catalog.schema.table). 2. Ask whether to reuse an existing Genie space or create a new one. 3. If creating: discover the warehouse, then create the space with databricks genie create-space (see Genie Guide for syntax and serialized space format). 4. If reusing: discover existing spaces with databricks genie list-spaces --profile <PROFILE> and let the user pick. 5. Scaffold or wire the space ID into the app — derive --set keys from databricks apps manifest.

Read the Genie Guide for configuration, SSE endpoints, and frontend integration.

Common Scaffolding Mistakes

# ❌ WRONG: name is NOT a positional argument
databricks apps init --features analytics my-app-name
# → "unknown command" error

# ✅ CORRECT: use --name flag
databricks apps init --name my-app-name --features analytics --set "..." --profile <PROFILE>

Directory Naming

databricks apps init creates directories in kebab-case matching the app name. App names must be lowercase with hyphens only (≤26 chars).

Other Frameworks (Streamlit, FastAPI, Flask, Gradio, Dash, Next.js, etc.)

Databricks Apps supports any framework that runs as an HTTP server. LLMs already know these frameworks — the challenge is Databricks platform integration.

READ [Other Frameworks Guide](references/other-frameworks.md) BEFORE building any non-AppKit app. It covers port/host configuration, app.yaml and databricks.yml setup, dependency management, networking, and framework-specific gotchas.

Post-Deploy Verification

After deploying, verify the app is running:

databricks apps get <app-name> --profile <PROFILE> -o json   # Check app_status.state: RUNNING; the `url` field is the app's URL
databricks apps logs <app-name> --follow --profile <PROFILE>  # Stream live logs (Ctrl+C to stop)

Note: databricks apps logs requires OAuth authentication and does not work with PAT. Use databricks apps get for status checks if using PAT auth.

Databricks App Kit SDK

TypeScript Import Rules

This template uses strict TypeScript settings with verbatimModuleSyntax: true. Always use `import type` for type-only imports.

Template enforces noUnusedLocals - remove unused imports immediately or build fails.

// ✅ CORRECT - use import type for types
import type { MyInterface, MyType } from './types';

// ❌ WRONG - will fail compilation
import { MyInterface, MyType } from './types';

Server Setup

For server configuration, see: npx @databricks/appkit docs ./docs/plugins.md

useAnalyticsQuery Hook

ONLY use when displaying data in a custom way that isn't a chart or table. For charts/tables, pass queryKey directly to the component — don't double-fetch. Charts also accept a format option ("json" | "arrow" | "auto", default "auto") to control the data transfer format.

Use cases:

• Custom HTML layouts (cards, lists, grids)
• Summary statistics and KPIs
• Conditional rendering based on data values
• Data that needs transformation before display

⚠️ Memoize Parameters to Prevent Infinite Loops

// ❌ WRONG - creates new object every render → infinite refetch loop
const { data } = useAnalyticsQuery('query', { id: sql.string(selectedId) });

// ✅ CORRECT - memoize parameters
const params = useMemo(() => ({ id: sql.string(selectedId) }), [selectedId]);
const { data } = useAnalyticsQuery('query', params);

Conditional Queries

// ❌ WRONG - `enabled` is NOT a valid option (this is a React Query pattern)
const { data } = useAnalyticsQuery('query', params, { enabled: !!selectedId });

// ✅ CORRECT - use autoStart: false
const { data } = useAnalyticsQuery('query', params, { autoStart: false });

// ✅ ALSO CORRECT - conditional rendering (component only mounts when data exists)
{selectedId && <DetailsComponent id={selectedId} />}

Type Inference

When appKitTypes.d.ts has been generated (via npm run typegen), types are inferred automatically:

// ✅ After typegen - types are automatic, no generic needed
const { data } = useAnalyticsQuery('my_query', params);

// ⚠️ Before typegen - data is `unknown`, you must provide type manually
const { data } = useAnalyticsQuery<MyRow[]>('my_query', params);

Common mistake — don't define interfaces that duplicate generated types:

// ❌ WRONG - manual interface may conflict with generated QueryRegistry
interface MyData { id: string; value: number; }
const { data } = useAnalyticsQuery<MyData[]>('my_query', params);

// ✅ CORRECT - run `npm run typegen` and let it provide types
const { data } = useAnalyticsQuery('my_query', params);

Basic Usage

import { useAnalyticsQuery, Skeleton } from '@databricks/appkit-ui/react';
import { sql } from '@databricks/appkit-ui/js';
import { useMemo } from 'react';

function CustomDisplay() {
  const params = useMemo(() => ({
    start_date: sql.date('2024-01-01'),
    category: sql.string("tools")
  }), []);

  const { data, loading, error } = useAnalyticsQuery('query_name', params);

  if (loading) return <Skeleton className="h-4 w-3/4" />;
  if (error) return <div className="text-destructive">Error: {error}</div>;
  if (!data) return null;

  return (
    <div className="grid gap-4">
      {data.map(row => (
        <div key={row.column_name} className="p-4 border rounded">
          <h3>{row.column_name}</h3>
          <p>{Number(row.value).toFixed(2)}</p>
        </div>
      ))}
    </div>
  );
}

Custom API Endpoints

CRITICAL: Do NOT add custom endpoints for SQL queries or warehouse data retrieval. Use config/queries/ + useAnalyticsQuery instead.

CRITICAL: Do NOT add custom endpoints for Unity Catalog file operations. Use the Files plugin instead.

When you need server-side logic that no plugin covers, extend the AppKit server in onPluginsReady and register Express routes with appkit.server.extend().

Use custom endpoints ONLY for:

• Mutations: Creating, updating, or deleting data (INSERT, UPDATE, DELETE)
• External APIs: Calling Databricks APIs not covered by a dedicated plugin (MLflow, Workspace API, etc.)
• Complex business logic: Multi-step operations that cannot be expressed in SQL
• File processing: Uploads, processing, transformations (when not covered by the Files plugin)
• Custom computations: Operations requiring TypeScript/Node.js logic

Before Adding Endpoints

ALWAYS complete these checks before registering routes:

1. Check AppKit Version

Read package.json to identify the installed @databricks/appkit version. Available server APIs and plugins differ across versions.

# From the project root
cat package.json | grep @databricks/appkit

2. Review Available Plugins

Check what plugins are already enabled and what server-side functionality they provide — avoid reimplementing what a plugin already handles.

# See plugin docs for the installed version
npx @databricks/appkit docs ./docs/plugins.md

# See all plugins available for a specific version
databricks apps manifest --version <VERSION> --profile <PROFILE>

# See plugins available for the default template
databricks apps manifest --profile <PROFILE>

Key plugins to check for:

• analytics — provides SQL warehouse query execution (do NOT reimplement with custom endpoints)
• lakebase — provides Lakebase plugin for PostgreSQL CRUD (use plugin in routes, don't create raw connections)
• genie — provides Genie AI-powered data exploration (check before building custom natural-language-to-SQL routes)
• files — provides file storage and retrieval helpers (check before writing custom file upload/download routes)
• serving — provides model serving endpoint proxy with invoke/stream (do NOT reimplement with custom endpoints)
• jobs — provides Lakeflow Job triggering and monitoring (do NOT reimplement with custom endpoints)

If a plugin already covers your use case, use the plugin's API instead of writing a custom route.

If a newer version of @databricks/appkit has a plugin that fits the use case, prompt the user for updating.

3. Check Existing Routes

Read server/server.ts to see what routes already exist. Add new handlers inside the existing onPluginsReady callback rather than creating a parallel server setup.

Server-side Pattern

Register routes inside onPluginsReady so plugins are initialized before the server accepts requests:

// server/server.ts
import { createApp, server } from "@databricks/appkit";
import { getExecutionContext } from "@databricks/appkit";
import { z } from "zod";

await createApp({
  plugins: [server()],
  async onPluginsReady(appkit) {
    appkit.server.extend((app) => {
      // Example: Call a Databricks API (e.g. MLflow)
      app.get("/api/experiments/:experimentId", async (req, res) => {
        const { experimentId } = req.params;
        const { serviceDatabricksClient: client } = getExecutionContext();
        const response = await client.experiments.getExperiment({
          experiment_id: experimentId,
        });
        res.json(response);
      });

      // Example: Mutation
      app.post("/api/records", async (req, res) => {
        const parsed = z.object({ name: z.string() }).safeParse(req.body);
        if (!parsed.success) {
          res.status(400).json({ error: "Invalid input" });
          return;
        }
        // Custom logic here
        res.status(201).json({ success: true, id: 123 });
      });
    });
  },
});

For Lakebase CRUD routes, schema initialization, and chat persistence, see Lakebase Guide.

Client-side Pattern

Call your endpoints with fetch from React components:

// client/src/components/MyComponent.tsx
import { useState, useEffect } from "react";

function MyComponent() {
  const [result, setResult] = useState(null);

  useEffect(() => {
    fetch("/api/experiments/123")
      .then((r) => r.json())
      .then(setResult)
      .catch(console.error);
  }, []);

  const handleCreate = async () => {
    await fetch("/api/records", {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({ name: "test" }),
    });
  };

  return <div>{/* component JSX */}</div>;
}

Decision Tree for Data Operations

1. Need to display data from SQL?

• Chart or Table? → Use visualization components (BarChart, LineChart, DataTable, etc.)
• Custom display (KPIs, cards, lists)? → Use useAnalyticsQuery hook
• Never add custom endpoints for SQL SELECT statements against the warehouse

2. Need to call a Databricks API?

• Serving endpoints → use serving() plugin (see Model Serving Guide)
• Jobs → use jobs() plugin (see Jobs Guide)
• MLflow, Workspace API, other APIs → custom endpoint via onPluginsReady

3. Need to modify data? → Custom endpoint in onPluginsReady

• INSERT, UPDATE, DELETE operations
• Multi-step transactions
• Business logic with side effects

4. Need non-SQL custom logic? → Custom endpoint in onPluginsReady

• File processing
• External API calls
• Complex computations in TypeScript

Summary:

• ✅ SQL queries → Visualization components or useAnalyticsQuery
• ✅ Databricks APIs without a plugin → custom endpoint via onPluginsReady
• ✅ Data mutations → custom endpoint via onPluginsReady
• ❌ SQL warehouse queries → custom endpoints (NEVER do this)
• ❌ Files operations → custom endpoints (NEVER do this — use Files plugin)

Files: Unity Catalog Volume Operations

For full Files plugin API (routes, types, config options): run npx @databricks/appkit docs ./docs/plugins/files.md.

Use the files() plugin when your app needs to browse, upload, download, or manage files in Databricks Unity Catalog Volumes. For analytics dashboards reading from a SQL warehouse, use config/queries/ instead. For persistent CRUD storage, use Lakebase.

When to Use Files vs Other Patterns

Scaffolding

databricks apps init --name <NAME> --features files \
  --run none --profile <PROFILE>

Files + analytics:

databricks apps init --name <NAME> --features analytics,files \
  --set "analytics.sql-warehouse.id=<WAREHOUSE_ID>" \
  --run none --profile <PROFILE>

Configure volume paths via environment variables in app.yaml or .env:

DATABRICKS_VOLUME_UPLOADS=/Volumes/catalog/schema/uploads
DATABRICKS_VOLUME_EXPORTS=/Volumes/catalog/schema/exports

The env var suffix (after DATABRICKS_VOLUME_) becomes the volume key, lowercased.

Plugin Setup

import { createApp, files, server } from "@databricks/appkit";

await createApp({
  plugins: [server(), files()],
});

Configuration Overrides

Only add plugin config when you need to override defaults from the discovered DATABRICKS_VOLUME_* env vars:

files({
  maxUploadSize: 5_000_000_000, // plugin-level default
  volumes: {
    uploads: {
      maxUploadSize: 100_000_000,
      policy: files.policy.allowAll(), // required for writes
    },
    user_data: {
      auth: "on-behalf-of-user", // HTTP routes run SDK calls as end user
    },
  },
});

Auto-discovered volumes merge with explicit config, so volumes: {} is only needed for overrides. Check the AppKit docs for the current IFilesConfig / VolumeConfig shape.

Permission Model

Three layers gate file access:

1. Unity Catalog grants — service-principal volumes need the app SP to hold WRITE_VOLUME; OBO volumes need each end user to hold it. 2. Execution identity — HTTP routes use the volume's auth mode. Programmatic user-driven handlers must call .asUser(req) to run SDK calls as the request user. 3. File policies — app-level allow/deny functions evaluated before every operation.

For SP volumes, removing a user's UC grant has no effect on HTTP access because the SDK call uses the SP. Use policies for per-user restrictions. For OBO volumes, UC grants gate the end user and policies stack on top.

Access Policies

Volumes without an explicit policy default to files.policy.publicRead() (reads allowed, writes denied) and log a startup warning. Set an explicit policy on every volume that accepts uploads, directory creation, or deletes.

import { files } from "@databricks/appkit";

files({
  volumes: {
    public_data: { policy: files.policy.publicRead() },
    uploads: { policy: files.policy.allowAll() },
    archive: { policy: files.policy.denyAll() },
  },
});

Use custom policies when access depends on the requesting user or action. For exact built-ins, combinators, FileAction, FileResource, FilePolicyUser, and PolicyDeniedError behavior, check npx @databricks/appkit docs ./docs/plugins/files.md.

Server-Side API (Programmatic)

Access volumes through the files() callable, which returns a VolumeHandle. Direct programmatic calls do not have request headers available, so they normally run as the service principal. Use .asUser(req) in user-driven route handlers when the SDK call must run as the request user.

// User-driven handler: SDK call runs as user; policy sees user.id from req.
await appkit.files("uploads").asUser(req).list();

// Background or trusted server code: runs as SP.
await appkit.files("uploads").list();

Use `.asUser(req)` in user-driven route handlers when you want UC grants enforced against the actual user. In production, asUser(req) throws AuthenticationError.missingToken if the forwarded user or access-token header is missing; in dev (NODE_ENV === "development") it logs a warning and falls back to SP. Policy denial throws PolicyDeniedError.

For method signatures, path rules, cache behavior, and retry/timeout defaults, check npx @databricks/appkit docs ./docs/plugins/files.md.

HTTP Routes

Mounted at /api/files/*. Routes use the volume's auth mode and run the policy before the operation. Use the AppKit docs for the exact route list, request bodies, response types, and /raw content-security behavior.

Frontend Components

Import file browser components from @databricks/appkit-ui/react. Full component props: npx @databricks/appkit docs ./docs/api/appkit-ui/files/DirectoryList.md and the related component pages.

File Browser Example

import type { DirectoryEntry, FilePreview } from '@databricks/appkit-ui/react';
import {
  DirectoryList,
  FileBreadcrumb,
  FilePreviewPanel,
} from '@databricks/appkit-ui/react';
import { useCallback, useEffect, useState } from 'react';

export function FilesPage() {
  const [volumeKey] = useState('uploads');
  const [currentPath, setCurrentPath] = useState('');
  const [entries, setEntries] = useState<DirectoryEntry[]>([]);
  const [selectedFile, setSelectedFile] = useState<string | null>(null);
  const [preview, setPreview] = useState<FilePreview | null>(null);

  const apiUrl = useCallback(
    (action: string, params?: Record<string, string>) => {
      const base = `/api/files/${volumeKey}/${action}`;
      if (!params) return base;
      return `${base}?${new URLSearchParams(params).toString()}`;
    },
    [volumeKey],
  );

  const loadDirectory = useCallback(async (path?: string) => {
    const url = path ? apiUrl('list', { path }) : apiUrl('list');
    const res = await fetch(url);
    if (!res.ok) {
      const errBody = await res.json().catch(() => null);
      console.error('Failed to load directory', errBody ?? res.statusText);
      return;
    }
    const data: DirectoryEntry[] = await res.json();
    // Sort: directories first, then alphabetically
    data.sort((a, b) => {
      if (a.is_directory && !b.is_directory) return -1;
      if (!a.is_directory && b.is_directory) return 1;
      return (a.name ?? '').localeCompare(b.name ?? '');
    });
    setEntries(data);
    setCurrentPath(path ?? '');
  }, [apiUrl]);

  useEffect(() => { loadDirectory(); }, [loadDirectory]);

  const segments = currentPath.split('/').filter(Boolean);

  return (
    <div className="flex gap-6">
      <div className="flex-2 min-w-0">
        <FileBreadcrumb
          rootLabel={volumeKey}
          segments={segments}
          onNavigateToRoot={() => loadDirectory()}
          onNavigateToSegment={(i) =>
            loadDirectory(segments.slice(0, i + 1).join('/'))
          }
        />
        <DirectoryList
          entries={entries}
          onEntryClick={(entry) => {
            const entryPath = currentPath
              ? `${currentPath}/${entry.name}`
              : entry.name ?? '';
            if (entry.is_directory) {
              loadDirectory(entryPath);
            } else {
              setSelectedFile(entryPath);
              fetch(apiUrl('preview', { path: entryPath }))
                .then(async (r) => {
                  if (!r.ok) {
                    const errBody = await r.json().catch(() => null);
                    console.error('Failed to load file preview', errBody ?? r.statusText);
                    return null;
                  }
                  return r.json();
                })
                .then((data) => {
                  if (data) {
                    setPreview(data);
                  }
                });
            }
          }}
          resolveEntryPath={(entry) =>
            currentPath ? `${currentPath}/${entry.name}` : entry.name ?? ''
          }
          isAtRoot={!currentPath}
          selectedPath={selectedFile}
        />
      </div>
      <FilePreviewPanel
        className="flex-1 min-w-0"
        selectedFile={selectedFile}
        preview={preview}
        onDownload={(path) =>
          window.open(apiUrl('download', { path }), '_blank', 'noopener,noreferrer')
        }
        imagePreviewSrc={(p) => apiUrl('raw', { path: p })}
      />
    </div>
  );
}

Upload Pattern

const handleUpload = async (file: File) => {
  const uploadPath = currentPath ? `${currentPath}/${file.name}` : file.name;
  const response = await fetch(apiUrl("upload", { path: uploadPath }), {
    method: "POST",
    body: file,
  });
  if (!response.ok) {
    const data = await response.json().catch(() => ({}));
    throw new Error(data.error ?? `Upload failed (${response.status})`);
  }
  // Reload directory after upload
  await loadDirectory(currentPath || undefined);
};

Delete Pattern

const handleDelete = async (filePath: string) => {
  const response = await fetch(
    `/api/files/${volumeKey}?path=${encodeURIComponent(filePath)}`,
    { method: "DELETE" },
  );
  if (!response.ok) {
    const data = await response.json().catch(() => ({}));
    throw new Error(data.error ?? `Delete failed (${response.status})`);
  }
};

Create Directory Pattern

const handleCreateDirectory = async (name: string) => {
  const dirPath = currentPath ? `${currentPath}/${name}` : name;
  const response = await fetch(apiUrl("mkdir"), {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ path: dirPath }),
  });
  if (!response.ok) {
    const data = await response.json().catch(() => ({}));
    throw new Error(
      data.error ?? `Create directory failed (${response.status})`,
    );
  }
};

Resource Requirements

The plugin auto-generates volume resource requirements from DATABRICKS_VOLUME_* env vars. Setting them in app.yaml is usually all you need.

Declare the volume explicitly in databricks.yml only when you need to pin it as a managed resource, then wire the env var via valueFrom in app.yaml:

# databricks.yml
resources:
  apps:
    my_app:
      user_api_scopes:
        - files.files        # Needed when using .asUser(req) programmatic API
      resources:
        - name: uploads-volume
          volume:
            path: /Volumes/catalog/schema/uploads
            permission: WRITE_VOLUME

Note: user_api_scopes is required for OBO volumes (auth: "on-behalf-of-user") and for any appkit.files("key").asUser(req) programmatic call. Pure SP volumes accessed only via HTTP routes don't need it. The plugin docs have the latest resource-requirement behavior.

# app.yaml
env:
  - name: DATABRICKS_VOLUME_UPLOADS
    valueFrom: uploads-volume

Troubleshooting

Frontend Guidelines

For full component API: run npx @databricks/appkit docs and navigate to the component you need.

Common Anti-Patterns

These mistakes appear frequently — check the official docs for actual prop names:

Always verify props against docs before using a component.

Chart Data Modes

All chart/data components support two modes:

• Query mode: pass queryKey + parameters — component fetches data automatically. parameters is REQUIRED even if empty (parameters={{}}).
• Data mode: pass static data via data prop (JSON array or Arrow Table) — no queryKey/parameters needed.

// Query mode (recommended for Databricks SQL)
<BarChart queryKey="sales_by_region" parameters={{}} />

// Data mode (static/pre-fetched data)
<BarChart data={myData} xKey="category" yKey="count" />

Chart Props Quick Reference

All charts accept these core props (verify full list via npx @databricks/appkit docs):

<BarChart
  queryKey="sales_by_region"   // SQL query filename without .sql
  parameters={{}}              // query params — REQUIRED in query mode, even if empty
  xKey="region"                // X axis field (auto-detected from schema if omitted)
  yKey="revenue"               // Y axis field(s) — string or string[] (auto-detected if omitted)
  format="auto"                // "json" | "arrow" | "auto" (default: "auto")
  transformer={(d) => d}       // transform raw data before rendering
  colors={['#40d1f5']}         // custom colors (overrides colorPalette)
  colorPalette="categorical"   // "categorical" | "sequential" | "diverging"
  title="Sales by Region"      // chart title
  showLegend                   // show legend
  options={{}}                 // additional ECharts options to merge
  height={400}                 // default: 300
  orientation="vertical"       // "vertical" | "horizontal" (BarChart/LineChart/AreaChart)
  stacked                      // stack bars/areas (BarChart/AreaChart)
/>

<LineChart queryKey="monthly_trend" parameters={{}} xKey="month" yKey={["revenue", "expenses"]}
  smooth showSymbol={false} />

Charts are ECharts-based — configure via props, not Recharts-style children. Components handle data fetching, loading, and error states internally.

⚠️ `parameters` is REQUIRED on all data components, even when the query has no params. Always include parameters={{}}.

// ❌ Don't double-fetch
const { data } = useAnalyticsQuery('sales_data', {});
return <BarChart queryKey="sales_data" parameters={{}} />;  // fetches again!

DataTable

DataTable auto-generates columns from data and handles fetching, loading, error, and empty states.

For full props: npx @databricks/appkit docs "DataTable".

// ❌ WRONG - missing required `parameters` prop
<DataTable queryKey="my_query" />

// ✅ CORRECT - minimal
<DataTable queryKey="my_query" parameters={{}} />

// ✅ CORRECT - with filtering and pagination
<DataTable
  queryKey="my_query"
  parameters={{}}
  filterColumn="name"
  filterPlaceholder="Filter by name..."
  pageSize={10}
  pageSizeOptions={[10, 25, 50]}
/>

// ✅ CORRECT - with row selection
<DataTable
  queryKey="my_query"
  parameters={{}}
  enableRowSelection
  onRowSelectionChange={(selection) => console.log(selection)}
/>

Custom column formatting — use the transform prop or format in SQL:

<DataTable
  queryKey="products"
  parameters={{}}
  transform={(data) => data.map(row => ({
    ...row,
    price: `$${Number(row.price).toFixed(2)}`,
  }))}
/>

Available Components (Quick Reference)

For full prop details: npx @databricks/appkit docs "appkit-ui API reference".

All data components support both query mode (queryKey + parameters) and data mode (static data prop). Common props across all charts: format, transformer, colors, colorPalette, title, showLegend, height, options, ariaLabel, testId.

Data Components (@databricks/appkit-ui/react)

UI Components (@databricks/appkit-ui/react)

All data components require `parameters={{}}` even when the query has no params.

Layout Structure

<div className="container mx-auto p-4">
  <h1 className="text-2xl font-bold mb-4">Page Title</h1>
  <form className="space-y-4 mb-8">{/* form inputs */}</form>
  <div className="grid gap-4">{/* list items */}</div>
</div>

Component Organization

• Shared UI components: @databricks/appkit-ui/react
• Feature components: client/src/components/FeatureName.tsx
• Split components when logic exceeds ~100 lines or component is reused

Gotchas

• SelectItem cannot have value="". Use sentinel value like "all" for "show all" options.
• Use <Skeleton> components instead of plain "Loading..." text
• Handle nullable fields: value={field || ''} for inputs
• For maps with React 19, use react-leaflet v5: npm install react-leaflet@^5.0.0 leaflet @types/leaflet

Databricks brand colors: ['#40d1f5', '#4462c9', '#EB1600', '#0B2026', '#4A4A4A', '#353a4a']

AppKit Genie Guide

Use Genie when your app needs a natural language query interface over Unity Catalog tables. For analytics dashboards, use config/queries/ instead. For persistent storage, use Lakebase.

When to Use

Architecture

User (browser) -> AppKit genie plugin (/api/genie/...) -> Databricks Genie API -> SQL Warehouse
               <- SSE stream (status, message_result, query_result) <-

The built-in genie() plugin from @databricks/appkit proxies requests via SSE streaming. It reads the space ID from the DATABRICKS_GENIE_SPACE_ID env var. Call genie() with no arguments.

Genie Space Creation

The databricks genie create-space command takes two positional arguments: WAREHOUSE_ID and SERIALIZED_SPACE (a JSON string).

databricks genie create-space <WAREHOUSE_ID> \
  '{"version":2,"data_sources":{"tables":[{"identifier":"catalog.schema.orders"},{"identifier":"catalog.schema.customers"}]}}' \
  --title "Sales Assistant" \
  --description "Answers sales analytics questions" \
  --profile <PROFILE>

The JSON must include version and data_sources.tables with each table as {"identifier":"catalog.schema.table"}. Optional flags: --title, --description, --parent-path.

To discover the full serialized space format (including optional fields), export an existing space:

databricks genie get-space <SPACE_ID> --include-serialized-space --profile <PROFILE>

Discover warehouse ID with:

databricks experimental aitools tools get-default-warehouse --profile <PROFILE>

Scaffolding a New Genie App

# 1. Discover warehouse
databricks experimental aitools tools get-default-warehouse --profile <PROFILE>

# 2. Create Genie space (see syntax above)
databricks genie create-space <WAREHOUSE_ID> '<SERIALIZED_SPACE_JSON>' \
  --title "My Space" --profile <PROFILE>

# 3. Check manifest for genie plugin keys
databricks apps manifest --profile <PROFILE>

# 4. Scaffold (derive --set keys from manifest output)
databricks apps init --name <APP_NAME> --features genie \
  --set "genie.<resourceKey>.<field>=<SPACE_ID>" \
  --run none --profile <PROFILE>

# 5. Set local env + develop
cd <APP_NAME>
echo "DATABRICKS_GENIE_SPACE_ID=<SPACE_ID>" >> server/.env
npm install && npm run dev

Do not guess --set flags — always derive from databricks apps manifest.

Adding Genie to an Existing App

`databricks.yml` — add Genie variables and resource:

variables:
  genie_space_id:
    description: Genie Space ID
  genie_space_name:
    description: Genie Space name

resources:
  apps:
    app:
      resources:
        # ... existing resources ...
        - name: genie-space
          genie_space:
            name: ${var.genie_space_name}
            space_id: ${var.genie_space_id}
            permission: CAN_RUN

targets:
  default:
    variables:
      genie_space_id: <space_id>
      genie_space_name: <space_name>

`app.yaml` — add env injection:

env:
  # ... existing env vars ...
  - name: DATABRICKS_GENIE_SPACE_ID
    valueFrom: genie-space

`server/server.ts` — register the plugin:

import { createApp, server, analytics, genie } from "@databricks/appkit";

createApp({
  plugins: [server(), analytics(), genie()],
}).catch(console.error);

Preserve existing plugins and add genie() to the array.

`server/.env` — for local development:

DATABRICKS_GENIE_SPACE_ID=<YOUR_SPACE_ID>

Frontend — add the chat component:

import { GenieChat } from "@databricks/appkit-ui/react";

function GeniePage() {
  return (
    <div style={{ height: 600 }}>
      <GenieChat />
    </div>
  );
}

Update smoke tests if headings or routes changed, then databricks apps validate.

For advanced Genie plugin usage, see npx @databricks/appkit docs ./docs/plugins/genie.md.

Multi-Space Deployment

For the spaces map API, GenieChat alias prop, and useGenieChat hook, see npx @databricks/appkit docs ./docs/plugins/genie.md.

This section covers the deployment-specific patterns for multi-space Genie apps (databricks.yml, app.yaml, stale conversation cleanup).

databricks.yml — add one variable + resource per space, plus target-level values:

variables:
  genie_space_id:
    description: Default Genie space ID (required by AppKit)
  genie_space_name:
    description: Default Genie space name
  genie_space_sales_id:
    description: Sales Genie space ID
  genie_space_support_id:
    description: Support Genie space ID

resources:
  apps:
    app:
      user_api_scopes:
        - dashboards.genie
      resources:
        - name: genie-space
          genie_space:
            name: ${var.genie_space_name}
            space_id: ${var.genie_space_id}
            permission: CAN_RUN
        - name: genie-space-sales
          genie_space:
            name: genie-space-sales
            space_id: ${var.genie_space_sales_id}
            permission: CAN_RUN
        - name: genie-space-support
          genie_space:
            name: genie-space-support
            space_id: ${var.genie_space_support_id}
            permission: CAN_RUN

targets:
  default:
    variables:
      genie_space_id: <any-space-id>
      genie_space_name: <space-name>
      genie_space_sales_id: <sales-space-id>
      genie_space_support_id: <support-space-id>

app.yaml — keep DATABRICKS_GENIE_SPACE_ID (AppKit validates it on startup). Add one valueFrom per UI space:

env:
  - name: DATABRICKS_GENIE_SPACE_ID
    valueFrom: genie-space
  - name: DATABRICKS_GENIE_SPACE_SALES
    valueFrom: genie-space-sales
  - name: DATABRICKS_GENIE_SPACE_SUPPORT
    valueFrom: genie-space-support

Critical gotcha: DATABRICKS_GENIE_SPACE_ID must always be set — AppKit validates it on startup even when using a custom spaces map.

Build version stamp — stamp every build so the page can detect a new deployment and clear stale conversation state:

// client/vite.config.ts
export default defineConfig({
  // ... existing config ...
  define: {
    "import.meta.env.VITE_APP_VERSION": JSON.stringify(Date.now().toString()),
  },
});

Stale conversation cleanup — GenieChat stores conversation IDs in URLs and localStorage that become stale across space switches or redeployments:

function clearConversationUrl() {
  const url = new URL(window.location.href);
  url.searchParams.delete("conversationId");
  window.history.replaceState({}, "", url.toString());
}

function initAlias(): string {
  const buildVersion = import.meta.env.VITE_APP_VERSION ?? "dev";
  if (localStorage.getItem("appkit:genie:version") !== buildVersion) {
    const savedAlias = localStorage.getItem("appkit:genie:alias");
    Object.keys(localStorage)
      .filter((k) => k.startsWith("appkit:genie:"))
      .forEach((k) => localStorage.removeItem(k));
    localStorage.setItem("appkit:genie:version", buildVersion);
    if (savedAlias) localStorage.setItem("appkit:genie:alias", savedAlias);
    clearConversationUrl();
  }
  // SPACES: array of {alias, spaceId} defined in your component
  return localStorage.getItem("appkit:genie:alias") ?? SPACES[0]?.alias ?? "";
}

Frontend

For full component API: run npx @databricks/appkit docs "GenieChat".

The GenieChat component handles SSE streaming, conversation state, history replay, and query result rendering. For custom UI, use the useGenieChat hook — see npx @databricks/appkit docs "useGenieChat".

Common anti-patterns:

HTTP Endpoints

The plugin mounts SSE endpoints under /api/genie:

SSE Event Types

Attachment Types

Troubleshooting

Jobs: Trigger Lakeflow Jobs from Apps

For full Jobs plugin API (routes, types, config options): run npx @databricks/appkit docs → Jobs plugin.

Use the jobs() plugin when your app needs to trigger or monitor pre-existing Databricks Lakeflow Jobs (notebooks, Python scripts, SQL, dbt, JARs) and surface their status to users. The jobs themselves still live as regular Lakeflow Jobs in the workspace — the plugin is the typed, resource-scoped accessor that lets app code start runs, poll status, and stream completion events.

The plugin is resource-scoped: only jobs declared via config or discovered from DATABRICKS_JOB_* env vars are accessible. It is not a generic Jobs SDK wrapper — to author or schedule jobs, use the databricks-jobs (Lakeflow) skill instead. See `overview.md` for the cross-plugin data-pattern selector.

Scaffolding

databricks apps init --name <NAME> --features jobs \
  --set "jobs.<resourceKey>.<field>=<JOB_ID>" \
  --run none --profile <PROFILE>

Do not guess --set keys — derive them from databricks apps manifest --profile <PROFILE> (look up the jobs plugin's resources.required entries).

Multi-job and analytics+jobs are common combinations:

databricks apps init --name <NAME> --features analytics,jobs \
  --set "analytics.sql-warehouse.id=<WAREHOUSE_ID>" \
  --set "jobs.<resourceKey>.<field>=<JOB_ID>" \
  --run none --profile <PROFILE>

Configure job IDs via environment variables in app.yaml (deployed) or server/.env (local dev):

# Single-job mode → exposed under the "default" key
DATABRICKS_JOB_ID=123456789

# Multi-job mode → exposed under lowercased keys ("etl", "ml")
DATABRICKS_JOB_ETL=123456789
DATABRICKS_JOB_ML=987654321

The env var suffix (after DATABRICKS_JOB_) becomes the job key, lowercased. Explicit jobs config in createApp() is merged with env-discovered jobs; explicit config wins on key collisions.

Plugin Setup

Minimal — discovers all jobs from the environment:

import { createApp, server, jobs } from "@databricks/appkit";

await createApp({
  plugins: [server(), jobs()],
});

With per-job validation and task-type mapping:

import { createApp, server, jobs } from "@databricks/appkit";
import { z } from "zod";

const appkit = await createApp({
  plugins: [
    server(),
    jobs({
      jobs: {
        etl: {
          taskType: "notebook",
          params: z.object({
            startDate: z.string(),
            endDate: z.string(),
            dryRun: z.boolean().optional(),
          }),
        },
      },
    }),
  ],
});

For the full IJobsConfig, JobConfig, and task-type → SDK parameter mapping, run npx @databricks/appkit docs Jobs plugin. Two non-obvious points: dbt accepts no parameters, and notebook/python_wheel/sql coerce all param values to strings before forwarding.

Server-Side API (Programmatic)

appkit.jobs(key) returns a JobHandle. All methods return ExecutionResult<T> — always check `.ok` before reading `.data`. Full method list and types: npx @databricks/appkit docs Jobs plugin.

const etl = appkit.jobs("etl");

// One-shot trigger
const result = await etl.runNow({ startDate: "2025-01-01" });
if (!result.ok) throw new Error(`Run failed: ${result.error}`);

// Trigger and stream status until completion (async iterable, SSE-backed)
for await (const status of etl.runAndWait({ startDate: "2025-01-01" })) {
  console.log(status.status); // "PENDING" | "RUNNING" | "TERMINATED" | ...
}

Read methods (lastRun, listRuns, getRun, getRunOutput, getJob) and cancelRun follow the same ExecutionResult<T> shape. Reads cache for 60s with 3 retries. runAndWait has a 600s server-side cap, but client-facing requests are bounded by the Apps platform's 120s reverse-proxy timeout (see Platform Guide, "HTTP Proxy & Streaming"). For runs longer than ~120s, use runNow and poll getRun (or GET /api/jobs/:jobKey/status) from separate short-lived requests instead of streaming.

Execution context

All operations run as the app's service principal. The resource binding in databricks.yml grants the SP CAN_MANAGE_RUN, so users trigger runs without needing their own grant. Per-run attribution in the Jobs UI shows the SP, not the human user. The plugin does not support on-behalf-of (OBO) user execution.

HTTP Endpoints

Routes mount at /api/jobs/:jobKey/... — full route list, request bodies, and SSE frame shape via npx @databricks/appkit docs Jobs plugin. The streaming endpoint (POST /api/jobs/:jobKey/run?stream=true) emits data: <json> events terminated by a blank line (\n\n), where the JSON is { status, timestamp, run }; clients must buffer until `\n\n` and reassemble across chunk boundaries before parsing. runAndWait (server-side) honors req.signal and aborts cleanly on client disconnect — but the platform's 120s reverse-proxy cap applies regardless.

Resource Requirements

Each job key requires a job resource with CAN_MANAGE_RUN in databricks.yml:

resources:
  apps:
    my_app:
      resources:
        - name: etl-job
          job:
            id: ${var.etl_job_id}
            permission: CAN_MANAGE_RUN

Wire the env var in app.yaml:

env:
  - name: DATABRICKS_JOB_ETL
    valueFrom: etl-job

Verify exact --set keys and resource shape via databricks apps manifest --profile <PROFILE>.

Troubleshooting

Lakebase: OLTP Database for Apps

Use Lakebase when your app needs persistent read/write storage — forms, CRUD operations, user-generated data. For analytics dashboards reading from a SQL warehouse, use config/queries/ instead.

When to Use Lakebase vs Analytics

Serving lakehouse data to apps? If your app needs low-latency reads of Delta/UC tables (entity lookups, product catalogs, feature serving), use Lakebase synced tables to materialize them into Lakebase instead of querying a SQL warehouse (which takes seconds to minutes). See Reading from Synced Tables below.

Scaffolding

Scaffolding is the fastest way to get started. If you already have an app, see Adding Lakebase to an Existing App below.

Lakebase only (no analytics SQL warehouse):

databricks apps init --name <NAME> --features lakebase \
  --set "lakebase.postgres.branch=<BRANCH_NAME>" \
  --set "lakebase.postgres.database=<DATABASE_NAME>" \
  --run none --profile <PROFILE>

Both Lakebase and analytics:

databricks apps init --name <NAME> --features analytics,lakebase \
  --set "analytics.sql-warehouse.id=<WAREHOUSE_ID>" \
  --set "lakebase.postgres.branch=<BRANCH_NAME>" \
  --set "lakebase.postgres.database=<DATABASE_NAME>" \
  --run none --profile <PROFILE>

Where <BRANCH_NAME> and <DATABASE_NAME> are full resource names (e.g. projects/<PROJECT_ID>/branches/<BRANCH_ID> and projects/<PROJECT_ID>/branches/<BRANCH_ID>/databases/<DB_ID>).

Use the databricks-lakebase skill to create a Lakebase project and discover branch/database resource names before running this command.

For multi-environment deployments (dev/prod), use variables: and targets: blocks in databricks.yml — see the `databricks-dabs` skill for patterns.

Naming conventions: Use domain names for user-facing code (ItemsPage.tsx, /api/items, item-routes.ts). Keep lakebase naming only for infrastructure config (lakebase() plugin, LAKEBASE_ENDPOINT, postgres app resource).

Get resource names (if you have an existing project):

# List branches → use the name field of a READY branch
databricks postgres list-branches projects/<PROJECT_ID> --profile <PROFILE>
# List databases → use the name field
databricks postgres list-databases projects/<PROJECT_ID>/branches/<BRANCH_ID> --profile <PROFILE>

Adding Lakebase to an Existing App

`databricks.yml` — add Lakebase variables and resource:

variables:
  lakebase_branch:
    description: Lakebase branch resource name
  lakebase_database:
    description: Lakebase database resource name

resources:
  apps:
    app:
      resources:
        # ... existing resources ...
        - name: postgres
          postgres:
            branch: ${var.lakebase_branch}
            database: ${var.lakebase_database}

targets:
  default:
    variables:
      lakebase_branch: projects/<PROJECT_ID>/branches/<BRANCH_ID>
      lakebase_database: projects/<PROJECT_ID>/branches/<BRANCH_ID>/databases/<DB_ID>

Use the databricks-lakebase skill to create a Lakebase project and discover branch/database resource names.

For per-user connections (OBO/RLS), also add postgres to user_api_scopes — see npx @databricks/appkit docs ./docs/plugins/lakebase.md for OBO setup.

`app.yaml` — add env injection:

env:
  # ... existing env vars ...
  - name: LAKEBASE_ENDPOINT
    valueFrom: postgres

Other Lakebase env vars (PGHOST, PGPORT, PGDATABASE, PGUSER, PGSSLMODE) are auto-injected by the platform when the postgres resource is configured. Only LAKEBASE_ENDPOINT must be set explicitly.

`server/server.ts` — register the plugin:

import { createApp, server, analytics, lakebase } from "@databricks/appkit";

createApp({
  plugins: [server(), analytics(), lakebase()],
}).catch(console.error);

Preserve existing plugins and add lakebase() to the array.

`server/.env` — for local development:

PGHOST=<host from endpoint>
PGPORT=5432
PGDATABASE=<your database name>
PGSSLMODE=require
LAKEBASE_ENDPOINT=projects/<PROJECT_ID>/branches/<BRANCH_ID>/endpoints/<ENDPOINT_ID>

Get connection details from databricks postgres get-endpoint. See Local Development below for the full workflow.

Deploy the app before local development — see Local Development > Prerequisites below. Update smoke tests if headings or routes changed, then databricks apps validate.

Project Structure (after databricks apps init --features lakebase)

my-app/
├── server/
│   └── server.ts       # Backend with Lakebase plugin + Express routes
├── client/
│   └── src/
│       └── App.tsx     # React frontend
├── app.yaml            # Manifest with database resource declaration
└── package.json        # Includes @databricks/lakebase dependency

Note: No `config/queries/` directory — Lakebase apps use server-side appkit.lakebase.query() calls, not SQL files.

Lakebase Plugin API

Scaffolding with --features lakebase (see above) generates this pattern. Access Lakebase through the plugin handle returned by createApp():

import { createApp, lakebase } from "@databricks/appkit";

const appkit = await createApp({
  plugins: [lakebase()],
});

// Query via the plugin handle — handles pooling and token refresh automatically
const result = await appkit.lakebase.query("SELECT * FROM users WHERE id = $1", [userId]);

The lakebase() plugin auto-configures from platform-injected env vars at deploy time. No manual pool setup needed.

Environment Variables (auto-set when deployed with database resource)

CRUD Routes Pattern

Always use server-side routes for Lakebase operations — do NOT call appkit.lakebase.query() from the client. Use onPluginsReady to initialize the schema and register Express routes:

// server/server.ts
import { createApp, server, lakebase } from "@databricks/appkit";
import { z } from 'zod';

await createApp({
  plugins: [server(), lakebase()],
  async onPluginsReady(appkit) {
    // Schema init (runs once before server accepts requests)
    await appkit.lakebase.query(`
      CREATE SCHEMA IF NOT EXISTS app_data;
      CREATE TABLE IF NOT EXISTS app_data.items (
        id SERIAL PRIMARY KEY,
        name TEXT NOT NULL,
        created_at TIMESTAMPTZ DEFAULT NOW()
      );
    `);

    // CRUD routes via Express
    appkit.server.extend((app) => {
      app.get('/api/items', async (_req, res) => {
        const { rows } = await appkit.lakebase.query(
          "SELECT * FROM app_data.items ORDER BY created_at DESC LIMIT 100"
        );
        res.json(rows);
      });

      app.post('/api/items', async (req, res) => {
        const parsed = z.object({ name: z.string().min(1) }).safeParse(req.body);
        if (!parsed.success) { res.status(400).json({ error: 'Invalid input' }); return; }
        const { rows } = await appkit.lakebase.query(
          "INSERT INTO app_data.items (name) VALUES ($1) RETURNING *",
          [parsed.data.name]
        );
        res.status(201).json(rows[0]);
      });

      app.delete('/api/items/:id', async (req, res) => {
        const id = parseInt(req.params.id, 10);
        if (isNaN(id)) { res.status(400).json({ error: 'Invalid id' }); return; }
        await appkit.lakebase.query("DELETE FROM app_data.items WHERE id = $1", [id]);
        res.status(204).send();
      });
    });
  },
});

Deploy first (App + Lakebase only)! When your Databricks App uses Lakebase, the Service Principal must create and own the schema. Run databricks apps deploy before any local development. See `databricks-lakebase` skill's Schema Permissions for Deployed Apps for details.

Schema Initialization

Always create a custom schema — the Service Principal cannot access any existing schemas (including public). It must create the schema itself to become its owner. See `databricks-lakebase` skill's Schema Permissions for Deployed Apps for the full permission model and deploy-first workflow. Initialize tables inside the onPluginsReady callback before registering routes (see CRUD pattern above):

// Inside onPluginsReady — runs once at startup before handling requests
await appkit.lakebase.query(`
  CREATE SCHEMA IF NOT EXISTS app_data;
  CREATE TABLE IF NOT EXISTS app_data.items (
    id SERIAL PRIMARY KEY,
    name TEXT NOT NULL,
    created_at TIMESTAMPTZ DEFAULT NOW()
  );
`);

ORM Integration (Optional)

The plugin exposes the raw pg.Pool via appkit.lakebase.pool — works with any PostgreSQL library:

// Drizzle ORM
import { drizzle } from "drizzle-orm/node-postgres";
const db = drizzle(appkit.lakebase.pool);

// Prisma (with @prisma/adapter-pg)
import { PrismaPg } from "@prisma/adapter-pg";
const adapter = new PrismaPg(appkit.lakebase.pool);
const prisma = new PrismaClient({ adapter });

For ORM-compatible config: appkit.lakebase.getOrmConfig().

Chat Persistence Pattern

Save AI chat conversations to Lakebase so users can resume sessions and scroll full message history.

Schema — create in a separate chat schema (not app) so the deploy-first ownership model stays clean:

CREATE SCHEMA IF NOT EXISTS chat;

CREATE TABLE IF NOT EXISTS chat.chats (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  user_id TEXT NOT NULL,
  title TEXT NOT NULL,
  created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE TABLE IF NOT EXISTS chat.messages (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  chat_id UUID NOT NULL REFERENCES chat.chats(id) ON DELETE CASCADE,
  role TEXT NOT NULL CHECK (role IN ('system', 'user', 'assistant', 'tool')),
  content TEXT NOT NULL,
  created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);

CREATE INDEX IF NOT EXISTS idx_messages_chat_id_created_at
  ON chat.messages(chat_id, created_at);

Bootstrap — run setup in onPluginsReady so tables exist before the server accepts requests:

await createApp({
  plugins: [server(), lakebase()],
  async onPluginsReady(appkit) {
    await setupChatTables(appkit);
    // then register routes via appkit.server.extend(...)
  },
});

Persistence helpers — use parameterized queries:

export async function createChat(appkit, input: { userId: string; title: string }) {
  const result = await appkit.lakebase.query(
    `INSERT INTO chat.chats (user_id, title) VALUES ($1, $2)
     RETURNING id, user_id, title, created_at, updated_at`,
    [input.userId, input.title],
  );
  return result.rows[0];
}

export async function appendMessage(appkit, input: { chatId: string; role: string; content: string }) {
  const result = await appkit.lakebase.query(
    `INSERT INTO chat.messages (chat_id, role, content) VALUES ($1, $2, $3)
     RETURNING id, chat_id, role, content, created_at`,
    [input.chatId, input.role, input.content],
  );
  return result.rows[0];
}

User identity: In deployed apps, use req.header("x-forwarded-email") (injected by the Databricks Apps platform proxy; for off-platform deployments, use your own auth middleware). For local dev, hardcode a test user ID.

History endpoints:

• GET /api/chats — list chats for current user
• GET /api/chats/:chatId/messages — load ordered history
• DELETE /api/chats/:chatId — delete chat (messages cascade)

AI SDK v6 integration: Use setMessages() from useChat return value for history loading (NOT initialMessages). To read response headers like X-Chat-Id, pass a custom fetch wrapper on the TextStreamChatTransport constructor.

Reading from Lakebase synced tables

Lakebase synced tables materialize Delta/UC tables into Lakebase Postgres for low-latency app reads. The lakehouse remains the source of truth; Lakebase serves as a read-optimized index.

Architecture:

Delta gold tables  →  Synced tables (read-only)  →  App reads via appkit.lakebase.query()
App writes         →  Lakebase OLTP tables        →  optional Lakehouse Sync → Delta

Use synced tables when data is curated in Delta, changes relatively slowly, and must be served at OLTP latency — operational consoles, user-facing apps on gold tables, feature serving, or hybrid read/write patterns. See the `databricks-lakebase` skill's synced-tables.md for the full decision checklist.

Security note: Synced tables do not propagate Unity Catalog fine-grained access control (row filters, column masks). If UC FGAC is critical, use DBSQL with user authorization instead.

How It Works

Synced tables (created via databricks postgres create-synced-table) appear as regular Postgres tables. From the app's perspective, use the same appkit.lakebase.query() pattern but read-only.

Key differences from CRUD tables:

Permission grant required: The app's SP has CAN_CONNECT_AND_CREATE but does not have pg_read_all_data. To read synced tables, the project owner must grant access — see the `databricks-lakebase` skill's SKILL.md "Grant app SP access to synced tables" section for the SQL commands and psql connection steps.

Example Express route reading synced taxi data:

// Inside onPluginsReady → appkit.server.extend((app) => { ... })
app.get('/api/top-pickups', async (_req, res) => {
  const { rows } = await appkit.lakebase.query(`
    SELECT pickup_zip, COUNT(*) AS trip_count, AVG(fare_amount) AS avg_fare
    FROM public.nyc_trips
    GROUP BY pickup_zip
    ORDER BY trip_count DESC
    LIMIT 10
  `);
  res.json(rows);
});

Do not write to synced tables. The sync pipeline manages the data — direct writes corrupt the sync state. For mixed read/write patterns, read from synced tables and write to separate app-owned tables. To create synced tables and grant the app's SP read access, see the `databricks-lakebase` skill's synced-tables.md and the "Grant app SP access to synced tables" section in its SKILL.md.

Key Differences from Analytics Pattern

NEVER use `useAnalyticsQuery` for Lakebase data — it queries the SQL warehouse, not Lakebase. NEVER put Lakebase SQL in `config/queries/` — those files are only for warehouse queries.

Local Development

Prerequisites (MUST verify before local development)

This applies when your Databricks App uses Lakebase. Run this check before any local development:

databricks apps get <APP_NAME> --profile <PROFILE>

Check the response for the active_deployment field. If it exists with status.state of SUCCEEDED, the app has been deployed. If active_deployment is missing, the app has never been deployed: 1. STOP — do not proceed with local development 2. Deploy first: databricks apps deploy <APP_NAME> --profile <PROFILE> 3. Wait for deployment to complete, then continue

If you skip this step, the Service Principal won't own the database schema. You'll create schemas under your credentials that the SP cannot access after deployment. See `databricks-lakebase` skill's Schema Permissions for Deployed Apps for the full workflow and recovery steps.

Lakebase project creators already have database access after the first deploy. Collaborators need databricks_superuser granted by the project creator via Branch Overview.

Project-owner note: If you are the Lakebase project owner, databricks_create_role may fail with "role already exists" and GRANT databricks_superuser may fail with "permission denied to grant role" — both errors are safe to ignore; the project owner already has the necessary access.

The Lakebase env vars (PGHOST, PGDATABASE, etc.) are auto-set only when deployed. For local development, get the connection details from your endpoint and set them manually:

# Get endpoint connection details
databricks postgres get-endpoint \
  projects/<PROJECT_ID>/branches/<BRANCH_ID>/endpoints/<ENDPOINT_ID> \
  --profile <PROFILE>

Then create server/.env with the values from the endpoint response:

PGHOST=<host from endpoint>
PGPORT=5432
PGDATABASE=<your database name>
PGUSER=<your service principal client ID>
PGSSLMODE=require
LAKEBASE_ENDPOINT=projects/<PROJECT_ID>/branches/<BRANCH_ID>/endpoints/<ENDPOINT_ID>

Load server/.env in your dev server (e.g. via dotenv or node --env-file=server/.env). Never commit .env files — add server/.env to .gitignore.

Troubleshooting

Model Serving: Calling ML Endpoints from Apps

Use Model Serving when your app needs AI features — chat, inference, embeddings, or predictions from a Databricks Model Serving endpoint. For analytics dashboards, use config/queries/ instead. For persistent storage, use Lakebase.

When to Use

Scaffolding

Check if the serving plugin is available in the AppKit template:

databricks apps manifest --profile <PROFILE>

If the manifest includes a `serving` plugin:

databricks apps init --name <APP_NAME> --features serving \
  --set "serving.serving-endpoint.name=<ENDPOINT_NAME>" \
  --run none --profile <PROFILE>

If adding to an existing app, see Adding Model Serving to an Existing App below.

Use the databricks-model-serving skill to create a serving endpoint first if one doesn't exist yet.

Adding Model Serving to an Existing App

`databricks.yml` — add serving endpoint resource and user_api_scopes:

resources:
  apps:
    app:
      user_api_scopes:
        # ... existing scopes ...
        - serving.serving-endpoints
      resources:
        # ... existing resources ...
        - name: serving-endpoint
          serving_endpoint:
            name: <ENDPOINT_NAME>
            permission: CAN_QUERY

`app.yaml` — add env injection:

env:
  # ... existing env vars ...
  - name: DATABRICKS_SERVING_ENDPOINT_NAME
    valueFrom: serving-endpoint

The injected value is the endpoint name (not a URL). Use it in server-side code to call the endpoint.

`server/server.ts` — register the plugin:

import { createApp, server, analytics, serving } from "@databricks/appkit";

createApp({
  plugins: [server(), analytics(), serving()],
}).catch(console.error);

Preserve existing plugins and add serving() to the array.

`server/.env` — for local development:

DATABRICKS_SERVING_ENDPOINT_NAME=<your-endpoint-name>

Update smoke tests if headings or routes changed, then databricks apps validate.

Serving Plugin API

Access model serving through the plugin handle returned by createApp():

import { createApp, server, serving } from "@databricks/appkit";

const appkit = await createApp({
  plugins: [server(), serving()],
});

// Non-streaming invocation
const result = await appkit.serving().invoke({
  messages: [{ role: "user", content: "Hello" }],
});

// Streaming invocation
for await (const chunk of appkit.serving().stream({
  messages: [{ role: "user", content: "Hello" }],
})) {
  console.log(chunk);
}

// On-behalf-of user (OBO) — uses the requesting user's identity
const result = await appkit.serving().asUser(req).invoke({
  messages: [{ role: "user", content: prompt }],
});

All serving routes execute on behalf of the authenticated user (OBO) by default. For programmatic access via exports(), use .asUser(req) to run in user context.

Named Endpoints

Use endpoint aliases to reference multiple serving endpoints by name:

serving({
  endpoints: {
    llm: { env: "DATABRICKS_SERVING_ENDPOINT_NAME" },
    classifier: { env: "DATABRICKS_SERVING_ENDPOINT_CLASSIFIER" },
  },
  timeout: 120000, // optional, default 2 min
})

Each alias maps to an environment variable holding the actual endpoint name. Access by alias:

const result = await appkit.serving("llm").invoke({ messages });
const classification = await appkit.serving("classifier").invoke({ inputs: ["text"] });

If an endpoint serves multiple models, use servedModel to target a specific model directly:

serving({
  endpoints: {
    llm: { env: "DATABRICKS_SERVING_ENDPOINT_NAME", servedModel: "llama-v2" },
  },
})

HTTP Endpoints

The plugin auto-registers routes under /api/serving:

Frontend

Use the built-in React hooks from @databricks/appkit-ui/react — do NOT call serving endpoints directly from the client.

Streaming (chat, real-time inference):

import { useServingStream } from "@databricks/appkit-ui/react";

function ChatStream() {
  const { stream, chunks, streaming, error, reset } = useServingStream(
    { messages: [{ role: "user", content: "Hello" }] },
    {
      alias: "llm",
      onComplete: (finalChunks) => console.log("Done:", finalChunks.length, "chunks"),
    },
  );

  return (
    <>
      <button onClick={stream} disabled={streaming}>Send</button>
      <button onClick={reset}>Reset</button>
      {chunks.map((chunk, i) => <pre key={i}>{JSON.stringify(chunk)}</pre>)}
      {error && <p>{error}</p>}
    </>
  );
}

Non-streaming (one-shot inference, classification):

import { useServingInvoke } from "@databricks/appkit-ui/react";

function Classify() {
  const { invoke, data, loading, error } = useServingInvoke(
    { inputs: ["sample text"] },
    { alias: "classifier" },
  );

  return (
    <>
      <button onClick={() => invoke()} disabled={loading}>Classify</button>
      {data && <pre>{JSON.stringify(data)}</pre>}
      {error && <p>{error}</p>}
    </>
  );
}

Both hooks accept autoStart: true to invoke automatically on mount.

For the full hook API and type generation details, see npx @databricks/appkit docs ./docs/plugins/model-serving.md.

For off-platform streaming (AI SDK v6 with Databricks AI Gateway), see the `databricks-model-serving` skill.

AppKit integrates with Model Serving endpoints. AI Gateway (beta) endpoints are not directly supported — use the underlying Model Serving endpoint name instead. AI Gateway features (rate limits, usage tracking) can be configured on Model Serving endpoints via the databricks-model-serving skill.

Troubleshooting

AppKit Overview

AppKit is the recommended way to build Databricks Apps - provides type-safe SQL queries, React components, and seamless deployment.

Choose Your Data Pattern FIRST

Before scaffolding, decide which data pattern the app needs:

See Lakebase Guide for full Lakebase scaffolding and app-code patterns. See Genie Guide for space creation, plugin setup, and frontend components.

Workflow

1. Scaffold: Run databricks apps manifest, then databricks apps init with --features and --set as in parent SKILL.md (App Manifest and Scaffolding) 2. Develop: cd <NAME> && npm install && npm run dev 3. Validate: databricks apps validate 4. Deploy: databricks apps deploy --profile <PROFILE> (⚠️ USER CONSENT REQUIRED)

Data Discovery (Before Writing SQL)

Use the parent `databricks-core` skill for data discovery (table search, schema exploration, query execution).

Pre-Implementation Checklist

Before writing App.tsx, complete these steps:

1. ✅ Create SQL files in config/queries/ 2. ✅ Run npm run typegen to generate query types 3. ✅ Read client/src/appKitTypes.d.ts to see available query result types 4. ✅ Verify component props via npx @databricks/appkit docs (check the relevant component page) 5. ✅ Plan smoke test updates (default expects "Minimal Databricks App")

DO NOT write UI code until types are generated and verified.

Post-Implementation Checklist

Before running databricks apps validate:

1. ✅ Update tests/smoke.spec.ts heading selector to match your app title 2. ✅ Update or remove the 'hello world' text assertion 3. ✅ Verify npm run typegen has been run after all SQL files are finalized 4. ✅ Ensure all numeric SQL values use Number() conversion in display code

Project Structure

my-app/
├── server/
│   ├── server.ts             # Backend entry point (AppKit)
│   └── .env                  # Optional local dev env vars (do not commit)
├── client/
│   ├── index.html
│   ├── vite.config.ts
│   └── src/
│       ├── main.tsx
│       └── App.tsx           # <- Main app component (start here)
├── config/
│   └── queries/
│       └── my_query.sql      # -> queryKey: "my_query"
├── app.yaml                  # Deployment config
├── package.json
└── tsconfig.json

Key files to modify:

Type Safety

For type generation details, see: npx @databricks/appkit docs ./docs/development/type-generation.md

Quick workflow: 1. Add/modify SQL in config/queries/ 2. Types auto-generate during dev via the Vite plugin (or run npm run typegen manually) 3. Types appear in client/src/appKitTypes.d.ts

Adding Visualizations

Step 1: Create SQL file config/queries/my_data.sql

SELECT category, COUNT(*) as count FROM my_table GROUP BY category

Step 2: Use component (types auto-generated!)

import { BarChart } from '@databricks/appkit-ui/react';
// Query mode: fetches data automatically
<BarChart queryKey="my_data" parameters={{}} />

// Data mode: pass static data directly (no queryKey/parameters needed)
<BarChart data={myData} xKey="category" yKey="count" />

AppKit Official Documentation

Always use AppKit docs as the source of truth for API details.

npx @databricks/appkit docs                              # show the docs index (start here)
npx @databricks/appkit docs <query>                      # look up a section by name or doc path

Do not guess paths — run without args first, then pick from the index.

References

Critical Rules

1. SQL for data retrieval: Use config/queries/ + visualization components. Never custom endpoints for warehouse SELECT. 2. Numeric types: SQL numbers may return as strings. Always convert: Number(row.amount) 3. Type imports: Use import type { ... } (verbatimModuleSyntax enabled). 4. Charts are ECharts: No Recharts children — use props (xKey, yKey, colors). xKey/yKey auto-detect from schema if omitted. 5. Two data modes: Charts/tables support query mode (queryKey + parameters) and data mode (static data prop). 6. Conditional queries: Use autoStart: false option or conditional rendering to control query execution.

Decision Tree

• Display data from SQL?
• Chart/Table → BarChart, LineChart, DataTable components
• Custom layout (KPIs, cards) → useAnalyticsQuery hook
• Call Databricks API? → Dedicated plugin (serving, jobs, files) or custom endpoint via onPluginsReady
• Modify data? → Express routes in onPluginsReady

Plugin Contract Reference

Concrete proto↔plugin mappings for the three core AppKit plugins.

Files Plugin Contract

Plugin manifest: files/manifest.json Resource: UC Volume with WRITE_VOLUME permission Env: DATABRICKS_VOLUME_FILES for volume path

Boundary: What the files plugin owns

The files plugin is the ONLY module that touches UC Volumes. Other modules interact with files through typed proto messages, never raw paths.

┌─────────────┐    UploadRequest     ┌──────────────┐
│ api module   │ ──────────────────→ │ files plugin  │
│              │ ←────────────────── │              │
│              │    StoredArtifact    │  UC Volumes   │
└─────────────┘                      └──────────────┘

Proto → Plugin Method Mapping

Volume Path Convention (from VolumeLayout proto)

/Volumes/{catalog}/{schema}/{volume}/
├── uploads/                    # User uploads (UploadRequest.destination_path)
├── results/                    # Computed outputs (StoredArtifact)
│   └── {run_id}/
│       ├── output.proto.bin    # Binary proto serialization
│       └── output.json         # JSON for debugging
└── artifacts/                  # Build artifacts, archives
    └── {app_name}/
        └── {version}/

Config ↔ Proto Mapping

---

Lakebase Plugin Contract

Plugin manifest: lakebase/manifest.json Resource: Postgres with CAN_CONNECT_AND_CREATE permission Env: PGHOST, PGDATABASE, PGPORT, PGSSLMODE, LAKEBASE_ENDPOINT

Boundary: What the lakebase plugin owns

Lakebase owns ALL structured data. Every table's schema is derived from a proto message in database.proto. No ad-hoc CREATE TABLE statements.

┌─────────────┐    RunRecord         ┌──────────────┐
│ compute mod  │ ──────────────────→ │ lakebase      │
│              │                     │ plugin        │
│              │    MetricRecord     │              │
│              │ ──────────────────→ │  Postgres     │
└─────────────┘                      └──────┬───────┘
                                            │
┌─────────────┐    SQL query               │
│ analytics   │ ←──────────────────────────┘
│ module      │    RunRecord[]
└─────────────┘

Proto → Table Mapping

Proto → DDL Type Mapping

Migration Convention

migrations/
├── 001_create_runs.sql
├── 002_create_metrics.sql
├── 003_create_configs.sql
└── 004_add_metrics_index.sql

Each migration is idempotent (CREATE TABLE IF NOT EXISTS, CREATE INDEX IF NOT EXISTS).

Config ↔ Proto Mapping

---

Jobs / Compute Contract

No plugin manifest — Jobs are invoked via @databricks/sdk-experimental Resource: Databricks Jobs API Auth: Workspace token or OAuth

Boundary: What the jobs module owns

The jobs module owns compute execution. It receives typed task inputs, runs them on Databricks clusters, and produces typed task outputs.

┌─────────────┐    JobConfig         ┌──────────────┐
│ api module   │ ──────────────────→ │ jobs module   │
│              │                     │              │
│              │    JobTaskInput     │  Databricks   │
│              │ ──────────────────→ │  Jobs API     │
│              │                     │              │
│              │    JobTaskOutput    │  Clusters     │
│              │ ←────────────────── │              │
└─────────────┘                      └──────────────┘

Proto → Jobs SDK Mapping

Task Parameter Convention

Job tasks receive their typed input via: 1. Small payloads (<256KB): Base64-encoded proto in task params 2. Large payloads: Proto binary written to UC Volume, path passed as param

// Producer (api module)
const input: JobTaskInput = { taskId, taskType, runId, inputPayload };
const encoded = Buffer.from(JobTaskInput.encode(input).finish()).toString('base64');
// Pass as notebook parameter: { "input": encoded }

// Consumer (job task code)
const decoded = JobTaskInput.decode(Buffer.from(params.input, 'base64'));

Task Output Convention

Job tasks write their typed output to:

/Volumes/{catalog}/{schema}/{volume}/results/{run_id}/{task_id}.output.bin

The output is a serialized JobTaskOutput proto. The orchestrator reads it back with the generated decoder.

Jobs API Patterns

// Create a multi-task job from JobConfig proto
const jobConfig: JobConfig = {
  jobName: `${appName}-${runId}`,
  clusterSpec: '{"num_workers": 1}',
  maxRetries: 2,
  timeoutSeconds: 3600,
  tasks: [
    { taskKey: 'generate', taskType: 'generate', dependsOn: [] },
    { taskKey: 'evaluate', taskType: 'evaluate', dependsOn: ['generate'] },
    { taskKey: 'aggregate', taskType: 'aggregate', dependsOn: ['evaluate'] },
  ],
};

Proto-First App Design

Schema-first approach for AppKit apps using protobuf data contracts. Define contracts BEFORE implementation — derive TypeScript types, Lakebase DDL, and Volume paths from .proto files.

When to use: New apps with multiple plugins (files + lakebase + jobs), or adding typed boundaries to existing apps. Skip for quick prototypes.

Requires: buf CLI for proto linting and code generation.

Rule: No implementation before contracts. No contracts without consumers.

Define protobuf data contracts FIRST, then derive everything else (TypeScript types, Lakebase DDL, Volume paths, API shapes) from those contracts.

When to Use

Core Principle

User intent → Module map → Proto contracts → Generated types → Implementation
                                  ↓                    ↓
                            Lakebase DDL        TypeScript interfaces
                                  ↓                    ↓
                            Migrations           Plugin code

The .proto file is the single source of truth. If it's not in a proto, it doesn't cross a module boundary.

Phase 1: Decompose into Modules

Every Databricks app decomposes into a combination of these plugin modules:

Decomposition Rules

1. Each module owns its data — files plugin never writes to lakebase, lakebase never writes to volumes. 2. Cross-module communication is typed — a proto message, never a raw JSON blob. 3. Every proto message has exactly one producer module. 4. Multiple modules can consume — but the producer defines the schema. 5. No god messages — if a message has >12 fields, split it.

Output: Module Map

Before proceeding, produce a module map for the user to confirm:

App: <app-name>
Modules:
  storage:   files plugin    → uploads/, results/, artifacts/
  db:        lakebase plugin → runs, metrics, configs tables
  compute:   jobs            → generation tasks, eval tasks
  api:       server plugin   → POST /run, GET /status, SSE /stream

Phase 2: Define Proto Contracts

Directory Structure

proto/
├── buf.yaml
├── buf.gen.yaml
└── <app>/
    └── v1/
        ├── common.proto     # Shared enums, IDs
        ├── storage.proto    # Files plugin boundary
        ├── database.proto   # Lakebase plugin boundary
        ├── compute.proto    # Jobs boundary
        └── api.proto        # Server/API boundary

Proto Style Rules

• Package: <app>.v1 (versioned from day one)
• One file per module boundary, not per message
• Every field has a consumer — if no code reads it, delete it
• snake_case for all field names
• proto3 syntax only

Files Plugin Boundary (storage.proto)

The files plugin operates on UC Volumes. Type every file path and payload:

syntax = "proto3";
package <app>.v1;

import "google/protobuf/timestamp.proto";

// StoredArtifact — produced by files plugin after upload.
message StoredArtifact {
  string volume_path = 1;
  string content_type = 2;
  int64 size_bytes = 3;
  google.protobuf.Timestamp created_at = 4;
  string checksum_sha256 = 5;
}

// UploadRequest — sent to files plugin by api module.
message UploadRequest {
  string destination_path = 1;
  string content_type = 2;
  bytes content = 3;
  map<string, string> metadata = 4;
}

// VolumeLayout — design-time contract for volume directory structure.
message VolumeLayout {
  string root = 1;          // /Volumes/catalog/schema/app_name
  string uploads_dir = 2;   // uploads/
  string results_dir = 3;   // results/
  string artifacts_dir = 4; // artifacts/
}

Lakebase Plugin Boundary (database.proto)

Every Lakebase table has a corresponding proto message. The message IS the schema:

syntax = "proto3";
package <app>.v1;

import "google/protobuf/timestamp.proto";

// RunRecord — one row in the `runs` table.
// Producer: compute module. Consumers: api, analytics.
message RunRecord {
  string run_id = 1;
  string app_name = 2;
  RunStatus status = 3;
  google.protobuf.Timestamp started_at = 4;
  google.protobuf.Timestamp completed_at = 5;
  string error_message = 6;
  string config_json = 7;
}

// MetricRecord — one row in the `metrics` table.
// Producer: compute module. Consumers: analytics, api.
message MetricRecord {
  string run_id = 1;
  string metric_name = 2;
  double value = 3;
  google.protobuf.Timestamp recorded_at = 4;
  map<string, string> dimensions = 5;
}

Jobs Boundary (compute.proto)

Type job task inputs and outputs:

syntax = "proto3";
package <app>.v1;

// JobTaskInput — typed payload sent to a Databricks job task.
// Producer: api module. Consumer: job task code.
message JobTaskInput {
  string task_id = 1;
  string task_type = 2;
  string run_id = 3;
  bytes input_payload = 4;
  map<string, string> env = 5;
}

// JobTaskOutput — typed result from a completed job task.
// Producer: job task code. Consumer: api module.
message JobTaskOutput {
  string task_id = 1;
  string run_id = 2;
  bool success = 3;
  string error = 4;
  bytes output_payload = 5;
  int64 duration_ms = 6;
  map<string, string> metrics = 7;
}

Phase 3: Generate Types and DDL

3a. Buf configuration

# buf.yaml
version: v2
lint:
  use:
    - STANDARD
breaking:
  use:
    - FILE

# buf.gen.yaml
version: v2
plugins:
  - remote: buf.build/connectrpc/es
    out: proto/gen
    opt: target=ts

3b. Generate TypeScript types

buf lint proto/
buf generate proto/

3c. Generate Lakebase DDL

For each message in database.proto, generate a numbered migration file.

Proto→SQL type mapping:

Example migration:

-- migrations/001_create_runs.sql
CREATE TABLE IF NOT EXISTS runs (
  run_id TEXT NOT NULL,
  app_name TEXT NOT NULL,
  status TEXT NOT NULL DEFAULT 'RUN_STATUS_PENDING',
  started_at TIMESTAMPTZ,
  completed_at TIMESTAMPTZ,
  error_message TEXT,
  config_json JSONB,
  PRIMARY KEY (run_id, app_name)
);

3d. Validate

npx tsc --noEmit  # all generated types compile
buf lint proto/    # proto style checks

Phase 4: Implement Against Contracts

NOW implementation begins. Each module uses ONLY its generated types:

import type { StoredArtifact, UploadRequest } from '../proto/gen/<app>/v1/storage';
import type { RunRecord, MetricRecord } from '../proto/gen/<app>/v1/database';
import type { JobTaskInput, JobTaskOutput } from '../proto/gen/<app>/v1/compute';

No any, no unknown, no JSON.parse() at module boundaries.

Validation Checklist

Before writing implementation code:

• [ ] Module map exists with clear data boundaries
• [ ] Proto files exist for every cross-boundary data structure
• [ ] buf lint proto/ passes
• [ ] buf generate proto/ produces TypeScript types
• [ ] Lakebase DDL derived from database.proto messages
• [ ] No proto message exceeds 12 fields
• [ ] Every field has at least one identified consumer
• [ ] Every message has exactly one producer module
• [ ] Volume layout documented (not freeform paths)
• [ ] Job inputs/outputs typed (no raw JSON params)

Common Traps

References

• Plugin Contract Details — proto↔plugin type mappings for files, lakebase, jobs

SQL Query Files

IMPORTANT: ALWAYS use SQL files in config/queries/ for data retrieval. NEVER add custom endpoints for warehouse SQL queries.

• Store ALL SQL queries in config/queries/ directory
• Name files descriptively: trip_statistics.sql, user_metrics.sql, sales_by_region.sql
• Reference by filename (without extension) in useAnalyticsQuery or directly in a visualization component passing it as queryKey
• App Kit automatically executes queries against configured Databricks warehouse
• Benefits: Built-in caching, proper connection pooling, better performance

Type Generation

For full type generation details, see: npx @databricks/appkit docs ./docs/development/type-generation.md

Type generation: Types are auto-regenerated during dev whenever SQL files change.

Quick workflow: Add SQL files → Types auto-generate during dev → Types appear in client/src/appKitTypes.d.ts

Query Schemas (Optional)

Create config/queries/schema.ts only if you need runtime validation with Zod.

import { z } from 'zod';

export const querySchemas = {
  my_query: z.array(
    z.object({
      category: z.string(),
      // Use z.coerce.number() - handles both string and number from SQL
      amount: z.coerce.number(),
    })
  ),
};

Why `z.coerce.number()`?

• Auto-generated types use number based on SQL column types
• But some SQL types (DECIMAL, large BIGINT) return as strings at runtime
• z.coerce.number() handles both cases safely

SQL Type Handling (Critical)

Understanding Type Generation vs Runtime:

1. Auto-generated types (appKitTypes.d.ts): Based on SQL column types

• BIGINT, INT, DECIMAL → TypeScript number
• These are the types you'll see in IntelliSense

2. Runtime JSON values: Some numeric types arrive as strings

• DECIMAL often returns as string (e.g., "123.45")
• Large BIGINT values return as string
• ROUND(), AVG(), SUM() results may be strings

Best Practice - Always convert before numeric operations:

// ❌ WRONG - may fail if value is string at runtime
<span>{row.total_amount.toFixed(2)}</span>

// ✅ CORRECT - convert to number first
<span>{Number(row.total_amount).toFixed(2)}</span>

Helper Functions:

Create app-specific helpers for consistent numeric formatting (for example in client/src/lib/formatters.ts):

// client/src/lib/formatters.ts
export const toNumber = (value: number | string): number => Number(value);
export const formatCurrency = (value: number | string): string =>
  `$${Number(value).toFixed(2)}`;
export const formatPercent = (value: number | string): string =>
  `${Number(value).toFixed(1)}%`;

Use them wherever you render query results:

import { toNumber, formatCurrency, formatPercent } from './formatters'; // adjust import path to your file layout

// Convert to number
const amount = toNumber(row.amount);  // "123.45" → 123.45

// Format as currency
const formatted = formatCurrency(row.amount);  // "123.45" → "$123.45"

// Format as percentage
const percent = formatPercent(row.rate);  // "85.5" → "85.5%"

Available sql.* Helpers

Full API reference: npx @databricks/appkit docs ./docs/api/appkit/Variable.sql.md — always check this for the latest available helpers.

import { sql } from "@databricks/appkit-ui/js";

// ✅ These exist:
sql.string(value)      // For STRING parameters
sql.number(value)      // For NUMERIC parameters (INT, BIGINT, DOUBLE, DECIMAL)
sql.boolean(value)     // For BOOLEAN parameters
sql.date(value)        // For DATE parameters (YYYY-MM-DD format)
sql.timestamp(value)   // For TIMESTAMP parameters
sql.binary(value)      // For BINARY (returns hex string, use UNHEX() in SQL)

// ❌ These DO NOT exist:
// sql.null()     - use sentinel values instead
// sql.array()    - use comma-separated sql.string() and split in SQL
// sql.int()      - use sql.number()
// sql.float()    - use sql.number()

For nullable string parameters, use sentinel values or empty strings. For nullable date parameters, use sentinel dates only (empty strings cause validation errors) — see "Optional Date Parameters" section below.

Databricks SQL Dialect

Databricks uses Databricks SQL (based on Spark SQL), NOT PostgreSQL/MySQL. Common mistakes:

Sample data date ranges — do NOT use CURRENT_DATE() on historical datasets:

• samples.tpch.* — historical dates, check with SELECT MIN(o_orderdate), MAX(o_orderdate) FROM samples.tpch.orders
• samples.nyctaxi.trips — NYC taxi data with specific date ranges
• samples.tpcds.* — data from 1998-2003

Always check date ranges before writing date-filtered queries.

Before Running npm run typegen

Verify each SQL file before running typegen:

• [ ] Uses Databricks SQL syntax (NOT PostgreSQL) — check dialect table above
• [ ] DATEDIFF has 3 arguments: DATEDIFF(DAY, start, end)
• [ ] Uses LOWER(col) LIKE LOWER(pattern) instead of ILIKE
• [ ] Column aliases in ORDER BY match SELECT aliases exactly
• [ ] Date columns are not passed to numeric functions like ROUND()
• [ ] Date range filters use actual data dates (NOT CURRENT_DATE() on historical data — check date ranges first)

Query Parameterization

SQL queries can accept parameters to make them dynamic and reusable.

Key Points:

• Parameters use colon prefix: :parameter_name
• Databricks infers types from values automatically
• For optional string parameters, use pattern: (:param = '' OR column = :param)
• For optional date parameters, use sentinel dates ('1900-01-01' and '9999-12-31') instead of empty strings

SQL Parameter Syntax

-- config/queries/filtered_data.sql
SELECT *
FROM my_table
WHERE column_value >= :min_value
  AND column_value <= :max_value
  AND category = :category
  AND (:optional_filter = '' OR status = :optional_filter)

Frontend Parameter Passing

import { sql } from "@databricks/appkit-ui/js";

const { data } = useAnalyticsQuery('filtered_data', {
  min_value: sql.number(minValue),
  max_value: sql.number(maxValue),
  category: sql.string(category),
  optional_filter: sql.string(optionalFilter || ''),  // empty string for optional params
});

Date Parameters

Use sql.date() for date parameters with YYYY-MM-DD format strings.

Frontend - Using Date Parameters:

import { sql } from '@databricks/appkit-ui/js';
import { useState } from 'react';

function MyComponent() {
  const [startDate, setStartDate] = useState<string>('2016-02-01');
  const [endDate, setEndDate] = useState<string>('2016-02-29');

  const queryParams = {
    start_date: sql.date(startDate),  // Pass YYYY-MM-DD string to sql.date()
    end_date: sql.date(endDate),
  };

  const { data } = useAnalyticsQuery('my_query', queryParams);

  // ...
}

SQL - Date Filtering:

-- Filter by date range using DATE() function
SELECT COUNT(*) as trip_count
FROM samples.nyctaxi.trips
WHERE DATE(tpep_pickup_datetime) >= :start_date
  AND DATE(tpep_pickup_datetime) <= :end_date

Date Helper Functions:

// Helper to get YYYY-MM-DD string for dates relative to today
const daysAgo = (n: number): string => {
  const date = new Date(Date.now() - n * 86400000);
  return date.toISOString().split('T')[0];  // "2024-01-15"
};

const params = {
  start_date: sql.date(daysAgo(7)),   // 7 days ago
  end_date: sql.date(daysAgo(0)),     // Today
};

Optional Date Parameters - Use Sentinel Dates

Databricks App Kit validates parameter types before query execution. DO NOT use empty strings (`''`) for optional date parameters as this causes validation errors.

✅ CORRECT - Use Sentinel Dates:

// Frontend: Use sentinel dates for "no filter" instead of empty strings
const revenueParams = {
  group_by: 'month',
  start_date: sql.date('1900-01-01'),  // Sentinel: effectively no lower bound
  end_date: sql.date('9999-12-31'),    // Sentinel: effectively no upper bound
  country: sql.string(country || ''),
  property_type: sql.string(propertyType || ''),
};

-- SQL: Simple comparison since sentinel dates are always valid
WHERE b.check_in >= CAST(:start_date AS DATE)
  AND b.check_in <= CAST(:end_date AS DATE)

Why Sentinel Dates Work:

• 1900-01-01 is before any real data (effectively no lower bound filter)
• 9999-12-31 is after any real data (effectively no upper bound filter)
• Always valid DATE types, so no parameter validation errors
• All real dates fall within this range, so no filtering occurs

Parameter Types Summary:

• ALWAYS use sql.* helper functions from the @databricks/appkit-ui/js package to define SQL parameters
• Strings/Numbers: Use directly in SQL with :param_name
• Dates: Use with CAST(:param AS DATE) in SQL
• Optional Strings: Use empty string default, check with (:param = '' OR column = :param)
• Optional Dates: Use sentinel dates (sql.date('1900-01-01') and sql.date('9999-12-31')) instead of empty strings