Databricks Dabs

Declarative Automation Bundles (DABs)

Use this skill for any bundle-related request including creating, configuring, validating, deploying, running, and managing Databricks resources through DABs.

Reference Documentation

The following reference files provide detailed guidance for specific bundle tasks:

• [Bundle Structure](references/bundle-structure.md) - Bundle structure, databricks.yml configuration, resource definitions, path resolution, variables, and multi-environment targets
• [SDP Pipelines](references/sdp-pipelines.md) - Spark Declarative Pipeline configurations for DABs
• [SQL Alerts](references/alerts.md) - SQL Alert schemas and configuration (critical - API differs from other resources)
• [Deploy and Run](references/deploy-and-run.md) - Validation, deployment, running resources, monitoring logs, and troubleshooting common issues
• [Resource Permissions](references/resource-permissions.md) - Permission levels and access control for bundle resources, per-resource-type levels, grants vs permissions

When to Use This Skill

Load this skill for any request involving:

• Creating new bundle projects or resources
• Configuring databricks.yml or resource YAML files
• Setting up multi-environment deployments (dev/prod targets)
• Deploying or running bundle resources
• Managing permissions for bundle resources
• Troubleshooting bundle validation or deployment errors
• Working with specific resource types (dashboards, jobs, pipelines, alerts, volumes, apps)

General Guidelines

1. Always validate after configuration changes - Use bundle validate --strict --target <target> after any change 2. Use reference documentation - Consult the appropriate reference file for detailed patterns and examples 3. Follow naming conventions - Resource files should use <name>.<resource_type>.yml format 4. Path resolution is critical - Paths differ based on file location (see Bundle Structure reference) 5. Preserve existing structure - Keep user comments and structure when editing YAML files 6. Use variables - Parameterize catalog, schema, and warehouse for multi-environment support

Documentation

• Declarative Automation Bundles
• Bundle Examples Repository - official end-to-end example bundles (jobs, pipelines, dashboards, apps, and more); use as working references for patterns not covered by the reference files above

SQL Alerts Resources for DABs

Critical: Schema Validation First

ALWAYS start by inspecting the schema:

databricks bundle schema | grep -A 100 'sql.AlertV2'

The Alert v2 API schema differs significantly from other resources. Don't assume field names.

Common Schema Mistakes

WRONG — These fields don't exist:

condition: # Should be "evaluation"
  op: LESS_THAN
  operand:
    column: # Wrong nesting
      name: 'r'

schedule:
  cron_schedule: # Should be direct fields under schedule
    quartz_cron_expression: '...'

subscriptions: # Should be under evaluation.notification
  - destination_type: 'EMAIL'

CORRECT — Alerts v2 API structure:

evaluation: # Not "condition"
  comparison_operator: 'LESS_THAN_OR_EQUAL'
  source: # Not nested under "operand.column"
    name: 'column_name'
    display: 'column_name'
  threshold:
    value:
      double_value: 100
  notification: # Subscriptions nested here
    notify_on_ok: false
    subscriptions:
      - user_email: '${workspace.current_user.userName}'

schedule: # Fields directly under schedule
  pause_status: 'UNPAUSED' # REQUIRED
  quartz_cron_schedule: '0 38 16 * * ?' # REQUIRED
  timezone_id: 'America/Los_Angeles' # REQUIRED

Alert Trigger Logic

Critical: Alerts trigger when condition evaluates to TRUE, not FALSE.

Example: Alert when count is NOT > 100 (i.e., <= 100):

# WRONG - This triggers when count IS > 100
comparison_operator: 'GREATER_THAN'

# CORRECT - This triggers when count IS <= 100
comparison_operator: 'LESS_THAN_OR_EQUAL'

Complete Alert Resource

resources:
  alerts:
    alert_name:
      display_name: '[${bundle.target}] Alert Name' # REQUIRED
      query_text: 'SELECT count(*) c FROM table' # REQUIRED
      warehouse_id: ${var.warehouse_id} # REQUIRED

      evaluation: # REQUIRED
        comparison_operator: 'LESS_THAN' # REQUIRED
        source: # REQUIRED
          name: 'c'
          display: 'c'
        threshold:
          value:
            double_value: 100
        notification:
          notify_on_ok: false
          subscriptions:
            - user_email: '${workspace.current_user.userName}'

      schedule: # REQUIRED
        pause_status: 'UNPAUSED' # REQUIRED
        quartz_cron_schedule: '0 0 9 * * ?' # REQUIRED
        timezone_id: 'America/Los_Angeles' # REQUIRED

      permissions:
        - level: CAN_RUN
          group_name: 'users'

Reference

Comparison operators: EQUAL, NOT_EQUAL, GREATER_THAN, GREATER_THAN_OR_EQUAL, LESS_THAN, LESS_THAN_OR_EQUAL

Permission levels: CAN_READ, CAN_RUN (recommended), CAN_EDIT, CAN_MANAGE

Quartz cron format: second minute hour day-of-month month day-of-week (use ? for day-of-week with * day-of-month)

Examples: '0 0 9 * * ?' (9 AM daily), '0 */30 * * * ?' (every 30 min)

Write Declarative Automation Bundles

Bundle Structure

project/
├── databricks.yml           # Main config + targets
├── resources/               # Resource definitions (one YAML file per resource)
│   ├── my_job.job.yml
│   ├── my_pipeline.pipeline.yml
│   └── my_dashboard.dashboard.yml
└── src/                     # Code/dashboard files
    ├── notebook.py
    └── pipeline.py

Resource file naming convention: <name>.<resource_type>.yml (e.g., my_job.job.yml, my_pipeline.pipeline.yml, my_dashboard.dashboard.yml)

Main Configuration (databricks.yml)

bundle:
  name: project-name

include:
  - resources/*.yml

variables:
  catalog:
    default: 'default_catalog'
  schema:
    default: 'default_schema'
  warehouse_id:
    lookup:
      warehouse: 'Shared SQL Warehouse'

targets:
  dev:
    default: true
    mode: development
    workspace:
      profile: dev-profile
    variables:
      catalog: 'dev_catalog'
      schema: 'dev_schema'

  prod:
    mode: production
    workspace:
      profile: prod-profile
    variables:
      catalog: 'prod_catalog'
      schema: 'prod_schema'

Path Resolution

Critical: Paths depend on file location:

Why: resources/ files are one level deep, so use ../ to reach bundle root. databricks.yml is at root, so use ./

Dashboard Resources

Support for dataset_catalog and dataset_schema parameters added in Databricks CLI 0.281.0 (January 2026)

resources:
  dashboards:
    dashboard_name:
      display_name: 'Dashboard Title'
      file_path: ../src/dashboards/dashboard.lvdash.json
      warehouse_id: ${var.warehouse_id}
      dataset_catalog: ${var.catalog}
      dataset_schema: ${var.schema}

Jobs Resources

resources:
  jobs:
    job_name:
      name: 'Job Name'
      tasks:
        - task_key: 'main_task'
          notebook_task:
            notebook_path: ../src/notebooks/main.ipynb
          new_cluster:
            spark_version: '17.3.x-scala2.13'
            node_type_id: 'i3.xlarge'
            num_workers: 2
      schedule:
        quartz_cron_expression: '0 0 9 * * ?'
        timezone_id: 'America/Los_Angeles'

Job Resource (Shared Classic Cluster)

When multiple tasks should reuse the same cluster, declare it once under job_clusters and reference it via job_cluster_key:

resources:
  jobs:
    sample_job:
      name: sample_job
      tasks:
        - task_key: notebook_task
          notebook_task:
            notebook_path: ../src/sample_notebook.ipynb
          job_cluster_key: job_cluster
          libraries:
            - whl: ../dist/*.whl

        - task_key: main_task
          depends_on:
            - task_key: notebook_task
          python_wheel_task:
            package_name: my_project
            entry_point: main
          job_cluster_key: job_cluster
          libraries:
            - whl: ../dist/*.whl

      job_clusters:
        - job_cluster_key: job_cluster
          new_cluster:
            spark_version: 16.4.x-scala2.12
            node_type_id: i3.xlarge
            data_security_mode: SINGLE_USER
            autoscale:
              min_workers: 1
              max_workers: 4

Registered Model Resources

resources:
  registered_models:
    customer_churn:
      name: '${var.catalog}.${var.schema}.customer_churn_model'
      description: 'Customer churn prediction model'

Volume Resources

resources:
  volumes:
    my_volume:
      catalog_name: ${var.catalog}
      schema_name: ${var.schema}
      name: 'volume_name'
      volume_type: 'MANAGED'

Volumes use `grants` not `permissions` — different format from other resources.

Apps Resources

Apps resource support added in Databricks CLI 0.239.0 (January 2025)

Apps have minimal configuration — environment variables are defined in app.yaml in the source directory, NOT in databricks.yml.

Generate from Existing App (Recommended)

databricks bundle generate app --existing-app-name my-app --key my_app --profile DEFAULT

Manual Configuration

resources/my_app.app.yml:

resources:
  apps:
    my_app:
      name: my-app-${bundle.target}
      description: 'My application'
      source_code_path: ../src/app

src/app/app.yaml:

command:
  - 'python'
  - 'dash_app.py'

env:
  - name: USE_MOCK_BACKEND
    value: 'false'
  - name: DATABRICKS_WAREHOUSE_ID
    value: 'your-warehouse-id'
  - name: DATABRICKS_CATALOG
    value: 'main'
  - name: DATABRICKS_SCHEMA
    value: 'my_schema'

Important: When source code is in project root (not src/app), use source_code_path: .. in the resource file.

Generate Configuration for Existing Resources

databricks bundle generate job <job-id>
databricks bundle generate pipeline <pipeline-id>
databricks bundle generate dashboard <dashboard-id>
databricks bundle generate app <app-name>

Other Resources

DABs supports schemas, models, experiments, clusters, warehouses, etc. Use databricks bundle schema to inspect schemas.

Substitutions

Substitutions are resolved at deploy time and are usable in any string field across databricks.yml, resource files, and variable defaults.

Variables themselves are declared in databricks.yml (with optional default: or lookup:) and overridden per target.

Key Principles

1. Path resolution: ../src/ in resources/\.yml, `./src/` in databricks.yml 2. Variables: Parameterize catalog, schema, warehouse 3. Mode: `development` for dev/staging, `production` for prod 4. Groups*: Use "users" for all workspace users

Deploy and Run Declarative Automation Bundles

Initialization

Start a new bundle interactively:

databricks bundle init

Built-in templates:

Pass a template name or a Git URL pointing at a template directory to skip the interactive picker.

Generate from Existing Resources

If a workspace already has the resource, generate its bundle YAML instead of writing it by hand:

databricks bundle generate job <job-id>
databricks bundle generate pipeline <pipeline-id>
databricks bundle generate dashboard <dashboard-id>
databricks bundle generate app <app-name>

This writes a resource file under resources/ plus any referenced source assets.

Validation

Validate bundle configuration:

• bundle validate --strict
• bundle validate --strict -t prod

Always validate with the `--strict` flag after any configuration change. The --strict flag ensures that warnings are treated as errors, catching issues that would otherwise be missed.

Deployment

Deploy:

• bundle deploy
• bundle deploy -t prod
• bundle deploy --auto-approve
• bundle deploy --force

For dev targets you can deploy without user consent. This allows you to run resources on the workspace too!

Skip validation before deployment for dev targets. Deployment itself will surface any issues, so a separate validation step is unnecessary.

Running Resources

Run resources:

• bundle run resource_name
• bundle run pipeline_name -t prod
• bundle run app_resource_key -t dev

View status: bundle summary

Destroy

bundle destroy removes everything the bundle previously deployed to the target workspace. It is destructive; confirm the target before running it.

• bundle destroy -t dev
• bundle destroy -t prod

Monitoring and Logs

databricks apps logs <app-name> --profile <profile-name>

Diagnosing Errors

• Read the error message from the CLI output to understand the issue, then inspect the relevant bundle files to diagnose the root cause.
• After diagnosing, provide a clear explanation and suggest concrete fixes.
• After fixing an error, validate the fix with the appropriate command:
• bundle summary if the error was in summary
• bundle deploy if the error was during deployment
• bundle validate --strict otherwise

Common Issues

DABs Resource Permissions

Permission Levels by Resource Type

Standard Permission Block

permissions:
  - level: CAN_VIEW
    group_name: 'users'

Use "users" for all workspace users.

Volume Grants (Different Format)

Volumes use grants not permissions:

resources:
  volumes:
    my_volume:
      catalog_name: ${var.catalog}
      schema_name: ${var.schema}
      name: 'volume_name'
      volume_type: 'MANAGED'
      # grants:
      #   - principal: "group_name"
      #     privileges:
      #       - "READ_VOLUME"

Common Mistakes

SDP Pipeline Configuration for DABs

Key Decisions (prompt if unclear)

1. Streaming or batch oriented? 2. Continuous or triggered execution? 3. Serverless (default) or classic compute?

Pipeline Resource Pattern

resources:
  pipelines:
    pipeline_name:
      name: 'Pipeline Name'

      catalog: ${var.catalog}
      target: ${var.schema}

      libraries:
        - glob:
            include: ../src/pipelines/<pipeline_folder>/transformations/**

      root_path: ../src/pipelines/<pipeline_folder>

      serverless: true

      configuration:
        source_catalog: ${var.source_catalog}
        source_schema: ${var.source_schema}

      continuous: false
      development: true
      photon: true

      channel: current

      permissions:
        - level: CAN_VIEW
          group_name: 'users'

Permission levels: CAN_VIEW, CAN_RUN, CAN_MANAGE

Best Practices

1. Use `root_path` and `libraries.glob` for newer organization structure 2. Default to serverless unless user specifies otherwise 3. Use variables for catalog/schema parameterization 4. Set `development: true` for dev/staging targets