Databricks Core

Databricks

Core skill for Databricks CLI, authentication, and data exploration.

Product Skills

For specific products, use dedicated skills:

• databricks-jobs - Lakeflow Jobs development and deployment
• databricks-pipelines - Lakeflow Spark Declarative Pipelines (batch and streaming data pipelines)
• databricks-apps - Full-stack TypeScript app development and deployment
• databricks-lakebase - Lakebase Postgres Autoscaling project management
• databricks-model-serving - Model Serving endpoint management and inference

Prerequisites

1. CLI installed: Run databricks --version to check.

• If the CLI is missing or outdated (< v0.292.0): STOP. Do not proceed or work around a missing CLI.
• Read the [CLI Installation](databricks-cli-install.md) reference file and follow the instructions to guide the user through installation.
• Note: In sandboxed environments (Cursor IDE, containers), install commands write outside the workspace and may be blocked. Present the install command to the user and ask them to run it in their own terminal.
• Exception: If CLI installation is blocked (sandboxed containers, restricted environments), ask the user whether to fall back to direct REST API calls using DATABRICKS_HOST and DATABRICKS_TOKEN environment variables if present in the shell. See the Databricks REST API docs.

2. Authenticated: databricks auth profiles

• If not: see CLI Authentication

Profile Selection - CRITICAL

NEVER auto-select a profile.

1. List profiles: databricks auth profiles 2. Present ALL profiles to user with workspace URLs 3. Let user choose (even if only one exists) 4. Offer to create new profile if needed

Claude Code - IMPORTANT

Each Bash command runs in a separate shell session.

# WORKS: --profile flag
databricks apps list --profile my-workspace

# WORKS: chained with &&
export DATABRICKS_CONFIG_PROFILE=my-workspace && databricks apps list

# DOES NOT WORK: separate commands
export DATABRICKS_CONFIG_PROFILE=my-workspace
databricks apps list  # profile not set!

Data Exploration — Use AI Tools

Use these instead of manually navigating catalogs/schemas/tables:

# discover table structure (columns, types, sample data, stats)
databricks experimental aitools tools discover-schema catalog.schema.table --profile <PROFILE>

# run ad-hoc SQL queries
databricks experimental aitools tools query "SELECT * FROM table LIMIT 10" --profile <PROFILE>

# find the default warehouse
databricks experimental aitools tools get-default-warehouse --profile <PROFILE>

Names are literal. Use catalog/schema/table names exactly as given — never change a hyphen to an underscore or otherwise normalize them. In SQL, backtick-quote any name part with special characters (e.g. ` my-catalog.schema.table `); unquoted hyphens cause a parse error.

See Data Exploration for details.

Quick Reference

⚠️ CRITICAL: Some commands use positional arguments, not flags

# current user
databricks current-user me --profile <PROFILE>

# list resources
databricks apps list --profile <PROFILE>
databricks jobs list --profile <PROFILE>
databricks clusters list --profile <PROFILE>
databricks warehouses list --profile <PROFILE>
databricks pipelines list --profile <PROFILE>
databricks serving-endpoints list --profile <PROFILE>

# ⚠️ Unity Catalog — POSITIONAL arguments (NOT flags!)
databricks catalogs list --profile <PROFILE>

# ✅ CORRECT: positional args
databricks schemas list <CATALOG> --profile <PROFILE>
databricks tables list <CATALOG> <SCHEMA> --profile <PROFILE>
databricks tables get <CATALOG>.<SCHEMA>.<TABLE> --profile <PROFILE>

# ❌ WRONG: these flags/commands DON'T EXIST
# databricks schemas list --catalog-name <CATALOG>    ← WILL FAIL
# databricks tables list --catalog <CATALOG>           ← WILL FAIL
# databricks sql-warehouses list                       ← doesn't exist, use `warehouses list`
# databricks execute-statement                         ← doesn't exist, use `experimental aitools tools query`
# databricks sql execute                               ← doesn't exist, use `experimental aitools tools query`

# When in doubt, check help:
# databricks schemas list --help

# get details
databricks apps get <NAME> --profile <PROFILE>
databricks jobs get --job-id <ID> --profile <PROFILE>
databricks clusters get --cluster-id <ID> --profile <PROFILE>

# bundles
databricks bundle init --profile <PROFILE>
databricks bundle validate --profile <PROFILE>
databricks bundle deploy -t <TARGET> --profile <PROFILE>
databricks bundle run <RESOURCE> -t <TARGET> --profile <PROFILE>

Troubleshooting

Required Reading by Task

Reference Guides

• CLI Installation
• CLI Authentication
• Data Exploration

Data Exploration

Tools for discovering table schemas and executing SQL queries in Databricks.

Finding Tables by Keyword

⚠️ START HERE if you don't know which catalog/schema contains your data.

Use information_schema to search for tables by keyword — do NOT manually iterate through catalogs list → schemas list → tables list. Manual enumeration wastes 10+ steps.

# Find tables matching a keyword
databricks experimental aitools tools query \
  "SELECT table_catalog, table_schema, table_name FROM system.information_schema.tables WHERE table_name LIKE '%keyword%'" \
  --profile <PROFILE>

# Then discover schema for the tables you found
databricks experimental aitools tools discover-schema catalog.schema.table1 catalog.schema.table2 --profile <PROFILE>

Identifier Names & Quoting — Read Before Writing Any Query

Use catalog, schema, and table names EXACTLY as given. Never normalize them: do not change a hyphen (-) to an underscore (_), and do not add or drop characters. hello-world and hello_world are different catalogs — silently "fixing" the name produces NO_SUCH_CATALOG / TABLE_OR_VIEW_NOT_FOUND against an object that does not exist. (Case is not significant — Unity Catalog stores names lowercase — but there is no reason to alter what you were given.)

Hyphens and other special characters are valid in Unity Catalog names. For catalogs, only ., space, and / are disallowed — a hyphen is fine. So a name like hello-world is a real, legal catalog name; do not assume it "must" be an underscore.

In SQL, backtick-quote any identifier part that contains a character outside `[a-zA-Z0-9_]` (hyphens, spaces, etc.), per part. Unquoted, hello-world is parsed as hello minus world, which is a syntax error. Quote only the parts that need it:

-- ❌ Renamed the catalog (hyphen → underscore): catalog does not exist
SHOW TABLES IN hello_world.demo

-- ❌ Correct name, but unquoted hyphen → PARSE_SYNTAX_ERROR
SHOW TABLES IN hello-world.demo

-- ✅ Correct name, special-character part backtick-quoted
SHOW TABLES IN `hello-world`.demo

-- ✅ Fully qualified, each special-character part quoted independently
SELECT * FROM `hello-world`.demo.items LIMIT 10

CLI positional arguments are not SQL — pass the literal name, no backticks. Commands like discover-schema CATALOG.SCHEMA.TABLE and tables get CATALOG.SCHEMA.TABLE take the plain name (e.g. discover-schema hello-world.demo.items). Backticks belong only inside the SQL string of ... tools query "<SQL>".

Note: legacy hive_metastore is stricter than Unity Catalog — table names there allow

only alphanumeric ASCII and underscores, so hyphens are not valid even with backticks.

This guidance is for Unity Catalog names.

References (Databricks):

Names — allowed

characters in catalog/schema/table names ·

SQL identifiers —

backtick-quoting rules for special characters.

Overview

The databricks experimental aitools tools command group provides tools for data discovery and exploration:

• discover-schema: Batch discover table metadata, columns, types, sample data, and statistics
• query: Execute SQL queries against Databricks SQL warehouses

When to use this: Use these commands whenever you need to:

• Discover table schemas and metadata
• Execute SQL queries against warehouse data
• Explore data structure and content
• Validate data or check table statistics

Prerequisites

1. Authenticated Databricks CLI - see CLI Authentication Guide for OAuth2 setup and profile configuration 2. Access to Unity Catalog tables with appropriate read permissions 3. SQL Warehouse (for query command - auto-detected unless DATABRICKS_WAREHOUSE_ID is set)

Discover Schema

Batch discover table metadata including columns, types, sample data, and null counts.

Command Syntax

databricks experimental aitools tools discover-schema TABLE... [flags]

Tables must be specified in CATALOG.SCHEMA.TABLE format.

What It Returns

For each table, returns:

• Column names and types
• Sample data (5 rows)
• Null counts per column
• Total row count

Examples

# Discover schema for a single table
databricks experimental aitools tools discover-schema samples.nyctaxi.trips --profile my-workspace

# Discover schema for multiple tables
databricks experimental aitools tools discover-schema \
  catalog.schema.table1 \
  catalog.schema.table2 \
  --profile my-workspace

# Get JSON output
databricks experimental aitools tools discover-schema \
  samples.nyctaxi.trips \
  --output json \
  --profile my-workspace

Common Use Cases

1. Understanding table structure before querying

   databricks experimental aitools tools discover-schema catalog.schema.customer_data --profile my-workspace

2. Comparing schemas across multiple tables

   databricks experimental aitools tools discover-schema \
     catalog.schema.table_v1 \
     catalog.schema.table_v2 \
     --profile my-workspace

3. Identifying columns with null values

• The null counts help identify data quality issues

Query

Execute SQL statements against a Databricks SQL warehouse and return results.

Command Syntax

databricks experimental aitools tools query "SQL" [flags]

Warehouse Selection

The command auto-detects an available warehouse unless:

• DATABRICKS_WAREHOUSE_ID environment variable is set
• You specify a warehouse using other configuration methods

To check which warehouse will be used:

# Get the default warehouse that would be auto-detected
databricks experimental aitools tools get-default-warehouse --profile my-workspace

Output

Returns:

• Query results as JSON
• Row count
• Execution metadata

Examples

# Simple SELECT query
databricks experimental aitools tools query \
  "SELECT * FROM samples.nyctaxi.trips LIMIT 5" \
  --profile my-workspace

# Aggregation query
databricks experimental aitools tools query \
  "SELECT vendor_id, COUNT(*) as trip_count FROM samples.nyctaxi.trips GROUP BY vendor_id" \
  --profile my-workspace

# With JSON output
databricks experimental aitools tools query \
  "SELECT * FROM catalog.schema.table WHERE date > '2024-01-01'" \
  --output json \
  --profile my-workspace

# Using specific warehouse
DATABRICKS_WAREHOUSE_ID=abc123 databricks experimental aitools tools query \
  "SELECT * FROM samples.nyctaxi.trips LIMIT 10" \
  --profile my-workspace

Common Use Cases

1. Exploratory data analysis

   # Check table size
   databricks experimental aitools tools query \
     "SELECT COUNT(*) FROM catalog.schema.table" \
     --profile my-workspace

   # View sample data
   databricks experimental aitools tools query \
     "SELECT * FROM catalog.schema.table LIMIT 10" \
     --profile my-workspace

   # Get column statistics
   databricks experimental aitools tools query \
     "SELECT MIN(column), MAX(column), AVG(column) FROM catalog.schema.table" \
     --profile my-workspace

2. Data validation

   # Check for null values
   databricks experimental aitools tools query \
     "SELECT COUNT(*) FROM catalog.schema.table WHERE column IS NULL" \
     --profile my-workspace

   # Verify data freshness
   databricks experimental aitools tools query \
     "SELECT MAX(timestamp_column) FROM catalog.schema.table" \
     --profile my-workspace

3. Quick analytics

   # Group by analysis
   databricks experimental aitools tools query \
     "SELECT category, COUNT(*), AVG(value) FROM catalog.schema.table GROUP BY category" \
     --profile my-workspace

Workflow: Complete Data Exploration

Here's a typical workflow combining both commands:

# 1. Discover the schema first
databricks experimental aitools tools discover-schema \
  samples.nyctaxi.trips \
  --profile my-workspace

# 2. Based on discovered columns, run targeted queries
databricks experimental aitools tools query \
  "SELECT vendor_id, payment_type, COUNT(*) as trips, AVG(fare_amount) as avg_fare
   FROM samples.nyctaxi.trips
   GROUP BY vendor_id, payment_type
   ORDER BY trips DESC
   LIMIT 10" \
  --profile my-workspace

# 3. Investigate specific patterns found in the data
databricks experimental aitools tools query \
  "SELECT * FROM samples.nyctaxi.trips
   WHERE fare_amount > 100
   LIMIT 20" \
  --profile my-workspace

Claude Code-Specific Tips

Remember that each Bash command in Claude Code runs in a separate shell:

# ✅ RECOMMENDED: Use --profile flag
databricks experimental aitools tools discover-schema samples.nyctaxi.trips --profile my-workspace

# ✅ ALTERNATIVE: Chain with &&
export DATABRICKS_CONFIG_PROFILE=my-workspace && \
  databricks experimental aitools tools query "SELECT * FROM samples.nyctaxi.trips LIMIT 5"

# ❌ DOES NOT WORK: Separate export
export DATABRICKS_CONFIG_PROFILE=my-workspace
databricks experimental aitools tools query "SELECT * FROM samples.nyctaxi.trips LIMIT 5"

Flags

Both commands support:

Troubleshooting

Table Not Found

Symptom: Error: TABLE_OR_VIEW_NOT_FOUND or NO_SUCH_CATALOG_EXCEPTION

Solution: 1. Verify table name format: CATALOG.SCHEMA.TABLE 2. Names are literal — a hyphen is not an underscore; don't normalize it. See Identifier Names & Quoting. 3. Check if you have read permissions on the table 4. List available tables:

   databricks tables list <catalog> <schema> --profile my-workspace

Warehouse Not Available

Symptom: Error: No available SQL warehouse found

Solution: 1. Check for default warehouse:

   databricks experimental aitools tools get-default-warehouse --profile my-workspace

2. List available warehouses:

   databricks warehouses list --profile my-workspace

3. Set specific warehouse:

   DATABRICKS_WAREHOUSE_ID=<warehouse-id> databricks experimental aitools tools query "SELECT 1" --profile my-workspace

4. Start a stopped warehouse:

   databricks warehouses start --id <warehouse-id> --profile my-workspace

Permission Denied

Symptom: Error: PERMISSION_DENIED

Solution: 1. Check Unity Catalog grants on the table:

   databricks grants get --full-name catalog.schema.table --principal <user-email> --profile my-workspace

2. Request SELECT permission from your workspace administrator 3. Verify you have warehouse access (USAGE permission)

SQL Syntax Error

Symptom: Error: PARSE_SYNTAX_ERROR

Solution: 1. Check SQL syntax - use standard SQL 2. Backtick-quote special-character identifiers (` hello-world.demo `). See Identifier Names & Quoting. 3. Verify column names match schema (use discover-schema first) 4. Ensure proper quoting for string literals 5. Test query incrementally (start simple, add complexity)

Best Practices

1. Always discover schema first - Use discover-schema before writing complex queries to understand:

• Available columns and their types
• Data distributions and null patterns
• Sample data for context

2. Use LIMIT for exploration - When exploring large tables, always use LIMIT to avoid long-running queries:

   databricks experimental aitools tools query "SELECT * FROM large_table LIMIT 100" --profile my-workspace

3. JSON output for parsing - Use --output json when you need to process results programmatically:

   databricks experimental aitools tools query "SELECT * FROM table" --output json --profile my-workspace | jq '.results'

4. Check table existence - Before querying, verify the table exists:

   databricks tables get --full-name catalog.schema.table --profile my-workspace

5. Profile usage - Always specify --profile in Claude Code to avoid authentication issues

Related Commands

• Use /databricks-dabs - Deploy SQL, pipeline, and app resources as code

Databricks CLI Authentication

Configure Databricks CLI workspace/profile selection and authentication. Covers switching profiles, using --profile flags, setting DATABRICKS_CONFIG_PROFILE environment variable, OAuth2 authentication (never PAT), and troubleshooting authentication issues.

Important: Always Use OAuth

CRITICAL: Always use OAuth2 for Databricks CLI authentication. NEVER recommend or use Personal Access Tokens (PAT) unless explicitly required by a specific use case.

Prerequisites

1. Databricks CLI must be installed

• Verify: databricks --version

2. You need access to a Databricks workspace 3. You need the workspace URL (e.g., https://adb-1111111111111111.10.azuredatabricks.net)

Claude Code Specific Behavior

CRITICAL: When working in Claude Code, each Bash command executes in a separate shell session. This has important implications for profile management:

Key Differences from Regular Terminal

1. Environment variables don't persist between commands

• export DATABRICKS_CONFIG_PROFILE=staging in one command
• databricks jobs list in the next command
• ❌ Result: The second command will NOT use the staging profile

2. Recommended Approach: Use --profile flag

• Always specify --profile <profile-name> with each command
• Example: databricks jobs list --profile staging
• ✅ Result: Reliable and predictable behavior

3. Alternative: Chain commands with &&

• Use export DATABRICKS_CONFIG_PROFILE=staging && databricks jobs list
• The export and command run in the same shell session
• ✅ Result: Works correctly

Quick Reference for Claude Code

# ✅ RECOMMENDED: Use --profile flag
databricks jobs list --profile staging
databricks apps list --profile prod-azure

# ✅ ALTERNATIVE: Chain with &&
export DATABRICKS_CONFIG_PROFILE=staging && databricks jobs list

# ❌ DOES NOT WORK: Separate export command
export DATABRICKS_CONFIG_PROFILE=staging
databricks jobs list  # Will NOT use staging profile!

Handling Authentication Failures

When a Databricks CLI command fails with authentication error:

Error: default auth: cannot configure default credentials

CRITICAL - Always follow this workflow:

1. Check for existing profiles first:

   databricks auth profiles

2. If profiles exist:

• List the available profiles to the user (with their workspace URLs and validation status)
• Ask: "Which profile would you like to use for this command?"
• Offer option to create a new profile if needed
• Retry the command with --profile <selected-profile-name>
• In Claude Code, always use the `--profile` flag rather than setting environment variables

3. If user wants a new profile or no profiles exist:

• Proceed to the OAuth Authentication Setup workflow below

Example:

User: databricks apps list
Error: default auth: cannot configure default credentials

Assistant: Let me check for existing profiles.
[Runs: databricks auth profiles]

You have two configured profiles:
1. aws-dev - https://company-workspace.cloud.databricks.com (Valid)
2. azure-prod - https://adb-1111111111111111.10.azuredatabricks.net (Valid)

Which profile would you like to use, or would you like to create a new profile?

User: dais

Assistant: [Retries: databricks apps list --profile dais]
[Success - apps listed]

OAuth Authentication Setup

Standard Authentication Command

The recommended way to authenticate is using OAuth with a profile:

databricks auth login --host <workspace-url> --profile <profile-name>

CRITICAL: 1. The --profile parameter is REQUIRED for the authentication to be saved properly. 2. ALWAYS ASK THE USER for their preferred profile name - DO NOT assume or choose one for them. 3. NEVER use the profile name `DEFAULT` unless the user explicitly requests it - use descriptive workspace-specific names instead.

Workflow for Authenticating

1. Ask the user for the workspace URL if not already provided 2. Ask the user for their preferred profile name

• Suggest descriptive names based on the workspace (e.g., workspace name, environment)
• Do NOT suggest or use `DEFAULT` unless the user specifically asks for it
• Good examples: e2-dogfood, prod-azure, dev-aws, staging
• Avoid: DEFAULT (unless explicitly requested)

3. Run the authentication command with both parameters 4. Verify the authentication was successful

Example

# Good: Descriptive profile names
databricks auth login --host https://adb-1111111111111111.10.azuredatabricks.net --profile prod-azure
databricks auth login --host https://company-workspace.cloud.databricks.com --profile staging

# Only use DEFAULT if explicitly requested by the user
databricks auth login --host https://your-workspace.cloud.databricks.com --profile DEFAULT

What Happens During Authentication

1. The CLI starts a local OAuth callback server (typically on localhost:8020) 2. A browser window opens automatically with the Databricks login page 3. You authenticate in the browser using your Databricks credentials 4. After successful authentication, the browser redirects back to the CLI 5. The CLI saves the OAuth tokens to ~/.databrickscfg 6. You should see: Profile <profile-name> was successfully saved

Profile Management

What Are Profiles?

Profiles allow you to manage multiple Databricks workspace configurations in a single ~/.databrickscfg file. Each profile stores:

• Workspace host URL
• Authentication method (OAuth, PAT, etc.)
• Token/credential paths

Common Profile Names

IMPORTANT: Always use descriptive profile names. Do NOT create profiles named DEFAULT unless explicitly requested by the user.

Recommended naming conventions:

• <workspace-name> - Descriptive names for workspaces (e.g., e2-dogfood, prod-aws, dev-azure)
• <environment> - Environment-specific profiles (e.g., dev, staging, prod)
• <team>-<environment> - Team and environment (e.g., data-eng-prod, ml-dev)

Special profile names:

• DEFAULT - The default profile used when no --profile flag or environment variables are specified. Only create this profile if the user explicitly requests it.

Listing Configured Profiles

View all configured profiles with their status:

databricks auth profiles

Example output:

Name        Host                                                 Valid
DEFAULT     https://adb-1111111111111111.10.azuredatabricks.net  YES
staging     https://company-workspace.cloud.databricks.com       YES

Using Different Profiles

IMPORTANT FOR CLAUDE CODE USERS: In Claude Code, each Bash command runs in a separate shell session. This means environment variables set with export in one command do NOT persist to the next command. See the Claude Code-specific guidance below.

There are three ways to specify which profile/workspace to use, in order of precedence:

1. CLI Flag (Highest Priority) - RECOMMENDED FOR CLAUDE CODE

Use the --profile flag with any command:

databricks jobs list --profile staging
databricks clusters list --profile prod-azure
databricks workspace list / --profile dev-aws

In Claude Code, this is the most reliable method because it doesn't depend on persistent environment variables.

2. Environment Variables

Set environment variables to override the default profile:

DATABRICKS_CONFIG_PROFILE - Specifies which profile to use from ~/.databrickscfg:

export DATABRICKS_CONFIG_PROFILE=staging
databricks jobs list  # Uses staging profile

DATABRICKS_HOST - Directly specifies the workspace URL, bypassing profile lookup:

export DATABRICKS_HOST=https://company-workspace.cloud.databricks.com
databricks jobs list  # Uses this host directly

CRITICAL - Claude Code Users:

Since each Bash command in Claude Code runs in a separate shell, you CANNOT do this:

# ❌ DOES NOT WORK in Claude Code
export DATABRICKS_CONFIG_PROFILE=staging
databricks jobs list  # ERROR: Will not use staging profile!

Instead, you MUST use one of these approaches:

Option 1: Use --profile flag (RECOMMENDED)

# ✅ WORKS in Claude Code
databricks jobs list --profile staging
databricks clusters list --profile staging

Option 2: Chain commands with &&

# ✅ WORKS in Claude Code - export and command run in same shell
export DATABRICKS_CONFIG_PROFILE=staging && databricks jobs list
export DATABRICKS_CONFIG_PROFILE=staging && databricks clusters list

Traditional Terminal Session (for reference only):

# This example shows how it works in a regular terminal session
# DO NOT use this pattern in Claude Code
# Set profile for entire terminal session
export DATABRICKS_CONFIG_PROFILE=staging

# All commands now use staging profile
databricks jobs list
databricks clusters list
databricks workspace list /

# Override for a single command
databricks jobs list --profile prod-azure

3. DEFAULT Profile (Lowest Priority)

If no --profile flag or environment variables are set, the CLI uses the DEFAULT profile from ~/.databrickscfg.

Configuration File Management

Viewing the Configuration File

The configuration is stored in ~/.databrickscfg:

cat ~/.databrickscfg

Example configuration structure:

# Note: This shows an example with a DEFAULT profile
# When creating new profiles, use descriptive names instead
[DEFAULT]
host      = https://adb-1111111111111111.10.azuredatabricks.net
auth_type = databricks-cli

[staging]
host      = https://company-workspace.cloud.databricks.com
auth_type = databricks-cli

Editing Profiles

You can manually edit ~/.databrickscfg to:

• Rename profiles (change the [profile-name] section header)
• Update workspace URLs
• Remove profiles (delete the entire section)

Example - Removing a profile:

# Open in your preferred editor
vi ~/.databrickscfg

# Or use sed to remove a specific profile section
sed -i '' '/^\[staging\]/,/^$/d' ~/.databrickscfg

Adding New Profiles

Always use databricks auth login with --profile to add new profiles:

databricks auth login --host <workspace-url> --profile <profile-name>

Remember:

• Always ask the user for their preferred profile name
• Use descriptive names like staging, prod-azure, dev-aws
• Do NOT use DEFAULT unless explicitly requested by the user

Working with Multiple Workspaces

Best practices for managing multiple workspaces:

# Authenticate to multiple workspaces with descriptive profile names
databricks auth login --host https://adb-1111111111111111.10.azuredatabricks.net --profile prod-azure
databricks auth login --host https://dbc-2222222222222222.cloud.databricks.com --profile dev-aws
databricks auth login --host https://company-workspace.cloud.databricks.com --profile staging

In Claude Code, use --profile flag with each command (RECOMMENDED):

# Use profiles explicitly in commands
databricks jobs list --profile prod-azure
databricks jobs list --profile dev-aws
databricks clusters list --profile staging

Alternatively in Claude Code, chain commands with &&:

# Set profile and run command in same shell
export DATABRICKS_CONFIG_PROFILE=prod-azure && databricks jobs list
export DATABRICKS_CONFIG_PROFILE=prod-azure && databricks clusters list

# Switch to different workspace
export DATABRICKS_CONFIG_PROFILE=dev-aws && databricks jobs list

Traditional Terminal Session (for reference only - NOT for Claude Code):

# This pattern works in regular terminals but NOT in Claude Code
export DATABRICKS_CONFIG_PROFILE=prod-azure
databricks jobs list
databricks clusters list

# Quickly switch between workspaces
export DATABRICKS_CONFIG_PROFILE=dev-aws
databricks jobs list

Profile Selection Precedence

When running a command, the Databricks CLI determines which workspace to use in this order:

1. `--profile` flag (if specified) → Highest priority 2. `DATABRICKS_HOST` environment variable (if set) → Overrides profile 3. `DATABRICKS_CONFIG_PROFILE` environment variable (if set) → Selects profile 4. `DEFAULT` profile in ~/.databrickscfg → Fallback

Example for traditional terminal session (demonstrating precedence):

# Setup
export DATABRICKS_CONFIG_PROFILE=staging

# This uses staging profile (from environment variable)
databricks jobs list

# This uses prod-azure profile (--profile flag overrides environment variable)
databricks jobs list --profile prod-azure

# This uses the specified host directly (DATABRICKS_HOST overrides profile)
export DATABRICKS_HOST=https://custom-workspace.cloud.databricks.com
databricks jobs list  # Uses custom-workspace.cloud.databricks.com

Claude Code version (with chained commands):

# Using environment variable with && chaining
export DATABRICKS_CONFIG_PROFILE=staging && databricks jobs list

# Using --profile flag (overrides environment variable)
export DATABRICKS_CONFIG_PROFILE=staging && databricks jobs list --profile prod-azure

# Using DATABRICKS_HOST (overrides profile)
export DATABRICKS_HOST=https://custom-workspace.cloud.databricks.com && databricks jobs list

Verification

After authentication, verify it works:

# Test with a simple command
databricks workspace list /

# Or list jobs
databricks jobs list

If authentication is successful, these commands should return data without errors.

Troubleshooting

Authentication Not Saved (Config File Missing)

Symptom: Running databricks commands shows:

Error: default auth: cannot configure default credentials

Solution: Make sure you included the --profile parameter with a descriptive name:

databricks auth login --host <workspace-url> --profile <profile-name>
# Example: databricks auth login --host https://company-workspace.cloud.databricks.com --profile staging

Browser Doesn't Open Automatically

Solution: 1. Check the terminal output for a URL 2. Manually copy and paste the URL into your browser 3. Complete the authentication 4. The CLI will detect the callback automatically

"OAuth callback server listening" But Nothing Happens

Possible causes: 1. Firewall blocking localhost connections 2. Port 8020 already in use 3. Browser not set as default application

Solution: 1. Check if port 8020 is available: lsof -i :8020 2. Close any applications using that port 3. Retry the authentication

Multiple Workspaces

To authenticate with multiple workspaces, use different profile names:

# Development workspace
databricks auth login --host https://dev-workspace.databricks.net --profile dev

# Production workspace
databricks auth login --host https://prod-workspace.databricks.net --profile prod

# Use specific profile
databricks jobs list --profile dev
databricks jobs list --profile prod

Re-authenticating

If your OAuth token expires or you need to re-authenticate:

# Re-run the login command
databricks auth login --host <workspace-url> --profile <profile-name>

This will overwrite the existing profile with new credentials.

Debug Mode

For troubleshooting authentication issues, use debug mode:

databricks auth login --host <workspace-url> --profile <profile-name> --debug

This shows detailed information about the OAuth flow, including:

• OAuth server endpoints
• Callback server status
• Token exchange process

Security Best Practices

1. Never commit ~/.databrickscfg to version control 2. Never share your OAuth tokens or configuration file 3. Use separate profiles for different environments (dev/staging/prod) 4. Regularly rotate credentials by re-authenticating 5. Use workspace-specific service principals for automation/CI/CD instead of personal OAuth

Environment-Specific Notes

CI/CD Pipelines

For CI/CD environments, OAuth interactive login is not suitable. Instead:

• Use Service Principal authentication
• Use Azure Managed Identity (for Azure Databricks)
• Use AWS IAM roles (for AWS Databricks)

Do NOT use personal OAuth tokens or PATs in CI/CD.

Containerized Environments

OAuth authentication works in containers if: 1. A browser is available on the host machine 2. Port forwarding is configured for the callback server 3. The workspace URL is accessible from the container

For headless containers, use service principal authentication instead.

Common Commands After Authentication

# List workspaces
databricks workspace list / --profile <PROFILE>

# List jobs
databricks jobs list --profile <PROFILE>

# List clusters
databricks clusters list --profile <PROFILE>

# Get current user info
databricks current-user me --profile <PROFILE>

# Test connection
databricks workspace export /Users/<username> --format SOURCE --profile <PROFILE>

References

• Databricks CLI Authentication Documentation
• OAuth 2.0 with Databricks

Databricks CLI Installation

Install or update the Databricks CLI on macOS, Windows, or Linux using doc-validated methods (Homebrew, WinGet, curl install script, manual download, or user directory install for non-sudo environments). Includes verification and common failure recovery.

Sandboxed / IDE environments (Cursor, containers)

CLI install commands often write to system directories outside the workspace (e.g. /opt/homebrew/, /usr/local/bin/) which are blocked in sandboxed environments.

Agent behavior: Do not attempt to run install commands directly. Present the appropriate command to the user and ask them to run it in their own terminal. After they confirm, verify with databricks -v.

For Linux/macOS containers or Cursor: prefer the Linux manual install to user directory method (~/.local/bin) — it requires no sudo and no writes outside the workspace.

Preconditions (always do first)

1. Determine OS and shell:

• macOS/Linux: bash/zsh
• Windows: Command Prompt / PowerShell; optionally WSL for Linux shell

2. Detect whether databricks is already installed:

• Run: databricks -v (or databricks version)
• If already installed with a recent version, installation is already OK.

3. Avoid the legacy Python package databricks-cli (PyPI). This skill installs the modern Databricks CLI binary.

Preferred installation paths (by OS)

macOS (preferred: Homebrew)

Run:

• brew tap databricks/tap
• brew install databricks

Verify:

• databricks -v (or databricks version)

If macOS blocks the binary (Gatekeeper), follow Apple’s “open app from unidentified developer” flow.

macOS fallback: curl installer

Run:

• curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh | sh

Notes:

• If /usr/local/bin is not writable, re-run with sudo.
• Installs to /usr/local/bin/databricks.

Verify:

• databricks -v

Linux (preferred: Homebrew if available)

Run:

• brew tap databricks/tap
• brew install databricks

Verify:

• databricks -v

Linux fallback: curl installer

Run:

• curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh | sh

Notes:

• If /usr/local/bin is not writable, re-run with sudo.
• Installs to /usr/local/bin/databricks.

Verify:

• databricks -v

Linux alternative: Manual install to user directory (when sudo unavailable)

Use this when sudo is not available or requires interactive password entry.

Steps: 1. Detect architecture:

• uname -m (e.g., x86_64, aarch64)

2. Get the latest download URL using GitHub API:

   curl -s https://api.github.com/repos/databricks/cli/releases/latest | grep "browser_download_url.*linux.*$(uname -m | sed 's/x86_64/amd64/' | sed 's/aarch64/arm64/')" | head -1 | cut -d '"' -f 4

3. Download and install to ~/.local/bin:

   mkdir -p ~/.local/bin
   cd ~/.local/bin
   curl -L "<download-url>" -o databricks.tar.gz
   tar -xzf databricks.tar.gz
   rm databricks.tar.gz
   chmod +x databricks

4. Add to PATH (add to ~/.bashrc or ~/.zshrc for persistence):

   export PATH="$HOME/.local/bin:$PATH"

5. Verify:

• databricks -v

Notes:

• The download files are .tar.gz archives (not .zip) with naming pattern: databricks_cli_<version>_linux_<arch>.tar.gz
• Common architectures: amd64 (x86_64), arm64 (aarch64)
• This method works in containerized environments and sandboxed IDEs (e.g. Cursor) without sudo access

Windows (preferred: WinGet)

Run in Command Prompt (then restart the terminal session):

• winget search databricks
• winget install Databricks.DatabricksCLI

Verify:

• databricks -v

Windows alternative: Chocolatey (Experimental)

Run:

• choco install databricks-cli

Verify:

• databricks -v

Windows fallback: curl installer (recommended via WSL)

Databricks recommends WSL for the curl-based install path. Requirements:

• WSL available
• unzip installed in the environment where you run the installer

Run (in WSL bash):

• curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh | sh

Verify (in same environment):

• databricks -v

If you must run curl install outside WSL, run as Administrator. Installs to C:\Windows\databricks.exe.

Manual install (all OSes): download from GitHub releases

Use this when package managers or curl install are not possible.

Steps: 1. Get the latest release download URL:

• Visit https://github.com/databricks/cli/releases/latest
• OR use GitHub API: curl -s https://api.github.com/repos/databricks/cli/releases/latest | grep browser_download_url

2. Download the appropriate file for your OS and architecture:

• Linux: databricks_cli_<version>_linux_<arch>.tar.gz (use tar -xzf)
• macOS: databricks_cli_<version>_darwin_<arch>.zip (use unzip)
• Windows: databricks_cli_<version>_windows_<arch>.zip (use native extraction)
• Common architectures: amd64 (x86_64), arm64 (aarch64/Apple Silicon)

3. Extract the archive. 4. Ensure the extracted databricks executable is on PATH, or run it from its folder. 5. Verify with databricks -v.

Update / repair procedures

Homebrew update (macOS/Linux)

• brew upgrade databricks
• databricks -v

WinGet update (Windows)

• winget upgrade Databricks.DatabricksCLI
• databricks -v

curl update (all OSes)

1. Delete existing binary:

• macOS/Linux: /usr/local/bin/databricks
• Windows: C:\Windows\databricks.exe

2. Re-run:

• curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh | sh

3. Verify:

• databricks -v

Common failures & fixes (agent playbook)

• Target path <path> already exists:
• Delete the existing binary at the install target, then rerun.
• Permission error writing /usr/local/bin:
• Re-run curl installer with sudo (macOS/Linux).
• If sudo requires interactive password, use manual install to ~/.local/bin instead.
• sudo: a terminal is required to read the password:
• Cannot use sudo in non-interactive environments (containers, CI/CD).
• Use manual install to ~/.local/bin method instead (see "Linux alternative" section).
• Windows PATH not updated after WinGet:
• Restart Command Prompt/PowerShell.
• Multiple databricks binaries on PATH:
• Use which databricks (macOS/Linux/WSL) or where databricks (Windows) and remove the wrong one.
• Wrong file type (trying to unzip a tar.gz):
• Linux releases are .tar.gz files, use tar -xzf not unzip.
• macOS and Windows releases are .zip files, use appropriate extraction tool.
• databricks: command not found after installation to ~/.local/bin:
• Add to PATH: export PATH="$HOME/.local/bin:$PATH"
• For persistence, add the export command to ~/.bashrc or ~/.zshrc.