Databricks Serverless Migration

Serverless Compute Migration

FIRST: Use the parent databricks-core skill for CLI basics, authentication, and profile selection.

Analyze existing Databricks code for serverless compute compatibility and guide migration from classic clusters. The skill follows a 4-step migration lifecycle: Ingest the workload → Analyze for compatibility → Test via A/B comparison → Validate and iterate.

When to Use This Skill

• Migrating notebooks, jobs, or pipelines from classic compute to serverless
• Checking if existing code is serverless-compatible
• Writing new code that targets serverless compute
• Troubleshooting serverless-specific errors after migration
• Choosing between Performance-Optimized and Standard mode

Where to Run This Skill

This skill is published as an Agent Skill (agentskills.io) and runs in any compatible client:

• Claude Code, Cursor, or any agentskills.io client on your laptop — the default. Install via databricks aitools install or follow the per-client docs.
• Inside a Databricks workspace via Genie Code Agent mode — drop the skill into /Workspace/Users/<you>/.assistant/skills/databricks-serverless-migration/ (per-user) or /Workspace/.assistant/skills/databricks-serverless-migration/ (workspace-wide, admin only). See Install in Databricks Genie Code for the three install methods and the important serverless-compute caveat (the Databricks CLI isn't pre-installed on serverless, so some deploy steps need adjustment).

If you finish a migration analysis for a user who's currently running you from a laptop client, mention the Genie Code option once at the end — many users prefer iterating on migrations inside the workspace where the workload lives.

Understanding Migration Blockers

Migration blockers fall into three categories. Focus your effort on category 2 — that's where this skill helps most.

Decision Tree: Is My Workload Ready?

Workload → Check language
├── R code → Category 3: keep on classic
├── Scala notebook cells → Category 2: port to PySpark/SQL, or compile as a JAR job task
├── Compiled Scala JAR (spark_jar_task, build.sbt/pom.xml/build.gradle) → see references/jar-migration.md
│     ├── Scala 2.12 build?             → recompile against 2.13.16
│     ├── Bundles spark-core/spark-sql? → use databricks-connect % Provided
│     └── Bundles libs on the kernel classpath? → mark % Provided (see the classpath table in references/jar-migration.md)
├── Python / SQL → Continue
    ├── Uses RDD APIs? → Category 2: rewrite to DataFrame API (see fixes below)
    ├── Uses DBFS paths? → Category 2: migrate to UC Volumes
    ├── Uses Hive Metastore? → Category 2: migrate to Unity Catalog (or use HMS Federation)
    ├── Uses df.cache/persist? → Category 1: remove and materialize to Delta (native support coming soon)
    ├── Uses streaming?
    │   ├── ProcessingTime trigger → Category 2: use AvailableNow or migrate to SDP
    │   ├── Continuous trigger → Category 2: use SDP continuous mode
    │   ├── No trigger specified → Category 2: add explicit .trigger(availableNow=True)
    │   └── AvailableNow / Once → Ready ✓
    ├── Uses init scripts? → Category 2: use Environments
    ├── Uses VPC peering? → Category 2: use NCCs / Private Link
    ├── Uses unsupported Spark configs? → Category 2: remove (serverless auto-tunes)
    ├── Uses custom JDBC drivers? → Category 2: use Lakehouse Federation or built-in JDBC
    ├── Uses Docker containers? → Category 3: use Environments for libs, or keep on classic
    └── All clear → Ready for serverless ✓

Migration Workflow

Step 1: Ingest — Gather Workload Context

Confirm the migration target is serverless compute. This skill is purpose-built for classic → serverless migrations. The checks, fixes, and workflow all target the serverless compute architecture (Spark Connect, Environments, NCCs). If the user wants to upgrade between classic DBR versions instead, this skill does not apply — classic DBR upgrades have a different compatibility surface and should follow the standard DBR upgrade guide.

Collect the full picture of what needs to migrate to serverless:

• Read the user's notebook/script files
• Identify the classic cluster configuration (instance type, DBR version, Spark configs, init scripts, libraries)
• Note the networking setup (VPC peering, instance profiles, mounts)
• Understand the workload type: batch job, streaming, interactive notebook, pipeline
• Determine the target: the output is always a serverless compute configuration, not a classic cluster with a newer DBR

Step 1.5: Handle Multi-Notebook Workloads

If the workload spans more than one user notebook (exclude _resources/ setup notebooks from the count), process them one at a time rather than all at once. The agent's context window is finite, and trying to hold the full source of an 8-notebook job in active context while doing analysis, fixes, and migration triggers autocompact thrashing (Claude Code) or equivalent context-overflow failures in other clients.

Procedure:

1. Enumerate first. List every user notebook with its path. Do not read the bodies in full yet beyond a few lines for orientation.

• Check for `bundle_config.py` (H4). If the source tree contains a bundle_config.py, parse it and follow any upstream-source declarations (git URLs, S3 paths, init_scripts, git_source, or custom upstream-repo declarations). Clone or fetch each referenced repo into a sibling location and include its notebooks in the enumeration. List "external sources" as a first-class artifact category in the migration plan. Without this step, multi-source demos (e.g., dbt-on-databricks references the upstream dbt-databricks-c360 repo) get mis-enumerated and the real task notebooks are missed.

2. For each notebook, in order: a. Read its full source. b. Run the Step 2 Analyze checklist scoped to this notebook only. c. Record a structured summary in your response or to a scratch file, then drop the raw source from active working memory. The summary must include:

• notebook_path (relative)
• detected_patterns[] (pattern IDs from this skill's catalog)
• blockers[] (Category-3 patterns this notebook hits, if any)
• migration_steps[] (concrete fixes ordered)
• unmigratable (bool — true if any blocker has no workaround)

d. If you have a writable scratch directory (~/.databricks-migration-skill/scratch/<run-id>/findings/), persist the summary as <notebook-basename>.json. This frees the source from context safely. If not, keep the summary compact in conversation history. 3. Synthesize. After all notebooks have a summary, produce the unified migration plan from the summaries alone. Apply fixes notebook-by-notebook, never re-reading the original source unless required to resolve an ambiguity in a summary. 4. Failure Reporting. Trigger the Failure Reporting Protocol exactly once per workload, not once per notebook. The detected_patterns array in the report aggregates across all notebooks; notebook_characteristics uses summed line counts and the union of language/streaming/ML flags.

Threshold guidance: apply this procedure when the user notebook count is ≥ 3, or when individual notebooks exceed ~5KB of source. For 1–2 small notebooks the single-pass workflow in Steps 2–4 is fine.

Anti-pattern: do NOT attempt a "merge all notebooks into one big file" or "read all notebooks, then act in one mega-turn" strategy. Both defeat the purpose by reintroducing the original context-pressure problem.

Step 2: Analyze — Scan for Serverless Readiness

Read notebooks before running them — do not rely on failed job runs to discover issues. A pre-run scan surfaces incompatibilities faster than iterating on error traces, and many serverless failures (hardcoded catalog references, init scripts, missing dependencies) are easy to spot statically but expensive to debug after a failed run.

Before creating or running any test job: 1. Read every notebook and source file referenced by the job 2. Scan for all hardcoded catalog/schema references (e.g., spark.table("main.schema.table"), spark.sql("... FROM main..."), catalog = "main") 3. Check for dependency patterns: init scripts, local wheel files, custom install functions, %pip install lines 4. Locate any requirements.txt or equivalent and resolve the full dependency set 5. Flag OS-level installs (apt install, yum install) for conversion or escalation

Scan the code for patterns that are incompatible with the serverless compute architecture. These checks are serverless-specific — most of these patterns work fine on classic compute regardless of DBR version. For each issue found, report:

• Category: Which of the 3 blocker categories it falls into
• Severity: Blocker (must fix for serverless) / Warning (should fix) / Info (awareness)
• Pattern: What was detected and where
• Fix: Specific remediation targeting serverless compute

Post-rewrite lint: Cell-magic boundary check (A1)

HIGH IMPACT. Before declaring a migrated notebook ready, run this lint pass on every output cell. Caused 3/7 demos to fail in the dbdemos E2E sweep (hls-readmission, fsi-fraud, retail-c360).

Detect: any cell that contains a # MAGIC %<word> line (e.g., # MAGIC %run, # MAGIC %sql, # MAGIC %md, # MAGIC %pip, # MAGIC %fs). Within that cell, every non-blank line must either start with # MAGIC or be a blank line. If a plain-Python comment (or any other Python code) precedes the # MAGIC %... directive in the same cell, the cell is corrupted: Databricks parses it as Python and %run falls back to IPython line magic, producing errors like File './00-global-setup-v2' not found.

Fix: never prepend plain-Python comments above a # MAGIC %... line within the same cell. Two valid options:

1. Preferred: put migration notes in a separate # MAGIC %md cell above (its own # COMMAND ---------- block). 2. Acceptable: drop the migration note entirely and rely on git/file history.

Example before (corrupted; %run fails with File not found):

# COMMAND ----------

# Migration: relative '%run ../../../_resources/00-global-setup-v2' may not
# resolve under serverless job tasks. Replaced with sibling reference.
# MAGIC %run ./00-global-setup-v2

Example after (clean; %run fires as cell magic):

# COMMAND ----------

# MAGIC %md
# MAGIC Migration: relative %run path replaced with sibling reference.

# COMMAND ----------

# MAGIC %run ./00-global-setup-v2

Category A: Unsupported APIs

Category B: Data Access

Category C: Streaming

Category D: Configuration

Category E: Libraries

Category F: Networking

Category G: Sizing & Debugging

Category H: Job-level config (dbt_task, SDP, multi-source, deploy preconditions)

These checks operate on the job/pipeline spec JSON and on deploy preconditions, not on notebook bodies. Apply them alongside the per-notebook checks in Categories A–G.

Required Output: Serverless Environment Specification

The migration output MUST include a Serverless Environment specification alongside migrated code. Generate this by:

1. Scanning all import statements and %pip install lines to detect required packages 2. Extracting init script pip install commands from the job configuration 3. Producing a JSON block suitable for the Jobs API environments field:

{
  "environment_key": "Default",
  "spec": {
    "client": "2",
    "dependencies": ["mlflow==2.12.1", "scikit-learn==1.3.0", "xgboost==2.0.3"]
  }
}

Important: ML runtime libraries (mlflow, scikit-learn, hyperopt, xgboost, tensorflow, torch, etc.) are NOT pre-installed on serverless compute. They MUST be listed explicitly in the environment spec dependencies. ML runtime is NOT available on serverless — always use Serverless Environments with explicit package dependencies instead.

Step 3: Test — Two-Branch Strategy

Use separate branches for testing and production to keep test-only workarounds out of the code that ships. The test branch is a safe sandbox for experimentation; the production branch contains only changes that production actually needs.

Test branch (serverless-test-{job_name}-{timestamp}): Temporary, no PR needed. 1. Create a branch from your current working branch 2. Set up test data: create sampled copies of upstream tables in a test catalog using job lineage (see test data setup below) 3. Parameterize the catalog so the notebook works with both test and production data (see catalog parameterization pattern below) 4. Apply all compatibility fixes discovered in Step 2 5. Create a serverless test job and run it 6. If it fails, get the error output, debug, fix, and retry 7. Document which changes are test workarounds vs. real compatibility fixes

Production branch (serverless-prod-{job_name}): PR required, created from master. 1. Create a new branch from master (NOT from the test branch) 2. Apply ONLY the real compatibility fixes — no test workarounds 3. Apply job config changes (see job config transformation below) 4. Commit and create a PR

Test Data Setup

When the job reads from production tables, do not point the test job at production data. Instead, create sampled copies of upstream tables in a dedicated test catalog and run the test job against those.

The recommended pattern: 1. Resolve the job's upstream tables from its lineage (or from a static scan of the notebook) 2. For each upstream table, run CREATE TABLE IF NOT EXISTS <test_catalog>.<schema>.<table> AS SELECT * FROM <prod_catalog>.<schema>.<table> LIMIT N (typical N: 10–1000 rows) 3. Keep the schema names identical to production — only the catalog changes 4. Make the operation idempotent: skip tables that already exist, so the setup step is safe to re-run 5. Require a running SQL warehouse and CREATE TABLE permission on the test catalog

With schema names preserved, the same notebook code runs in both environments — only the catalog widget value changes.

Decision Tree: Should This Change Go to Production?

Simple test: Would production fail without this change on serverless? If yes → include. If no → test branch only.

A/B Comparison

After both branches are ready, compare outputs:

# Compare outputs between classic and serverless runs
classic_df = spark.read.table("main.output.classic_results")
serverless_df = spark.read.table("main.output.serverless_results")

assert classic_df.count() == serverless_df.count(), "Row count mismatch"
assert classic_df.schema == serverless_df.schema, "Schema mismatch"
diff = classic_df.exceptAll(serverless_df)
assert diff.count() == 0, f"Found {diff.count()} differing rows"

Temporary bridge configs: If the serverless run fails, you may temporarily set supported Spark configs (like spark.sql.shuffle.partitions) to bridge gaps. Mark these as temporary — remove once the workload stabilizes.

Step 4: Validate — Confirm and Monitor

Once the A/B comparison passes: 1. Merge the production branch PR 2. Switch the production job to serverless compute 3. Monitor cost via system tables (system.billing.usage) and budget policies 4. Remove any temporary bridge configurations 5. Set up budget alerts for cost visibility

Migration Deliverables

At the end of a successful migration run, surface these artifacts so the user can verify the work and inspect the results:

If any deliverable is missing, the migration is incomplete — do not mark it as done.

Stopping Conditions

Do not attempt workarounds for these — surface them to the user and stop:

• Permission failures on source tables, the test catalog, or the workspace
• Category 3 blockers (R code, custom Spark data source JARs, features that require classic compute)
• SQL warehouse or test catalog not available
• Repeated failures (typically 5+) with no new information in the error trace — generate a failure report instead (see Failure Reporting Protocol)

Quick Fixes Reference

Replace DBFS paths with UC Volumes

# BEFORE (classic)
df = spark.read.csv("dbfs:/mnt/datalake/sales/data.csv", header=True)
df.write.parquet("dbfs:/mnt/output/results")

# AFTER (serverless)
df = spark.read.csv("/Volumes/main/sales/raw_data/data.csv", header=True)
df.write.parquet("/Volumes/main/analytics/output/results")

# Replace mounts with external volumes (SQL):
# CREATE EXTERNAL VOLUME main.data.raw_files LOCATION 's3://my-bucket/data/';
# Then use: /Volumes/main/data/raw_files/

# Pandas paths too:
# BEFORE: pd.read_csv("/dbfs/mnt/data/file.csv")
# AFTER:  pd.read_csv("/Volumes/main/data/volume/file.csv")

Replace RDD operations with DataFrames

from pyspark.sql import functions as F

# parallelize + map
# BEFORE:
rdd = sc.parallelize([1, 2, 3])
result = rdd.map(lambda x: x * 2).collect()
# AFTER:
df = spark.createDataFrame([(1,), (2,), (3,)], ["value"])
result = df.select((F.col("value") * 2).alias("value")).collect()

# flatMap (word splitting)
# BEFORE:
words = sc.parallelize(["hello world"]).flatMap(lambda l: l.split(" ")).collect()
# AFTER:
df = spark.createDataFrame([("hello world",)], ["line"])
words = df.select(F.explode(F.split("line", " ")).alias("word")).collect()

# groupByKey
# BEFORE:
rdd = sc.parallelize([("a", 1), ("b", 2), ("a", 3)])
grouped = rdd.groupByKey().mapValues(list).collect()
# AFTER:
df = spark.createDataFrame([("a", 1), ("b", 2), ("a", 3)], ["key", "value"])
grouped = df.groupBy("key").agg(F.collect_list("value").alias("values")).collect()

# mapPartitions → applyInPandas
# BEFORE:
def process_partition(iterator):
    yield sum(iterator)
result = sc.parallelize(range(100), 4).mapPartitions(process_partition).collect()
# AFTER:
import pandas as pd
def process_group(pdf: pd.DataFrame) -> pd.DataFrame:
    return pd.DataFrame({"total": [pdf["id"].sum()]})
result = (spark.range(100).repartition(4)
    .groupBy(F.spark_partition_id())
    .applyInPandas(process_group, schema="total long")
    .collect())

# textFile
# BEFORE: rdd = sc.textFile("/mnt/data/file.txt")
# AFTER:  df = spark.read.text("/Volumes/catalog/schema/volume/file.txt")

# wholeTextFiles
# BEFORE: rdd = sc.wholeTextFiles("/mnt/data/dir/")
# AFTER:  df = spark.read.format("binaryFile").load("/Volumes/catalog/schema/volume/dir/")

Fix streaming triggers

# CRITICAL: Omitting .trigger() defaults to ProcessingTime(0) — not supported on serverless

# BEFORE (fails on serverless — no trigger = ProcessingTime default):
query = df.writeStream.format("delta").outputMode("append").start(path)

# BEFORE (fails — explicit ProcessingTime):
query = df.writeStream.trigger(processingTime="10 seconds").start(path)

# AFTER (serverless compatible):
query = (df.writeStream
    .format("delta")
    .outputMode("append")
    .trigger(availableNow=True)
    .option("checkpointLocation", "/Volumes/main/data/checkpoints/stream1")
    .start("/Volumes/main/data/output/stream1"))
query.awaitTermination()

# With OOM prevention (recommended for large sources):
query = (spark.readStream.format("delta")
    .option("maxFilesPerTrigger", 100)          # Delta/Parquet sources
    .option("maxBytesPerTrigger", "10g")         # Limit data per micro-batch
    .load(input_path)
    .writeStream
    .trigger(availableNow=True)
    .option("checkpointLocation", checkpoint_path)
    .start(output_path))

# Kafka: use maxOffsetsPerTrigger
query = (spark.readStream.format("kafka")
    .option("kafka.bootstrap.servers", "broker:9092")
    .option("subscribe", "topic1")
    .option("maxOffsetsPerTrigger", 100000)      # Kafka-specific
    .load()
    .writeStream.trigger(availableNow=True).start(output_path))

# Auto Loader: use cloudFiles.maxFilesPerTrigger (note the prefix)
query = (spark.readStream.format("cloudFiles")
    .option("cloudFiles.format", "json")
    .option("cloudFiles.maxFilesPerTrigger", 1000)  # cloudFiles. prefix
    .load(landing_path)
    .writeStream.trigger(availableNow=True).start(output_path))

Remove caching

# BEFORE (classic):
df = spark.read.parquet(path)
df.cache()
df.count()  # materialize cache
result1 = df.filter("status = 'active'")
result2 = df.groupBy("region").agg(F.sum("revenue"))

# AFTER (serverless — remove .cache(); native support coming soon):
df = spark.read.parquet(path)
result1 = df.filter("status = 'active'")
result2 = df.groupBy("region").agg(F.sum("revenue"))

# For truly expensive intermediate results, materialize to Delta:
expensive_df.write.format("delta").mode("overwrite").saveAsTable("main.scratch.intermediate")
result = spark.table("main.scratch.intermediate")

# SQL equivalent:
# BEFORE: CACHE TABLE my_table
# AFTER:  (just remove the CACHE TABLE statement)

Other quick fixes

Detect serverless at runtime

import os
is_serverless = os.getenv("IS_SERVERLESS", "").lower() == "true"

Transform job config from classic to serverless

Remove job_clusters/new_cluster, add environments with serverless spec, replace job_cluster_key with environment_key, remove init_scripts. See Configuration Guide for full before/after JSON and environment version mapping.

Environment version mapping (match to the DBR version the workload was on):

Job Definition Migration

When migrating a job, the job configuration JSON must be transformed alongside notebook code. The agent should perform all of the following:

Init scripts to Serverless Environments: Detect init_scripts in the job JSON. Extract all pip install commands and convert them to Environment dependencies. For OS-level packages (apt install/yum install) that have pip equivalents (e.g., apt install python3-opencv becomes opencv-python), convert them. Flag OS-level packages without pip equivalents as serverless-incompatible (Category 3).

Cluster libraries (Maven/JAR) to Environment or Volumes: Maven coordinates for Python-wrapping JARs should be replaced with their PyPI equivalent in the Environment spec. Custom JARs on DBFS need to be moved to /Volumes/<your_catalog>/schema/volume/ and referenced there. Custom Spark data source JARs (v1/v2) are a Category 3 blocker — flag them for classic retention.

job_clusters to serverless compute: Remove job_clusters / new_cluster blocks entirely. Add an environments array with the serverless spec. Replace job_cluster_key in each task with environment_key. Remove init_scripts, num_workers, node_type_id, spark_version. See Configuration Guide for a complete before/after example.

spark_conf migration: Scan all spark.conf.set(...) calls in the notebook and spark_conf entries in the job JSON. For each:

• Supported (keep): spark.sql.shuffle.partitions, spark.sql.session.timeZone, spark.sql.ansi.enabled, spark.sql.files.maxPartitionBytes, spark.sql.legacy.timeParserPolicy, spark.databricks.execution.timeout
• Auto-tuned (remove with comment): AQE configs, Delta auto-compact, executor/driver sizing, parallelism configs
• Credential configs (remove): fs.s3a.*, fs.azure.* — replaced by UC external locations
• Add a code comment at each removal explaining why: # Removed: auto-tuned on serverless or # Removed: use UC external locations instead

Parameterize catalogs for testing

dbutils.widgets.text("catalog", "main")  # Default to production
catalog = dbutils.widgets.get("catalog")
df = spark.table(f"{catalog}.sales.orders")
# Pass catalog="test_catalog" as a job parameter during testing

See Configuration Guide for mock table catalog mapping and test job creation patterns.

Debug failed serverless runs

Always get the actual error with w.jobs.get_run_output(run_id=...) before guessing. Common errors:

See Configuration Guide for the full error reference and SDK code examples.

Performance Mode Selection

Standard mode is NOT available for notebooks. Notebooks always use Performance-Optimized.

Serverless Defaults to Know

Failure Reporting Protocol

When migration fails or hits an unmigratable pattern, generate a structured failure report and offer the user a one-step path to file it as a GitHub issue. This feedback loop is how the skill learns — without it, gaps go undiscovered.

See Failure Reporting for the full redaction checklist, URL-encoding recipe, and example pre-filled link.

Decision tree: when to offer to file an issue

Offer to file a GitHub issue any time the workload cannot be fully migrated to serverless as-is. Reports from "known" patterns (R, Scala, custom JAR data sources, JVM access, third-party connectors) are just as valuable as reports from unknown patterns — they tell maintainers which gaps users hit most often, which drives prioritization.

Concretely, ALWAYS offer if any of these is true:

1. Workload contains any Category 3 (classic-only) blocker — R, Scala notebook cells, custom JAR data sources, JVM/Py4J access, third-party connectors without serverless equivalents, native binary dependencies, etc. The fact that the pattern is "documented as Cat 3" is not a reason to skip the offer. 2. Retries exhausted — retry_count >= max_retries (typically 5) and final status is FAILED 3. Unknown pattern — a classic-compute construct was detected that isn't in the skill's catalog 4. Fix didn't resolve — a known fix was applied but the workload still fails on serverless 5. Explicit request — the user invokes /migration-report

Do NOT offer to file only when:

• The migration succeeded fully (even after retries), or
• The workload is already serverless-compatible and required no changes.

How to generate the report

Write a JSON file to ~/.databricks-migration-skill/reports/failure-<ISO-timestamp>.json. Create the directory if it doesn't exist.

Schema (strictly follow — no free-text code or identifiers):

{
  "report_version": "1.1",
  "report_id": "<uuid-v4>",
  "skill_version": "<from SKILL.md frontmatter metadata.version>",
  "timestamp": "<ISO 8601 UTC>",
  "failure_phase": "analyze | migrate | test | validate",
  "detected_patterns": [
    {"category": "A", "pattern_id": "rdd_parallelize", "count": 3}
  ],
  "attempted_fixes": [
    {"pattern_id": "rdd_parallelize", "fix_applied": "<fix_id>", "attempt_number": 1, "outcome": "failed"}
  ],
  "final_error_category": "unknown_api | missing_library | data_access | permission | custom_data_source | jvm_access | unsupported_language | other",
  "final_error_signature": "<SHA256 of top 3 stack frames, NOT the frames themselves>",
  "retry_count": 5,
  "total_duration_seconds": 245,
  "notebook_characteristics": {
    "lines_of_code": 180,
    "language": "python | sql | scala | r",
    "uses_streaming": false,
    "uses_ml_libraries": true,
    "databricks_runtime_source": "<DBR version only, no cluster identifiers>"
  }
}

What the report MUST NOT contain

Hard requirement — the report must be safe to share publicly on GitHub Issues:

• No code content — pattern IDs only (e.g., rdd_parallelize), never code snippets, function bodies, or even single-line examples
• No file paths — no notebook names, directory paths, workspace URLs, or DBFS paths
• No error message text — only the error category enum and a hashed signature
• No identifiers — no table names, column names, catalog names, schema names, secret scope names, user emails, workspace IDs, or account IDs
• No internal Databricks references — no Databricks employee names, internal codenames (e.g., product code names not in public docs), go/ links, Confluence page IDs, Google Doc IDs, Slack user or channel IDs (U…, C…), PROD- / SEV- / SC-* ticket numbers
• No customer references — no company names, product names of customer systems, or anything that would identify the workspace's owning organization
• No credentials — no tokens, API keys, connection strings, JDBC URLs, or service principal IDs
• No data descriptions — no column value samples, row counts tied to specific tables, or schema fingerprints beyond the notebook_characteristics fields

Anonymization safety pass

Before writing the report, scan every string field against this pattern checklist. If any pattern matches, drop the offending field (do not redact partially — empty string is safer than risking leakage):

The notebook_characteristics fields are the only safe surface for workload metadata. Do not add new fields without expanding this checklist.

Deterministic post-serialization scrub (required)

The redaction above happens at generation time and depends on the model applying every rule. That is not enough on a path that ends in a public GitHub issue. After serializing the report to JSON:

1. Re-run the MUST-NOT-CONTAIN regex set as a literal text search over the final JSON file. Match the same patterns from the table above, plus any catalog/schema/table names that were referenced in the analyzed notebook. 2. If any pattern matches, refuse to display the pre-filled URL or the `gh issue create` command. Print the local file path, list the patterns that matched, and tell the user to redact the file manually before sharing. 3. Only when the post-serialization scrub is clean may the pre-filled URL be shown.

This deterministic check is non-negotiable; do not skip it even when you are confident the generation-time redaction was applied.

After generating the report — output template

When the decision tree above says "offer to file", the default output is local-only. The pre-filled URL and gh command are shown only after the user has acknowledged the local file and the post-serialization scrub (see above) is clean.

Default response — always include:

1. The local report file path (~/.databricks-migration-skill/reports/failure-<timestamp>.json, with <timestamp> filled in). 2. A one-line note that the report contains pattern IDs and notebook characteristics only, no code or identifiers. 3. A prompt: "This is a draft. Open the file, confirm the redaction looks right, then tell me to share it. I'll generate the pre-filled GitHub issue URL once you've confirmed."

Only after the user explicitly confirms (e.g. "share it", "file the issue", "looks good") AND the deterministic post-serialization scrub above returned clean, then produce:

• Option A — a complete pre-filled https://github.com/databricks/databricks-agent-skills/issues/new?template=migration-feedback.md&title=<…>&body=<…> URL, both parameters URL-encoded.
• Option B — the literal gh issue create --repo databricks/databricks-agent-skills … command, body-file pointing at the local report.

Do not produce the URL or gh command in the same turn as the file write. Two turns: write + offer review, then publish after explicit confirmation.

Use this exact wrap-up template, replacing <…> placeholders:

Migration could not complete. A failure report has been generated at:

  ~/.databricks-migration-skill/reports/failure-<timestamp>.json

The report contains anonymized diagnostic data (detected pattern IDs, error
category, retry count, notebook characteristics) and no code content or PII.
Submission is optional and opt-in.

To help improve this skill, file the report as a GitHub issue:

  Option A — One-click in browser (pre-filled):
    <PREFILLED_ISSUE_URL>

  Option B — From the terminal (if you have the GitHub CLI installed):
    gh issue create \
      --repo databricks/databricks-agent-skills \
      --title "<TITLE>" \
      --body-file ~/.databricks-migration-skill/reports/failure-<timestamp>.json \
      --label migration-skill

Before submitting, please open the JSON and confirm nothing sensitive
slipped through. We never transmit reports automatically.

Build <PREFILLED_ISSUE_URL> like this:

1. Title: [migration-skill] <final_error_category> in <failure_phase> phase Example: [migration-skill] custom_data_source in migrate phase 2. Body: the issue template's markdown skeleton (Category, Environment, Description, Failure report JSON fenced in `json ) with the report JSON inlined. 3. URL-encode both title and body (%20 for spaces, %23 for #, %0A for newline, etc.). 4. Final URL: https://github.com/databricks/databricks-agent-skills/issues/new?template=migration-feedback.md&title=<URL-encoded title>&body=<URL-encoded body>

If your runtime cannot actually write the file (sandboxed, no filesystem write), still show the path the file WOULD be at and produce Options A and B. The user can write the JSON to disk themselves.

The full recipe with a worked example is in Failure Reporting.

Never transmit the report automatically. The user owns their data and must review before sharing. If the user declines, do not press them — log the local report path and move on.

Success report

When migration succeeds, generate a per-workload report with the same structure as the failure report (minus the redaction-to-publish step), listing the patterns detected and the fixes applied for each workload. Write to ~/.databricks-migration-skill/reports/success-<timestamp>.json and tell the user the file path.

Multiple workload migration

When several independent workloads are in scope, migrate them in parallel (e.g. one subagent per workload). Keep the per-notebook steps within a single workload sequential, as in Step 1.5.

Reference Guides

For detailed workarounds and code examples beyond the quick fixes above:

• Compatibility Checks — Full pattern detection table with all 40+ checks
• Streaming Migration — Trigger migration, SDP continuous mode, continuous jobs
• Networking and Security — VPC peering to NCCs, Private Link, firewall setup
• Code Patterns — Complete before/after code examples for every migration pattern
• Configuration Guide — Supported Spark configs, Environments setup, budget policies
• MLflow on UC — AutoML rewrite to sklearn, UC-aware registration, latest_versions replacement, spark_udf closure-bug fix, prediction-column dtype alignment
• Multi-source enumeration — Parsing bundle_config.py and fetching upstream git sources before per-notebook analysis
• Failure Reporting — Redaction checklist + pre-filled GitHub issue URL recipe (for when migration cannot complete)
• Install in Databricks Genie Code — Run this skill inside a Databricks workspace
• JAR Migration — Migrating a compiled Scala JAR (spark_jar_task): Scala 2.13.16, Databricks Connect, dependency conflicts vs the serverless kernel classpath, sbt fixes, build/test/deploy

Documentation

• Serverless compute overview: https://docs.databricks.com/en/compute/serverless/
• Migration guide: https://docs.databricks.com/en/compute/serverless/migration
• Limitations: https://docs.databricks.com/en/compute/serverless/limitations
• Best practices: https://docs.databricks.com/en/compute/serverless/best-practices
• Serverless notebooks: https://docs.databricks.com/en/compute/serverless/notebooks
• Serverless jobs: https://docs.databricks.com/en/jobs/run-serverless-jobs
• Serverless SDP: https://docs.databricks.com/en/ldp/serverless
• Spark Connect vs. classic: https://docs.databricks.com/en/spark/connect-vs-classic
• Unity Catalog upgrade: https://docs.databricks.com/en/data-governance/unity-catalog/upgrade/
• Supported Spark configs: https://docs.databricks.com/en/spark/conf#serverless

Code Patterns Reference

Complete before/after code examples for every migration pattern from classic compute to serverless compute.

RDD to DataFrame

All RDD APIs and sc.* methods are not supported on serverless (which uses the Spark Connect architecture). Replace with DataFrame operations.

sc.parallelize

# BEFORE:
rdd = sc.parallelize([1, 2, 3, 4, 5])
result = rdd.collect()

# AFTER:
df = spark.createDataFrame([(i,) for i in [1, 2, 3, 4, 5]], ["value"])
result = df.collect()

rdd.map

# BEFORE:
rdd = sc.parallelize([1, 2, 3])
mapped = rdd.map(lambda x: x * 2).collect()

# AFTER:
from pyspark.sql import functions as F
df = spark.createDataFrame([(1,), (2,), (3,)], ["value"])
result = df.select((F.col("value") * 2).alias("value")).collect()

rdd.filter

# BEFORE:
rdd = sc.parallelize([1, 2, 3, 4, 5])
filtered = rdd.filter(lambda x: x > 3).collect()

# AFTER:
from pyspark.sql import functions as F
df = spark.createDataFrame([(i,) for i in range(1, 6)], ["value"])
result = df.filter(F.col("value") > 3).collect()

rdd.reduce

# BEFORE:
rdd = sc.parallelize([1, 2, 3, 4, 5])
total = rdd.reduce(lambda a, b: a + b)

# AFTER:
from pyspark.sql import functions as F
df = spark.createDataFrame([(i,) for i in range(1, 6)], ["value"])
total = df.agg(F.sum("value")).collect()[0][0]

rdd.flatMap

# BEFORE:
rdd = sc.parallelize(["hello world", "foo bar"])
words = rdd.flatMap(lambda line: line.split(" ")).collect()

# AFTER:
from pyspark.sql import functions as F
df = spark.createDataFrame([("hello world",), ("foo bar",)], ["line"])
words = df.select(F.explode(F.split(F.col("line"), " ")).alias("word")).collect()

rdd.groupByKey

# BEFORE:
rdd = sc.parallelize([("a", 1), ("b", 2), ("a", 3)])
grouped = rdd.groupByKey().mapValues(list).collect()

# AFTER:
from pyspark.sql import functions as F
df = spark.createDataFrame([("a", 1), ("b", 2), ("a", 3)], ["key", "value"])
grouped = df.groupBy("key").agg(F.collect_list("value").alias("values")).collect()

rdd.reduceByKey

# BEFORE:
rdd = sc.parallelize([("a", 1), ("b", 2), ("a", 3)])
summed = rdd.reduceByKey(lambda a, b: a + b).collect()

# AFTER:
from pyspark.sql import functions as F
df = spark.createDataFrame([("a", 1), ("b", 2), ("a", 3)], ["key", "value"])
summed = df.groupBy("key").agg(F.sum("value").alias("value")).collect()

rdd.mapPartitions to applyInPandas

# BEFORE:
rdd = sc.parallelize(range(100), 4)
def process_partition(iterator):
    yield sum(iterator)
result = rdd.mapPartitions(process_partition).collect()

# AFTER:
import pandas as pd
from pyspark.sql import functions as F

def process_group(pdf: pd.DataFrame) -> pd.DataFrame:
    return pd.DataFrame({"total": [pdf["id"].sum()]})

df = spark.range(100).repartition(4)
result = df.groupBy(F.spark_partition_id()).applyInPandas(
    process_group, schema="total long"
).collect()

sc.textFile

# BEFORE:
rdd = sc.textFile("/mnt/data/file.txt")
lines = rdd.collect()

# AFTER:
df = spark.read.text("/Volumes/catalog/schema/volume/file.txt")
lines = df.collect()

sc.wholeTextFiles

# BEFORE:
rdd = sc.wholeTextFiles("/mnt/data/directory/")
files = rdd.collect()

# AFTER:
df = spark.read.format("binaryFile").load("/Volumes/catalog/schema/volume/directory/")
files = df.collect()

rdd.sortByKey

# BEFORE:
rdd = sc.parallelize([("b", 2), ("a", 1), ("c", 3)])
sorted_rdd = rdd.sortByKey().collect()

# AFTER:
df = spark.createDataFrame([("b", 2), ("a", 1), ("c", 3)], ["key", "value"])
sorted_df = df.orderBy("key").collect()

rdd.join

# BEFORE:
rdd1 = sc.parallelize([("a", 1), ("b", 2)])
rdd2 = sc.parallelize([("a", "x"), ("b", "y")])
joined = rdd1.join(rdd2).collect()

# AFTER:
df1 = spark.createDataFrame([("a", 1), ("b", 2)], ["key", "val1"])
df2 = spark.createDataFrame([("a", "x"), ("b", "y")], ["key", "val2"])
joined = df1.join(df2, "key").collect()

SparkContext.getOrCreate()

# BEFORE:
from pyspark import SparkContext
sc = SparkContext.getOrCreate()
data = sc.parallelize([1, 2, 3, 4, 5])
result = data.map(lambda x: x * 2).collect()

# AFTER (serverless — SparkContext.getOrCreate() raises RuntimeError):
# Use the pre-existing `spark` SparkSession directly
df = spark.createDataFrame([(i,) for i in [1, 2, 3, 4, 5]], ["value"])
from pyspark.sql import functions as F
result = df.select((F.col("value") * 2).alias("value")).collect()

# For simple ranges, spark.range() is even simpler:
df = spark.range(1, 6)  # Creates DataFrame with column "id" = [1, 2, 3, 4, 5]

%run Inlining

When %run ./relative/path causes issues in serverless job tasks, inline the referenced notebook or convert to an absolute path.

Inlining a Config Notebook

# BEFORE (main notebook):
# Cell 1:
# %run ./config

# Cell 2:
df = spark.table(f"{catalog}.{schema}.orders")
result = df.filter(f"region = '{region}'")

# Where ./config notebook contains:
# catalog = "main"
# schema = "sales"
# region = "US"
# FEATURE_FLAG_V2 = True

# AFTER (inline the config — preferred when <500 lines):
# Cell 1:
catalog = "main"
schema = "sales"
region = "US"
FEATURE_FLAG_V2 = True

# Cell 2:
df = spark.table(f"{catalog}.{schema}.orders")
result = df.filter(f"region = '{region}'")

Converting to Absolute Path

# BEFORE:
# %run ./utils/helpers

# AFTER (when the notebook is too large to inline):
dbutils.notebook.run("<workspace_path>/utils/helpers", timeout_seconds=300)
# Note: Variables defined in the called notebook are NOT available in the caller
# when using dbutils.notebook.run(). Use dbutils.notebook.exit() to return a value.

DBFS to Volumes

Reading Files (Spark)

# BEFORE:
df = spark.read.csv("dbfs:/mnt/datalake/sales/data.csv", header=True)

# AFTER:
df = spark.read.csv("/Volumes/main/sales/raw_data/data.csv", header=True)

Reading Files (Pandas)

# BEFORE:
import pandas as pd
pdf = pd.read_csv("/dbfs/mnt/datalake/sales/data.csv")

# AFTER:
import pandas as pd
pdf = pd.read_csv("/Volumes/main/sales/raw_data/data.csv")

Writing Files

# BEFORE:
df.write.parquet("dbfs:/mnt/datalake/output/results.parquet")

# AFTER:
df.write.parquet("/Volumes/main/analytics/output/results.parquet")

Listing Files

# BEFORE:
files = dbutils.fs.ls("dbfs:/mnt/datalake/sales/")

# AFTER:
files = dbutils.fs.ls("/Volumes/main/sales/raw_data/")

Replacing Mounts with External Volumes

# BEFORE:
dbutils.fs.mount(source="s3://my-bucket/data/", mount_point="/mnt/datalake")
df = spark.read.csv("/mnt/datalake/file.csv")

# AFTER (one-time SQL setup):
# CREATE EXTERNAL VOLUME main.data.raw_files LOCATION 's3://my-bucket/data/';
# Then in code:
df = spark.read.csv("/Volumes/main/data/raw_files/file.csv")

Non-Sequential File Writes (Excel, Images, etc.)

Some libraries require writing to local disk first. Use /local_disk0/tmp/ as a staging area:

import xlsxwriter
from shutil import copyfile

# Write to local temp
workbook = xlsxwriter.Workbook('/local_disk0/tmp/report.xlsx')
worksheet = workbook.add_worksheet()
worksheet.write('A1', 'Report Title')
# ... more writes ...
workbook.close()

# Copy to Volumes
copyfile('/local_disk0/tmp/report.xlsx', '/Volumes/main/reports/output/report.xlsx')

Volume Limitations to Know

• dbutils.fs operations on Volumes work from the driver only (not from executors/RDDs)
• Use dbutils.fs.mv instead of %sh mv when moving files between Volumes
• Temporary files should use /local_disk0/tmp/ (cleaned up when session ends)

SparkContext to SparkSession

broadcast to broadcast join

# BEFORE:
lookup = sc.broadcast({"US": "United States", "UK": "United Kingdom"})
# ... use in UDF or RDD operation

# AFTER:
from pyspark.sql.functions import broadcast
lookup_df = spark.createDataFrame(
    [("US", "United States"), ("UK", "United Kingdom")],
    ["code", "name"])
result = main_df.join(broadcast(lookup_df), main_df.country_code == lookup_df.code)

accumulator to DataFrame aggregation

# BEFORE:
error_count = sc.accumulator(0)
def process(row):
    if row.status == "error":
        error_count.add(1)
rdd.foreach(process)
print(f"Errors: {error_count.value}")

# AFTER:
error_count = df.filter(F.col("status") == "error").count()
print(f"Errors: {error_count}")

# For multiple aggregations:
stats = df.agg(
    F.count(F.when(F.col("status") == "error", 1)).alias("error_count"),
    F.sum("amount").alias("total_amount"),
    F.count("*").alias("total_rows")
).collect()[0]

sqlContext.sql to spark.sql

# BEFORE:
result = sqlContext.sql("SELECT * FROM hive_metastore.default.my_table")

# AFTER:
result = spark.sql("SELECT * FROM main.my_schema.my_table")

sc.hadoopConfiguration to UC External Locations

# BEFORE:
sc.hadoopConfiguration.set("fs.s3a.access.key", "AKIAEXAMPLE")
sc.hadoopConfiguration.set("fs.s3a.secret.key", dbutils.secrets.get("scope", "key"))
df = spark.read.parquet("s3a://bucket/path/")

# AFTER (no credential config needed — UC handles auth):
df = spark.read.parquet("s3://bucket/path/")
# Or via Volumes:
df = spark.read.parquet("/Volumes/main/data/volume/path/")

sc.getConf to spark.conf.get

# BEFORE:
value = sc.getConf().get("spark.sql.shuffle.partitions")

# AFTER:
value = spark.conf.get("spark.sql.shuffle.partitions")

Cache/Persist to Remove or Materialize

Simple Cache Removal

# BEFORE:
df = spark.read.parquet(path)
df.cache()
df.count()  # Materialize cache
result1 = df.filter("status = 'active'")
result2 = df.groupBy("region").agg(F.sum("revenue"))

# AFTER (just remove .cache()):
df = spark.read.parquet(path)
result1 = df.filter("status = 'active'")
result2 = df.groupBy("region").agg(F.sum("revenue"))
# df.cache is not yet supported — native support coming soon
# For now, materialize expensive intermediate results to Delta (see below)

Materialize Expensive Intermediate Results

# BEFORE:
expensive_df = (spark.read.parquet(path)
    .join(other_df, "key")
    .groupBy("region").agg(F.sum("amount").alias("total")))
expensive_df.cache()
report1 = expensive_df.filter("total > 1000")
report2 = expensive_df.orderBy("total")

# AFTER (materialize to Delta for explicit reuse):
expensive_df = (spark.read.parquet(path)
    .join(other_df, "key")
    .groupBy("region").agg(F.sum("amount").alias("total")))
expensive_df.write.format("delta").mode("overwrite").saveAsTable("main.scratch.region_totals")
intermediate = spark.table("main.scratch.region_totals")
report1 = intermediate.filter("total > 1000")
report2 = intermediate.orderBy("total")

SQL Cache Removal

-- BEFORE:
CACHE TABLE my_temp_table;
SELECT * FROM my_temp_table WHERE status = 'active';

-- AFTER (just remove CACHE TABLE):
SELECT * FROM my_temp_table WHERE status = 'active';

HMS to Unity Catalog

HMS Federation (Bridge Approach)

Use when you need existing code to keep working while planning a full migration:

-- One-time setup: create federation connection
CREATE CONNECTION hms_connection TYPE hive_metastore;
CREATE FOREIGN CATALOG hms_catalog USING CONNECTION hms_connection;

-- Existing queries work with a catalog prefix:
SELECT * FROM hms_catalog.default.my_table;

Direct Migration

-- Migrate a table:
CREATE TABLE main.sales.customers AS
  SELECT * FROM hive_metastore.default.customers;

-- For Delta tables, use SYNC to avoid data copy:
-- (preserves the underlying data files)
SYNC TABLE main.sales.customers
FROM hive_metastore.default.customers;

Update Code References

# BEFORE (2-level namespace):
df = spark.table("my_database.my_table")

# AFTER (3-level namespace with UC):
df = spark.table("main.my_database.my_table")

# Set default catalog to avoid changing every reference:
spark.sql("USE CATALOG main")
df = spark.table("my_database.my_table")  # Now resolves to main.my_database.my_table

Automation with UCX

For large-scale migrations, use the UCX tool to automate HMS to UC migration:

• Scans all notebooks, jobs, and tables for HMS references
• Generates migration scripts
• Handles table, view, and function migration

Init Scripts to Environments

Basic Library Installation

# BEFORE (init script):
#!/bin/bash
pip install numpy==2.2.2 pandas==2.2.3 scikit-learn==1.4.0
pip install /dbfs/libs/my_package-1.0.0.whl

# AFTER (Environment panel or requirements.txt):
numpy==2.2.2
pandas==2.2.3
scikit-learn==1.4.0
/Volumes/main/libs/packages/my_package-1.0.0-py3-none-any.whl

Private Packages

# BEFORE (init script):
#!/bin/bash
pip install --index-url https://pypi.example.com/simple/ my-internal-package==1.0.0

# AFTER (Environment panel):
# Configure private PyPI mirror in workspace admin settings
# Then add to Environment:
my-internal-package==1.0.0

Workspace Utility Packages

# BEFORE (init script installing shared utils):
#!/bin/bash
pip install /dbfs/shared/utils-1.0.0.whl

# AFTER (use workspace files):
# Upload package to workspace:
#   /Workspace/shared/helper_utils/ (with pyproject.toml)
# Add to Environment:
/Workspace/shared/helper_utils

Init Script with apt install to Environment Spec

# BEFORE (init script — setup_ml.sh):
#!/bin/bash
apt-get update
apt-get install -y libgomp1 python3-opencv tesseract-ocr
pip install mlflow==2.12.1 scikit-learn==1.3.0 xgboost==2.0.3
pip install opencv-python==4.9.0.80
pip install /dbfs/libs/internal_ml_utils-2.0.0.whl
export MODEL_REGISTRY=production

// AFTER (Serverless Environment spec):
{
  "environment_key": "ml_env",
  "spec": {
    "client": "2",
    "dependencies": [
      "mlflow==2.12.1",
      "scikit-learn==1.3.0",
      "xgboost==2.0.3",
      "opencv-python==4.9.0.80",
      "pytesseract==0.3.10",
      "/Volumes/<your_catalog>/libs/packages/internal_ml_utils-2.0.0.whl"
    ]
  }
}
// Notes:
// - libgomp1: bundled with xgboost pip package — no separate install needed
// - python3-opencv → opencv-python (pip equivalent)
// - tesseract-ocr → pytesseract (pip wrapper; the OCR engine itself is NOT
//   available on serverless — if Tesseract binary is required, this is a
//   Category 3 blocker)
// - DBFS wheel path → moved to Volumes
// - export MODEL_REGISTRY=production → pass as job parameter, read with
//   dbutils.widgets.get("model_registry")
// - ML runtime is NOT available on serverless — all ML libs must be listed
//   explicitly in dependencies

Critical Rules for Environments

• Never install PySpark as a dependency — this breaks the serverless session
• Always pin versions for reproducibility (numpy==2.2.2, not just numpy)
• Place wheel files in UC Volumes, not DBFS
• Admins can configure default package repositories in workspace settings
• ML runtime is NOT available on serverless — list all ML libraries (mlflow, scikit-learn, xgboost, hyperopt, tensorflow, torch, etc.) explicitly in the environment spec

Custom JDBC to Alternatives

Lakehouse Federation (Recommended)

Query external databases in place with predicate pushdown:

-- One-time setup:
CREATE CONNECTION pg_connection TYPE POSTGRESQL
OPTIONS (
    host 'db.example.com',
    port '5432',
    user secret('scope', 'pg-user'),
    password secret('scope', 'pg-password')
);

CREATE FOREIGN CATALOG pg_catalog
USING CONNECTION pg_connection
OPTIONS (database 'mydb');

-- Query directly (pushdown to remote DB):
SELECT * FROM pg_catalog.public.customers WHERE region = 'US';

Supported connection types: MySQL, PostgreSQL, Oracle, Redshift, SQL Server, BigQuery, Snowflake, Databricks-to-Databricks, and more.

Built-in JDBC (No Custom JARs)

Works on serverless for common databases without custom driver JARs:

df = (spark.read.format("jdbc")
    .option("url", "jdbc:postgresql://host:5432/mydb")
    .option("dbtable", "public.customers")
    .option("user", dbutils.secrets.get("scope", "user"))
    .option("password", dbutils.secrets.get("scope", "password"))
    .load())

PySpark Custom Data Sources

For unsupported formats, implement a PySpark data source:

from pyspark.sql.datasource import DataSource, DataSourceReader
from pyspark.sql.types import StructType, StructField, StringType

class MyCustomReader(DataSourceReader):
    def __init__(self, schema, options):
        self.options = options

    def read(self, partition):
        # Implement your read logic here
        yield ("row1_col1", "row1_col2")

class MyCustomSource(DataSource):
    @classmethod
    def name(cls):
        return "my_custom_source"

    def schema(self):
        return StructType([
            StructField("col1", StringType()),
            StructField("col2", StringType())
        ])

    def reader(self, schema):
        return MyCustomReader(schema, self.options)

# Register and use:
spark.dataSource.register(MyCustomSource)
df = spark.read.format("my_custom_source").option("key", "value").load()

Query Profile (Spark UI Replacement)

Interactive Usage

Serverless compute does not have a Spark UI. Use Query Profile instead:

1. Run your query in a notebook 2. Click "See performance" under the cell output 3. Click "See query profile" for the full execution DAG

Key Features

• Top Operators: Identify the most expensive operations
• Scan Statistics: Row counts, bytes read per table scan
• Shuffle Statistics: Data shuffled between stages
• Export/Import: Share profiles as JSON via the kebab menu

Programmatic Plan Analysis

# Physical plan
df.explain(True)

# Extended plan with statistics
df.explain("extended")

# Cost-based plan
df.explain("cost")

# Formatted plan
df.explain("formatted")

Runtime Detection

Detect whether code is running on serverless to conditionally use Query Profile vs Spark UI:

import os
is_serverless = os.getenv("IS_SERVERLESS", "").lower() == "true"

if is_serverless:
    # Use explain() for plan analysis
    df.explain(True)
else:
    # Can also use Spark UI on classic
    df.explain(True)

Global Temporary View to Session-Scoped Temporary View

CREATE GLOBAL TEMPORARY VIEW is not supported on serverless. Global temp views use a special global_temp database that doesn't exist on serverless. Replace with session-scoped temporary views.

# BEFORE (Classic)
spark.sql("CREATE GLOBAL TEMPORARY VIEW monthly_stats AS SELECT month, SUM(revenue) FROM sales GROUP BY month")
result = spark.sql("SELECT * FROM global_temp.monthly_stats")

# AFTER (Serverless)
spark.sql("CREATE OR REPLACE TEMPORARY VIEW monthly_stats AS SELECT month, SUM(revenue) FROM sales GROUP BY month")
result = spark.sql("SELECT * FROM monthly_stats")

os.environ Restrictions

On serverless, the environment is managed by the platform. Variables like SPARK_HOME, JAVA_HOME, and custom env vars set via cluster configs are NOT available. Direct os.environ["VAR"] will raise KeyError.

# BEFORE (Classic) — crashes on serverless with KeyError
spark_home = os.environ["SPARK_HOME"]
java_home = os.environ["JAVA_HOME"]
custom_var = os.environ["MY_CUSTOM_VAR"]

# AFTER (Serverless)
import os
spark_home = os.environ.get("SPARK_HOME", "N/A")  # Not guaranteed on serverless
java_home = os.environ.get("JAVA_HOME", "/usr/lib/jvm/default")
custom_var = dbutils.widgets.get("MY_CUSTOM_VAR")  # Use widget parameter instead
# Or for Spark-specific info:
spark_version = spark.version  # "14.3" etc.

Hive Variable Substitution to SQL Session Variables

Hive variable substitution (SET hivevar:name = value + ${hivevar:name}) is not supported on serverless. The Spark Connect backend doesn't process Hive variable interpolation. Replace with SQL session variables (available in DBR 14.1+).

-- BEFORE (Classic) — uses Hive variable substitution
SET hivevar:target_date = '2024-01-01';
SET hivevar:min_amount = 100;
SELECT * FROM transactions 
WHERE date >= '${hivevar:target_date}' 
AND amount > ${hivevar:min_amount};

-- AFTER (Serverless) — uses SQL session variables (DBR 14.1+)
DECLARE OR REPLACE VARIABLE target_date STRING = '2024-01-01';
DECLARE OR REPLACE VARIABLE min_amount INT = 100;
SELECT * FROM transactions 
WHERE date >= target_date 
AND amount > min_amount;

For Python notebooks, use Python variables and f-strings instead of Hive substitution:

# BEFORE (Classic) — Hive substitution in spark.sql
spark.sql("SET hivevar:target_date = '2024-01-01'")
result = spark.sql("SELECT * FROM transactions WHERE date >= '${hivevar:target_date}'")

# AFTER (Serverless) — Python f-strings
target_date = '2024-01-01'
result = spark.sql(f"SELECT * FROM transactions WHERE date >= '{target_date}'")

Documentation

• Serverless limitations: https://docs.databricks.com/en/compute/serverless/limitations
• Spark Connect vs classic: https://docs.databricks.com/en/spark/connect-vs-classic
• UC Volumes: https://docs.databricks.com/en/volumes/volume-files
• UC external locations: https://docs.databricks.com/en/connect/unity-catalog/external-locations.html
• UC upgrade guide: https://docs.databricks.com/en/data-governance/unity-catalog/upgrade/
• Serverless dependencies: https://docs.databricks.com/en/compute/serverless/dependencies
• Lakehouse Federation: https://docs.databricks.com/en/query-federation/
• Query Profile: https://docs.databricks.com/en/sql/user/queries/query-profile.html
• PySpark data sources: https://docs.databricks.com/en/pyspark/datasource.html

Compatibility Checks Reference

Complete pattern detection table for serverless compute compatibility scanning. Each entry includes the detection pattern, severity, fix summary, and a link to authoritative documentation.

How to Use This Reference

Scan user code for the regex/grep patterns below. When a match is found, report the severity and provide the fix. Patterns are grouped by category matching the SKILL.md analysis categories (A through G).

Category A: Unsupported APIs

RDD and SparkContext Operations

SparkContext Methods

Caching Operations

Language Restrictions

Category B: Data Access

Category C: Streaming

Category D: Configuration

Category E: Libraries

Category F: Networking

Category G: Sizing and Debugging

Comprehensive Regex for Quick Scan

Use these combined patterns for a fast first-pass scan:

# Blockers - must fix before migration
blocker_patterns = [
    r'sc\.(parallelize|textFile|wholeTextFiles|broadcast|accumulator)\(',
    r'SparkContext\.getOrCreate\(\)',
    r'spark\.sparkContext',
    r'sqlContext\.',
    r'sc\.hadoopConfiguration',
    r'\.mapPartitions\(',
    r'\.groupByKey\(',
    r'\.reduceByKey\(',
    r'\.flatMap\(',
    r'dbfs:/',
    r'"/dbfs/',
    r'dbutils\.fs\.mount\(',
    r'\.trigger\(processingTime',
    r'\.trigger\(continuous',
    r'%scala',
    r'%r\b',
    r'%pip install.*pyspark',
    r'CREATE\s+(DATABASE|SCHEMA)\s+',  # Check for missing USE CATALOG
    r'"init_scripts"',  # In job JSON
    r'CREATE\s+GLOBAL\s+TEMP',  # Global temp views not supported
    r'SET\s+hivevar:',  # Hive variable substitution not supported
    r'\$\{hivevar:',  # Hive variable references not supported
]

# Warnings - should fix for best results
warning_patterns = [
    r'\.cache\(\)',
    r'\.persist\(',
    r'\.checkpoint\(',
    r'hive_metastore\.',
    r'spark\.conf\.set\(',
    r'CACHE\s+TABLE',
    r'\$\{[^}]+\}',  # Hive variable syntax in SQL
    r'%run\s+\.\.?\/',  # Relative %run paths
    r'file:///dbfs/',  # FUSE mount paths
    r'global_temp\.',  # Global temp view references
    r'os\.environ\[',  # Direct env var access (may KeyError on serverless)
]

# Info - awareness items
info_patterns = [
    r'spark\.executor\.',
    r'spark\.driver\.memory',
    r'spark\.ui\.',
    r'format\("kafka"\)',
    r'format\("cloudFiles"\)',
]

Configuration Guide

Supported Spark configurations, environment setup, budget policies, and performance mode selection for serverless compute.

Supported Spark Configurations

Serverless compute supports only 6 Spark configurations. All other configurations are auto-tuned and cannot be overridden.

Setting Supported Configurations

# In Python:
spark.conf.set("spark.sql.shuffle.partitions", "200")
spark.conf.set("spark.sql.session.timeZone", "America/New_York")
spark.conf.set("spark.sql.ansi.enabled", "false")

-- In SQL:
SET spark.sql.shuffle.partitions = 200;
SET spark.sql.session.timeZone = 'America/New_York';
SET spark.sql.ansi.enabled = false;

Important: ANSI Mode is Enabled by Default

Serverless compute enables ANSI SQL mode by default. This means:

• Integer overflow throws an error instead of wrapping
• Division by zero throws an error instead of returning NULL
• Invalid casts throw errors instead of returning NULL

If your existing code relies on non-ANSI behavior, you may need to:

# Option 1: Disable ANSI mode (quick fix)
spark.conf.set("spark.sql.ansi.enabled", "false")

# Option 2: Fix code to handle edge cases (recommended)
from pyspark.sql import functions as F
# Use try_cast instead of cast for safe conversions
df.select(F.expr("try_cast(col AS INT)"))
# Use try_divide for safe division
df.select(F.expr("try_divide(a, b)"))

Common Unsupported Configurations and What to Do

Auto-Tuned (Remove — Serverless Handles Automatically)

Resource Configs (Remove — Serverless Auto-Scales)

Credential Configs (Remove — Use UC Instead)

Temporary Bridge Strategy

If removing a configuration breaks your workload, check if it maps to one of the 6 supported configs. Common mappings:

# BEFORE (classic — many configs):
spark.conf.set("spark.sql.shuffle.partitions", "2000")
spark.conf.set("spark.sql.adaptive.enabled", "true")
spark.conf.set("spark.sql.adaptive.coalescePartitions.enabled", "true")
spark.conf.set("spark.executor.memory", "8g")
spark.conf.set("spark.default.parallelism", "2000")

# AFTER (serverless — only the supported config):
spark.conf.set("spark.sql.shuffle.partitions", "2000")
# Everything else is auto-tuned. You can likely remove shuffle.partitions too
# since AQE will optimize it, but keep it temporarily if needed for validation.

Environment Setup

Managing Dependencies

Serverless compute uses Environments instead of init scripts, cluster libraries, or dbutils.library.install. Dependencies can be configured via:

1. Notebook Environment panel — click the Environment icon in the notebook sidebar 2. Job Environment spec — in the job task configuration 3. `%pip install` — inline in notebooks (works but Environment panel is preferred for reproducibility)

Pinning Versions

Always pin dependency versions for reproducible builds:

# Good — pinned versions:
numpy==2.2.2
pandas==2.2.3
scikit-learn==1.4.0
requests==2.31.0

# Bad — unpinned (may break on next run):
numpy
pandas
scikit-learn

Using Private Package Repositories

# In workspace admin settings, configure a default PyPI mirror:
# Admin Console → Compute → Environment Settings → PyPI Repository

# Then packages resolve from your private mirror:
my-internal-package==1.0.0

Installing Wheel Files

# From UC Volumes (recommended):
/Volumes/main/libs/packages/my_package-1.0.0-py3-none-any.whl

# From workspace files:
/Workspace/shared/packages/my_package-1.0.0-py3-none-any.whl

Workspace Utility Packages

For shared utility code used across notebooks:

# Structure in workspace:
# /Workspace/shared/helper_utils/
#   ├── pyproject.toml
#   ├── helper_utils/
#   │   ├── __init__.py
#   │   └── transforms.py

# Add to Environment:
/Workspace/shared/helper_utils

# Use in notebook:
from helper_utils.transforms import clean_data

Critical Rules

• Never install PySpark — this overwrites the managed PySpark installation and breaks the session
• Never install packages that bundle their own Spark (e.g., some older ML packages)
• Pin all versions — floating versions cause non-reproducible results
• Use UC Volumes for wheel files, not DBFS paths

Budget Policies

Budget policies replace cluster policies for cost attribution and control on serverless.

Creating Budget Policies

Budget policies are configured by workspace admins via the Databricks UI or API:

# List existing budget policies
databricks budget-policies list

# Assign a budget policy to a job
databricks jobs update <job-id> \
  --json '{"budget_policy_id": "<policy-id>"}'

Using Budget Policies for Cost Attribution

• Each serverless workload can be tagged with a budget policy
• Cost data flows to system.billing.usage with the policy tag
• Use this for team-level or project-level cost tracking

-- Query cost by budget policy
SELECT
  budget_policy_id,
  SUM(usage_quantity) as total_dbus,
  SUM(usage_quantity * list_price) as estimated_cost
FROM system.billing.usage
WHERE usage_type = 'SERVERLESS_COMPUTE'
  AND usage_date >= current_date() - INTERVAL 30 DAYS
GROUP BY budget_policy_id
ORDER BY estimated_cost DESC;

Monitoring Cost After Migration

-- Compare classic vs serverless cost for the same workload
SELECT
  sku_name,
  usage_type,
  SUM(usage_quantity) as total_dbus,
  MIN(usage_date) as first_seen,
  MAX(usage_date) as last_seen
FROM system.billing.usage
WHERE workspace_id = '<workspace-id>'
  AND usage_date >= current_date() - INTERVAL 90 DAYS
GROUP BY sku_name, usage_type
ORDER BY total_dbus DESC;

Performance Mode Selection

Serverless compute offers two performance modes. Choose based on workload characteristics.

Performance-Optimized

Standard

Standard mode is NOT available for notebooks. Notebooks always use Performance-Optimized.

Selection Guide

Workload → What type?
├── Interactive notebook → Performance-Optimized (only option)
├── Job / Pipeline
│   ├── Time-sensitive (SLA < 10 min)? → Performance-Optimized
│   ├── Dev/test iteration? → Performance-Optimized
│   ├── Batch ETL, scheduled? → Standard (cost savings)
│   ├── Large-scale data processing? → Standard (cost savings)
│   └── Default for production pipelines → Standard (unless latency matters)

Setting Performance Mode

# Via Databricks CLI for a job:
databricks jobs update <job-id> \
  --json '{"tasks": [{"task_key": "etl", "environment_key": "default", "compute": {"serverless": {"performance_mode": "STANDARD"}}}]}'

In the Databricks UI: 1. Open the Job configuration 2. Under Compute, select "Serverless" 3. Choose "Performance-Optimized" or "Standard"

Serverless Compute Defaults

Job Configuration Transformation

Before (Classic Compute)

{
  "job_clusters": [{
    "job_cluster_key": "my_cluster",
    "new_cluster": {
      "spark_version": "15.4.x-scala2.12",
      "num_workers": 2,
      "node_type_id": "Standard_DS3_v2",
      "init_scripts": [{ "dbfs": { "destination": "dbfs:/init/setup.sh" } }]
    }
  }],
  "tasks": [{
    "task_key": "main_task",
    "job_cluster_key": "my_cluster",
    "notebook_task": { "notebook_path": "/path/to/notebook" }
  }]
}

After (Serverless)

{
  "environments": [{
    "environment_key": "serverless_env",
    "spec": {
      "client": "2",
      "dependencies": ["pandas==2.2.3", "numpy==2.2.2"]
    }
  }],
  "tasks": [{
    "task_key": "main_task",
    "environment_key": "serverless_env",
    "notebook_task": { "notebook_path": "/path/to/notebook" }
  }]
}

Key Changes

• Remove job_clusters / new_cluster — serverless manages infrastructure. Removing the cluster reference is what makes a task run on serverless compute.
• Remove job_cluster_key (and any existing_cluster_id / new_cluster) from each task. Tasks with no cluster reference and no compute block run on serverless compute by default.
• Add an environments array at the job level if the task needs Python dependencies, and reference one via environment_key per task. environment_key declares dependencies (PyPI / workspace files / etc.), it does not replace job_cluster_key as a compute selector. A task can have environment_key set without specifying compute, and it will still run on serverless.
• Remove init_scripts — move to environment dependencies
• Remove num_workers, node_type_id, spark_version — all auto-managed

Job Definition Migration (Complete Example)

This section shows the full transformation from a classic job with init scripts, cluster libraries, and spark_conf to a serverless job with Environments.

Before (Classic Job with Init Scripts, Libraries, and spark_conf)

{
  "name": "daily_etl_pipeline",
  "job_clusters": [{
    "job_cluster_key": "etl_cluster",
    "new_cluster": {
      "spark_version": "15.4.x-ml-scala2.12",
      "num_workers": 4,
      "node_type_id": "i3.xlarge",
      "init_scripts": [
        { "dbfs": { "destination": "dbfs:/init/install_packages.sh" } },
        { "dbfs": { "destination": "dbfs:/init/setup_env.sh" } }
      ],
      "spark_conf": {
        "spark.sql.shuffle.partitions": "400",
        "spark.sql.adaptive.enabled": "true",
        "spark.executor.memory": "8g",
        "spark.databricks.delta.autoCompact.enabled": "true",
        "fs.s3a.access.key": "{{secrets/scope/s3-key}}"
      },
      "custom_tags": { "team": "data-eng" }
    }
  }],
  "tasks": [{
    "task_key": "etl_main",
    "job_cluster_key": "etl_cluster",
    "notebook_task": { "notebook_path": "/Repos/team/project/etl_main" },
    "libraries": [
      { "maven": { "coordinates": "com.example:connector:1.0" } },
      { "jar": "dbfs:/libs/custom_udf.jar" }
    ]
  }]
}

Suppose the init scripts contain:

# install_packages.sh
#!/bin/bash
pip install mlflow==2.12.1 scikit-learn==1.3.0 xgboost==2.0.3
pip install /dbfs/libs/internal_utils-1.0.0.whl

# setup_env.sh
#!/bin/bash
apt-get install -y libgomp1
export MODEL_ENV=production

After (Serverless Job with Environments)

{
  "name": "daily_etl_pipeline",
  "environments": [{
    "environment_key": "etl_env",
    "spec": {
      "client": "2",
      "dependencies": [
        "mlflow==2.12.1",
        "scikit-learn==1.3.0",
        "xgboost==2.0.3",
        "/Volumes/<your_catalog>/libs/packages/internal_utils-1.0.0.whl"
      ]
    }
  }],
  "tasks": [{
    "task_key": "etl_main",
    "environment_key": "etl_env",
    "notebook_task": {
      "notebook_path": "/Repos/team/project/etl_main",
      "base_parameters": { "model_env": "production" }
    }
  }]
}

Transformation Notes

Environment Version Mapping

Match the serverless environment version to the DBR version the workload was running on:

Example environment configuration:

{
  "environments": [{
    "environment_key": "serverless_env",
    "spec": {
      "client": "2",
      "dependencies": ["pandas==2.2.3"]
    }
  }]
}

For test jobs, disable auto optimization for faster feedback:

from databricks.sdk import WorkspaceClient
from databricks.sdk.service import jobs

w = WorkspaceClient()
w.jobs.create(
    name="serverless-test-job",
    tasks=[jobs.Task(
        task_key="test_task",
        notebook_task=jobs.NotebookTask(notebook_path="/path/to/notebook"),
        environment_key="serverless_env",
        disable_auto_optimization=True,  # Faster startup for testing
        max_retries=0,  # No retries — surface failures immediately
    )],
    environments=[jobs.JobEnvironment(
        environment_key="serverless_env",
        spec=jobs.JobEnvironmentSpec(client="2", dependencies=[]),
    )],
)

Catalog Parameterization

Make notebooks work with both production and test catalogs during migration:

# Add at the top of the notebook
dbutils.widgets.text("catalog", "main")  # Default to production
catalog = dbutils.widgets.get("catalog")

# Use the parameter in all table references
df = spark.table(f"{catalog}.sales.orders")
df_accounts = spark.table(f"{catalog}.sfdc_bronze.account")

When creating mock tables for testing, the schema names stay the same — only the catalog changes:

• main.sfdc_bronze.case → test_catalog.sfdc_bronze.case
• main.sales.orders → test_catalog.sales.orders

Pass catalog="test_catalog" as a job parameter during testing. In production, the default "main" is used.

Test Data Setup Pattern

When migrating jobs that read from production tables, do not run the test job against production. Create sampled copies in a dedicated test catalog instead — this keeps the test loop fast, safe, and idempotent.

Prerequisites

• A running SQL warehouse (needed to execute the CREATE TABLE AS SELECT statements)
• A test catalog that already exists, with CREATE TABLE and CREATE SCHEMA permissions granted to the user
• SELECT permission on each source table — flag any permission gaps early, do not silently skip tables

Setup Steps

1. Discover upstream tables from the job — either from the job's lineage (if available) or by static scan of the notebook 2. For each upstream table, run:

CREATE SCHEMA IF NOT EXISTS <test_catalog>.<schema>;
CREATE TABLE IF NOT EXISTS <test_catalog>.<schema>.<table>
AS SELECT * FROM <prod_catalog>.<schema>.<table> LIMIT <N>;

3. Typical N is 10 rows for smoke tests, up to 1000 for more realistic validation 4. Keep schema names identical between prod and test — only the catalog changes. This lets the same notebook code run in both environments via the catalog widget

Idempotency

Use CREATE TABLE IF NOT EXISTS so the setup is safe to re-run. Skip tables that already exist rather than recreating them — this keeps test data stable across migration iterations.

Post-Setup

Once mock tables exist, update the notebook's catalog references (or set the catalog job parameter) to point to the test catalog before creating the test job. Do not hardcode the test catalog into the notebook — use the widget parameter so the same file ships to production unchanged.

Test Job Settings

For the test job, disable auto-optimization and retries to surface failures quickly:

from databricks.sdk import WorkspaceClient
from databricks.sdk.service import jobs

w = WorkspaceClient()
w.jobs.create(
    name="serverless-test-<job_name>",
    tasks=[jobs.Task(
        task_key="test_task",
        notebook_task=jobs.NotebookTask(
            notebook_path="/path/to/notebook",
            base_parameters={"catalog": "<test_catalog>"},
        ),
        environment_key="serverless_env",
        disable_auto_optimization=True,  # Faster startup while iterating
        max_retries=0,                    # Surface failures immediately — no retries during testing
    )],
    environments=[jobs.JobEnvironment(
        environment_key="serverless_env",
        spec=jobs.JobEnvironmentSpec(client="2", dependencies=[]),
    )],
)

• disable_auto_optimization=True — skip auto-optimization for faster test-loop feedback
• max_retries=0 — any failure surfaces immediately instead of being masked by retries
• Do not carry these settings into the production job config — they are test-only

Debugging Failed Serverless Runs

Always get the actual error before guessing:

from databricks.sdk import WorkspaceClient
w = WorkspaceClient()

run_id = 12345  # replace with the failed run_id from the Jobs UI
run_output = w.jobs.get_run_output(run_id=run_id)
print(run_output.error)        # Error message
print(run_output.error_trace)  # Full stack trace

Common Error Messages

Documentation

• Supported Spark configs: https://docs.databricks.com/en/spark/conf#serverless
• Serverless environments/dependencies: https://docs.databricks.com/en/compute/serverless/dependencies
• Budget policies: https://docs.databricks.com/en/admin/usage/budget-policies.html
• System billing tables: https://docs.databricks.com/en/admin/system-tables/billing.html
• Serverless best practices: https://docs.databricks.com/en/compute/serverless/best-practices
• Serverless compute overview: https://docs.databricks.com/en/compute/serverless/
• Databricks SDK: https://docs.databricks.com/en/dev-tools/sdk-python.html