Databricks Vector Search

Databricks Vector Search

FIRST: Use the parent databricks-core skill for CLI basics, authentication, and profile selection.

Patterns for creating, managing, and querying vector search indexes for RAG and semantic search applications.

When to Use

Use this skill when:

• Building RAG (Retrieval-Augmented Generation) applications
• Implementing semantic search or similarity matching
• Creating vector indexes from Delta tables
• Choosing between storage-optimized and standard endpoints
• Querying vector indexes with filters

Overview

Databricks Vector Search provides managed vector similarity search with automatic embedding generation and Delta Lake integration.

Endpoint Types

Index Types

Quick Start

Create Endpoint

from databricks.sdk import WorkspaceClient

w = WorkspaceClient()

# Create a standard endpoint
endpoint = w.vector_search_endpoints.create_endpoint(
    name="my-vs-endpoint",
    endpoint_type="STANDARD"  # or "STORAGE_OPTIMIZED"
)
# Note: Endpoint creation is asynchronous; check status with get_endpoint()

Create Delta Sync Index (Managed Embeddings)

# Source table must have: primary key column + text column
index = w.vector_search_indexes.create_index(
    name="catalog.schema.my_index",
    endpoint_name="my-vs-endpoint",
    primary_key="id",
    index_type="DELTA_SYNC",
    delta_sync_index_spec={
        "source_table": "catalog.schema.documents",
        "embedding_source_columns": [
            {
                "name": "content",  # Text column to embed
                "embedding_model_endpoint_name": "databricks-gte-large-en"
            }
        ],
        "pipeline_type": "TRIGGERED"  # or "CONTINUOUS"
    }
)

Query Index

results = w.vector_search_indexes.query_index(
    index_name="catalog.schema.my_index",
    columns=["id", "content", "metadata"],
    query_text="What is machine learning?",
    num_results=5
)

for doc in results.result.data_array:
    score = doc[-1]  # Similarity score is last column
    print(f"Score: {score}, Content: {doc[1][:100]}...")

Common Patterns

Create Storage-Optimized Endpoint

# For large-scale, cost-effective deployments
endpoint = w.vector_search_endpoints.create_endpoint(
    name="my-storage-endpoint",
    endpoint_type="STORAGE_OPTIMIZED"
)

Delta Sync with Self-Managed Embeddings

# Source table must have: primary key + embedding vector column
index = w.vector_search_indexes.create_index(
    name="catalog.schema.my_index",
    endpoint_name="my-vs-endpoint",
    primary_key="id",
    index_type="DELTA_SYNC",
    delta_sync_index_spec={
        "source_table": "catalog.schema.documents",
        "embedding_vector_columns": [
            {
                "name": "embedding",  # Pre-computed embedding column
                "embedding_dimension": 768
            }
        ],
        "pipeline_type": "TRIGGERED"
    }
)

Direct Access Index

import json

# Create index for manual CRUD
index = w.vector_search_indexes.create_index(
    name="catalog.schema.direct_index",
    endpoint_name="my-vs-endpoint",
    primary_key="id",
    index_type="DIRECT_ACCESS",
    direct_access_index_spec={
        "embedding_vector_columns": [
            {"name": "embedding", "embedding_dimension": 768}
        ],
        "schema_json": json.dumps({
            "id": "string",
            "text": "string",
            "embedding": "array<float>",
            "metadata": "string"
        })
    }
)

# Upsert data
w.vector_search_indexes.upsert_data_vector_index(
    index_name="catalog.schema.direct_index",
    inputs_json=json.dumps([
        {"id": "1", "text": "Hello", "embedding": [0.1, 0.2, ...], "metadata": "doc1"},
        {"id": "2", "text": "World", "embedding": [0.3, 0.4, ...], "metadata": "doc2"},
    ])
)

# Delete data
w.vector_search_indexes.delete_data_vector_index(
    index_name="catalog.schema.direct_index",
    primary_keys=["1", "2"]
)

Query with Embedding Vector

# When you have pre-computed query embedding
results = w.vector_search_indexes.query_index(
    index_name="catalog.schema.my_index",
    columns=["id", "text"],
    query_vector=[0.1, 0.2, 0.3, ...],  # Your 768-dim vector
    num_results=10
)

Hybrid Search (Semantic + Keyword)

Hybrid search combines vector similarity (ANN) with BM25 keyword scoring. Use it when queries contain exact terms that must match — SKUs, error codes, proper nouns, or technical terminology — where pure semantic search might miss keyword-specific results. See references/search-modes.md for detailed guidance on choosing between ANN and hybrid search.

# Combines vector similarity with keyword matching
results = w.vector_search_indexes.query_index(
    index_name="catalog.schema.my_index",
    columns=["id", "content"],
    query_text="SPARK-12345 executor memory error",
    query_type="HYBRID",
    num_results=10
)

Filtering

Standard Endpoint Filters (Dictionary)

# filters_json uses dictionary format
results = w.vector_search_indexes.query_index(
    index_name="catalog.schema.my_index",
    columns=["id", "content"],
    query_text="machine learning",
    num_results=10,
    filters_json='{"category": "ai", "status": ["active", "pending"]}'
)

Storage-Optimized Filters (SQL-like)

Storage-Optimized endpoints use SQL-like filter syntax via the databricks-vectorsearch package's filters parameter (accepts a string):

from databricks.vector_search.client import VectorSearchClient

vsc = VectorSearchClient()
index = vsc.get_index(endpoint_name="my-storage-endpoint", index_name="catalog.schema.my_index")

# SQL-like filter syntax for storage-optimized endpoints
results = index.similarity_search(
    query_text="machine learning",
    columns=["id", "content"],
    num_results=10,
    filters="category = 'ai' AND status IN ('active', 'pending')"
)

# More filter examples
# filters="price > 100 AND price < 500"
# filters="department LIKE 'eng%'"
# filters="created_at >= '2024-01-01'"

Trigger Index Sync

# For TRIGGERED pipeline type, manually sync
w.vector_search_indexes.sync_index(
    index_name="catalog.schema.my_index"
)

Scan All Index Entries

# Retrieve all vectors (for debugging/export)
scan_result = w.vector_search_indexes.scan_index(
    index_name="catalog.schema.my_index",
    num_results=100
)

Reference Files

CLI Quick Reference

# List endpoints
databricks vector-search-endpoints list-endpoints

# Create endpoint (positional args: NAME ENDPOINT_TYPE)
databricks vector-search-endpoints create-endpoint my-endpoint STANDARD

# List indexes on endpoint (positional arg: ENDPOINT_NAME)
databricks vector-search-indexes list-indexes my-endpoint

# Get index status (positional arg: INDEX_NAME)
databricks vector-search-indexes get-index catalog.schema.my_index

# Sync index (positional arg: INDEX_NAME)
databricks vector-search-indexes sync-index catalog.schema.my_index

# Delete index (positional arg: INDEX_NAME)
databricks vector-search-indexes delete-index catalog.schema.my_index

Common Issues

Embedding Models

Databricks provides built-in embedding models:

# Use with managed embeddings
embedding_source_columns=[
    {
        "name": "content",
        "embedding_model_endpoint_name": "databricks-gte-large-en"
    }
]

Notes

• Storage-Optimized is newer — better for most use cases unless you need <100ms latency
• Delta Sync recommended — easier than Direct Access for most scenarios
• Hybrid search — available for both Delta Sync and Direct Access indexes
• `columns_to_sync` matters — only synced columns are available in query results; include all columns you need
• Filter syntax differs by endpoint — Standard uses dict-format filters, Storage-Optimized uses SQL-like string filters. Use the databricks-vectorsearch package's filters parameter which accepts both formats
• Management vs runtime — CLI and SDK handle lifecycle management; for agent tool-calling at runtime, use VectorSearchRetrieverTool

Related Skills

• databricks-model-serving - Deploy agents that use VectorSearchRetrieverTool
• [databricks-agent-bricks](../databricks-agent-bricks/SKILL.md) - Knowledge Assistants use RAG over indexed documents
• [databricks-unstructured-pdf-generation](../databricks-unstructured-pdf-generation/SKILL.md) - Generate documents to index in Vector Search
• [databricks-unity-catalog](../databricks-unity-catalog/SKILL.md) - Manage the catalogs and tables that back Delta Sync indexes
• databricks-pipelines - Build Delta tables used as Vector Search sources

End-to-End RAG with Vector Search

Build a complete Retrieval-Augmented Generation pipeline: prepare documents, create a vector index, query it, and wire it into an agent.

CLI Commands Used

---

Step 1: Prepare Source Table

The source Delta table needs a primary key column and a text column to embed.

CREATE TABLE IF NOT EXISTS catalog.schema.knowledge_base (
    doc_id STRING,
    title STRING,
    content STRING,
    category STRING,
    updated_at TIMESTAMP DEFAULT current_timestamp()
);

INSERT INTO catalog.schema.knowledge_base VALUES
('doc-001', 'Getting Started', 'Databricks is a unified analytics platform...', 'overview', current_timestamp()),
('doc-002', 'Unity Catalog', 'Unity Catalog provides centralized governance...', 'governance', current_timestamp()),
('doc-003', 'Delta Lake', 'Delta Lake is an open-source storage layer...', 'storage', current_timestamp());

Or via CLI:

databricks experimental aitools tools query --warehouse WAREHOUSE_ID "
    CREATE TABLE IF NOT EXISTS catalog.schema.knowledge_base (
        doc_id STRING,
        title STRING,
        content STRING,
        category STRING,
        updated_at TIMESTAMP DEFAULT current_timestamp()
    )
"

Step 2: Create Vector Search Endpoint

manage_vs_endpoint(
    action="create",
    name="my-rag-endpoint",
    endpoint_type="STORAGE_OPTIMIZED"
)

Endpoint creation is asynchronous. Check status:

manage_vs_endpoint(action="get", name="my-rag-endpoint")
# Wait for state: "ONLINE"

Step 3: Create Delta Sync Index

manage_vs_index(
    action="create",
    name="catalog.schema.knowledge_base_index",
    endpoint_name="my-rag-endpoint",
    primary_key="doc_id",
    index_type="DELTA_SYNC",
    delta_sync_index_spec={
        "source_table": "catalog.schema.knowledge_base",
        "embedding_source_columns": [
            {
                "name": "content",
                "embedding_model_endpoint_name": "databricks-gte-large-en"
            }
        ],
        "pipeline_type": "TRIGGERED",
        "columns_to_sync": ["doc_id", "title", "content", "category"]
    }
)

Key decisions:

• `embedding_source_columns`: Databricks computes embeddings automatically from the content column
• `pipeline_type`: TRIGGERED for manual sync (cheaper), CONTINUOUS for auto-sync on table changes
• `columns_to_sync`: Only sync columns you need in query results (reduces storage and improves performance)

Step 4: Sync and Verify

# Trigger initial sync
manage_vs_index(action="sync", index_name="catalog.schema.knowledge_base_index")

# Check status
manage_vs_index(action="get", index_name="catalog.schema.knowledge_base_index")
# Wait for state: "ONLINE"

Step 5: Query the Index

# Semantic search
query_vs_index(
    index_name="catalog.schema.knowledge_base_index",
    columns=["doc_id", "title", "content", "category"],
    query_text="How do I govern my data?",
    num_results=3
)

With Filters

The filter syntax depends on the endpoint type used when creating the index.

# Storage-Optimized endpoint (used in this walkthrough): SQL-like filter syntax
query_vs_index(
    index_name="catalog.schema.knowledge_base_index",
    columns=["doc_id", "title", "content"],
    query_text="How do I govern my data?",
    num_results=3,
    filters="category = 'governance'"
)

# Standard endpoint (if you created a Standard endpoint instead): JSON filters_json
query_vs_index(
    index_name="catalog.schema.my_standard_index",
    columns=["doc_id", "title", "content"],
    query_text="How do I govern my data?",
    num_results=3,
    filters_json='{"category": "governance"}'
)

Hybrid Search (Vector + Keyword)

query_vs_index(
    index_name="catalog.schema.knowledge_base_index",
    columns=["doc_id", "title", "content"],
    query_text="Delta Lake ACID transactions",
    num_results=5,
    query_type="HYBRID"
)

---

Step 6: Use in an Agent

As a Tool in a ChatAgent

Use VectorSearchRetrieverTool to wire the index into an agent deployed on Model Serving:

from databricks.agents import ChatAgent
from databricks.agents.tools import VectorSearchRetrieverTool
from databricks.sdk import WorkspaceClient

# Define the retriever tool
retriever_tool = VectorSearchRetrieverTool(
    index_name="catalog.schema.knowledge_base_index",
    columns=["doc_id", "title", "content"],
    num_results=3,
)

class RAGAgent(ChatAgent):
    def __init__(self):
        self.w = WorkspaceClient()

    def predict(self, messages, context=None):
        query = messages[-1].content

        results = self.w.vector_search_indexes.query_index(
            index_name="catalog.schema.knowledge_base_index",
            columns=["title", "content"],
            query_text=query,
            num_results=3,
        )

        context_docs = "\n\n".join(
            f"**{row[0]}**: {row[1]}"
            for row in results.result.data_array
        )

        response = self.w.serving_endpoints.query(
            name="databricks-meta-llama-3-3-70b-instruct",
            messages=[
                {"role": "system", "content": f"Answer using this context:\n{context_docs}"},
                {"role": "user", "content": query},
            ],
        )

        return {"content": response.choices[0].message.content}

---

Updating the Index

Add New Documents

INSERT INTO catalog.schema.knowledge_base VALUES
('doc-004', 'MLflow', 'MLflow is an open-source platform for ML lifecycle...', 'ml', current_timestamp());

Then sync:

manage_vs_index(action="sync", index_name="catalog.schema.knowledge_base_index")

Delete Documents

DELETE FROM catalog.schema.knowledge_base WHERE doc_id = 'doc-001';

Then sync — the index automatically handles deletions via Delta change data feed.

---

Common Issues

Vector Search Index Types

Comparison Matrix

Delta Sync with Managed Embeddings

Databricks automatically computes embeddings from your text column.

Requirements

• Source Delta table with:
• Primary key column (unique identifier)
• Text column (content to embed)
• Embedding model endpoint (or use built-in)

Create Index

from databricks.sdk import WorkspaceClient

w = WorkspaceClient()

index = w.vector_search_indexes.create_index(
    name="catalog.schema.docs_index",
    endpoint_name="my-vs-endpoint",
    primary_key="doc_id",
    index_type="DELTA_SYNC",
    delta_sync_index_spec={
        "source_table": "catalog.schema.documents",
        "embedding_source_columns": [
            {
                "name": "content",
                "embedding_model_endpoint_name": "databricks-gte-large-en"
            }
        ],
        "pipeline_type": "TRIGGERED",  # or "CONTINUOUS"
        "columns_to_sync": ["doc_id", "content", "title", "category"]
    }
)

Pipeline Types

Source Table Example

CREATE TABLE catalog.schema.documents (
    doc_id STRING,
    title STRING,
    content STRING,  -- Text to embed
    category STRING,
    created_at TIMESTAMP
);

Delta Sync with Self-Managed Embeddings

You pre-compute embeddings and store them in the source table.

Requirements

• Source Delta table with:
• Primary key column
• Embedding vector column (array of floats)

Create Index

index = w.vector_search_indexes.create_index(
    name="catalog.schema.custom_index",
    endpoint_name="my-vs-endpoint",
    primary_key="id",
    index_type="DELTA_SYNC",
    delta_sync_index_spec={
        "source_table": "catalog.schema.embedded_docs",
        "embedding_vector_columns": [
            {
                "name": "embedding",
                "embedding_dimension": 768
            }
        ],
        "pipeline_type": "TRIGGERED"
    }
)

Compute Embeddings

from databricks.sdk import WorkspaceClient
import pandas as pd

w = WorkspaceClient()

def get_embeddings(texts: list[str]) -> list[list[float]]:
    """Call embedding endpoint for texts."""
    response = w.serving_endpoints.query(
        name="databricks-gte-large-en",
        input=texts
    )
    return [item.embedding for item in response.data]

# Add embeddings to your data
df = spark.table("catalog.schema.documents").toPandas()
df["embedding"] = get_embeddings(df["content"].tolist())

# Write back to Delta
spark.createDataFrame(df).write.mode("overwrite").saveAsTable(
    "catalog.schema.embedded_docs"
)

Source Table Example

CREATE TABLE catalog.schema.embedded_docs (
    id STRING,
    content STRING,
    embedding ARRAY<FLOAT>,  -- Pre-computed embedding
    metadata STRING
);

Direct Access Index

Full control over vector data via CRUD API. No Delta table sync.

Requirements

• Define schema upfront
• Manage upsert/delete operations yourself

Create Index

import json

index = w.vector_search_indexes.create_index(
    name="catalog.schema.realtime_index",
    endpoint_name="my-vs-endpoint",
    primary_key="id",
    index_type="DIRECT_ACCESS",
    direct_access_index_spec={
        "embedding_vector_columns": [
            {"name": "embedding", "embedding_dimension": 768}
        ],
        "schema_json": json.dumps({
            "id": "string",
            "text": "string",
            "embedding": "array<float>",
            "category": "string",
            "score": "float"
        })
    }
)

Upsert Data

import json

# Insert or update vectors
w.vector_search_indexes.upsert_data_vector_index(
    index_name="catalog.schema.realtime_index",
    inputs_json=json.dumps([
        {
            "id": "doc-001",
            "text": "Machine learning basics",
            "embedding": [0.1, 0.2, 0.3, ...],  # 768 floats
            "category": "ml",
            "score": 0.95
        },
        {
            "id": "doc-002",
            "text": "Deep learning overview",
            "embedding": [0.4, 0.5, 0.6, ...],
            "category": "dl",
            "score": 0.88
        }
    ])
)

Delete Data

w.vector_search_indexes.delete_data_vector_index(
    index_name="catalog.schema.realtime_index",
    primary_keys=["doc-001", "doc-002"]
)

Attach Embedding Model (Optional)

For Direct Access with text queries:

# Create index with embedding model for query-time embedding
index = w.vector_search_indexes.create_index(
    name="catalog.schema.hybrid_index",
    endpoint_name="my-vs-endpoint",
    primary_key="id",
    index_type="DIRECT_ACCESS",
    direct_access_index_spec={
        "embedding_vector_columns": [
            {"name": "embedding", "embedding_dimension": 768}
        ],
        "embedding_model_endpoint_name": "databricks-gte-large-en",  # For query_text
        "schema_json": json.dumps({...})
    }
)

Choosing the Right Type

Start here:
│
├─ Do you have pre-computed embeddings?
│   ├─ Yes → Do you want auto-sync from Delta?
│   │         ├─ Yes → Delta Sync (Self-Managed)
│   │         └─ No  → Direct Access
│   │
│   └─ No → Delta Sync (Managed Embeddings)
│
└─ Do you need real-time updates (<1 sec)?
    ├─ Yes → Direct Access
    └─ No  → Delta Sync (any type)

Endpoint Selection

After choosing index type, choose endpoint:

Vector Search Modes

Databricks Vector Search supports three search modes: ANN (semantic, default), HYBRID (semantic + keyword), and FULL_TEXT (keyword only, beta). ANN and HYBRID work with Delta Sync and Direct Access indexes.

Semantic Search (ANN)

ANN (Approximate Nearest Neighbor) is the default search mode. It finds documents by vector similarity — matching the meaning of your query against stored embeddings.

When to use

• Conceptual or meaning-based queries ("How do I handle errors in my pipeline?")
• Paraphrased input where exact terms may not appear in the documents
• Multilingual scenarios where query and document languages may differ
• General-purpose RAG retrieval

Example

# ANN is the default — no query_type parameter needed
results = w.vector_search_indexes.query_index(
    index_name="catalog.schema.my_index",
    columns=["id", "content"],
    query_text="How do I handle errors in my pipeline?",
    num_results=5
)

Hybrid Search

Hybrid search combines vector similarity (ANN) with BM25 keyword scoring. It retrieves documents that are both semantically similar and contain matching keywords, then merges the results.

When to use

• Queries containing exact terms that must appear: SKUs, product codes, error codes, acronyms
• Proper nouns — company names, people, specific technologies
• Technical documentation where terminology precision matters
• Mixed-intent queries combining concepts with specific terms

Example

results = w.vector_search_indexes.query_index(
    index_name="catalog.schema.my_index",
    columns=["id", "content"],
    query_text="SPARK-12345 executor memory error",
    query_type="HYBRID",
    num_results=10
)

Decision Guide

Start with ANN. Switch to HYBRID if you notice relevant documents being missed because they don't share vocabulary with the query.

Combining Search Modes with Filters

Both search modes support filters. The filter syntax depends on your endpoint type:

• Standard endpoints → filters as dict (or filters_json as JSON string via databricks-sdk)
• Storage-Optimized endpoints → filters as SQL-like string (via databricks-vectorsearch package)

Standard endpoint with hybrid search

results = w.vector_search_indexes.query_index(
    index_name="catalog.schema.my_index",
    columns=["id", "content", "category"],
    query_text="SPARK-12345 executor memory error",
    query_type="HYBRID",
    num_results=10,
    filters_json='{"category": "troubleshooting", "status": ["open", "in_progress"]}'
)

Storage-Optimized endpoint with hybrid search

from databricks.vector_search.client import VectorSearchClient

vsc = VectorSearchClient()
index = vsc.get_index(endpoint_name="my-storage-endpoint", index_name="catalog.schema.my_index")

results = index.similarity_search(
    query_text="SPARK-12345 executor memory error",
    columns=["id", "content", "category"],
    query_type="hybrid",
    num_results=10,
    filters="category = 'troubleshooting' AND status IN ('open', 'in_progress')"
)

Using with Pre-Computed Embeddings

If you compute embeddings yourself, use query_vector instead of query_text for ANN search:

# ANN with pre-computed embedding (default)
results = w.vector_search_indexes.query_index(
    index_name="catalog.schema.my_index",
    columns=["id", "content"],
    query_vector=[0.1, 0.2, 0.3, ...],  # Your embedding vector
    num_results=10
)

For hybrid search with self-managed embeddings (indexes without an associated model endpoint), you must provide both query_vector and query_text. The vector is used for the ANN component and the text for the BM25 keyword component:

# HYBRID with self-managed embeddings — requires both vector AND text
results = w.vector_search_indexes.query_index(
    index_name="catalog.schema.my_index",
    columns=["id", "content"],
    query_vector=[0.1, 0.2, 0.3, ...],  # For ANN similarity
    query_text="executor memory error",   # For BM25 keyword matching
    query_type="HYBRID",
    num_results=10
)

Notes:

• For ANN queries: provide either query_text or query_vector, not both.
• For HYBRID queries on managed embedding indexes: provide only query_text (the system handles both components).
• For HYBRID queries on self-managed indexes without a model endpoint: provide both query_vector and query_text.
• When using query_text alone, the index must have an associated embedding model (managed embeddings or embedding_model_endpoint_name on a Direct Access index).

Parameter Reference

Vector Search Troubleshooting & Operations

Operational guidance for monitoring, cost optimization, capacity planning, and migration of Databricks Vector Search resources.

Monitoring Endpoint Status

Use databricks vector-search-endpoints get-endpoint ENDPOINT_NAME (CLI) or w.vector_search_endpoints.get_endpoint() (SDK) to check endpoint health.

Endpoint fields

Example

endpoint = w.vector_search_endpoints.get_endpoint(endpoint_name="my-endpoint")
print(f"State: {endpoint.endpoint_status.state.value}")
print(f"Indexes: {endpoint.num_indexes}")

What to do per state:

• PROVISIONING → Wait. Endpoint creation is asynchronous and can take several minutes.
• ONLINE → Ready to serve queries and host indexes.
• OFFLINE → Check the message field for error details. May require recreation.
• YELLOW_STATE → Endpoint is degraded but still serving. Investigate the message field.
• RED_STATE → Endpoint is unhealthy. Check message for details; may need support intervention.

Monitoring Index Status

Use databricks vector-search-indexes get-index INDEX_NAME (CLI) or w.vector_search_indexes.get_index() (SDK) to check index health.

Index fields

Example

index = w.vector_search_indexes.get_index(index_name="catalog.schema.my_index")
if index.status.ready:
    print("Index is ONLINE")
else:
    print(f"Index is NOT_READY: {index.status.message}")

Pipeline Type Trade-offs

Delta Sync indexes use a DLT pipeline to sync data from the source Delta table. The pipeline type determines sync behavior:

Triggering a sync

# For TRIGGERED pipelines only
w.vector_search_indexes.sync_index(index_name="catalog.schema.my_index")
# Check sync progress with get_index()

Tip: CONTINUOUS pipelines cannot be synced manually — they sync automatically. Calling sync_index() on a CONTINUOUS index will raise an error.

Cost Optimization

Endpoint type selection

Recommendation: Start with Storage-Optimized unless you need sub-100ms latency. It handles most RAG workloads well.

Reducing storage costs

• Use columns_to_sync to limit which columns are synced to the index. Only synced columns are available in query results, so include only what you need.
• Choose TRIGGERED pipelines for batch workloads to avoid continuous compute costs.

# Only sync the columns you actually need in query results
delta_sync_index_spec={
    "source_table": "catalog.schema.documents",
    "embedding_source_columns": [
        {"name": "content", "embedding_model_endpoint_name": "databricks-gte-large-en"}
    ],
    "pipeline_type": "TRIGGERED",
    "columns_to_sync": ["id", "content", "title"]  # Exclude large unused columns
}

Capacity Planning

Estimating needs:

• One document typically maps to one vector (or multiple if chunked)
• If chunking at ~512 tokens, expect 2-5 vectors per page of text
• Monitor num_indexes on your endpoint to understand utilization

Migration Patterns

Changing endpoint type

Endpoints are immutable after creation — you cannot change the type (Standard ↔ Storage-Optimized) of an existing endpoint. To migrate:

1. Create a new endpoint with the desired type 2. Recreate indexes on the new endpoint pointing to the same source tables 3. Wait for sync to complete (check index state) 4. Update applications to query the new index names 5. Delete old indexes, then delete the old endpoint

# Step 1: Create new endpoint
w.vector_search_endpoints.create_endpoint(
    name="my-endpoint-storage-optimized",
    endpoint_type="STORAGE_OPTIMIZED"
)

# Step 2: Recreate index on new endpoint (same source table)
w.vector_search_indexes.create_index(
    name="catalog.schema.my_index_v2",
    endpoint_name="my-endpoint-storage-optimized",
    primary_key="id",
    index_type="DELTA_SYNC",
    delta_sync_index_spec={
        "source_table": "catalog.schema.documents",
        "embedding_source_columns": [
            {"name": "content", "embedding_model_endpoint_name": "databricks-gte-large-en"}
        ],
        "pipeline_type": "TRIGGERED"
    }
)

# Step 3: Trigger sync and wait for ONLINE state
w.vector_search_indexes.sync_index(index_name="catalog.schema.my_index_v2")

# Step 4: Update your application to use "catalog.schema.my_index_v2"
# Step 5: Clean up old resources
w.vector_search_indexes.delete_index(index_name="catalog.schema.my_index")
w.vector_search_endpoints.delete_endpoint(endpoint_name="my-endpoint")

Expanded Troubleshooting