Build a Delta Sync Index on Databricks AI Search for Your RAG App, Step by Step

Kamal Thapa

8 min read

·

3 days ago

--

Listen

Share

From a chunked Delta table to hybrid search with filters, kept fresh automatically

A few days ago I wrote RAG in Plain English, an explainer on how retrieval-augmented generation lets an AI answer from your own data. I ended that post with a promise: a hands-on walkthrough of building a Delta Sync Index on Databricks. This is that post.

Quick note on names first. The product used to be called Databricks Vector Search. It’s now Databricks AI Search. You’ll still see “vector search” in REST API paths, the CLI, Terraform resources, and the vector_search() SQL function. That's expected, and those are still the current interfaces.

Here’s what we’ll build:

1. A chunked Delta table with a primary key and Change Data Feed turned on
2. An AI Search endpoint
3. A Delta Sync Index where Databricks computes the embeddings for us
4. Queries using similarity search, hybrid search, and filters
5. A simple way to keep the index fresh

Then I’ll share the lessons I’ve learned along the way.

The big picture

Press enter or click to view image in full size

The idea is simple. Your pipelines already keep Delta tables up to date. A Delta Sync Index watches one of those tables and keeps a search index in step with it. When a row is inserted, updated, or deleted, the index follows. Everything lives in Unity Catalog, so the index is governed like any other table.

Step 0: Prerequisites

Before you start, check these with your workspace admin:

• A Unity Catalog-enabled workspace.
• Serverless compute enabled. AI Search requires it.
• CREATE TABLE privilege on the schema where the index will live.
• Permission to create an AI Search endpoint. Endpoint permissions are managed with access control lists.
• Access to an embedding model endpoint. I’ll use the pay-per-token Foundation Model API endpoint databricks-gte-large-en.

I’ll use a catalog called rag_demo and a schema called knowledge. Swap in your own names.

Step 1: Prepare a chunked source table

I assume you already have a table of parsed documents, rag_demo.knowledge.docs_parsed, with one row per document: doc_id, title, department, source_url, updated_at, and the full text in body.

First, create the chunk table. Two things matter here. Every row needs a stable, unique primary key, and the table needs Change Data Feed so the index can pick up only what changed.

CREATE TABLE IF NOT EXISTS rag_demo.knowledge.docs_chunks (
  chunk_id    STRING NOT NULL,
  doc_id      STRING,
  chunk_no    INT,
  title       STRING,
  department  STRING,
  source_url  STRING,
  updated_at  TIMESTAMP,
  chunk_text  STRING
)
TBLPROPERTIES (delta.enableChangeDataFeed = true);

If the table already exists, turn Change Data Feed on like this:

ALTER TABLE rag_demo.knowledge.docs_chunks
SET TBLPROPERTIES (delta.enableChangeDataFeed = true);

Databricks docs also note that tables with row tracking enabled get a change data feed automatically, without this setting. Either way, the index needs one.

Now split each document into chunks. This deliberately simple version packs paragraphs into chunks of about 1,500 characters. Real projects often split on headings instead, but the shape is the same.

from pyspark.sql import functions as F
from pyspark.sql.types import ArrayType, StringType
@F.udf(ArrayType(StringType()))
def split_into_chunks(text):
    chunks, current = [], ""
    for para in (text or "").split("\n\n"):
        if current and len(current) + len(para) > 1500:
            chunks.append(current.strip())
            current = ""
        current += para + "\n\n"
    if current.strip():
        chunks.append(current.strip())
    return chunks
chunks = (
    spark.table("rag_demo.knowledge.docs_parsed")
    .select("doc_id", "title", "department", "source_url", "updated_at",
            F.posexplode(split_into_chunks("body")).alias("chunk_no", "chunk_text"))
    .withColumn("chunk_id", F.concat_ws("-", "doc_id", "chunk_no"))
)

The chunk_id combines the document ID and the chunk's position, so it stays the same from run to run as long as the document doesn't change.

Next, write the chunks with a MERGE instead of an overwrite. I'll explain why in the lessons section.

from delta.tables import DeltaTable
target = DeltaTable.forName(spark, "rag_demo.knowledge.docs_chunks")
(target.alias("t")
    .merge(chunks.alias("s"), "t.chunk_id = s.chunk_id")
    .whenMatchedUpdateAll(condition="t.chunk_text <> s.chunk_text OR t.updated_at <> s.updated_at")
    .whenNotMatchedInsertAll()
    .whenNotMatchedBySourceDelete()
    .execute())

Rows that didn’t change are left alone. New chunks are inserted. Chunks that no longer exist in the source are deleted, which also removes them from the index on the next sync. (That last clause assumes docs_parsed holds the full current set of documents. If it only holds a daily batch, drop it and handle deletes separately.)

Step 2: Install the SDK and create an endpoint

The current Python package is databricks-ai-search, and the client class is AISearchClient. Older tutorials use databricks-vectorsearch and VectorSearchClient. The new package keeps VectorSearchClient as an alias, but I'd write new code with the new names.

In a notebook:

%pip install databricks-ai-search
dbutils.library.restartPython()

Then create a Standard endpoint, which serves the index:

from databricks.ai_search.client import AISearchClient
client = AISearchClient()  # picks up notebook credentials automatically
ENDPOINT = "rag-endpoint"
if not client.endpoint_exists(ENDPOINT):
    client.create_endpoint_and_wait(name=ENDPOINT, endpoint_type="STANDARD")

There’s also a STORAGE_OPTIMIZED endpoint type built for much larger indexes. It works a bit differently: it only supports triggered sync, and its filters use a SQL-like string instead of a dictionary. For a first RAG app, Standard is the easy place to start.

Step 3: Create the Delta Sync Index with managed embeddings

Now the main event. We point the index at the chunk table, tell it which column holds the text, and name the embedding model endpoint. Databricks computes the embeddings for every row and for every query.

INDEX = "rag_demo.knowledge.docs_chunks_index"
index = client.create_delta_sync_index_and_wait(
    endpoint_name=ENDPOINT,
    source_table_name="rag_demo.knowledge.docs_chunks",
    index_name=INDEX,
    pipeline_type="TRIGGERED",
    primary_key="chunk_id",
    embedding_source_column="chunk_text",
    embedding_model_endpoint_name="databricks-gte-large-en",
    columns_to_sync=["doc_id", "title", "department", "source_url", "updated_at", "chunk_text"],
)

A few notes on those parameters:

• pipeline_type is either "TRIGGERED" or "CONTINUOUS". More on that in a moment.
• columns_to_sync controls which columns land in the index. The primary key and embedding columns are always included. Only synced columns can be returned in results or used as filters, so include your metadata.
• create_delta_sync_index_and_wait blocks until the index is ready. Use create_delta_sync_index if you don't want to wait.

A note on the embedding model: for production on Standard endpoints, the docs currently recommend databricks-qwen3-embedding-0-6b on a provisioned throughput endpoint. At the time of writing, that model is in Public Preview, so check its status before you commit. Whichever you choose, pick it deliberately. The model is part of the index definition, so changing it means building a new index.

You can also do this in the UI: open the source table in Catalog Explorer, click Create, and choose the vector search index option. I prefer code because it goes into source control.

Triggered vs. continuous sync

This choice is about freshness versus cost.

• Triggered: the index updates only when you ask it to. On Standard endpoints, each sync is incremental, so only changed rows are processed. This is the right default for most document collections that change daily or hourly.
• Continuous: the index stays in sync with seconds of latency. The trade-off is cost, because a compute cluster runs the streaming sync pipeline the whole time.

My rule: start with triggered. Move to continuous only when a real person needs answers that reflect changes within seconds, and the budget owner agrees.

Step 4: Query the index

Let’s ask a question. A plain similarity search (approximate nearest neighbor, the default) looks like this:

index = client.get_index(index_name=INDEX)
results = index.similarity_search(
    query_text="How much can I expense for a home office setup?",
    columns=["chunk_id", "title", "source_url", "chunk_text"],
    num_results=5,
)
for row in results["result"]["data_array"]:
    print(row)

Each row comes back in the same order as columns, with a similarity score added at the end.

For RAG, I usually start with hybrid search. It runs vector search and keyword search together and merges the results. That matters when people type product codes, policy numbers, or acronyms that embeddings don’t capture well.

results = index.similarity_search(
    query_text="Policy HR-204 home office stipend",
    columns=["chunk_id", "title", "source_url", "chunk_text"],
    num_results=5,
    query_type="HYBRID",
    filters={"department": ["HR", "All Employees"]},
)

That filters argument is how you scope results. On Standard endpoints it's a dictionary. A list of values means "match any of these," and operators go in the key, such as {"department NOT": "Legal"}. On Storage-Optimized endpoints, you write a SQL-like string instead, such as "department IN ('HR', 'All Employees')".

If your analysts live in SQL, there’s also a vector_search() function (in Public Preview at the time of writing):

SELECT * FROM vector_search(
  index      => 'rag_demo.knowledge.docs_chunks_index',
  query_text => 'home office stipend',
  query_type => 'HYBRID',
  num_results => 5
);

It doesn’t accept filters during the preview and isn’t available on classic SQL warehouses, so I use it for exploration, not as the app’s main path.

Step 5: Keep it fresh

With a triggered index, the simplest pattern is to sync right after your chunking job finishes. Add one more task to the same Lakeflow Job:

from databricks.ai_search.client import AISearchClient
client = AISearchClient()
index = client.get_index(index_name="rag_demo.knowledge.docs_chunks_index")
index.sync()

Now the flow is: documents land, the chunking task merges changes into docs_chunks, and the sync task pushes only those changes into the index. You can also click Sync now on the index's Overview tab in Catalog Explorer when you're testing.

Note that sync() only applies to triggered indexes. Continuous indexes update on their own.

Lessons and gotchas from the field

1. Merge, don’t overwrite. If your chunking job overwrites the whole table every night, every row shows up as changed, and the sync has to process and re-embed all of them. A MERGE that only touches changed rows keeps syncs small and fast.

2. Treat the primary key like a contract. If chunk IDs shift every run, for example because you numbered rows globally, the index sees a flood of deletes and inserts. Build IDs from stable parts like document ID plus position.

3. Schema changes mean a new index. The index schema is fixed when you create it. To add or change a column, create a new index on the updated table, switch your app over when it’s ready, then delete the old one. Plan the metadata columns you’ll want to filter on before day one.

4. Don’t name a column _id. It's reserved. Rename it before you create the index.

5. Warm up custom embedding endpoints. If you serve your own embedding model, Databricks recommends turning off scale-to-zero. A cold endpoint can cause the first query to time out. The docs also suggest GPU endpoints for larger datasets, because index initialization can time out on undersized CPU endpoints.

6. Security goes in the filter, not the prompt. Store who can see each chunk as metadata, and filter on it at query time. The model should never receive a chunk the user isn’t allowed to read. Also remember that querying requires USE CATALOG, USE SCHEMA, and SELECT on the index, and production apps should use a service principal, not a personal token.

7. Test retrieval before you blame the model. Keep a small set of real questions with the chunks that should come back. Rerun it every time you change chunk size, the embedding model, or the query type. If quality still isn’t there, try the built-in reranker before rebuilding everything.

Wrapping up

A Delta Sync Index turns RAG freshness into a data engineering problem your team already knows how to solve. Keep a clean chunk table with a stable key and Change Data Feed, point an index at it, and let your existing pipelines do the rest.

If you build this, I’d love to hear how it goes. What chunking strategy worked for your documents? Did you go triggered or continuous? Leave a comment and tell me what you’d like me to cover next.