Spring AI 2.0 RAG Missed 94 of 100 Filtered Searches — pgvector Had the Docs

HobbiesPark

9 min read

·

Oct 26, 2026

--

Listen

Share

Press enter or click to view image in full size

A docs assistant held 200 chunks for one library in pgvector, a user asked a question those docs answer, and Spring AI 2.0.1 sent the model zero documents plus an instruction to apologize. Across 100 libraries, 94 searches ended in the same polite refusal. No exception, no WARN, HTTP 200. Three more defaults hide behind that answer: the inner product distance never uses its index, re-ingesting a page adds a second copy, and an empty search reads exactly like missing knowledge. When your RAG says it can’t answer, count what retrieval returned before you blame the model or the documents.

This article was written with help from an AI tool.

• With a metadata filter, the default HNSW index hands PostgreSQL its 40 nearest candidates (hnsw.ef_search is 40) and the filter runs afterwards, so a value that matches 1% of the rows usually gets 0 of its 4 documents.
• NEGATIVE_INNER_PRODUCT , which the Spring AI docs suggest for normalized embeddings, orders by 1 + (embedding <#> ?) , so its vector_ip_ops index is never used and every query reads the whole table.
• TokenTextSplitter gives every chunk a new random ID, even when the page has a fixed ID and fits in one chunk, so ON CONFLICT (id) never matches and each sync appends a full copy.
• When retrieval returns nothing, RetrievalAugmentationAdvisor replaces the user's question with a fixed instruction to decline, and logs that only at DEBUG.

All of it ran on Spring AI 2.0.1 defaults: HNSW and cosine distance in PgVectorStore , topK 4 and threshold 0.0 in SearchRequest . The corpus is synthetic documentation for 100 HTTP client libraries, 200 chunks each, embedded with all-MiniLM-L6-v2 through Spring AI's ONNX model and normalized like OpenAI embeddings. The plans are what PostgreSQL 16.15 logged for Spring AI's own JDBC calls.

Rows Removed by Filter: 40 — the metadata filter runs after HNSW stops looking

Every search asked how to set the connection timeout, which each library’s docs cover in 20 passages. For a plain SearchRequest filtered on library == 'lib-037' , PostgreSQL logged this plan (costs, timings and the query vector trimmed):

Query Text: SELECT *, embedding <=> $1 AS distance FROM public.vector_store WHERE embedding <=> $2 < $3  AND metadata::jsonb @@ '$."library" == "lib-037"'::jsonpath  ORDER BY distance LIMIT $4
Limit  (actual rows=0 loops=1)
  ->  Index Scan using spring_ai_vector_index on vector_store  (actual rows=0 loops=1)
        Order By: (embedding <=> '[...]'::vector)
        Filter: (((embedding <=> '[...]'::vector) < '1'::double precision) AND ((metadata)::jsonb @@ '($."library" == "lib-037")'::jsonpath))
        Rows Removed by Filter: 40

HNSW is an approximate index. To serve ORDER BY distance LIMIT 4 it returns the 40 nearest candidates it found, because hnsw.ef_search defaults to 40. The library condition is not in the index, so PostgreSQL checks it on those 40 afterwards. All 40 came from other libraries, whose docs answer the same question in nearly the same words. lib-037 had 200 chunks and got none.

The same question, asked once for each library under three settings:

defaults                               0 docs:  94 libraries | 1-3 docs:   1 | 4 docs:   5
hnsw.iterative_scan = relaxed_order    0 docs:   0 libraries | 1-3 docs:   0 | 4 docs: 100
enable_indexscan = off                 0 docs:   0 libraries | 1-3 docs:   0 | 4 docs: 100

The last line is exact search with index scans off: every library had four good matches. The five lucky ones had passages among the 40 nearest in the whole table, and each library you add shrinks everyone’s share of those 40.

pgvector’s README documents this: at the default ef_search , a condition true for 10% of rows yields about four matches, so a value that matches 1% averages 0.4. Since 0.8.0, hnsw.iterative_scan keeps scanning until enough rows pass, but it is off by default. Spring AI 2.0.1 doesn't set it, has no property for it, and its pgvector page sells metadata filtering on efficiency without saying what an approximate index does to a filter.

On Spring AI 2.0.1 with the default HNSW index, a filtered search sees at most 40 candidates before the filter runs, however many matching rows the table holds.

Seq Scan on vector_store: the inner product distance never touches its index

Spring AI’s pgvector reference says normalized vectors can use NEGATIVE_INNER_PRODUCT "for best performance", and OpenAI's embeddings are normalized. On these 20,000 chunks it returned the same four IDs as cosine, and schema initialization even builds a matching vector_ip_ops index. Every search then logs this, next to cosine on the same data (costs and vector trimmed):

LOG:  duration: 19.686 ms  plan:
  Query Text: SELECT *, (1 + (embedding <#> $1)) AS distance FROM public.vector_store WHERE (1 + (embedding <#> $2)) < $3  ORDER BY distance LIMIT $4
  Limit  (actual time=19.517..19.519 rows=4 loops=1)
    ->  Sort  (actual time=19.515..19.517 rows=4 loops=1)
          Sort Key: (('1'::double precision + (embedding <#> '[...]'::vector)))
          ->  Seq Scan on vector_store  (actual time=0.038..16.069 rows=17208 loops=1)
                Rows Removed by Filter: 2792
LOG:  duration: 0.524 ms  plan:
  Query Text: SELECT *, embedding <=> $1 AS distance FROM public.vector_store WHERE embedding <=> $2 < $3  ORDER BY distance LIMIT $4
  Limit  (actual time=0.386..0.392 rows=4 loops=1)
    ->  Index Scan using spring_ai_vector_index on vector_store  (actual time=0.385..0.390 rows=4 loops=1)

pgvector uses an HNSW index only when the query orders by the bare operator, as in ORDER BY embedding <#> ? . Spring AI's template orders by 1 + (embedding <#> ?) , which the index can't produce in order, so the planner reads every row and sorts. The scan kept 17,208 rows and removed 2,792 with the distance check: all 20,000. Results stay correct, since adding 1 keeps the order. The cost grows with the table: about 20 ms at 20,000 rows, against under 1 ms for cosine.

This one has a production report. Spring AI issue 5713, filed against 1.1.3 on PostgreSQL 16, shows the query at 11.3 seconds on a full table scan, against about 1.1 seconds ordered by the bare operator. Pull request 5723 proposes moving the arithmetic into the row mapper. As of October 13, the template is unchanged in 2.0.1 and on the 2.0.x and main branches.

In Spring AI 2.0.1, NEGATIVE_INNER_PRODUCT returns the same rows as cosine and reads every row in the table to find them.

ON CONFLICT (id) DO UPDATE: a nightly re-sync that never updates anything

A nightly sync reads one library’s documentation pages, each with a stable ID derived from its path, and calls vectorStore.add(splitter.apply(pages)) . Spring AI inserts with ON CONFLICT (id) DO UPDATE , so re-running looks safe. A week of syncs on that library's 11 pages:

night 1: 11 pages, rows in table 11
night 2: 11 pages, rows in table 22
night 7: 11 pages, rows in table 77
   score=0.6644  page=3  id=e0114e82-13b5-4f3f-be81-85eb70c30526
   score=0.6644  page=3  id=077bebce-cdab-4022-ba41-8fb35d7c4db5
   score=0.6644  page=3  id=e7168d75-3f4b-4315-806e-f365272697e1
   score=0.6644  page=3  id=e88962ae-23f9-4cee-9acf-d5d8d7609599
top 4, distinct texts 1
page id fbeccb36-0c23-3cef-9b74-051994651d58 -> chunk ids f95c6047-de29-40a0-ac52-3e771846c48a / 488b1fa5-c8cd-452e-8622-14632e7a6981

The last line is one page split twice. TextSplitter builds each chunk as a new Document with a random UUID and keeps the page's ID only as parent_document_id in the metadata. It does this even when a page fits in one chunk, as all 11 did, and then its only log line, an INFO about splitting into several chunks, never prints. ON CONFLICT (id) never matches, every sync appends a full copy, and after seven nights the top four were the timeout page four times. The document joiner in RetrievalAugmentationAdvisor drops repeats by ID, and these IDs all differ.

Spring AI issue 1167, filed in August 2024, asks splitters to keep the input ID. The maintainers’ answer is that chunks are new documents and the parent ID belongs in metadata, which is how 2.0.1 behaves.

In Spring AI 2.0.1, a document that goes through TokenTextSplitter never keeps its ID, so the pgvector upsert inserts a new row for every chunk on every run.

The user query is outside your knowledge base: zero documents become a polite answer

Put the empty search from the first section behind RetrievalAugmentationAdvisor , with the library filter on its VectorStoreDocumentRetriever . A stub chat model records the user message it receives:

== RetrievalAugmentationAdvisor, documents retrieved: 0
-- user message the model received:
The user query is outside your knowledge base.
Politely inform the user that you can't answer it.

The default augmenter, ContextualQueryAugmenter , doesn't allow an empty context. With zero documents it doesn't annotate the question. It replaces it. The model never sees what the user asked, follows the instruction to apologize, and the reply goes back with HTTP 200 and one DEBUG line. QuestionAnswerAdvisor keeps the question but sends an empty context and a prompt that says to decline when the answer isn't there, which ends the same way.

Spring AI’s RAG reference documents this default and offers allowEmptyContext(true) , which only trades the refusal for an answer from the model's own memory, for a library whose answer was in the table all along.

With Spring AI 2.0.1 defaults, a search that returned 0 documents and a knowledge base that has no answer produce the same sentence for the user.

The fix: count what retrieval returned, then change the defaults

See the plans Spring AI actually runs. On a staging database, have PostgreSQL log every plan for your application’s role:

--sql
ALTER ROLE app SET session_preload_libraries = 'auto_explain';
ALTER ROLE app SET auto_explain.log_min_duration = 0;
ALTER ROLE app SET auto_explain.log_analyze = on;

Ask a filtered question on a new connection and search the server log for Rows Removed by Filter under spring_ai_vector_index , and for Seq Scan on vector_store . Reset the settings afterwards, since log_analyze slows every statement.

Let filtered scans keep going. Turn on iterative scans for every connection the pool opens:

#properties
spring.datasource.hikari.connection-init-sql=SET hnsw.iterative_scan = relaxed_order

ALTER ROLE app SET hnsw.iterative_scan = relaxed_order; does the same server-side, and either one brought all 100 libraries back to four documents. Relaxed order may return rows slightly out of order, which the RetrievalAugmentationAdvisor joiner re-sorts by score. Scans stop at hnsw.max_scan_tuples (20,000 by default), and lib-037 needed about 230 rows. When one filter value is a tiny share of a huge table, pgvector suggests a partial index or partitioning by that value.

Stay on cosine, and rebuild the index if you switched. For unit-length embeddings, cosine ranks the same as inner product. Setting spring.ai.vectorstore.pgvector.distance-type back to COSINE_DISTANCE isn't enough on an existing table: schema initialization runs CREATE INDEX IF NOT EXISTS , the vector_ip_ops index stays, and the cosine query falls back to a sequential scan too. Swap the index without a gap:

--sql
CREATE INDEX CONCURRENTLY vector_store_cosine_idx ON vector_store USING hnsw (embedding vector_cosine_ops);
DROP INDEX CONCURRENTLY spring_ai_vector_index;
ALTER INDEX vector_store_cosine_idx RENAME TO spring_ai_vector_index;

Replace a source’s chunks instead of appending them. Put a source key in each page’s metadata, which the splitter copies to every chunk, and delete by it in the same transaction as the add:

//java
TokenTextSplitter splitter = TokenTextSplitter.builder().build();
transactionTemplate.executeWithoutResult(status -> {
    vectorStore.delete(new FilterExpressionBuilder().eq("source", "docs/lib-037").build());
    vectorStore.add(splitter.apply(pages));
});

Both calls go through the same JdbcTemplate , so readers see the old chunks until the new ones commit. In the reproduction, the table stayed at 11 rows for seven nights, with four different pages on top.

Record retrieval next to every answer. Both advisors put the documents they used on the response metadata:

//java
ChatResponse response = chatClient.prompt(question).advisors(rag).call().chatResponse();
List<Document> docs = response.getMetadata().get(RetrievalAugmentationAdvisor.DOCUMENT_CONTEXT);
long distinct = docs.stream().map(Document::getText).distinct().count();
if (docs.isEmpty()) {
    log.warn("RAG retrieved 0 documents for library {}", libraryId);
}

Export both counts per filter value. A rising share of zero-document answers points at the index, and distinct below docs.size() points at ingestion. QuestionAnswerAdvisor uses the key QuestionAnswerAdvisor.RETRIEVED_DOCUMENTS instead.

When a RAG pipeline says it can’t answer, that sentence reports on retrieval first. Until you log how many documents stood behind each answer, you can’t tell a knowledge base that lacks the answer from a search that never reached it.

Versions tested against

Read and run in Spring AI 2.0.1 with Spring Boot 4.1.1 dependency management, PostgreSQL 16.15 and pgvector 0.8.7. The 2.0.x and main branches have the same templates and defaults as of October 13, and 2.0.2 is not out. The corpus is synthetic, so your counts will differ; the mechanisms won’t. The inner product scan is an observed incident (issue 5713). The filter behavior is documented by pgvector, the empty-context prompt by Spring AI, and the chunk IDs in issue 1167; their numbers come from the reproduction, not from an incident I am reporting.

More in this series

From the Spring Boot 4 Production Traps series:

I Streamed a Tool Call on Spring AI 2.0 — Half the Cached Tokens Disappeared
https://medium.com/codetodeploy/spring-ai-streaming-cached-tokens-lost-ae9c0a19b973

My Spring AI Tool Never Fired. There Was No Error to Read
https://medium.com/codetodeploy/spring-ai-tool-annotation-not-firing-5-silent-failures-548afc021365

If this saved you a support ticket

Clap so it reaches the next team whose RAG keeps apologizing, follow for the rest of the series, repost it to whoever owns your ingestion job, and subscribe to get the next one by email.

Sources

pgvector README, sections Filtering and Iterative Index Scans
https://github.com/pgvector/pgvector

Spring AI reference, PGvector
https://docs.spring.io/spring-ai/reference/api/vectordbs/pgvector.html

Spring AI reference, Retrieval Augmented Generation
https://docs.spring.io/spring-ai/reference/api/retrieval-augmented-generation.html

Spring AI issue 5713, PGVectorStore: Index not used for negative inner product
https://github.com/spring-projects/spring-ai/issues/5713

Spring AI pull request 5723, Make SQL Statement for negative inner product index friendly
https://github.com/spring-projects/spring-ai/pull/5723

Spring AI issue 1167, TokenTextSplitter creates document with different id than it’s input document
https://github.com/spring-projects/spring-ai/issues/1167