Cutting Embeddings to a Quarter

Atharva Gaikwad

6 min read

·

14 hours ago

--

Listen

Share

How Matryoshka representations made my search system 4× cheaper to store, and why I expect relevance to hold up

Embedding dimensions feel like a detail when you start building semantic search. You pick a model, use the size it gives you, and move on.

Then the corpus grows.

Every document carries a vector. Every vector takes memory. And in most vector databases, memory is what you pay for.

I ran into this while building a procurement search system: a tender search platform that indexes government tender notices from many countries. My test set holds about 88,000 notices. The live system will take in 85,000–100,000 new notices a day, so it will reach millions of documents quickly.

At that scale, the size of each vector stops being a detail and becomes a big part of the hosting bill.

This is how I used Matryoshka Representation Learning (MRL) to cut my embeddings from 4,096 dimensions to 1,024, why the architecture around it matters as much as the technique, and how I’m checking that it holds up.

The problem: documents × dimensions

A dense embedding is a list of floating-point numbers. Stored as float32, each one takes 4 bytes. So the raw size of a vector index is roughly:

documents × dimensions × 4 byte

It grows on both axes. Double the corpus and it doubles. Double the dimensions and it doubles again.

My embedding model, Qwen3-Embedding-8B, outputs 4,096-dimensional vectors by default. That’s a rich representation, but a heavy one: 16 KB of vector data per tender notice, before any index overhead.

For a few thousand documents, nobody notices. For millions, those 16 KB per document set the size of machine you need.

What MRL actually is

Matryoshka Representation Learning was introduced by Kusupati et al. in the 2022 paper Matryoshka Representation Learning (NeurIPS 2022). The name comes from Russian nesting dolls, and the image fits.

A normal embedding model spreads information across all its dimensions in no particular order. Chop off half the vector and you lose information at random. Quality usually drops badly.

An MRL-trained model is trained differently. The loss is calculated not only on the full vector but also on several prefixes of it: the first 64, 128, 256, 512 dimensions and so on. The model learns to pack the most important information into the earliest dimensions.

The first N values of the vector are already a usable embedding on their own.

Full vector: 4,096 dims

┌─────┬─────┬─────────┬──────────────────────────────────┐

│ 256 │ 512 │  1,024  │              4,096               │

└─────┴────┴─────────┴──────────────────────────────────┘

◄256►

◄───512───►

◄──────1,024───────►   ← what I store

◄──────────────────── 4,096 ───────────────────────────►

Each prefix is a smaller, self-contained embedding nested inside the larger one.

To use a smaller size, you keep the first N values and re-normalize the vector to unit length, so cosine similarity and dot product still behave correctly. Many embedding APIs do this for you through a dimensions parameter. The API I use returns the first 1,024 values, re-normalized.

Two caveats:

• It only works for models trained this way. Truncate a model that wasn’t trained with MRL and you lose information unpredictably. Check the model card first.
• It isn’t the only way to shrink vectors. PCA reduces dimensions after the fact, but it’s an extra fitted step to maintain as your data changes, and it isn’t free in quality either. Quantization (int8 or binary instead of float32) cuts the bytes per dimension rather than the number of dimensions. Quantization and MRL work on different axes, so you can combine them.

The paper reports that MRL embeddings can be much smaller than full-size ones while staying close in accuracy on the tasks it tested. How close depends on the model, the task and how far you cut. That’s why I treat it as something to measure, not assume.

How I used it

The change was a single API parameter: I request 1,024 dimensions instead of 4,096. No retraining. No extra pipeline step. No PCA model to keep updated.

Here is what that does to raw float32 storage:

Corpus size          4,096 dims    1,024 dims    Saving

───────────────────────────────────────────────────────

88K tenders (test)   ~1.45 GB      ~362 MB       4×

1M tenders           ~16 GB        ~4 GB         4×

5M tenders           ~82 GB        ~20 GB        4×

These are raw vector sizes. Real indexes such as HNSW add graph overhead on top, but the ratio still points the same way. Because vector database pricing is driven largely by memory, a 4× smaller footprint means a smaller hosting tier. Shorter vectors also make each distance calculation cheaper, so search gets faster too.

Why the quality risk is smaller in my design

This is the part that matters most: in my system, the embedding doesn’t decide the final ranking.

The pipeline is hybrid and filter-first:

1. Hard filters run first: country, publication dates, contract value, notice type.
2. Keyword search (BM25) and vector search run inside the filtered set, and their results are blended.
3. A reranker re-scores the top 50 candidates, reading the full text of each notice, and decides the final order.

That changes what truncation can damage. The vectors’ job is to get relevant tenders into the top 50, not to order them. So the realistic risk of shorter vectors is a relevant notice falling just outside the candidate set, not a worse ranking on the page the user sees.

The keyword half helps too. Tender searches are full of exact terms: CPV codes, product names, specific services. BM25 catches those regardless of what the vectors do. The vectors earn their place on paraphrases and concept-level matches, and those depend mostly on the early dimensions MRL protects.

The vectors only need to get good candidates into the top 50. The reranker decides the order.

[RESULT: candidate recall@50 at 1,024 vs 4,096 dims]

[RESULT: top-10 relevance at 1,024 vs 4,096 dims]

Does it hold across models?

One model surviving truncation isn’t proof. So I’m also testing a second open embedding model, F2LLM-v2–4B (2,560 dimensions natively), at full size and truncated to 1,024. If the gap is small for both, I can be more confident that 1,024 isn’t just luck with one model.

[RESULT: F2LLM-v2–4B 2,560 vs 1,024 dims — recall@50 and top-10 relevance]

When not to cut aggressively

MRL isn’t free compression. I’d be more careful with:

• Very small dimensions. 4,096 → 1,024 keeps a lot of capacity. Going to 64 or 128 is a different decision. Quality tends to fall faster at the small end, and fine distinctions blur first.
• Models not trained for MRL. If the model card doesn’t mention flexible dimensions, don’t truncate. Use quantization or PCA, and measure.
• Systems without a reranker. If vector similarity is your final ranking, every bit of lost precision shows up directly in front of users.
• Pure semantic search. Without BM25 alongside, nothing catches the exact-match cases shorter vectors might miss.
• Changing your mind later. Vectors at different sizes aren’t comparable, so moving from 1,024 to 2,048 means re-embedding the corpus. Keep full vectors in cheap cold storage and you can re-truncate without calling the model again.

How to evaluate it yourself

Published benchmarks won’t tell you how your corpus behaves. A quick, honest test:

1. Collect real queries. A few dozen to a few hundred, from logs or from people who know the domain. Mix short keyword queries with longer descriptive ones.
2. Run the full pipeline twice, at full and truncated dimensions. Keep filters, BM25 weights and reranker identical.
3. Measure candidate recall. How much of the full-dimension top 50 also appears in the truncated top 50? That’s the step truncation actually touches.
4. Measure final relevance. Grade the top 10 from each run, ideally by people who know the domain, or an LLM judge with a clear rubric you’ve spot-checked against human grades.
5. Read the disagreements. The queries where the runs differ tell you what kind of relevance you’re losing, and that matters more than an average score.

If candidate recall stays high and top-10 relevance barely moves, you have your answer. If it moves, try the next size up before giving up on the idea.

Takeaways

• Vector storage scales with documents × dimensions. At millions of documents, it becomes a real share of infrastructure cost.
• MRL turns a prefix of the embedding into a smaller embedding, as long as the model was trained for it. For me, that was one API parameter and a 4× cut in raw storage.
• Architecture decides how much truncation hurts. Filters, hybrid keyword search and a reranker mean the vectors only need to find good candidates, not rank them perfectly.
• Measure on your own data. Candidate recall and top-10 relevance at full vs truncated dimensions beat any general benchmark.

I’ll update this post with measured results once the comparison runs are finished. If you’ve tried MRL truncation in your own retrieval stack, I’d like to hear how far you could cut before relevance moved. Leave a response below.

Reference: Kusupati, A., et al. (2022). Matryoshka Representation Learning. NeurIPS 2022. https://arxiv.org/abs/2205.13147