Retrieving document pages without OCR: late interaction and visual grounding
I built a ColPali-style retriever that scores query tokens against page patches and draws a box around the evidence. Here is how it works and what I did and did not measure.
Most PDF retrieval pipelines start the same way: render the page, run OCR, flatten the result into text, chunk it, embed it. That works until the page is a ten-column financial table or a chart. OCR reads across column dividers, charts turn into nothing, and one dense vector per chunk has to carry every number on it.
ColPali (Faysse et al., 2024) skips the OCR step. A vision-language model turns each page image into a grid of patch embeddings, and the query is matched against those patches directly. I built DocuVision-ColPali to understand that pipeline end to end, including the part I cared about most for audit work: showing the reader exactly where on the page an answer came from.
One thing up front. The default encoder in this repo is a deterministic surrogate, not the real ColPali model, and I have not run the ViDoRe benchmark. This article covers the retrieval and grounding logic, which is real and testable, and says clearly where the evidence stops.
The retrieval step: MaxSim
Each page becomes a 32 x 32 grid of patches, so 1,024 vectors of 128 dimensions. The query becomes one vector per token. The score is the sum, over query tokens, of the best-matching patch:
Score(Q, D) = sum over query tokens i of ( max over patches j of q_i . d_j )A query such as “operating income 2024” lets each word find its own patch. Nothing is averaged away, which is the point of late interaction. The implementation is a single matrix product:
sim_matrix = torch.matmul(q_norm, d_norm.transpose(0, 1)) # (|Q|, N_patches)token_max_scores, _ = torch.max(sim_matrix, dim=1) # best patch per query token
total_score = float(torch.sum(token_max_scores).item())patch_max_sims, _ = torch.max(sim_matrix, dim=0) # best query token per patch
The last line matters for grounding. Taking the max over query tokens for each patch gives a 1,024-value heatmap of where the page responds to the query.
From heatmap to bounding box
A citation is only useful if a person can check it. extract_bounding_boxes reshapes the heatmap into the 32 x 32 grid, normalizes it, smooths it with a small Gaussian blur, and thresholds it with Otsu’s method so the cut-off adapts to each page. A short morphological close joins neighbouring patches, then OpenCV contours become normalized boxes:
heatmap_2d = scores.reshape((grid_size, grid_size))
norm_map = (heatmap_2d - h_min) / (h_max - h_min)
blurred = cv2.GaussianBlur(norm_map, (3, 3), sigmaX=1.0)
_, binary_mask = cv2.threshold(blurred_u8, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)
closed_mask = cv2.morphologyEx(binary_mask, cv2.MORPH_CLOSE, kernel)
contours, _ = cv2.findContours(closed_mask, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)Each box carries a confidence, the mean attention inside it, and a Pydantic model keeps the coordinates inside [0, 1] so they can be scaled to any render size. Small contours are dropped, and if nothing survives the filter the code falls back to a box around the single strongest patch.
What the surrogate encoder is, and why it exists
The real ColPali model needs a GPU and several gigabytes of weights. To keep the whole pipeline runnable on a laptop and on free Streamlit hosting, ColPaliEmbedder defaults to model_name_or_path=”surrogate”. It builds each patch vector from the colour statistics of that patch, then adds a hashed vector for every word of the PDF’s text layer at that word’s position on the grid. Query tokens get the same hashed vectors.
Get Muhammad Zainurrahman’s stories in your inbox
Join Medium for free to get updates from this writer.
That makes the surrogate a text-position matcher, not a vision model. It exercises MaxSim, the grid geometry, the grounding code and the UI honestly, but it says nothing about how well ColPali reads charts or scanned pages. If byaldi and the model weights are installed, the embedder loads the real model instead.
Below is a real output from the demo, a generated three-page financial report. It shows the grounding path working end to end, and it shows its limits: the boxes land on the right region of the table but are loose, and two of them sit on the page title. A box that is approximately right is a long way from a reliable citation.
Answer synthesis and the demo
Retrieved pages and their boxes go to a vision-capable model (Gemini 2.5 Flash by default) to write the answer. If the API call fails, a deterministic fallback synthesizer answers so the demo never shows an empty screen. The Streamlit app includes a one-click sample report, so a visitor can try a query without finding a PDF first. Pages are rasterized with PyMuPDF at 150 DPI, and no OCR engine is involved on the real-model path.
What I have not measured
The repo contains a benchmark report with NDCG@5 figures against an OCR text-RAG baseline. Those numbers are projections, not results: I did not run ViDoRe, and the surrogate encoder could not produce a meaningful score on it anyway. For measured ColPali results on ViDoRe, read the original paper.
A fair evaluation would use the real ColPali weights on a ViDoRe subset, compare against a Tesseract plus BGE-M3 baseline on the same queries, and report NDCG@5, recall and latency per document type. That is the next piece of work.
What I took from building it
MaxSim is simple enough to write in four lines, and the same similarity matrix gives both the ranking and the heatmap, so grounding costs almost nothing extra. The harder part is making boxes tight enough to trust. A 32 x 32 grid over a full page means one patch covers several table rows. Better citations will need either finer patches or a second pass that snaps the box to the underlying text layout.
Source code: github.com/MuhZainur/DocuVision-ColPali
