Advanced RAG: Layout Analysis for Document Ingestion
Locating text, tables, images, and preserving the layout clues that connect them.
A document is not just a sequence of words. A price belongs to a row. A caption belongs beside a figure. Two paragraphs at the same height may belong to different columns.
People use those clues almost without noticing. A pipeline that turns the whole document into one string can lose them before it creates its first chunk.
Imagine asking a RAG system:
“How much did we spend on shipping labels?”
The document contains the answer. But if the ingestion process separated the prices from their item names, retrieving the right page may still give the model the wrong evidence.
Layout analysis identifies the parts of a document and where they belong.
In a RAG pipeline, that map helps later readers handle paragraphs, tables, and images separately, then assemble useful context without relying only on the order in which bytes happened to be stored.
We’ll look at different layout-analysis cases, from locating tables and figures to suggesting reading order and identifying repeated headers.
We’ll use Raggae, a Python SDK with independent workers for document processing. We’ll use its layout-analysis workers to demonstrate how these problems can be handled.
pip install raggaeThe PDF that mixes columns, a table, and a figure
A document can mix paragraphs, tables, figures, and captions in many arrangements. Text may run in two columns, a table may span both, and a chart may occupy the next page.
Flattening that into one string creates several problems at once: columns can become interleaved, table values can lose their headers, and the chart can disappear entirely. Sending everything to both a text reader and a table reader can also duplicate the table’s content.
Here is complex-report.pdf, a two-page report we’ll use. Page 1 switches between columns and a wide table. Page 2 contains a chart, its caption, and follow-up text.
NativePdfLayoutAnalyzer inspects the PDF’s stored characters, ruling lines, and raster-image placements. It groups text into candidate blocks, locates the ruled table, and records the chart image’s position.
from raggae.documents.layout_analysis import NativePdfLayoutAnalyzer
pdf_worker = NativePdfLayoutAnalyzer()from collections import Counter
pdf_layout = pdf_worker.analyze("complex-report.pdf")
print(pdf_layout.outcome)
print("pages:", len(pdf_layout.surfaces))
print("regions:", dict(Counter(region.kind for region in pdf_layout.regions)))
for region in pdf_layout.regions:
if region.kind in ("table", "image"):
print(region.id, region.kind, region.bbox)success
pages: 2
regions: {'text': 22, 'table': 1, 'image': 1}
page-1-r8 table (48.0, 312.0, 564.0, 420.0)
page-2-r5 image (48.0, 180.0, 564.0, 402.0)That response gives later readers specific places to work: the text blocks, one table area, and one picture area. The table’s cell text is excluded from the ordinary text regions, avoiding duplicate extraction through those two routes.
To see those regions on the original pages, use Raggae’s DocumentRegionRenderer. It prepares a Pillow image for each page with the returned boxes drawn on it; it does not display anything itself. PDF rendering requires:
pip install 'raggae[layout-visual]' from raggae.documents import DocumentRegionRenderer
renderer = DocumentRegionRenderer()In a Jupyter notebook:
from IPython.display import display
previews = renderer.render(pdf_source, layout=pdf_layout)
try:
for page_number, preview in previews.items():
print(f"Page {page_number}")
display(preview)
finally:
for preview in previews.values():
preview.close()Here, display shows the images in a notebook; another application could save them with preview.save("page.png") before closing them. The original PDF stays unchanged.
Here are the returned boxes. Blue is text, green is the table, and orange is the image.
We can now add reading-order suggestions and connect the figure to its nearby text.
Which region comes next?
The boxes tell us where the content is, but not how to move through it. For this report, we also want a suggested sequence through the columns, the wide table, and the figure on the next page.
GeometricReadingOrderResolver takes the existing layout result and looks for gaps between its rectangles. It returns links saying “read this region before that one,” without reading the PDF again:
from raggae.documents.layout_analysis import GeometricReadingOrderResolver
resolver = GeometricReadingOrderResolver()order = resolver.resolve_order(pdf_layout, direction="ltr")
print(order.outcome, "links:", len(order.edges))
print("unresolved:", len(order.unresolved_region_ids))
for edge in order.edges:
if edge.before_region_id in ("page-1-r8", "page-2-r5"):
print(edge.before_region_id, "->", edge.after_region_id, edge.basis)success links: 23
unresolved: 0
page-1-r8 -> page-1-r9 horizontal_gap
page-2-r5 -> page-2-r6 horizontal_gapFor example, the image page-2-r5 comes before the nearby text page-2-r6. horizontal-gap records the geometric evidence for that suggestion. This is an ordering link, not a claim that the text captions the image.
Now draw those links with DocumentReadingOrderRenderer:
from raggae.documents import DocumentReadingOrderRenderer
order_renderer = DocumentReadingOrderRenderer()from raggae.documents import DocumentReadingOrderRenderer
order_previews = order_renderer.render(
pdf_source, layout=pdf_layout, order=order,
)
try:
display(order_previews[2])
finally:
for preview in order_previews.values():
preview.close()Here is the suggested sequence for the figure page. Arrows show returned before/after links; numbers show the suggested order within this page.
A later assembly step can use those links when arranging the extracted content.
Common PDF layout patterns
A purchase table between ordinary paragraphs
This purchase report puts Shipping labels / 600 / $90 in a ruled table between an introduction and a delivery note. The table should stay together without absorbing the surrounding prose.
Raggae identifies the table as one area and keeps the surrounding paragraphs separate. A later table reader can inspect the item, quantity, and amount columns together.
Three columns below a full-width introduction
This page starts with an introduction across its full width, switches to three team updates, and ends with a shared conclusion. Joining lines simply because they appear at the same height would mix the updates together.
Raggae groups the text within each column rather than merging across the page. The introduction and conclusion also retain their own regions for later reading and assembly.
A rotated page
This page displays in landscape even though its underlying PDF page is stored in portrait orientation. Its content locations need to follow the displayed rotation.
Raggae accounts for the rotation and returns boxes aligned with the displayed text and table. Later readers can use those positions to select the intended content.
Two figures with their own nearby text
This report page contains two embedded chart images, each with a nearby caption. The pictures need separate locations so an image reader can process them individually while the text reader handles their labels.
Raggae returns a separate image region for each chart and text regions for the nearby labels. Their locations give later readers the pieces needed to extract the images and associate them with the appropriate text.
What is actually in the map?
All discovery workers return LayoutAnalysisResult. The names are easier to understand with this example:
surfaces: The places containing content: PDF pages, slides, sheets, image frames, or a continuous document body.regions: The pieces we found, each with an ID, a kind, and a location in the original file.bbox: A rectangle, when physical coordinates are available: left, top, right, bottom.findings: Explanations of missing coverage, limits, or problems.
PDF rectangles use points measured from the displayed page’s top-left. Image rectangles use pixels. An unrendered Word document or worksheet instead has structural locations; inventing page coordinates would be misleading.
The map contains neither the final paragraph text nor extracted image files. Region IDs identify pieces; their numbering does not establish reading order. Keep the exact source alongside the map so a later reader can resolve those locations.
The Word report with a picture inside a table
A purchasing report can mix headings, explanatory paragraphs, a cost table, and a reference table containing both a nested table and a picture. Reading only the document’s top-level paragraphs would miss important content; recursively treating every cell paragraph as body text would duplicate it.
Here is receiving-report.docx. The first page contains a purchase table; the second contains a reference table with a nested table on the left and an embedded picture on the right. The teal rectangle is the sample picture.
Word stores these as actual elements, so DocxLayoutAnalyzer can locate them without guessing their roles from pixels.
from raggae.documents.layout_analysis import DocxLayoutAnalyzer
analyzer = DocxLayoutAnalyzer()word_layout = analyzer.analyze("receiving-report.docx")
print(word_layout.outcome)
print(dict(Counter(region.kind for region in word_layout.regions)))
for region in word_layout.regions:
if region.kind in ("table", "image"):
print(region.id, region.kind, region.location.child_indexes)success
{'heading': 2, 'text': 3, 'table': 2, 'image': 1}
body-r3 table (2,)
body-r6 table (6,)
body-r7 image (6, 2, 1, 2, 0, 0)The highlights below connect the returned element IDs to the tables and picture in the rendered preview. The worker’s source locations are the element paths printed above.
An element path such as (2,) means “the third body element”. The longer image path continues through children inside the table. Each location also names the internal document part, here word/document.xml, so another reader can find the same element in the same file.
The result represents the Word body as one continuous flow. Its element locations let later readers follow the document’s structure across explicit page breaks.
Other Word cases
- An employee handbook with custom heading styles: built-in headings and styles inherited from them become
heading, preserving useful section boundaries for later chunk assembly. - A reference table containing a picture: both the table and embedded image retain exact locations, so a table reader and an image reader can work on their respective content.
- A report with repeated wording: element paths identify the original paragraph or table directly, so later readers can select the right occurrence without searching for matching text.
The presentation whose shapes are not a reading sequence
A shipment briefing contains a title, an introduction, a stock table, a picture, and a note underneath. Extracting only text misses the picture and the table’s structure. Following the stored shape list creates a different risk: it describes stacking, not necessarily the presenter’s intended sequence.
Here is the first slide in shipment-briefing.pptx, illustrated from its stored shapes. It mixes text, a table, and a sample blue picture; we’ll use the second slide in the reading-order example below.
PptxLayoutAnalyzer returns both the source shape locations and their positions on each slide:
from raggae.documents.layout_analysis import PptxLayoutAnalyzer
analyzer = PptxLayoutAnalyzer()slide_layout = analyzer.analyze("shipment-briefing.pptx")
print(slide_layout.outcome)
print("slides:", len(slide_layout.surfaces))
print(dict(Counter(region.kind for region in slide_layout.regions)))
for region in slide_layout.regions:
if region.kind in ("table", "image"):
print(region.id, region.kind, region.bbox)success
slides: 2
{'heading': 1, 'text': 4, 'table': 1, 'image': 1}
slide-1-s2 table (40.0, 200.0, 340.0, 340.0)
slide-1-s3 image (400.0, 200.0, 580.0, 320.0)
The table and picture now have separate locations for their respective readers. Each region also identifies its original slide part and shape path. A path such as (3, 1) means the second child inside the fourth shape, which is a group.
Boxes describe shape rectangles measured from the slide’s top-left. Reading-order suggestions are kept separately from those source locations.
Other presentation cases
Grouped, resized, or mirrored diagrams. Children store coordinates relative to their group. In our mirrorer-group example, the first child ends up on the right at (380, 200, 580, 260), while the second is on the left. The worker applies the group transforms before returning slide coordinates, including nested groups. It returns the leaf regions rather than extra overlapping group containers.
A lower text box created before the upper one. The second slide in our briefing keeps both source locations and their actual positions. The geometric resolver can then suggest upper-before-lower rather than following the shape-creation sequence.
A briefing spread across several slides. Each slide has its own surface and shape locations, keeping a table on one slide distinct from a picture on another.
The workbook with several independent content areas
A warehouse workbook can contain a stock table, a separate cost block, and a merged label on the same sheet. Reading the whole used range as one table mixes those areas together, including the blank columns between them.
Get Okan Yenigün’s stories in your inbox
Join Medium for free to get updates from this writer.
Here is stock-and-costs.xlsx. On the Warehouse sheet, stock sits on the left, costs on the right, and a merged dispatch label below. The second sheet contains a filled grid and two diagonal labels.
from raggae.documents.layout_analysis import XlsxLayoutAnalyzer
analyzer = XlsxLayoutAnalyzer()sheet_layout = analyzer.analyze("stock-and-costs.xlsx")
print(sheet_layout.outcome)
print(dict(Counter(region.kind for region in sheet_layout.regions)))
for region in sheet_layout.regions:
print(region.kind, region.location.sheet_name, region.location.cell_range)success
{'table': 1, 'unknown': 5}
table Warehouse A1:B4
unknown Warehouse A7:D7
unknown Warehouse F2:G4
unknown No declared table A1:B3
unknown No declared table F1
unknown No declared table G2
The declared table retains its table role. Other occupied areas receive unknown as their content kind, together with exact ranges that a later reader can inspect.
The useful result is the separation: costs stay apart from stock, the merged label appears once, and the diagonal labels remain two independent regions. Each location names its sheet, so identical cell addresses on different sheets remain distinct.
Other spreadsheet cases
- A logo attached at E6: the image receives a drawing location containing its sheet, image index, and anchor cell, ready for an image reader.
- A1 contains
0, and Excel’s last cell containsFalse: the sparse example returns two occupied regions without scanning every cell between them. Both values count as content; formatting-only cells are ignored. - A workbook with blank sheets: those sheets retain their own surfaces without introducing empty content regions, preserving the original workbook structure.
For worksheets, locations are cell ranges and drawing references rather than page rectangles.
The web article whose structure is already written down
A saved web report may already distinguish headings, paragraphs, a stock table, and a figure caption through HTML tags. Stripping every tag immediately discards useful evidence before a reader uses it.
Here is a content preview of warehouse-report.html, illustrated from its stored elements. It includes two heading levels, a stock table, an image reference, a figure caption, and closing text.
HtmlLayoutAnalyzer locates those elements without fetching the picture or executing the page:
from raggae.documents.layout_analysis import HtmlLayoutAnalyzer
analyzer = HtmlLayoutAnalyzer()html_layout = analyzer.analyze("warehouse-report.html")
print(html_layout.outcome)
for region in html_layout.regions:
print(region.kind, region.location.tag, region.location.child_indexes)success
heading h1 (0, 1, 0, 0)
text p (0, 1, 0, 1)
heading h2 (0, 1, 0, 2, 0)
table table (0, 1, 0, 2, 1)
image img (0, 1, 0, 3, 0)
caption figcaption (0, 1, 0, 3, 1)
text p (0, 1, 0, 4)
The outlines show the returned element roles and IDs on the same content preview. Their source locations remain the HTML element paths printed above.
The table and its own caption stay inside one table region. The figure’s figcaption is separate, ready for a relationship analysis. The element paths count element children from the parsed document root; they are not screen coordinates. A later reader can resolve the table element rather than searching the whole page again.
Other HTML cases
An article with several heading levels. Heading tags preserve the distinction between the main title and section headings, while ordinary paragraphs remain text regions.
A table containing paragraphs and pictures. The table stays one region for its cell contents, while embedded image elements can retain their own locations for an image reader.
An HTML fragment rather than a whole web page. A stored article fragment can be analyzed directly; its element paths still start at the parsed document root.
The scanned report that is one enormous picture
A scanned PDF may look exactly like a digital report. Inside, however, its paragraphs, tables, and charts can all be pixels in one image. There are no stored text boxes or table lines for a native analyzer to recover.
OCR can turn those pixels into words. But we also need to identify which area should go to a table reader and which is a chart. That is a visual layout problem, not just a text-recognition problem.
Here is scanned-report.pdf. Its first page contains native PDF content; the second stores a similar report as one full-page image. Both visibly contain a heading, body text, a table, a chart, a caption, and footers. Here is the scanned page:
VisualLayoutAnalyzer renders PDF pages, or opens image files, and sends their pixels to an explicitly selected local detection service. The service predicts labeled rectangles. The worker translates them back into the document's coordinates.
Two available services use Paddle’s layout detector and Docling’s Heron model. Both upstream projects describe layout detection as locating and classifying document regions; that is distinct from running their broader document-processing pipelines.
Install the backend you want:
pip install 'raggae[layout-visual,layout-docling]'
# Alternative: pip install 'raggae[layout-visual,layout-paddle]'Download the chosen model’s weights separately from its model repository. RAGGAE_DOCLING_MODEL_DIR below must name an existing local Heron model folder; constructing the service is not a download step. Document inference runs locally.
import os
from raggae.documents.layout_analysis import (
DoclingLayoutService, VisualLayoutAnalyzer,
)
service = DoclingLayoutService(
model_path=Path(os.environ["RAGGAE_DOCLING_MODEL_DIR"]),
device="cpu",
)
visual_worker = VisualLayoutAnalyzer(service=service)scan_pdf = Path("scanned-report.pdf")
visual_layout = visual_worker.analyze(scan_pdf)
print(visual_layout.outcome)
for surface in visual_layout.surfaces:
kinds = Counter(
region.kind for region in visual_layout.regions
if region.surface_id == surface.id_
)
print(surface.id_, dict(kinds))success
page-1 {'table': 1, 'text': 5, 'image': 1, 'caption': 1, 'heading': 1}
page-2 {'text': 5, 'table': 1, 'image': 1, 'caption': 1, 'heading': 1}Each page now has candidate text, table, and picture regions, even though one page has no native text.
To use Paddle, change the injected service, not the worker interface:
from raggae.documents.layout_analysis import PaddleLayoutService
paddle_service = PaddleLayoutService(
model_path=Path(os.environ["RAGGAE_PADDLE_MODEL_DIR"]),
device="cpu",
)
paddle_layout = VisualLayoutAnalyzer(service=paddle_service).analyze(scan_pdf)The worker interface and result schema stay the same when the service changes. This lets an application reuse the same downstream readers and renderers while choosing its local detection backend.
The scanned page now has separate areas for text, table, and image reading. Those specialized readers can use the returned locations to extract the corresponding content.
Other visual-analysis cases
A receipt or report supplied as a PNG. Use the same worker directly, without wrapping the image in a PDF:
image_layout = visual_worker.analyze("scanned-page.png")
surface = image_layout.surfaces[0]
print(image_layout.outcome, surface.kind, surface.unit)success image pxA multi-frame TIFF or a mixed native/scanned PDF. Frames and pages are inspected individually. The visual worker can analyze the whole input, but it is an alternative to native discovery — not an automatic second pass that every PDF must run through.
A reusable local detection service. Create the service once and pass it to successive worker calls, keeping model initialization separate from individual document processing.
The two columns that become one broken conversation
Finding every text block is not enough. A research article might continue down the left column before moving to the right. Sorting blocks only by height can instead alternate between them.
Geometry can help: an empty vertical strip suggests separate columns; a horizontal gap can separate a title from the body. GeometricReadingOrderResolver applies those rules to an existing map. It returns before/after suggestions, without opening the document again.
We already applied this worker to the complex report in the first example. Its links kept the table atomic and placed the figure before nearby text, while preserving the original region locations.
Other ordering cases
A slide whose lower box was created first. On the second slide of our briefing, the source stores the bottom box before the top one. We can ask the resolver for a sequence based on their actual positions:
slide_order = resolver.resolve_order(slide_layout)
for edge in slide_order.edges:
if edge.before_region_id.startswith("slide-2-") and edge.after_region_id.startswith("slide-2-"):
print(edge.before_region_id, "->", edge.after_region_id)slide-2-s1 -> slide-2-s0The suggested link goes from the upper box to the lower one, independently of when the presenter created them.
Right-to-left columns. Pass direction="rtl" to choose right-to-left traversal across separated columns. This is an explicit layout convention chosen for the document.
A wide table between text sections. The table remains a single region in the proposed sequence, so its individual cells are not interleaved with the surrounding paragraphs.
A document spanning several pages. Page-order links connect populated surfaces, allowing later assembly to follow the document across page boundaries.
The handbook that repeats its title in every chunk
A 100-page handbook may repeat the company name, an internal-use notice, and a page number on every page. If those fragments keep entering the extracted body text, they consume chunk space and can interrupt useful passages.
Deleting every repeated sentence is not a safe substitute. “Always record the shipment ID” might be an important instruction repeated deliberately in several sections.
Here is handbook.pdf: two pages with different section titles, a repeated handbook header, an internal-use footer, changing page numbers, and the same reminder in the middle of both pages. We want to distinguish position as well as repetition.
from raggae.documents.layout_analysis import PdfRepeatedMarginDetector
analyzer = NativePdfLayoutAnalyzer()handbook_pdf = Path("handbook.pdf")
handbook_layout = analyzer.analyze(handbook_pdf)
margins = PdfRepeatedMarginDetector().detect(handbook_layout, handbook_pdf)
for group in margins.groups:
print(group.kind, group.basis, len(group.region_ids))header repeated_text 2
footer repeated_text 2
footer page_number 2
This worker reads short native-text samples from the same PDF used to create the map. It compares text and relative positions across pages, including supported changing page-number patterns.
The candidate groups stay separate from the original content. A later assembly step can omit repeated furniture from body text, retain it as metadata, or label it explicitly.
Other repeated-margin cases
Changing page numbers beside changing invoice IDs. In our numbered-invoice example, Page 1 of 2 and Page 2 of 2 form a page-number group. Invoice 1001 and Invoice 1002 do not. The worker does not erase every changing number to manufacture a match. A repeated Release 2026 label can still match as ordinary text.
A reminder repeated inside the body. Our handbook repeats the same instruction on both pages, but its central position keeps it out of the header/footer groups. Repetition and position work together to distinguish it from page furniture.
Consistently positioned headers across different page sizes. Positions are compared relative to each page’s dimensions, rather than by assuming that every page has the same pixel or point size.
The chart that needs to stay with its explanation
A chart shows four bars. The line beneath it explains what those bars represent, while a more distant paragraph discusses another point. Keeping that nearby label associated with the chart gives later content assembly useful context.
Here is caption-report.pdf. Page 1 has a chart with a short label below it; page 2 has a table with its label above it.
RuleBasedRegionRelationshipDetector receives the existing layout result. It uses proximity and alignment to propose caption links, and structural paths or enclosing rectangles to establish containment links.
from raggae.documents.layout_analysis import RuleBasedRegionRelationshipDetector
caption_layout = pdf_worker.analyze("caption-report.pdf")
relationship_worker = RuleBasedRegionRelationshipDetector()
links = relationship_worker.detect(caption_layout)
print(links.outcome)
for link in links.relationships:
print(link.kind, link.source_region_id, "->", link.target_region_id, link.basis)success
caption_candidate page-1-r5 -> page-1-r4 proximity
caption_candidate page-2-r4 -> page-2-r5 proximity
Each nearby label is linked to its corresponding target, whether it appears above or below it. The result records proximity as the evidence, giving a later assembly step a concrete caption candidate to use.
Running the useful parts together
We have used separate workers to make their responsibilities visible. A normal ingestion call should not need to repeat that wiring.
DocumentLayoutAnalyzer selects a discovery worker, then runs the enabled analyses that apply to its map:
from raggae.documents.layout_analysis import DocumentLayoutAnalyzer
report = DocumentLayoutAnalyzer().analyze(handbook_pdf)
print(report.outcome)
print("regions:", len(report.layout.regions))
print("margin groups:", len(report.repeated_margins.groups))success
regions: 16
margin groups: 3The report keeps the original layout alongside reading_order, relationships, and repeated_margins. These analyses inspect the same map independently: detecting a header does not secretly remove it before the order resolver runs.
Default routing uses the file extension for PDF, DOCX, PPTX, XLSX, and HTML. For raw bytes, give source_format="pdf" to select the PDF worker explicitly.
Need just the content locations? Disable the additional jobs:
discovery_only = DocumentLayoutAnalyzer().analyze(
handbook_pdf,
reading_order=False,
relationships=False,
repeated_margins=False,
)For images, configure visual_service=service. To use visual discovery for a scanned PDF, inject that analyzer directly:
scan_report = DocumentLayoutAnalyzer(
analyzer=VisualLayoutAnalyzer(service=service),
).analyze(scan_pdf, repeated_margins=False)Layout analysis keeps information that plain text alone cannot express: a table’s boundary, a paragraph’s place on a page, a shape’s location on a slide, a caption’s possible target.
That gives later readers a better starting point. Text can be read as text, tables as tables, and images as images — with enough location information to bring their contents back together.
The next stages can extract the content, use the layout links when assembling it, and carry the relevant context into searchable chunks.
Locate the evidence first. Extract it next. Use its meaning to decide how it belongs together. A RAG system has a better chance of answering from the right evidence when ingestion has not already discarded the structure that made it understandable.
