Extracting Structured Data from Text

Tabish alam

7 min read

·

5 days ago

--

Listen

Share

Most of the text a business receives is not meant for a database. An invoice arrives as a PDF, a support request arrives as an email, a contract arrives as ten pages of prose. Somebody still needs the invoice number, the date and the total in a table.

Press enter or click to view image in full size

Turning free text into fields is called structured extraction. It sounds like a small task, and it is where many NLP projects quietly fail. The extraction step is rarely the hardest part. The hard part is knowing when an extraction is wrong.

This article builds a small, honest example to show where extraction breaks and what to put around it so a wrong record does not reach your database.

What Structured Extraction Means

You start with text, and you want a record with fixed fields:

{
  "invoice_number": "INV-8737",
  "date": "2026-11-13",
  "total": 15795.59,
  "vendor": "Lucerne Foods"
}

Two ideas matter here.

First, the output has a schema: a list of fields, each with a type. Everything downstream (a spreadsheet, an accounting tool, a dashboard) relies on that schema being respected.

Second, extraction is more than finding spans of text. Sequence labeling (Part 4) can mark where a name or a date sits in a sentence. A record also needs the date written as a date and the total written as a number. Finding the span is half the job. Normalizing it is the other half.

The Pipeline

A safe extraction system is not one model call. It is a short chain, where every step is allowed to say “stop”.

Press enter or click to view image in full size

The extractor can be a set of rules, a trained tagger or a language model. The rest of the chain does not care which. It parses the output, repairs small formatting slips, checks the schema, normalizes values and finally checks that each value really appears in the source text. Anything that fails goes to a person instead of the database.

Why Rules Break Quietly

To see the problem, the demo uses synthetic invoices in four layouts. The rules were written after looking at layouts A and B only.

• Layout A: Invoice No: INV-2033, an ISO date and a $ total.
• Layout B: a single sentence with a date like 13 November 2026 and USD before the amount.
• Layout C: a European style, with 27/01/2026 for the date and 7.857,20 EUR for the total.
• Layout D: a layout the rules never saw, with Doc number: 1034-B, Dec 15, 2026 and 5 399 INR.

Each layout had 50 generated invoices, so 200 fields each. The results:

Press enter or click to view image in full size

On A and B, all 50 records were fully correct. On C, none of the 50 records were fully correct. On D, none were either. That is expected, since the rules were never designed for them. What matters is how they failed.

On layout C, 102 of 200 fields were correct. 27 were missing, and 71 were wrong values that looked fine. The details explain why:

• The totals were wrong in all 50 records. The text 7.857,20 was read as 7.8572, because the rule assumed a comma separates thousands. The real total was 7,857.20.
• Dates written day first were handled as month first. When the day was above 12, the rule found no valid date and returned nothing (27 cases, a loud failure). When the day was 12 or below, the rule invented a valid but wrong date (21 cases, a silent failure). Only 2 of 50 were right, and both were dates where day and month happen to be equal (06/06 and 02/02).

On layout D, 150 of 200 fields were missing, and 50 were wrong. The total 5 399 INR became 5.0, because the rule stopped at the space.

The lesson is not that rules are bad. It is that a missing value is visible, and a wrong value is not. A pipeline that only counts “did we get something back?” will report success while storing 5.0 as the total of a five-thousand-unit invoice. Evaluating NLP Systems (Part 12) covers evaluation more broadly, and the number-format problem here connects to the multilingual issues in Part 16.

Language Models Change the Extractor, Not the Problem

A language model can handle new layouts far better than hand-written rules, because it reads the text instead of matching patterns. It is a strong choice for the extractor step. It does not remove the need for the rest of the chain.

Model output is text. Even when you ask for JSON, you may get:

• the JSON wrapped in a markdown code fence;
• a trailing comma that breaks the parser;
• the total as a string, such as "1,250.50 USD", instead of a number;
• the date as 14 March 2026 instead of 2026-03-14;
• a plausible value that is not in the document at all;
• a sentence of prose with no JSON.

Each of these is common enough that a pipeline has to expect it.

Validate Before You Trust

The validation layer in the demo does five things, in order.

1. Parse and repair. Strip code fences and trailing commas, then parse the JSON. If there is no JSON, reject.
2. Check the schema. All four keys must be present.
3. Check types. The total must become a number, so "1,250.50 USD" is converted to 1250.5.
4. Normalize. Dates are converted to ISO format. A date the code cannot read with confidence is rejected, not guessed. This matters for formats like 03/04/2026, which could be March or April.
5. Check against the source. The invoice number and vendor must appear in the text, and the total must appear in one of its usual written forms.

The last step is the one that is easy to skip and the one that catches invented values:

def grounded(value, text):
    forms = {f"{value:,.2f}", f"{value:.2f}"}
    return any(f in text for f in forms)

If the model returns a total that never appears in the document, the record is rejected, however neat the JSON looks.

What Validation Bought

To test this, the demo generated 400 model-style outputs for layout A and B invoices. They were produced by a script, not a real model, and about 55% were clean while the rest carried the slips listed above. Two approaches were compared.

Press enter or click to view image in full size

Trusting the output: 249 of 400 were exactly right. 89 could not be parsed at all, and 62 parsed but had the wrong form or value, such as a total stored as a string.

Validating first: 249 were accepted as they were, 111 were repaired and accepted, and 40 were rejected. Of the rejections, 18 had no usable JSON and 22 contained a value that was not in the source text. All 360 accepted records matched the correct answer.

That is 360 usable records instead of 249, which is 90% against about 62%. More importantly, the 40 rejected outputs did not turn into bad rows. They became a short review queue.

What This Demo Does Not Prove

Be careful with these numbers.

• The invoices and the model outputs are synthetic. The error mix was chosen by hand, so the percentages will not carry over to your data. Real documents have scans, tables and odd wording that this demo does not include.
• The model outputs came from a script. The demo shows what a validation layer does with typical slips. It says nothing about how often a real model makes them.
• Grounding catches values that are not in the text. It does not catch a value that is in the text but is the wrong one, for example reading a subtotal instead of the total. Those errors need spot checks against human labels, and the label quality issues in Part 17 apply here as well.
• All 360 accepted records were correct in this test. That is a result of this setup, not a guarantee.

Practical Advice

• Define the schema first, with types, and treat anything outside it as an error.
• Count missing and wrong separately. Wrong values are the dangerous ones.
• Never guess an ambiguous date or number format. Reject it, or ask for the locale.
• Ground every extracted value in the source text where you can.
• Keep a review queue. A rejected record that a person fixes in thirty seconds is cheaper than a bad row found at month end.
• Test on layouts you did not build the system around. If you add a new document type, measure again.
• When the extractor sits behind an API, the confidence and “uncertain” handling from Part 13 and the step limits from Part 14 apply in the same way.

Key Takeaways

• Structured extraction is finding the value and converting it to the right type.
• Rules written for known layouts failed on new ones. On a layout with different date and number formats, 102 of 200 fields were correct and 71 were silently wrong.
• A missing value is visible. A wrong value is not, so measure both.
• Language models improve the extractor but still produce fences, trailing commas, string numbers and invented values.
• A validation chain (parse, schema, types, normalize, check against source) turned 249 usable outputs into 360 out of 400 in this synthetic test, with no wrong records accepted.
• Reject what you cannot verify, and send it to a person.

Written by Tabish Alam, Proxima.ai
Hire me on Upwork