Document extraction at scale: invoices to structured data

If you've ever tried to get a machine to reliably read invoices, you know the pain: scanned PDFs, crooked photos, vendor-specific layouts, handwritten notes, line items buried in tables. Templates break the moment a new supplier shows up, and regex can't cope with "Total", "Amount due", and "Balance payable" meaning the same thing.

The modern answer is a pipeline, not a single model: render each page as an image, get exact text out with specialized OCR (or a vision model), have a language model map that text to a strict schema, then validate the output programmatically before it ever touches your database. Here's how to build it.

The architecture, in one diagram#

Invoice PDF
    ↓
Render each page as an image (PNG, ~300 DPI)
    ↓
Text layer: dedicated OCR (Google Vision, PaddleOCR, Tesseract)
    ↓
Intelligence layer: LLM maps text → JSON with a strict schema
    ↓
Validation layer: Pydantic checks, arithmetic checks, confidence scores
    ↓
Pass → database.  Fail → retry with the error, then human review

The key insight, borne out by practitioners' real-world testing, is that the vision/OCR layer and the intelligence layer do different jobs. OCR is about precision — reading characters exactly. Extraction is about semantics — knowing that "Total due: $1,247.50" belongs in total_amount. One practitioner found that sending invoice images straight to a vision LLM for everything returned incorrect fields across the board, while splitting the job into a two-step pipeline (Google Vision API for OCR, then GPT-4.1 for field extraction) fixed it. Probablistic models are great at understanding; dedicated OCR is great at reading.

That doesn't mean vision models have no role. Some modern stacks send page images directly to a multimodal model (Claude Sonnet, GPT-4-class models) and skip OCR entirely — and small specialized models like TinyDoc-VLM (256M parameters, CPU-friendly) are built specifically for document Q&A. But for invoices at scale, where a single misread digit corrupts a total, the two-step approach is the safest default. Pick your architecture deliberately.

Step 1: Ingest and render#

Raw PDF text extraction is unreliable for scanned or visually complex invoices. Render each page as an image first — 300 DPI is the commonly recommended resolution for OCR accuracy.

from pdf2image import convert_from_path

pages = convert_from_path("invoice.pdf", dpi=300)
pages[0].save("page_1.png", "PNG")

Multi-page invoices are the norm, so loop over all pages and track a page field in your extraction schema. Also record the filename and page count at ingest time — you'll want them for audit trails later.

Step 2: OCR — the precision layer#

Use a dedicated OCR engine rather than asking an LLM to read characters. Two widely used options:

• Cloud OCR (Google Cloud Vision's DOCUMENT_TEXT_DETECTION, Azure Document Intelligence): best accuracy, pay-per-use, often with a free tier (Google Vision gives the first 1,000 pages/month free). In one reported test on 500 invoices, Google Vision hit roughly 98.7% accuracy on clean invoices and 92.3% on handwritten notes.
• Local OCR (PaddleOCR, Tesseract): zero per-page cost, keeps sensitive documents on your hardware, slightly lower accuracy and more tuning.

Always capture per-word confidence scores from the OCR engine — you'll need them in Step 4.

Step 3: Extraction — the intelligence layer#

Now feed the OCR text (not the image) to an LLM with a strict output schema. The single biggest upgrade you can make here is schema enforcement: instead of asking for JSON and hoping it's valid, force the model's output to match your schema. Anthropic's Structured Outputs feature (behind the structured-outputs-2025-11-13 beta header as of late 2025) and OpenAI's structured outputs both guarantee the response parses as JSON conforming to your schema. A simple prompt becomes:

System: You are a precise invoice data extractor. Return ONLY valid JSON.
Do not hallucinate fields. If a field is not present, use null.

User: Extract these fields from the invoice text below:
- invoice_number (string)
- vendor_name (string)
- invoice_date (YYYY-MM-DD)
- due_date (YYYY-MM-DD)
- line_items (array of {description, quantity, unit_price})
- subtotal, tax, total_amount (numbers)

Invoice text:
{ocr_text}

Keep temperature low (0–0.2) for extraction tasks. Since you're sending text rather than images, you can also use cheaper, smaller models (like GPT-4.1 Mini-class models) for simple invoice formats, and long-context models handle large multi-page documents without losing track of fields.

Step 4: Validation — where pipelines are won or lost#

Even with perfect OCR and schema enforcement, extraction can still be semantically wrong: a tax rate misread, the shipping line picked as the total. Never write LLM output directly to your database. Build a validation layer with:

• Type validation with Pydantic. Model every field with types, required keys, and value ranges. This catches the malformed or out-of-range outputs that slip through.

from pydantic import BaseModel, field_validator

class LineItem(BaseModel):
    description: str
    quantity: float
    unit_price: float

class Invoice(BaseModel):
    invoice_number: str
    vendor_name: str
    total_amount: float
    line_items: list[LineItem]

    @field_validator("total_amount")
    @classmethod
    def positive(cls, v):
        assert v > 0, "total must be positive"
        return v

• Arithmetic cross-checks. The most powerful validation is also the simplest: subtotal + tax == total_amount (within rounding tolerance), and sum(quantity × unit_price) ≈ subtotal. One community invoice-extraction project pairs these numeric range checks with fuzzy name matching against a master vendor list. Arithmetic fails are a strong signal the model grabbed the wrong number.
• Confidence scoring. Combine OCR word confidence for the fields the model used with the model's own output. Below a threshold (say 0.9), route to human review instead of the database.
• Self-correction loop. When Pydantic validation fails, send the response back to the LLM along with the validation error message and ask it to fix it. Practitioners report this self-correction loop catching the large majority of production errors before they reach the database.

Step 5: Evaluate like it matters#

"Vibes" is not an evaluation metric. The open-source VLM invoice-extraction tutorial series built on the CORD dataset (Naver's receipt/invoice dataset on Hugging Face) measures field-level accuracy, precision, recall, and F1 — not just token-level match. For your pipeline, track:

A reasonable starting set: 100–200 representative invoices covering your ugliest formats. Label them once, re-run on every pipeline change. If your human-review rate is above ~10–15%, your extraction or OCR layer needs work, not more reviewers.

What it costs#

Splitting OCR and intelligence makes costs predictable and independently optimizable. One practitioner's breakdown for 10,000 invoice pages/month: Google Vision (first 1,000 pages free, then ~$1.50 per 1,000) comes to about $13.50/month; the LLM cost stays small because you're sending text, not images — roughly $2 per million input tokens for text-class extraction models. Total: well under $50/month. Going fully local with PaddleOCR plus a self-hosted or on-device model drops the per-document cost to zero, at the price of hardware and tuning time.

Common failure modes and fixes#

• Handwritten or fax-quality scans. OCR accuracy degrades fast. Raise your confidence threshold for these, route more to review, and consider a dedicated handwriting pass.
• Line items spanning pages. Chunking per page loses context. Extract per page, then consolidate with an LLM pass that merges partial results (the docling-graph project uses exactly this extract-then-consolidate pattern).
• The model "helpfully" inventing fields. "Do not hallucinate fields; use null" plus schema enforcement fixes most of this.
• New vendor layouts. This is the whole point of the pipeline over templates — but monitor your review rate per vendor. A vendor that always lands in review may deserve a layout-specific prompt tweak.
• Silent arithmetic drift. Totals that are off by pennies usually mean OCR misread a digit (5 vs 6). Cross-checks catch these; log which field failed so you can improve.

The takeaway#

Document extraction at scale is a systems problem, not a model problem. Render pages as images, let dedicated OCR do the reading, let an LLM with a strict JSON schema do the understanding, and validate everything — types, arithmetic, confidence — before it hits your database. The pieces are all off-the-shelf; the craft is in the validation loop and the evaluation set. Build those two well, and messy PDFs become clean records at a cost of well under a cent per invoice.