OCR and document extraction: what developers should know

OCR and document extraction: what developers should know

Every day, millions of receipts, invoices, and forms flow through business systems—but most arrive as unstructured images or PDFs. Getting that data into your database cleanly is harder than it looks. OCR and document extraction is not a solved problem; it's a spectrum of tools, each with trade-offs around accuracy, cost, and complexity. In this guide, we'll walk through the practical approaches we've used across e-commerce, travel, and SaaS—from Tesseract to cloud AI to vision LLMs—and show you how to validate the output so you don't ship garbage data.

Classic OCR with Tesseract: the open-source baseline

Tesseract is the industry workhorse for text extraction. It's been around for decades, it's free, and it runs anywhere. For us, that means predictable costs and no cloud vendor lock-in. The catch is that Tesseract is a pure OCR engine; it doesn't understand structure or semantics. It reads text, period.

Setup is straightforward. On most systems, you install the Tesseract binary and call it from Python or Node.js:

import pytesseract
from PIL import Image

# Load and extract text from an image
image = Image.open('receipt.jpg')
extracted_text = pytesseract.image_to_string(image)
print(extracted_text)

For a clean, well-lit receipt, you might get 95%+ accuracy. But real-world receipts are crumpled, faded, or photographed at odd angles. Preprocessing helps—rotate, contrast-enhance, deskew:

import cv2
import numpy as np

# Load and preprocess
img = cv2.imread('receipt.jpg', cv2.IMREAD_GRAYSCALE)

# Threshold to binary
_, binary = cv2.threshold(img, 150, 255, cv2.THRESH_BINARY)

# Denoise
denoised = cv2.medianBlur(binary, 5)

# Extract text
text = pytesseract.image_to_string(denoised)
print(text)

We've found that spending 30 minutes tuning preprocessing can jump accuracy from 70% to 85% for a specific document type. The trap is over-tuning for one style; if your input varies widely, you'll need multiple pipelines or a more robust approach.

Tesseract also provides bounding box data, which is useful for layout reconstruction:

import pytesseract

# Get detailed output including coordinates
data = pytesseract.image_to_data(Image.open('receipt.jpg'), output_type=pytesseract.Output.DICT)

# data['text'], data['left'], data['top'], data['width'], data['height']
for i, text in enumerate(data['text']):
    if text.strip():
        print(f"{text}: ({data['left'][i]}, {data['top'][i]})")

Use this for grouping words into lines, identifying headers, or detecting columns—but be aware that coordinates alone won't tell you semantic meaning. A date and an amount might have similar positioning; you need parsing logic downstream.

Cloud document AI for forms and invoices

When you need to extract structured fields—invoice number, total, line items, payment terms—cloud providers offer pre-trained models that cut through the layout mess. Google Cloud Document AI, AWS Textract, and Azure Form Recognizer all excel here.

The advantage is obvious: these models have seen millions of invoices and receipts. They understand that "Total" is a label and the number to its right or below is the value. They handle tables, multi-column layouts, and handwriting. The trade-off is cost (typically $0.01–$0.10 per page) and latency (a few seconds round-trip).

Here's a practical example using Google Document AI to extract invoice fields:

from google.cloud import documentai

# Initialize the client
client = documentai.DocumentProcessorServiceClient()
project_id = "my-project"
processor_id = "invoice-processor-id"
processor_name = client.processor_path(project_id, "us", processor_id)

# Load and process a PDF
with open('invoice.pdf', 'rb') as f:
    pdf_content = f.read()

raw_document = documentai.RawDocument(content=pdf_content, mime_type='application/pdf')
request = documentai.ProcessRequest(name=processor_name, raw_document=raw_document)
result = client.process_document(request=request)
document = result.document

# Extract entities
for entity in document.entities:
    field_name = entity.type_
    field_value = entity.text_anchor.content if entity.text_anchor else ""
    confidence = entity.confidence
    print(f"{field_name}: {field_value} ({confidence:.2%} confidence)")

One thing we learned the hard way: confidence scores vary wildly between cloud providers and document types. A 95% confidence score from one model might mean something completely different from another. Always sample outputs and validate against ground truth before deploying.

Also, cloud APIs typically work best on clean, well-lit images. If you're processing user-uploaded smartphone photos, preprocess locally with Tesseract first or run a simple quality check to reject blurry or rotated images.

Vision LLMs for messy layouts and edge cases

When your documents don't fit standard templates—handwritten forms, scanned historical documents, or unusual layouts—vision LLMs like GPT-4 Vision or Claude 3 can be remarkably effective. They're not trained on your specific form, but they understand context and can reason about relationships between text elements.

The cost is higher than cloud document AI ($0.01–$0.05 per image for GPT-4V), and latency is slower, but the accuracy on truly messy cases is often better. We've used this approach for travel receipts with logos, handwriting, and multi-language text.

Here's a practical pattern: define a JSON schema for what you want to extract, pass the image and schema to the LLM, and parse the response:

import anthropic
import base64
import json

client = anthropic.Anthropic()

# Load image and encode as base64
with open('receipt.jpg', 'rb') as f:
    image_data = base64.standard_b64encode(f.read()).decode('utf-8')

# Define extraction schema
schema = {
    "type": "object",
    "properties": {
        "merchant_name": {"type": "string"},
        "date": {"type": "string", "format": "YYYY-MM-DD"},
        "total_amount": {"type": "number"},
        "currency": {"type": "string"},
        "line_items": {
            "type": "array",
            "items": {
                "type": "object",
                "properties": {
                    "description": {"type": "string"},
                    "quantity": {"type": "number"},
                    "unit_price": {"type": "number"}
                }
            }
        }
    }
}

# Call the LLM
message = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=1024,
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image",
                    "source": {
                        "type": "base64",
                        "media_type": "image/jpeg",
                        "data": image_data,
                    },
                },
                {
                    "type": "text",
                    "text": f"Extract receipt data and return valid JSON matching this schema: {json.dumps(schema)}"
                }
            ],
        }
    ],
)

# Parse response
try:
    extracted = json.loads(message.content[0].text)
    print(extracted)
except json.JSONDecodeError:
    print("LLM returned invalid JSON; manual review needed")

One critical detail: LLMs are non-deterministic. Ask the same question twice and you might get slightly different answers. For business-critical data, always add a round-trip validation step (see next section).

Validate extracted totals and dates

Raw extraction output is worthless without validation. You need to check that data is coherent before storing or using it. Build a validation layer that catches common errors:

Date parsing and normalization. A receipt might say "12/25/23" or "25.12.2023" or "25 Dec 2023". Parse flexibly and normalize to ISO 8601:

from dateutil import parser
from datetime import datetime

def parse_receipt_date(date_str):
    try:
        # dateutil is forgiving; it handles most common formats
        dt = parser.parse(date_str)
        return dt.strftime('%Y-%m-%d')
    except:
        return None

# Test with various formats
print(parse_receipt_date("12/25/23"))      # 2023-12-25
print(parse_receipt_date("25.12.2023"))    # 2023-12-25
print(parse_receipt_date("25 Dec 2023"))   # 2023-12-25

Amount parsing and rounding. Extracted amounts might be "USD 123.456789" or "123,45" (European format). Strip currency symbols, handle locales, and round to cents:

import re
from decimal import Decimal

def parse_amount(amount_str):
    # Remove common currency symbols and whitespace
    cleaned = re.sub(r'[A-Z$€£¥\s]', '', amount_str.upper())
    
    # Handle European decimal separator (comma)
    if cleaned.count(',') == 1 and cleaned.count('.') == 0:
        cleaned = cleaned.replace(',', '.')
    elif cleaned.count(',') > 1:
        # Comma as thousands separator; remove it
        cleaned = cleaned.replace(',', '')
    
    try:
        amount = Decimal(cleaned).quantize(Decimal('0.01'))
        return float(amount)
    except:
        return None

# Test
print(parse_amount("USD 123.45"))        # 123.45
print(parse_amount("€ 123,45"))          # 123.45
print(parse_amount("123,456.78"))        # 123456.78

Sanity checks for line items. If you extracted line items, verify that they sum to the total (within rounding error):

def validate_receipt(extracted_data):
    errors = []
    
    # Check date is recent and not in future
    receipt_date = datetime.strptime(extracted_data['date'], '%Y-%m-%d').date()
    if receipt_date > datetime.now().date():
        errors.append(f"Receipt date {receipt_date} is in the future")
    if (datetime.now().date() - receipt_date).days > 365:
        errors.append(f"Receipt date {receipt_date} is over a year old")
    
    # Check total matches line item sum
    line_total = sum(item['quantity'] * item['unit_price'] 
                     for item in extracted_data.get('line_items', []))
    if abs(line_total - extracted_data['total_amount']) > 0.05:
        errors.append(
            f"Line item sum {line_total} doesn't match total {extracted_data['total_amount']}"
        )
    
    # Check currency code is valid (ISO 4217)
    valid_currencies = {'USD', 'EUR', 'GBP', 'JPY', 'AUD', 'CAD'}  # Simplified
    if extracted_data.get('currency') not in valid_currencies:
        errors.append(f"Unknown currency: {extracted_data.get('currency')}")
    
    return len(errors) == 0, errors

# Test
data = {
    'date': '2024-01-15',
    'total_amount': 123.45,
    'currency': 'USD',
    'line_items': [
        {'quantity': 2, 'unit_price': 50.00},
        {'quantity': 1, 'unit_price': 23.45}
    ]
}
is_valid, errors = validate_receipt(data)
print(f"Valid: {is_valid}, Errors: {errors}")

Confidence scoring and flagging. If an OCR or LLM confidence score is below a threshold, or if validation catches issues, mark the record for human review instead of silently failing:

def process_receipt(image_path, confidence_threshold=0.85):
    # Extract (using method of choice)
    extracted = extract_with_vision_llm(image_path)
    
    # Validate
    is_valid, errors = validate_receipt(extracted)
    
    # Determine status
    if not is_valid or extracted.get('confidence', 0) < confidence_threshold:
        return {
            'status': 'needs_review',
            'data': extracted,
            'errors': errors,
            'confidence': extracted.get('confidence', 0)
        }
    else:
        return {
            'status': 'approved',
            'data': extracted
        }

Common mistakes

• Trusting Tesseract on low-quality images. Tesseract shines on clean, high-contrast text. On crumpled receipts or faded photocopies, it falls apart. Preprocess or use a cloud/LLM approach instead.
• Ignoring layout context. OCR gives you words, but not structure. A line that reads "Price 99.99" means something different if it's in a table column header versus a line item. Use bounding boxes and spatial relationships.
• Over-relying on confidence scores. High confidence doesn't mean correct. A model can confidently misread "1" as "I" or "0" as "O". Always validate against business rules.
• Batch processing without error handling. Processing 1,000 receipts in a loop without catching failures will leave you stuck. Wrap each document in a try-catch, log errors, and queue failures for later retry or review.
• Storing unvalidated data. Once bad data enters your database, it's expensive to clean up. Validate before insert, period.
• Using the wrong tool for the job. A simple, clean invoice? Use Tesseract or cloud document AI. A messy handwritten form? Use a vision LLM. Picking the wrong approach wastes time and money.

Key takeaways

• Tesseract is free and reliable for clean documents but requires preprocessing and won't extract structure automatically.
• Cloud document AI (Google, AWS, Azure) is trained on standard forms and invoices; use it for structured extraction when cost is not the primary concern.
• Vision LLMs handle messy, non-standard layouts and can reason about context; they cost more and are slower but excel on edge cases.
• Always validate extracted data—parse dates and amounts defensively, check line-item totals, and flag low-confidence extractions for human review.
• Preprocessing (rotate, contrast, denoise) improves Tesseract accuracy significantly; it's worth the effort for high-volume pipelines.
• Combine tools: use Tesseract for quality checks, cloud API for structured forms, and LLMs for exceptions.