Named entity recognition: a AI & ML guide for developers

Introduction

Every day, your applications process millions of text messages, support tickets, product reviews, and customer inquiries. Manually labeling who said what, which products are mentioned, or which locations matter becomes impossible at scale. Named entity recognition: a AI & ML guide for developers will walk you through extracting names, places and products from text programmatically, using tools that range from battle-tested pretrained models to custom fine-tuned systems and LLMs.

This is Day 94 of the 100-day Practical AI & Machine Learning series. We're past the foundation phase; now we're solving real extraction problems that power search, recommendation systems, compliance workflows, and customer insights pipelines.

What is Named Entity Recognition?

Named entity recognition (NER) is the task of identifying and classifying named entities—people, organizations, locations, products, monetary values, dates—within unstructured text. Instead of asking a machine learning model to classify an entire document ("is this email about a refund?"), NER asks it to pinpoint *which tokens* in that email refer to entities and *what type* they are.

The output is typically a sequence of tokens with their entity labels. For example, given the text "John Smith at Acme Corp requested a refund for Product X on January 15th," NER extracts:

• John Smith → PERSON
• Acme Corp → ORG
• Product X → PRODUCT
• January 15th → DATE

This structured extraction enables downstream systems to reason about what matters in unstructured text.

Pretrained NER Models from spaCy

spaCy's pretrained models are the starting point for most teams. They ship with statistical NER pipelines trained on standard corpora and handle common entity types well out of the box. No GPUs required, and latency is measured in milliseconds.

Here's how to get started:

import spacy

# Load a pretrained English model
nlp = spacy.load("en_core_web_sm")

text = "Sarah Chen from Microsoft launched a new product in Seattle on March 10th."
doc = nlp(text)

for ent in doc.ents:
    print(f"{ent.text} → {ent.label_}")

Output:

Sarah Chen → PERSON
Microsoft → ORG
Seattle → GPE
March 10th → DATE

spaCy's pretrained models use a statistical approach (not deep learning) to assign entity labels. The advantage is speed and predictability; the trade-off is that accuracy is capped around 85–90% on standard benchmarks. If your domain is e-commerce, finance, or biotech, the model will struggle with domain-specific entity types.

For production use, wrap it in a simple function that handles edge cases:

def extract_entities(text: str, entity_types: set = None):
    """Extract entities, optionally filtered by type."""
    doc = nlp(text)
    entities = []
    for ent in doc.ents:
        if entity_types is None or ent.label_ in entity_types:
            entities.append({
                "text": ent.text,
                "label": ent.label_,
                "start": ent.start_char,
                "end": ent.end_char
            })
    return entities

# Usage
result = extract_entities(
    "John Smith bought 500 shares at Nasdaq.",
    entity_types={"PERSON", "ORG"}
)
print(result)

Pretrained NER Models from Hugging Face

When you need higher accuracy or specialized entity types, Hugging Face transformers offer models trained on larger, more diverse datasets. These are typically deep learning models (BERT-based or similar), so they require more compute but deliver better results, especially on complex text.

from transformers import pipeline

# Load a pretrained NER pipeline
ner = pipeline("ner", model="dbmdz/bert-base-cased-finetuned-conll03-english")

text = "John Smith works at Apple in Cupertino."
results = ner(text)

for entity in results:
    print(f"{entity['word']} → {entity['entity']}")

Hugging Face models often fragment tokens (due to subword tokenization), so you'll see output like:

John → B-PER
Smith → I-PER
Apple → B-ORG
Cupertino → B-LOC

The B- and I- prefixes indicate "beginning" and "inside" tags for multi-token entities. Clean this up before downstream use:

from transformers import pipeline

def clean_ner_output(raw_results: list):
    """Merge subword tokens into complete entities."""
    entities = []
    current_entity = None
    
    for item in raw_results:
        label = item["entity"].replace("B-", "").replace("I-", "")
        is_beginning = item["entity"].startswith("B-")
        
        if is_beginning and current_entity:
            entities.append(current_entity)
            current_entity = {"text": item["word"], "label": label}
        elif is_beginning:
            current_entity = {"text": item["word"], "label": label}
        elif current_entity and current_entity["label"] == label:
            current_entity["text"] += item["word"].replace("##", "")
    
    if current_entity:
        entities.append(current_entity)
    
    return entities

Fine-tuning NER for Custom Entities

Pretrained models work, but they recognize only standard entity types. If your domain needs extraction of SKUs, deal stages, disease codes, or custom jargon, fine-tuning is necessary.

Real-world scenario: you're building a CRM system that must extract deal data from sales emails. Standard PERSON and ORG tags aren't enough; you need DEAL_STAGE, PRODUCT_CATEGORY, and DEAL_VALUE.

Gather labeled training data first. Create a dataset where you've manually annotated examples:

# Example training data in spaCy format
training_data = [
    ("We closed a $50K enterprise deal for Sales Cloud.", 
     {"entities": [(10, 14, "DEAL_VALUE"), (15, 24, "DEAL_STAGE"), (29, 39, "PRODUCT_CATEGORY")]}),
    ("Lost a $10K SMB opportunity in Data Analytics.",
     {"entities": [(8, 11, "DEAL_VALUE"), (12, 16, "DEAL_STAGE"), (32, 45, "PRODUCT_CATEGORY")]})
]

Then fine-tune spaCy's model:

import spacy
from spacy.training import Example

nlp = spacy.load("en_core_web_sm")
ner = nlp.get_pipe("ner")

# Add custom entity labels
ner.add_label("DEAL_STAGE")
ner.add_label("PRODUCT_CATEGORY")
ner.add_label("DEAL_VALUE")

# Training loop (simplified)
optimizer = nlp.create_optimizer()

for epoch in range(30):
    losses = {}
    for text, annotations in training_data:
        doc = nlp.make_doc(text)
        example = Example.from_dict(doc, annotations)
        nlp.update([example], sgd=optimizer, drop=0.5, losses=losses)
    print(f"Epoch {epoch}: {losses['ner']:.4f}")

nlp.to_disk("./custom_ner_model")

After fine-tuning, your model recognizes domain-specific entities with better precision. Expect 50–100 labeled examples to see meaningful improvement; 500+ for robust results.

Using LLMs for Flexible Extraction

For highly unstructured or variable text, LLMs offer flexibility that static NER models can't match. They understand context, handle typos, and can extract entities that don't fit predetermined schemas.

Use case: extracting variable data from support tickets that describe products, issues, and customer frustration levels in free-form language.

from openai import OpenAI

client = OpenAI()

def extract_entities_with_llm(text: str, entity_types: list):
    """Use LLM to extract entities with flexible schemas."""
    prompt = f"""Extract the following entity types from the text below.
Return JSON only, no explanation.

Entity types: {', '.join(entity_types)}

Text: {text}"""
    
    response = client.chat.completions.create(
        model="gpt-4-turbo",
        messages=[{"role": "user", "content": prompt}],
        temperature=0
    )
    
    import json
    return json.loads(response.choices[0].message.content)

text = "Customer John is absolutely furious about the billing bug in our API."
result = extract_entities_with_llm(text, ["PERSON", "ISSUE", "SENTIMENT"])
print(result)

Expected output:

{
  "PERSON": ["John"],
  "ISSUE": ["billing bug"],
  "SENTIMENT": ["furious"]
}

LLMs excel at semantic understanding but are slower and costlier than statistical models. For production systems, use LLMs to extract complex or novel entity types, and use spaCy for high-volume standard extraction.

Validating Extracted Values

Extraction is only as good as your validation. A model might identify a string as a PERSON when it's actually a job title, or extract a PHONE_NUMBER that's malformed. Always validate before storing or acting on extracted data.

Build a validation layer:

import re
from typing import dict

def validate_entity(text: str, label: str) -> bool:
    """Check if extracted entity conforms to expected format."""
    if label == "EMAIL":
        return bool(re.match(r"^[\w\.-]+@[\w\.-]+\.\w+$", text))
    elif label == "PHONE":
        return bool(re.match(r"^\+?1?\d{10,}$", text.replace("-", "").replace(" ", "")))
    elif label == "DATE":
        try:
            from dateutil import parser
            parser.parse(text)
            return True
        except:
            return False
    elif label == "CURRENCY":
        return bool(re.match(r"^\$?\d+(?:,\d{3})*(?:\.\d{2})?$", text))
    return True  # Permissive default for PERSON, ORG, etc.

def extract_and_validate(text: str, model_type: str = "spacy"):
    """Extract entities and filter by validation."""
    if model_type == "spacy":
        doc = nlp(text)
        entities = doc.ents
    else:
        ner = pipeline("ner", model="dbmdz/bert-base-cased-finetuned-conll03-english")
        entities = ner(text)
    
    validated = []
    for ent in entities:
        entity_text = ent.text if hasattr(ent, 'text') else ent['word']
        entity_label = ent.label_ if hasattr(ent, 'label_') else ent['entity']
        
        if validate_entity(entity_text, entity_label):
            validated.append({"text": entity_text, "label": entity_label})
    
    return validated

This prevents downstream systems from acting on malformed or obviously incorrect extractions.

Common Mistakes

• Trusting extraction without validation: NER is probabilistic. Always validate that extracted entities match your expected format or business rules before storing or acting on them.
• Using pretrained models on domain-specific text: A general model trained on Wikipedia and news won't recognize product SKUs, medical codes, or internal jargon. Fine-tune or use LLMs for specialized domains.
• Ignoring case sensitivity and normalization: "John Smith", "john smith", and "JOHN SMITH" are the same person. Normalize text before extraction or after, but be consistent.
• Not measuring performance: Extract entities on a held-out test set, compute precision and recall, and track degradation over time as your data distribution shifts.
• Over-relying on LLMs for high-volume extraction: LLMs are expensive and slower. Use statistical models for routine extraction, and reserve LLMs for edge cases or novel entity types.

Key Takeaways

• Start with spaCy for speed and simplicity on standard entity types; switch to Hugging Face transformers when you need higher accuracy or domain adaptation.
• Fine-tune pretrained models on your domain-specific data to recognize custom entity types; 50–100 labeled examples often suffice for measurable improvement.
• Use LLMs for flexible, unstructured extraction when entity types are variable or semantic understanding is critical; they're powerful but expensive.
• Always validate extracted entities against expected formats, regex patterns, and business rules before downstream use.
• Measure performance (precision, recall, F1) on a held-out test set and monitor for data drift in production.