Structured output from LLMs: what developers should know

Why structured output matters in production

Every team that's integrated LLMs into production has faced the same problem: you ask Claude or GPT for JSON, and sometimes you get malformed output, missing fields, or values in the wrong type. Structured output from LLMs: what developers should know starts with recognizing that LLMs are probabilistic—they can hallucinate, deviate from instructions, and produce output that looks right until your parser chokes on it. Building reliable systems means treating LLM responses as untrusted input and constraining their output at the API level before it ever reaches your validation logic.

We've learned this across support ticket triage, data extraction pipelines, and e-commerce product classification. The difference between "ask nicely in a prompt" and "enforce structure at the model level" is the difference between systems that work 85% of the time and systems you can ship to production. Let's walk through how to get JSON you can trust.

Ask for a schema — describe the exact fields and types

The foundation of structured output is clarity. You don't ask an LLM to "extract some data"—you define exactly what fields you need, their names, their types, and what values are allowed. This becomes your schema.

In real work, this might look like a support ticket classifier. Instead of asking the model to "categorize this ticket," you describe a Ticket class with specific fields:

from pydantic import BaseModel
from typing import Literal

class Ticket(BaseModel):
    category: Literal["billing", "bug", "other"]
    urgency: int  # 1-5
    subject: str
    summary: str

When you pass this schema to an LLM API, you're not just asking—you're constraining. The model knows it can only return one of three categories. The urgency must be an integer. The subject and summary must be strings. This constraint reduces the surface area for errors.

A well-written schema also documents itself. Six months from now, a new engineer reading your code sees exactly what the LLM is expected to produce. No guessing. No "I think the urgency might be a string or a number, let me check the logs."

Start by listing what you actually need. In a ride-hailing scenario, you might extract ride details from a driver complaint:

class RideExtraction(BaseModel):
    ride_id: str
    issue_type: Literal["safety", "payment", "navigation", "vehicle", "driver"]
    severity: Literal["low", "medium", "high"]
    resolution_recommended: str

Notice: we don't ask for "all possible fields"—we ask for what the downstream system needs. Fewer fields means fewer points of failure.

Use tool or JSON modes — most APIs support constrained structured output

Modern LLM APIs no longer require you to cross your fingers and parse JSON from plain text. OpenAI's JSON mode, Anthropic's tool_choice parameter, and similar features in other providers enforce structural guarantees at the model level.

With OpenAI's API, you'd pass your schema as a function definition:

import json
from openai import OpenAI

client = OpenAI()

tools = [
    {
        "type": "function",
        "function": {
            "name": "classify_ticket",
            "description": "Classify a support ticket",
            "parameters": {
                "type": "object",
                "properties": {
                    "category": {
                        "type": "string",
                        "enum": ["billing", "bug", "other"]
                    },
                    "urgency": {
                        "type": "integer",
                        "minimum": 1,
                        "maximum": 5
                    },
                    "subject": {"type": "string"},
                    "summary": {"type": "string"}
                },
                "required": ["category", "urgency", "subject", "summary"]
            }
        }
    }
]

response = client.chat.completions.create(
    model="gpt-4",
    messages=[
        {
            "role": "user",
            "content": "Please classify this ticket: Customer reports billing charge appearing twice."
        }
    ],
    tools=tools,
    tool_choice={"type": "function", "name": "classify_ticket"}
)

# Extract the tool call
tool_call = response.choices[0].message.tool_calls[0]
structured_data = json.loads(tool_call.function.arguments)
print(structured_data)

With Anthropic's Claude, you'd use native JSON mode:

import anthropic
import json

client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=1024,
    messages=[
        {
            "role": "user",
            "content": "Classify this ticket as JSON: Customer reports billing charge appearing twice."
        }
    ],
    system="""You are a support ticket classifier. Always respond with valid JSON matching this schema:
{
  "category": "billing" | "bug" | "other",
  "urgency": 1-5,
  "subject": "string",
  "summary": "string"
}""",
)

structured_data = json.loads(response.content[0].text)
print(structured_data)

The key difference from plain prompting: the API guarantees the model will attempt to return valid JSON or a tool call matching your schema. It's not foolproof, but it's vastly more reliable than hoping the model follows instructions in text.

In a trading scenario where you're extracting market signals from news, using structured output means you get consistent, parseable data even under load or with novel news sources.

Validate — parse with Pydantic and retry on failure

Structured output at the API level is a strong constraint, but it's not a guarantee. Networks fail, edge cases slip through, and sometimes the model produces output that's technically valid JSON but doesn't match your business logic. Validation is your second line of defense.

Pydantic makes this straightforward:

from pydantic import BaseModel, ValidationError, Field
from typing import Literal

class Ticket(BaseModel):
    category: Literal["billing", "bug", "other"]
    urgency: int = Field(ge=1, le=5)
    subject: str = Field(min_length=1, max_length=200)
    summary: str

# Assume structured_data came from the LLM
try:
    ticket = Ticket(**structured_data)
    print(f"Valid ticket: {ticket.category}, urgency {ticket.urgency}")
except ValidationError as e:
    print(f"Validation failed: {e}")
    # Retry logic here

Pydantic's ValidationError tells you exactly what failed: urgency might be 6 (out of range), subject might be empty, or category might be a value we didn't expect. This is actionable debugging information.

For production systems, pair validation with retry logic. If the LLM produces invalid output, send it back with the error details:

def classify_ticket_with_retry(ticket_text: str, max_retries: int = 3) -> Ticket:
    for attempt in range(max_retries):
        response = client.chat.completions.create(
            model="gpt-4",
            messages=[
                {
                    "role": "user",
                    "content": f"Classify this ticket: {ticket_text}"
                }
            ],
            tools=tools,
            tool_choice={"type": "function", "name": "classify_ticket"}
        )
        
        try:
            structured_data = json.loads(
                response.choices[0].message.tool_calls[0].function.arguments
            )
            ticket = Ticket(**structured_data)
            return ticket
        except (ValidationError, json.JSONDecodeError, IndexError) as e:
            if attempt == max_retries - 1:
                raise
            # On retry, include the error in the prompt
            ticket_text += f"\n\nPrevious attempt failed: {str(e)}"
    
    raise ValueError("Failed to classify ticket after retries")

We've seen this pattern work well in CRM systems where you're extracting customer intent from support emails. The first pass often works; when it doesn't, telling the model what went wrong usually fixes it on the second try.

Keep it small — fewer fields means fewer errors

There's a temptation to ask the LLM for everything at once. Extract the ticket category, urgency, sentiment, tags, customer segment, product ID, and recommended workflow all in one call. Resist this urge.

Each additional field is another opportunity for the model to:

• Misunderstand what you're asking
• Hallucinate a value that sounds plausible but is wrong
• Fail validation and require a retry

We've found that LLM accuracy degrades measurably as schema complexity grows. A schema with 3-4 fields has near-production reliability. A schema with 10+ fields starts to feel fragile.

Better approach: pipeline your extractions. First, classify the ticket into a category. Then, in a separate call, extract the customer sentiment. Then, in a third call, tag it with relevant products. Each step is a small, focused task:

class TicketClassification(BaseModel):
    category: Literal["billing", "bug", "other"]
    urgency: int = Field(ge=1, le=5)

class TicketSentiment(BaseModel):
    sentiment: Literal["positive", "neutral", "negative"]
    confidence: float = Field(ge=0, le=1)

class TicketTags(BaseModel):
    tags: list[str] = Field(max_items=5)

# Three separate LLM calls, each focused
classification = classify_ticket(ticket_text)
sentiment = analyze_sentiment(ticket_text)
tags = extract_tags(ticket_text, classification.category)

This feels like more API calls, but it's actually more reliable. Each model invocation has a clear, narrow task. Error rates drop. Retry logic becomes simpler. And if one extraction fails, you don't need to re-run the entire pipeline—just retry that step.

In a travel booking system, this means extracting the booking ID separately from extracting the customer complaint separately. Simpler, more reliable, easier to test.

Common mistakes

Mistake 1: Trusting the LLM to remember your schema. Passing your Pydantic class to the LLM and hoping it follows the structure is not enough. Convert your schema to JSON schema explicitly. Make it part of the system prompt or tool definition. Be redundant. The model won't remember.

Mistake 2: Skipping validation because the API returned structured output. Just because OpenAI's API returned a function call doesn't mean the arguments are valid. The model might respect the type constraint (integer) but violate your business constraint (1-5). Always validate.

Mistake 3: Making the schema too permissive. Using string or null for fields that should be enums, or omitting constraints on numbers, invites garbage data. If urgency must be 1-5, enforce it in the schema, not in downstream code.

Mistake 4: Not testing with edge cases. Your schema works great on the ticket examples you tested. But what about very long ticket text? Tickets in languages other than English? Retry the entire pipeline with adversarial examples before shipping.

Mistake 5: Building a monolithic extraction. If your schema has 15 fields, the LLM's error rate goes up exponentially. Split it into smaller, independent extractions, even if it means more API calls. Reliability trumps efficiency here.

Key takeaways

• Define your schema explicitly, using Pydantic or JSON schema. Be specific about types, constraints, and allowed values.
• Use tool_choice, JSON mode, or similar API constraints to enforce structure at the model level. Structured output APIs are more reliable than plain-text prompting.
• Always validate LLM output before using it, even if the API returned structured data. Pydantic makes this painless; ValidationError tells you exactly what failed.
• Pipeline complex extractions into multiple smaller calls. A focused extraction is more reliable and easier to debug than a monolithic one.
• When validation fails, retry with the error details included in the prompt. The model usually corrects itself on the second pass.
• Test your schemas with edge cases, long text, multiple languages, and adversarial input before deploying to production.