Synthetic data for testing ML: a practical AI & ML guide

Introduction

In production ML systems across e-commerce, fraud detection, and recommendation engines, we often face a critical bottleneck: real training data is either scarce, expensive to label, or too sensitive to share. Synthetic data for testing ML: a practical AI & ML guide explores how to generate realistic datasets that let us validate pipelines, test edge cases, and catch bugs before they reach production. We'll walk through generating synthetic data with LLMs, validating distributions, and—most importantly—understanding when synthetic data is useful and when it isn't.

Why synthetic data matters for pipeline testing

Before we train any model on real data, we need confidence that the entire pipeline works end-to-end. Imagine a ride-hailing platform that processes millions of trip records daily. We can't wait for real data to accumulate before testing the data validation layer, feature engineering, or model serving infrastructure. Synthetic data lets us start testing immediately.

Consider a typical workflow:

• Validate schema and data types
• Test feature transformations and aggregations
• Verify model inference endpoints handle edge cases
• Load-test data pipelines with realistic volume
• Check for data quality issues (nulls, duplicates, outliers)

With synthetic data, we can generate 10 million records in seconds, apply them to our pipeline, and catch schema mismatches or performance bottlenecks before production. A team working on a travel booking platform might generate thousands of fake bookings, user profiles, and payment records to stress-test the feature store and ensure the model serving layer responds within SLA.

The key insight: synthetic data is a testing tool, not a training tool (at least not primarily). It's about validating infrastructure and catching integration bugs early.

Using LLMs to generate realistic text examples

When your model ingests text—customer reviews, chat logs, search queries, product descriptions—you need synthetic examples that feel natural and represent the distribution you expect. Large language models are excellent for this task.

Here's a practical example for generating synthetic customer support tickets:

import anthropic
import json

client = anthropic.Anthropic()

def generate_support_tickets(num_tickets: int = 100, category: str = "billing"):
    """Generate synthetic support tickets for testing a ticket classification model."""
    
    prompt = f"""Generate {num_tickets} realistic customer support ticket titles and descriptions for the '{category}' category.
    
Output as a JSON array where each element has:
- "title": a 5-15 word title
- "body": a 50-150 word description
- "sentiment": one of ["positive", "neutral", "negative"]
- "urgency": one of ["low", "medium", "high"]

Examples of {category} category tickets to match the tone:
- Billing issues, subscription changes, payment failures
- Refund requests, invoice disputes, pricing questions

Be diverse in language, structure, and specific scenarios. Make them realistic enough that a classifier trained on real support tickets would recognize them."""

    message = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=4096,
        messages=[
            {"role": "user", "content": prompt}
        ]
    )
    
    response_text = message.content[0].text
    # Extract JSON from markdown code blocks if present
    if "```json" in response_text:
        response_text = response_text.split("```json")[1].split("```")[0]
    
    tickets = json.loads(response_text)
    return tickets

# Generate 50 billing tickets and 50 technical tickets
billing_tickets = generate_support_tickets(50, "billing")
technical_tickets = generate_support_tickets(50, "technical")

print(f"Generated {len(billing_tickets)} billing tickets")
print(f"Generated {len(technical_tickets)} technical tickets")

# Use these for testing your classification pipeline
for ticket in billing_tickets[:2]:
    print(f"Title: {ticket['title']}")
    print(f"Body: {ticket['body'][:100]}...")
    print(f"Urgency: {ticket['urgency']}")
    print()

This approach generates diverse, contextually accurate text in seconds. Each ticket has natural language variation—different phrasings, complaint styles, and urgency levels. A team building a CRM system can generate hundreds of realistic customer interactions to test intent detection, sentiment analysis, and routing logic without touching actual customer data.

Beyond text, you can generate synthetic data for other modalities:

• Structured data: User profiles, transactions, product catalogs using random sampling and realistic distributions
• Time-series data: Sensor readings, stock prices, website traffic using statistical models or interpolation
• Images: Diffusion models or GANs for computer vision testing (more specialized, but increasingly accessible)

Validating that synthetic data matches real distributions

Generating synthetic data is straightforward; ensuring it's representative is the hard part. Synthetic data that's too clean or unrealistic can give you false confidence that your pipeline works. You might test with perfectly formatted records and miss the fact that production data is 15% null values and malformed timestamps.

Here's how to validate synthetic data against real distributions:

import pandas as pd
import numpy as np
from scipy import stats

def compare_distributions(real_df: pd.DataFrame, synthetic_df: pd.DataFrame, column: str):
    """Compare distributions of a column between real and synthetic data."""
    
    real_values = real_df[column].dropna()
    synthetic_values = synthetic_df[column].dropna()
    
    # For numerical columns, use KS test
    if pd.api.types.is_numeric_dtype(real_values):
        statistic, p_value = stats.ks_2samp(real_values, synthetic_values)
        print(f"\n{column} (numerical)")
        print(f"  Real: mean={real_values.mean():.2f}, std={real_values.std():.2f}")
        print(f"  Synthetic: mean={synthetic_values.mean():.2f}, std={synthetic_values.std():.2f}")
        print(f"  KS test p-value: {p_value:.4f} (>0.05 suggests similar distributions)")
        return p_value > 0.05
    
    # For categorical columns, use chi-squared test
    else:
        real_counts = real_values.value_counts(normalize=True)
        synthetic_counts = synthetic_values.value_counts(normalize=True)
        
        # Align categories
        all_categories = set(real_counts.index) | set(synthetic_counts.index)
        real_counts = real_counts.reindex(all_categories, fill_value=0)
        synthetic_counts = synthetic_counts.reindex(all_categories, fill_value=0)
        
        # Chi-squared test
        chi2, p_value = stats.chisquare(
            f_exp=synthetic_counts * len(synthetic_values),
            f_obs=real_counts * len(real_values)
        )
        
        print(f"\n{column} (categorical)")
        print(f"  Real categories: {dict(real_counts.head())}")
        print(f"  Synthetic categories: {dict(synthetic_counts.head())}")
        print(f"  Chi-squared p-value: {p_value:.4f}")
        return p_value > 0.05

# Load your real training data (sanitized/anonymized)
real_data = pd.read_csv("real_transactions.csv")

# Load or generate synthetic data
synthetic_data = pd.read_csv("synthetic_transactions.csv")

# Validate key columns
columns_to_check = ["amount", "merchant_category", "country", "hour_of_day"]
all_pass = all(
    compare_distributions(real_data, synthetic_data, col)
    for col in columns_to_check
)

if all_pass:
    print("\n✓ Synthetic data distributions match real data")
else:
    print("\n✗ Regenerate synthetic data—distributions don't match")

This validation catches common pitfalls:

• Missing edge cases: Your synthetic data lacks fraud patterns, extreme outliers, or seasonal trends
• Skewed distributions: LLM-generated text might overrepresent polite language and underrepresent slang or misspellings
• Correlation loss: Features that should be correlated in reality (zip code and currency) might be independent in synthetic data

For a SaaS analytics platform, you'd validate that synthetic user session data has realistic session lengths, contains the right ratio of returning vs. new users, and preserves temporal patterns (e.g., traffic spikes on weekdays). If synthetic data passes distribution tests, you can trust it for infrastructure testing.

Never evaluate final models only on synthetic data

This is the critical guardrail: synthetic data is for testing, not for validation or evaluation. A model that performs perfectly on synthetic data can fail spectacularly on real data because of distribution shift, edge cases, or adversarial inputs that the synthetic data doesn't capture.

Here's why this matters in practice:

• Overfitting to synthetic patterns: If you tune hyperparameters on synthetic data, you're optimizing for the generator's biases, not real-world signals
• Missing real-world noise: Real transactions have typos, delays, incomplete information, and adversarial behavior that synthetic data might sanitize away
• False confidence: An 98% accuracy score on synthetic data doesn't guarantee 80% on real data

The correct workflow is:

1. Development: Use synthetic data to test pipelines, validate schema, stress-test infrastructure
2. Baseline training: Train models on real historical data (or a mix of real + synthetic if real data is very scarce)
3. Validation: Evaluate on held-out real data, never on synthetic
4. Testing: Use synthetic data to test specific edge cases after validation on real data confirms the model is viable

For a fraud detection model in a fintech platform, you might generate synthetic legitimate transactions to test your pipeline handles volume, but you absolutely validate model performance on real historical fraud labels. Conversely, you could generate synthetic fraud patterns (if they're rare) to augment training, but you'd still evaluate on real fraud data to ensure the model generalizes.

Common mistakes

• Skipping distribution validation: Assuming synthetic data is "good enough" without testing it statistically. Always compare before using.
• Using synthetic data for model selection: Tuning hyperparameters or choosing between models based only on synthetic data leads to overfitting to the generator.
• Not including realistic failures: Synthetic data often omits null values, timeouts, rate limits, and other operational realities that will hit production.
• Generating too little data: 100 synthetic records won't catch performance issues that appear at 100K records. Generate at least as much synthetic data as you'll see in your first week of production.
• Forgetting temporal aspects: If your real data has seasonal patterns or trends, synthetic data should too. A static synthetic dataset doesn't simulate time-series behavior.

Key takeaways

• Synthetic data accelerates pipeline testing and infrastructure validation without waiting for real data to accumulate.
• LLMs can generate realistic, diverse text examples in seconds; use them to test NLP pipelines before touching customer data.
• Always validate synthetic data against real distributions using statistical tests (KS test for numerical, chi-squared for categorical) to catch representativeness gaps.
• Reserve synthetic data for testing and development; never evaluate final model performance only on synthetic data—always use real held-out data for validation.
• Treat synthetic data as a complement to real data, not a replacement. The strongest pipelines use synthetic data early (infrastructure testing) and real data for decisions that matter (model evaluation).