Batch vs real-time predictions: best practices explained

We've all felt the pressure: build a model, wrap it in an API, deploy it, and watch the predictions flow in real time. But that approach isn't always the right fit. The question of batch vs real-time predictions: best practices explained cuts to the heart of how we architect ML systems, and the answer depends entirely on your data constraints, latency tolerance, and operational budget.

Understanding the fundamental difference

Batch predictions and real-time predictions represent two distinct operational models for scoring data with trained models. In batch processing, we score a large volume of data at scheduled intervals—typically nightly or weekly—and store the results in a database or cache for later retrieval. Real-time prediction, by contrast, scores data on demand when a request arrives, returning results synchronously to the caller.

The choice between these approaches feels binary at first glance, but it's really about matching your infrastructure to your business requirements. A recommendation engine for an e-commerce platform might use batch predictions computed every evening, while a fraud detection system needs real-time scoring as transactions arrive. Most production systems don't choose one exclusively; they use both.

Batch: score everything nightly, store results

Batch prediction is the workhorse of data science teams. You load a trained model, iterate over your entire dataset (or a relevant subset), generate predictions, and persist them to a table or cache. The next morning, your application queries those pre-computed results instead of invoking a model service.

Consider a ride-hailing platform that needs to estimate driver earnings for the upcoming week. Every night at 2 AM, we load our XGBoost model trained on historical trip data, score every active driver account against their recent activity patterns, and write the earnings predictions to PostgreSQL. By 6 AM, the mobile app queries those predictions and displays them to drivers. No API latency, no model service scaling issues.

Here's a typical batch pipeline in Python:

import joblib
import pandas as pd
import psycopg2
from datetime import datetime

# Load the trained model
model = joblib.load('/models/earnings_estimator.pkl')

# Fetch input features for all active drivers
conn = psycopg2.connect("dbname=production user=ml_service")
cursor = conn.cursor()
cursor.execute("""
  SELECT driver_id, trips_last_7d, avg_rating, vehicle_type, 
         miles_driven, acceptance_rate 
  FROM drivers WHERE is_active = true
""")
rows = cursor.fetchall()
df = pd.DataFrame(rows, columns=[
    'driver_id', 'trips_last_7d', 'avg_rating', 'vehicle_type',
    'miles_driven', 'acceptance_rate'
])
cursor.close()

# Generate predictions
predictions = model.predict(df[['trips_last_7d', 'avg_rating', 
                                'miles_driven', 'acceptance_rate']])

# Write results back
df['predicted_earnings'] = predictions
df['prediction_timestamp'] = datetime.utcnow()

cursor = conn.cursor()
for _, row in df.iterrows():
    cursor.execute("""
      INSERT INTO driver_earnings_predictions 
      (driver_id, predicted_earnings, prediction_timestamp)
      VALUES (%s, %s, %s)
      ON CONFLICT (driver_id) DO UPDATE SET
      predicted_earnings = EXCLUDED.predicted_earnings,
      prediction_timestamp = EXCLUDED.prediction_timestamp
    """, (row['driver_id'], row['predicted_earnings'], 
          row['prediction_timestamp']))
conn.commit()
cursor.close()
conn.close()

print(f"Batch complete: {len(df)} predictions written")

This pattern scales beautifully. Process 10 million records in one job. Use distributed compute if needed (Spark, Dask). The model service never experiences peak load from user requests. Costs are predictable: you run compute once per day, then the database handles lightweight lookups for the next 24 hours.

Real-time: score on request

Real-time prediction inverts the flow. Your application receives a request, extracts features from the incoming data, calls a model service (API or local library), and returns the prediction immediately. No pre-computation, no stored results.

Real-time is essential when the input features are only known at request time. In a fraud detection system, we don't know tomorrow's transactions today. When a payment attempt arrives, we assemble features on the fly—transaction amount, merchant category, user's geolocation, historical patterns for that card—and score the transaction immediately. A 50-millisecond delay is acceptable; a 24-hour batch window is not.

Here's a simplified fraud detection endpoint:

from fastapi import FastAPI
from pydantic import BaseModel
import joblib
import numpy as np
from datetime import datetime

app = FastAPI()
model = joblib.load('/models/fraud_classifier.pkl')

class TransactionRequest(BaseModel):
    card_id: str
    amount: float
    merchant_category: str
    user_country: str
    timestamp: str

@app.post("/score-transaction")
async def score_transaction(txn: TransactionRequest):
    # Feature engineering happens at request time
    features = extract_features(
        card_id=txn.card_id,
        amount=txn.amount,
        merchant_category=txn.merchant_category,
        user_country=txn.user_country,
        timestamp=datetime.fromisoformat(txn.timestamp)
    )
    
    # Score immediately
    fraud_probability = model.predict_proba(
        np.array([features])
    )[0, 1]
    
    decision = "BLOCK" if fraud_probability > 0.8 else "APPROVE"
    
    return {
        "decision": decision,
        "fraud_probability": float(fraud_probability),
        "timestamp_utc": datetime.utcnow().isoformat()
    }

def extract_features(card_id, amount, merchant_category, 
                     user_country, timestamp):
    # Query recent transaction history for this card
    recent_txns = get_card_history(card_id, days=7)
    avg_txn_amount = np.mean([t['amount'] for t in recent_txns])
    
    # Build feature vector
    return [
        amount,
        amount / (avg_txn_amount + 1),  # Amount ratio
        1 if merchant_category == "HIGH_RISK" else 0,
        1 if user_country == "US" else 0,
        len(recent_txns),  # Transaction count last 7 days
    ]

Real-time prediction demands infrastructure investment: load balancing, horizontal scaling, caching, circuit breakers, and monitoring. But it gives you the responsiveness needed for interactive or synchronous workflows.

Batch is cheaper and simpler

From an operational standpoint, batch wins on cost and complexity. You schedule a job, it runs once, and you're done. Compute is bursty and predictable. You don't need auto-scaling groups or health checks. No one gets paged because a model endpoint is down during off-peak hours.

The database read-after-batch-write pattern is trivial: fetch the pre-computed score from a table indexed on your key. A single PostgreSQL instance can handle millions of concurrent lookups. Compare that to a real-time model API: you need at least two instances for redundancy, a load balancer, request queuing, timeout handling, and on-call rotations when inference latency spikes.

We've seen this play out across e-commerce and SaaS systems repeatedly. A product recommendation model? Batch every night, serve from Redis. Customer churn scoring? Batch weekly, write to a normalized column in the users table. Both patterns work beautifully because the inputs (user purchase history, subscription metrics) don't change hourly.

Cost scales linearly with batch volume but is amortized across all lookups. Real-time cost grows with request volume and latency requirements—more replicas, more sophisticated caching, more operational overhead.

Real-time when inputs are only known at request time

The decision tree is actually quite clear: if you can compute the input features before the prediction is needed, batch is simpler. If you cannot, you need real-time.

A lending platform might use batch predictions for existing customers: score your credit risk daily based on public data, open accounts, and transaction history. All of that exists in the system already. But when a new applicant applies, we need real-time scoring because we've never seen them before and we have only their application form.

A content moderation system works in real-time because user-generated content arrives continuously and unpredictably. You can't batch-score comments that haven't been posted yet. You score them as they arrive.

Here's the critical insight: some workflows naturally accommodate latency, and some don't. A subscription renewal email sent at 2 AM that says "based on yesterday's churn prediction, here's a discount" is fine. A user clicking "checkout" and waiting for a real-time fraud decision is also fine, because milliseconds matter, not hours. Choose based on your product requirements, not on what seems "more advanced."

Hybrid architectures

In practice, production systems rarely choose one approach exclusively. A travel booking platform might use batch predictions for hotel recommendations (based on historical search behavior) but real-time predictions for dynamic pricing (based on current inventory and demand). A lending system batches credit risk scoring but scores fraud risk in real-time.

The pattern is simple: batch for stable, pre-computable features; real-time for dynamic, request-specific data.

Common mistakes

• Building real-time infrastructure prematurely. We often engineer for real-time prediction because it sounds more impressive, then discover that batch would have solved the problem more cheaply. Start with batch. Add real-time only when the business demands it.
• Ignoring feature freshness in batch systems. If you batch-score daily but your features are 7 days stale, your predictions degrade. Ensure feature recency matches your prediction window and business tolerance.
• Not handling model updates in real-time systems. Reloading a model mid-request causes issues. Use feature flags or blue-green deployments to roll out new models without downtime.
• Treating batch and real-time as mutually exclusive. Use both. Batch for the stable cases, real-time for the edge cases and novel data.
• Forgetting to monitor prediction staleness. If a batch job fails silently, you'll serve yesterday's predictions tomorrow. Alert on stale predictions; implement fallback logic.

Key takeaways

• Batch prediction is simpler, cheaper, and sufficient whenever inputs are known before the prediction is needed. Score everything nightly; serve from cache or database the next day.
• Real-time prediction is required when input features are only known at request time, or when latency is critical to the user experience (fraud, ride matching, content moderation).
• Batch scales to millions of records with a single scheduled job; real-time scales with infrastructure complexity and per-request overhead.
• Start with batch. Add real-time only when business requirements demand it, not because it sounds more impressive.
• Most production systems use both: batch for predictable, pre-computable scenarios; real-time for dynamic, request-dependent ones.
• If you choose batch, ensure feature freshness and monitor for stale predictions. If you choose real-time, invest in observability, scaling, and deployment safety.