Deploying ML models with FastAPI for AI & ML developers

Deploying ML models with FastAPI for AI & ML developers

Once you've trained a model that performs well in your notebook, the real work begins: getting predictions to the systems that need them. Serving machine learning models over HTTP using FastAPI has become the standard approach for teams shipping AI in production, because it's fast, type-safe, and forces you to think about versioning and input validation from day one. In this guide, we'll walk through the practical patterns we've learned deploying classification, regression, and ranking models across e-commerce, CRM, and SaaS platforms.

Why FastAPI for model serving?

Before we dive into the patterns, let's be clear about why FastAPI stands out. It validates request bodies automatically using Pydantic, generates OpenAPI documentation without extra work, handles async requests efficiently, and runs behind Uvicorn with near-native performance. Most importantly, it keeps you honest: the gap between training a model on 15 features and accidentally accepting a request with 14 or 16 is caught at the API boundary, not deep in your scoring pipeline.

We've seen teams waste weeks debugging predictions that diverged from validation metrics. The root cause? A feature was added to training but the service contract wasn't updated. FastAPI and Pydantic make that mistake impossible.

Load once — Load the model at startup, not per request

The most common mistake we see is loading the model inside the request handler. On day one it feels fine; six months later when requests are slow and your team is paged at 2 AM, you'll realize the model was being unpickled on every single call.

Instead, load it once when the application starts and hold it in memory:

import pickle
from fastapi import FastAPI
from contextlib import asynccontextmanager

model = None

@asynccontextmanager
async def lifespan(app: FastAPI):
    global model
    with open("model.pkl", "rb") as f:
        model = pickle.load(f)
    print("Model loaded at startup")
    yield
    print("Shutting down")

app = FastAPI(lifespan=lifespan)

@app.post("/predict")
def predict(req: Features):
    proba = model.predict_proba([req.as_row()])[0, 1]
    return {"churn_probability": float(proba)}

This pattern uses FastAPI's lifespan context manager, available in version 0.93+. When the application starts, the context manager runs before the first request arrives. The model stays in memory for the lifetime of the process. If you're using an older version of FastAPI, you can accomplish the same thing with @app.on_event("startup").

In production, we typically run several instances of this service behind a load balancer. Each process loads its own copy of the model into memory. For very large models (multi-gigabyte neural networks), this becomes expensive, and you might consider a dedicated model server like TensorFlow Serving or Seldon, but for models under 500 MB, in-process loading is simpler and lower-latency.

One practical detail: always load from an absolute path or an environment variable, never a relative path. Your production runner might have a different working directory than your development machine.

Validate input — Pydantic models mirror training features

Your model was trained on a specific set of features with specific names and types. The request body must match exactly. This is where Pydantic shines: it validates the shape, types, and sometimes the business logic of incoming requests before they touch your model.

Start by creating a Pydantic model that mirrors your training features:

from pydantic import BaseModel, Field
from datetime import datetime

class Features(BaseModel):
    customer_lifetime_value: float = Field(..., gt=0, description="CLV in USD")
    days_since_last_purchase: int = Field(..., ge=0)
    total_purchases: int = Field(..., ge=1)
    email_opened_rate: float = Field(..., ge=0.0, le=1.0)
    support_tickets: int = Field(..., ge=0)
    joined_date: datetime
    
    def as_row(self) -> list:
        """Convert to feature vector in training order."""
        return [
            self.customer_lifetime_value,
            self.days_since_last_purchase,
            self.total_purchases,
            self.email_opened_rate,
            self.support_tickets,
            (datetime.now() - self.joined_date).days,
        ]

Notice the as_row() method. This is critical: your model expects features in a specific order. It was trained with that order baked in. Converting the request to a list in the exact same order prevents shape mismatches and off-by-one errors that can silently corrupt predictions.

The Field constraints (gt, ge, le) aren't just documentation—Pydantic enforces them. If someone sends customer_lifetime_value: -50, the request fails validation before reaching your model. This saves you from pathological predictions that your training data never saw.

In a real system, you might also add custom validators:

from pydantic import field_validator

class Features(BaseModel):
    customer_lifetime_value: float = Field(..., gt=0)
    days_since_last_purchase: int = Field(..., ge=0)
    
    @field_validator('days_since_last_purchase')
    @classmethod
    def check_recency(cls, v):
        if v > 730:  # Business rule: 2-year silence means data stale
            raise ValueError("Customer data too old")
        return v

This pattern catches business logic errors at the boundary, making your logs and monitoring much clearer. Instead of debugging a weird prediction, you get a clear 422 Unprocessable Entity response with a message the caller can act on.

Return confidence — Probabilities let callers decide thresholds

Many junior ML engineers return only the class prediction: "churn" or "no churn". But different callers have different thresholds. The retention team might want to intervene at 60% churn probability; the data science team might need 80% for statistical significance.

Always return the probability (confidence) alongside the decision:

from pydantic import BaseModel

class PredictionResponse(BaseModel):
    churn_probability: float
    predicted_class: str
    model_version: str
    
@app.post("/predict")
def predict(req: Features) -> PredictionResponse:
    proba = model.predict_proba([req.as_row()])[0, 1]
    predicted_class = "churn" if proba >= 0.5 else "retain"
    
    return PredictionResponse(
        churn_probability=float(proba),
        predicted_class=predicted_class,
        model_version="v2.3.1",
    )

Notice we also return model_version. When you have multiple models in production (which you will), you need to know which one made each prediction. This is essential for debugging if a model degrades or for cohort analysis.

In production, you might batch predictions or add latency tracking:

import time
from typing import List

@app.post("/predict/batch")
def predict_batch(reqs: List[Features]) -> List[PredictionResponse]:
    start = time.time()
    
    rows = [req.as_row() for req in reqs]
    probas = model.predict_proba(rows)[:, 1]
    
    latency_ms = (time.time() - start) * 1000
    
    return [
        PredictionResponse(
            churn_probability=float(p),
            predicted_class="churn" if p >= 0.5 else "retain",
            model_version="v2.3.1",
        )
        for p in probas
    ]

Batch endpoints let callers amortize HTTP overhead. A single request with 100 examples is faster and cheaper than 100 individual requests, and your model's vectorized operations run more efficiently on a larger matrix.

Containerize — Ship the model with its exact library versions

Your model works in your environment today. Tomorrow, NumPy releases a minor version that changes a floating-point operation slightly, and predictions drift by 2%. This is not theoretical; we've lived it.

Package everything—code, model, and dependencies—into a Docker image:

# Dockerfile
FROM python:3.11-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY model.pkl .
COPY app.py .

CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "8000"]

And your requirements.txt must pin exact versions:

fastapi==0.104.1
uvicorn==0.24.0
scikit-learn==1.3.2
pydantic==2.5.0
numpy==1.26.2
pandas==2.1.1

Build the image and push to your container registry. When you deploy, you're guaranteed that the inference code, library versions, and model are all compatible. No surprises at 3 AM.

In practice, we also store the model outside the Docker image and mount it at runtime:

# app.py
import os
model_path = os.getenv("MODEL_PATH", "/models/model.pkl")
with open(model_path, "rb") as f:
    model = pickle.load(f)

This lets you upgrade the model without rebuilding the Docker image. You can run the same image in staging and production, pointing each to their own model version. This is especially useful for A/B testing: two replicas of the same service code, different models, different traffic splits.

Store models in a versioned artifact repository (S3, GCS, Azure Blob) with metadata that ties each model to its training commit and performance metrics. When you need to roll back due to model drift, you can pull a known-good version from a few weeks ago.

Health checks and monitoring

Your service needs to tell orchestrators (Kubernetes, ECS) when it's healthy and ready to receive traffic:

@app.get("/health")
def health():
    return {"status": "ok"}

@app.get("/ready")
def ready():
    if model is None:
        return {"ready": False}, 503
    return {"ready": True, "model_version": "v2.3.1"}

The difference: /health runs frequently (Kubernetes liveness probe), /ready runs once on startup (readiness probe). A healthy container might not be ready if the model is still loading.

Instrument predictions with structured logging:

import logging
import json

logger = logging.getLogger(__name__)

@app.post("/predict")
def predict(req: Features) -> PredictionResponse:
    proba = model.predict_proba([req.as_row()])[0, 1]
    result = PredictionResponse(
        churn_probability=float(proba),
        predicted_class="churn" if proba >= 0.5 else "retain",
        model_version="v2.3.1",
    )
    
    logger.info(json.dumps({
        "event": "prediction",
        "clv": req.customer_lifetime_value,
        "probability": result.churn_probability,
        "class": result.predicted_class,
    }))
    
    return result

Send these logs to a centralized logging system. Aggregate them to track prediction distribution over time. If the mode probability suddenly shifts from 0.3 to 0.7, you have a signal that model performance has drifted and you need to investigate.

Common mistakes

• Loading the model per request: Profiling will show 80% of latency is I/O. Move it to startup.
• Feature order mismatches: Your model expects [clv, days_since, purchases, ...] but you send [days_since, clv, purchases, ...]. Silent corruption. Always convert via a method that documents the order.
• Returning only class predictions: Downstream teams need probabilities to set their own thresholds. Always return confidence scores.
• Forgetting model version in responses: Six months later, you're looking at a prediction and have no idea which model made it. Include version in every response.
• Not validating input bounds: Your model was trained on CLV in [0, 50000]. A request comes in with 5 million. It doesn't error; it predicts nonsense. Use Pydantic validators.
• Unpinned dependencies: numpy==1.26 works great until your container rebuilds and pulls numpy==1.27, and predictions drift slightly. Pin everything.
• No health checks: Your model loads but it's corrupted. The container runs, but predictions are garbage. Implement readiness checks that actually test the model.

Key takeaways

• Load models at startup using lifespan context managers, not per request. Hold them in memory for the lifetime of the process.
• Create Pydantic models that mirror your training features, including validation constraints and a method to convert to the exact row order your model expects.
• Always return probabilities alongside class predictions, and include the model version so callers know what generated each prediction.
• Containerize with pinned dependency versions so you get the same inference behavior in dev, staging, and production.
• Implement /health and /ready endpoints and instrument predictions with structured logging for monitoring and debugging.
• Store models as versioned artifacts outside the Docker image, letting you update models without rebuilding code.
• Batch endpoints save callers network overhead and let your model run vectorized operations on larger matrices.