Feature stores explained: a AI & ML guide for developers

Feature stores explained: a AI & ML guide for developers. Share features between training and serving.

The training-serving gap

You've just deployed a recommendation model to production that performed beautifully in your notebook. Six months later, it's underperforming by 15%. Your team investigates and discovers that the feature logic used during training differs subtly from what runs in your serving pipeline. The model learned patterns that don't generalize to real traffic because the features themselves changed. This is the core problem that feature stores solve—they're the single source of truth for feature logic shared between offline training and online inference.

Same feature logic offline and online

When you compute features separately for training and serving, you're maintaining two codebases that must stay in sync. In practice, they don't. A data engineer might optimize a SQL query for training performance by using a different aggregation window than the one originally planned. A serving engineer might implement the same feature in Go for speed and introduce a subtle rounding difference. Six months later, nobody remembers why they diverged.

A feature store eliminates this duplication by materializing features once and reusing them everywhere. You define how to compute a customer's average purchase value, how many days since their last order, or what products they've viewed. The feature store handles both the batch computation (for training datasets) and the real-time lookup (for live predictions).

Let's say you're building a churn prediction model for an e-commerce platform. Without a feature store, your workflow looks like this:

# Training: compute features in Python/pandas
import pandas as pd

def compute_customer_features_training(customer_ids):
    # Query warehouse, aggregate, compute features
    avg_order_value = query_warehouse(
        "SELECT customer_id, AVG(amount) FROM orders GROUP BY customer_id"
    )
    days_since_purchase = query_warehouse(
        "SELECT customer_id, DATEDIFF(NOW(), MAX(order_date)) FROM orders GROUP BY customer_id"
    )
    return pd.merge(avg_order_value, days_since_purchase, on='customer_id')

# Serving: re-implement features in production code (maybe different language)
def get_customer_features_serving(customer_id):
    # Different query, different aggregation logic
    result = redis_cache.get(f"features:{customer_id}")
    if not result:
        # Fallback to database with slightly different logic
        result = fetch_from_features_table(customer_id)
    return result

With a feature store, both paths call the same logic:

# Feature store definition (single source of truth)
class CustomerFeatures:
    def avg_order_value(self, customer_id):
        return query("SELECT AVG(amount) FROM orders WHERE customer_id = ?", customer_id)
    
    def days_since_purchase(self, customer_id):
        return query("SELECT DATEDIFF(NOW(), MAX(order_date)) FROM orders WHERE customer_id = ?", customer_id)

# Training uses the feature store
features_store = FeatureStore()
training_data = features_store.get_batch(['customer_1', 'customer_2', ...])

# Serving uses the same feature store
predicted_churn = model.predict(features_store.get_online(customer_id))

The feature store handles the infrastructure differences: batch mode materializes features into a data warehouse for training datasets, while online mode serves from a low-latency cache (usually Redis or DynamoDB) for predictions.

Avoid training-serving skew

Training-serving skew occurs when the distribution of features differs between your training data and production traffic. This happens for three reasons: different feature logic, data drift over time, and stale cached features. A feature store addresses all three.

Consider a fraud detection system where you train on historical data to predict whether a transaction is fraudulent. Your training features include "average transaction amount per merchant over the last 30 days." But in serving, your cache of pre-computed features updates every 6 hours. A sudden spike in transactions means the model sees outdated features, and its predictions become unreliable.

The feature store forces consistency by defining the feature once and controlling how it's computed and cached:

# Feature store: unified definition with explicit freshness guarantees
@feature_store.define(
    name="merchant_avg_transaction_amount_30d",
    freshness="1 hour",  # Online features refresh hourly
    lookback_window="30 days"
)
def compute_merchant_avg_amount(merchant_id):
    return query(
        "SELECT AVG(amount) FROM transactions "
        "WHERE merchant_id = ? AND timestamp > NOW() - INTERVAL 30 DAY",
        merchant_id
    )

# Training: explicitly reads historical snapshot
training_features = feature_store.get_training_data(
    entity_ids=fraud_cases,
    timestamp="2024-01-15",  # Consistent point in time
    features=["merchant_avg_transaction_amount_30d"]
)

# Serving: reads fresh data with controlled staleness
online_features = feature_store.get_online(
    entity_id=current_transaction['merchant_id'],
    features=["merchant_avg_transaction_amount_30d"]
)

By making freshness explicit, you either accept the staleness in serving (and train on equally stale features) or invest in infrastructure to keep features fresher. The skew becomes visible and measurable instead of a hidden bug.

Point-in-time correct training data

Building a training dataset without a feature store is deceptively hard. You need to join historical events with the features that existed at that moment in time. Get the join window wrong, and you're training on features that were computed after the label occurred—data leakage.

Imagine a subscription churn model. A customer canceled on January 15. You need their features as they existed on January 14, not January 16. Without careful handling, it's easy to accidentally include their post-cancellation feature state.

Feature stores solve this with explicit point-in-time semantics. You specify an event timestamp and a feature timestamp, and the store retrieves the correct feature state:

# Events with timestamps (when did churn occur?)
events = [
    {"customer_id": 123, "churn": True, "event_timestamp": "2024-01-15 14:30:00"},
    {"customer_id": 456, "churn": False, "event_timestamp": "2024-01-16 09:15:00"},
]

# Feature store materializes features at specific timestamps
training_df = feature_store.get_training_data(
    events=events,
    event_timestamp_col="event_timestamp",
    features=[
        "customer:days_since_purchase",
        "customer:avg_order_value",
        "customer:support_tickets_count"
    ]
)

# Result: each row has features as they existed at event_timestamp
# customer_id | churn | days_since_purchase | avg_order_value | ...
# 123         | True  | 45                  | 87.50           | ...
# 456         | False  | 3                   | 156.20          | ...

The feature store maintains historical feature values (feature versioning) so you can always reconstruct the exact state at any point in time. This eliminates the class of data leakage bugs that plague machine learning.

Useful once many models share features

A feature store isn't necessary when you're building a single model with a small team. You can hard-code feature logic in your training script and serving API. The operational overhead isn't justified.

But the moment you have multiple models (churn, recommendation, fraud, personalization) or multiple teams sharing code, feature stores become invaluable. Each team can define features once and share them:

Without a feature store, each model team maintains their own feature computation. With one, you define avg_order_value once and all three models use it. When you discover a bug in the computation, you fix it once and all models benefit.

A typical feature store platform like Feast or Tecton provides:

• Feature definition registry: A central place to declare what features exist and how to compute them.
• Batch ingestion: Compute features offline and store them in a data warehouse for training.
• Online serving: Low-latency retrieval from a cache (Redis, DynamoDB) for live predictions.
• Feature versioning: Track changes to feature logic and training data reproducibility.
• Monitoring: Alert when feature distributions drift or freshness SLAs are violated.

Common mistakes

Over-engineering too early: Don't introduce a feature store for one model. The operational cost often outweighs the benefit. Wait until you have multiple models or teams sharing feature logic.

Ignoring latency in serving: A feature store query that takes 500ms is useless if your serving SLA is 100ms. Design for the tail latency you need, not the average case.

Not versioning features: When you change how a feature is computed, the old model might perform worse on new data. Version your features explicitly so you can roll back if needed.

Skipping monitoring: A feature store gives you visibility into data quality and distribution drift. If you don't monitor it, you lose that advantage. Track freshness, cardinality, and statistical properties of each feature.

Mixing real-time and batch logic: Compute features only once. If you're computing a feature in batch for training and also in real-time for serving, you'll diverge. Pick the primary source and stick with it.

Key takeaways

• Feature stores eliminate training-serving skew by using identical feature logic for both offline and online paths.
• Point-in-time correct training data requires explicit timestamp handling, preventing data leakage bugs that are easy to miss.
• A shared feature store becomes valuable when multiple models or teams need to reuse feature definitions, reducing maintenance burden.
• Start simple: hard-code feature logic until you hit the pain of duplication across teams, then invest in a feature store.
• Design for the latency your production serving requires, not the query time you measure on a quiet database.