Experiment tracking with MLflow: lessons from production

Introduction: why experiment tracking matters

Six months into a production recommendation system, your team discovers that last month's model performed better on a key segment than the current deployment—but nobody can remember which hyperparameters were used or why the switch was made. Experiment tracking with MLflow: lessons from production is the difference between having a reproducible history of your work and living in a fog of uncertainty. Without proper tracking, you're essentially guessing when you need to roll back, compare approaches, or explain model behavior to stakeholders.

MLflow solves this by capturing every decision: parameters you tuned, metrics you measured, and artifacts you generated. Over dozens or hundreds of experiments, this becomes your source of truth—what ran, what worked, and why it mattered.

Log parameters, metrics and artifacts

Every experiment is a combination of three things: what you tried (parameters), how well it worked (metrics), and what it produced (artifacts). MLflow gives you a structured way to record all three so that later—whether it's next week or next quarter—you can answer the question "what was I actually doing?"

Parameters are the knobs you turn: learning rate, batch size, tree depth, regularization strength. Metrics are the numbers that matter: accuracy, F1, AUC, precision, recall. Artifacts are the files: your serialized model, training plots, evaluation reports, even raw predictions for debugging.

Here's a practical example from a production classification task:

import mlflow
import mlflow.sklearn
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, precision_score, recall_score
from datetime import datetime

# Start a new MLflow experiment run
mlflow.set_experiment("customer_churn_v2")
with mlflow.start_run():
    # Log parameters
    params = {
        "n_estimators": 100,
        "max_depth": 15,
        "min_samples_split": 10,
        "random_state": 42,
        "timestamp": datetime.now().isoformat()
    }
    for key, value in params.items():
        mlflow.log_param(key, value)
    
    # Train model
    model = RandomForestClassifier(
        n_estimators=params["n_estimators"],
        max_depth=params["max_depth"],
        min_samples_split=params["min_samples_split"],
        random_state=params["random_state"]
    )
    model.fit(X_train, y_train)
    
    # Log metrics
    y_pred = model.predict(X_test)
    accuracy = accuracy_score(y_test, y_pred)
    precision = precision_score(y_test, y_pred)
    recall = recall_score(y_test, y_pred)
    
    mlflow.log_metric("accuracy", accuracy)
    mlflow.log_metric("precision", precision)
    mlflow.log_metric("recall", recall)
    
    # Log artifacts
    import matplotlib.pyplot as plt
    from sklearn.metrics import confusion_matrix
    import seaborn as sns
    
    cm = confusion_matrix(y_test, y_pred)
    plt.figure(figsize=(8, 6))
    sns.heatmap(cm, annot=True, fmt='d')
    plt.savefig('/tmp/confusion_matrix.png')
    mlflow.log_artifact('/tmp/confusion_matrix.png')
    
    # Log the model itself
    mlflow.sklearn.log_model(model, "churn_classifier")

When this run completes, MLflow stores everything in a single experiment record. You've created a complete snapshot of one experiment. Multiply this by 50 iterations—different hyperparameters, different feature engineering approaches, different train-test splits—and suddenly you have a searchable lab notebook.

Compare runs side by side

The real power emerges when you need to choose between approaches. Did increasing max_depth improve generalization or just overfit? Was the boost in recall worth the precision drop? MLflow's UI makes these comparisons immediate and visual.

You can view runs in a parallel coordinates plot, showing how different parameter choices correlate with performance metrics. You can filter by metric ranges ("show me runs where F1 is above 0.82") or by parameter values ("show me what happened with learning_rate between 0.001 and 0.01"). You can download comparison tables as CSV for sharing with non-technical stakeholders.

In a real scenario: your team ran 12 experiments with different feature combinations. Run #7 had the highest accuracy but took 45 minutes to train. Run #11 had slightly lower accuracy but trained in 8 minutes. Run #4 had excellent precision but poor recall. Using MLflow's UI, you can pull up all three side by side, see exactly which features were included in each, and make a principled decision about which trade-off is right for your use case.

To access this locally, run:

mlflow ui

This opens a web interface on localhost:5000 where you can explore all experiments, drill into individual runs, and see parameter-metric correlations.

Register the best model

Once you've found a winner—whether through accuracy, inference speed, or business impact—you need to formalize it. MLflow's Model Registry lets you do this: mark a run's model as a candidate, then promote it to staging, then to production. This creates a clear lineage and audit trail.

The registry handles versioning automatically. If you later register a new model from a different run, it gets version 2. You can see exactly when each transition happened, who made it (if integrated with your auth system), and what metrics justified the change.

Here's how to register a model from an existing run:

from mlflow.tracking import MlflowClient

client = MlflowClient()

# Assume we know the run_id of our best experiment
run_id = "abc123def456"
model_uri = f"runs:/{run_id}/churn_classifier"

# Register the model
model_version = mlflow.register_model(
    model_uri=model_uri,
    name="churn_classifier_prod"
)

# Transition to staging for validation
client.transition_model_version_stage(
    name="churn_classifier_prod",
    version=model_version.version,
    stage="Staging"
)

After validation, your deployment pipeline can check the Model Registry and pull the production version automatically:

import mlflow.pyfunc

# Load the production model
production_model = mlflow.pyfunc.load_model(
    model_uri="models:/churn_classifier_prod/production"
)

# Use it for inference
predictions = production_model.predict(new_data)

This decouples experiment tracking from serving. Your data science team can experiment with new models, but your deployment doesn't change until something is explicitly promoted. It's a buffer between exploration and production.

Works locally or with a tracking server

MLflow is agnostic about where you run it. For exploration, you can log everything to a local directory and use mlflow ui to browse. No infrastructure required.

When you're ready to scale—sharing experiments across a team, running thousands of experiments in parallel, or maintaining a permanent audit trail—you can point to a remote MLflow Tracking Server. This can run on a small EC2 instance, a container in Kubernetes, or a managed service.

The client code stays identical. Only the configuration changes:

import mlflow

# Local mode (default)
# Logs to ./mlruns directory

# Remote mode
mlflow.set_tracking_uri("http://mlflow-server.internal:5000")
mlflow.set_experiment("customer_churn_v2")
with mlflow.start_run():
    # Code is the same
    mlflow.log_param("learning_rate", 0.001)
    mlflow.log_metric("accuracy", 0.94)

A shared tracking server becomes especially valuable when multiple team members run experiments. Everyone can see everyone else's runs, compare across experiments, and avoid duplicating effort. It's a central nervous system for your ML work.

Common mistakes

• Not logging enough metadata: Parameter values without feature names or data dates make it hard to reproduce or learn from old runs. Include anything that might matter later: data version, preprocessing steps, hardware specs, library versions.
• Mixing train and test metrics without clarity: Always name metrics clearly. Use train_loss and val_loss, not just loss. When you're comparing 50 runs, ambiguous names cause expensive mistakes.
• Logging too granularly: Logging every epoch or every batch creates noise. Usually you want final metrics, maybe a few checkpoints. Otherwise the UI becomes unusable.
• Forgetting to version your data: An experiment is reproducible only if you know what data it ran on. Log a data version, hash, or date alongside your runs. A great model on stale data is useless.
• Ignoring the registry: Dumping trained models in S3 without structure makes hand-offs to engineering teams painful. Use the registry. It's designed for this.
• Running experiments without an experiment name: MLflow puts everything under a default experiment if you don't specify one. A month later, finding which runs solved which problem becomes a detective story. Name your experiments clearly: "churn_v2_feature_engineering", "fraud_hyperparameter_sweep", etc.

Key takeaways

• Log parameters, metrics, and artifacts systematically so every experiment is a complete, searchable record.
• Use MLflow's UI to compare runs visually and spot patterns across dozens or hundreds of experiments.
• Register the best model in the Model Registry and promote it through stages (Staging, Production) to create clear lineage and audit trails.
• Start locally with mlflow ui for solo work; scale to a remote tracking server when your team or volume grows.
• Include metadata like data versions, preprocessing steps, and library versions so runs remain reproducible months later.