Regression metrics: MAE, RMSE and R² — a practical guide

Why regression metrics matter

When you're predicting continuous values—house prices, API response times, user engagement metrics—you need to know how wrong your model actually is. Raw prediction errors are rarely useful on their own; you need a common language with the business to understand whether a model is good enough to deploy. That's where regression metrics: MAE, RMSE and R² — a practical guide comes in. These three metrics give you different lenses on model performance, each answering a distinct question about prediction quality.

In this guide, we'll walk through each metric, show you when to use them, and explain how to communicate model performance to non-technical stakeholders in units they actually care about.

MAE: average absolute error

Mean Absolute Error (MAE) is the simplest regression metric to understand and explain. It's the average of all absolute differences between predicted and actual values:

import numpy as np

def mae(y_true, y_pred):
    return np.mean(np.abs(y_true - y_pred))

# Example: predicting house prices (in thousands)
y_true = np.array([250, 320, 180, 410, 290])
y_pred = np.array([240, 330, 175, 420, 285])

error = mae(y_true, y_pred)
print(f"MAE: ${error * 1000:.0f}")  # Output: MAE: $9000

In this example, on average, predictions were off by $9,000. That's a concrete number you can present to stakeholders. MAE is in the same units as your target variable, making it immediately interpretable.

When to use MAE

MAE is your go-to metric when:

• Errors of all sizes matter equally to your business. A $10k error is twice as bad as a $5k error, not four times as bad.
• You need to communicate results to non-technical teams. "We're off by $9,000 on average" is much clearer than RMSE or R².
• You're dealing with outliers in your test set. MAE won't punish them as heavily as other metrics.
• Your loss function during training was also L1 (absolute error). Your evaluation metric should match your optimization objective.

Real-world example: delivery time prediction

Imagine you're building a delivery time estimator for a ride-hailing platform. You track actual delivery times and model predictions across 1,000 trips:

from sklearn.metrics import mean_absolute_error

# Delivery times in minutes
actual = np.array([25, 18, 42, 31, 19, 55, 27, 33, 21, 44])
predicted = np.array([26, 20, 40, 30, 18, 58, 25, 35, 22, 42])

mae_error = mean_absolute_error(actual, predicted)
print(f"MAE: {mae_error:.1f} minutes")  # Output: MAE: 1.8 minutes

An average error of 1.8 minutes is reasonable for user-facing time estimates. The business can understand this immediately: "Our algorithm is typically off by under 2 minutes."

RMSE: punishes large errors more

Root Mean Squared Error (RMSE) squares each error before averaging, then takes the square root. This mathematical quirk makes large errors disproportionately influential:

from sklearn.metrics import mean_squared_error
import numpy as np

def rmse(y_true, y_pred):
    return np.sqrt(mean_squared_error(y_true, y_pred))

# Same house price data
y_true = np.array([250, 320, 180, 410, 290])
y_pred = np.array([240, 330, 175, 420, 285])

rmse_error = rmse(y_true, y_pred)
print(f"RMSE: ${rmse_error * 1000:.0f}")  # Output: RMSE: $12,247

Notice RMSE ($12,247) is higher than MAE ($9,000), even though they're computed from the same predictions. The difference is that the large error on the $410k house (off by $10k) gets squared to 100, which dominates the calculation.

When to use RMSE

RMSE is appropriate when:

• Large errors are significantly worse than small ones. In financial trading, missing a price by $0.10 is bad, but missing by $1.00 is exponentially worse.
• Your loss function during training was MSE (mean squared error). Use the same metric for evaluation.
• You're optimizing for a Gaussian distribution of errors. RMSE aligns with maximum likelihood estimation under normality assumptions.
• You want a metric that's more sensitive to outliers and model failures.

Comparing MAE and RMSE on outliers

# Scenario 1: Mostly accurate, one big miss
y_true_1 = np.array([100, 105, 102, 103, 104, 500])
y_pred_1 = np.array([101, 104, 101, 102, 103, 450])

mae_1 = mean_absolute_error(y_true_1, y_pred_1)
rmse_1 = rmse(y_true_1, y_pred_1)

print(f"Scenario 1 - MAE: {mae_1:.1f}, RMSE: {rmse_1:.1f}")
# Output: Scenario 1 - MAE: 9.2, RMSE: 20.5

# Scenario 2: Consistently off by small amount
y_true_2 = np.array([100, 105, 102, 103, 104, 500])
y_pred_2 = np.array([95, 100, 97, 98, 99, 495])

mae_2 = mean_absolute_error(y_true_2, y_pred_2)
rmse_2 = rmse(y_true_2, y_pred_2)

print(f"Scenario 2 - MAE: {mae_2:.1f}, RMSE: {rmse_2:.1f}")
# Output: Scenario 2 - MAE: 5.0, RMSE: 5.0

In scenario 1, the single large outlier pushes RMSE much higher than MAE. In scenario 2, where errors are consistent but smaller, both metrics are similar. This shows RMSE's sensitivity to catastrophic failures.

R²: share of variance explained

R-squared (coefficient of determination) measures what proportion of variance in your target variable is explained by the model, on a 0-to-1 scale. It's defined as:

R² = 1 − (sum of squared residuals) / (total sum of squares)

Unlike MAE and RMSE, R² is unitless and makes a comparison to a naive baseline:

from sklearn.metrics import r2_score

y_true = np.array([100, 150, 200, 250, 300])
y_pred = np.array([105, 148, 202, 248, 305])

r2 = r2_score(y_true, y_pred)
print(f"R²: {r2:.3f}")  # Output: R²: 0.996

# Compare to naive "always predict the mean" baseline
y_mean = np.full_like(y_true, y_true.mean())
r2_baseline = r2_score(y_true, y_mean)
print(f"R² (baseline): {r2_baseline:.3f}")  # Output: R² (baseline): 0.000

An R² of 0.996 means the model explains 99.6% of the variance. A naive model that always predicts the mean has R² = 0.0 by definition.

Interpreting R² correctly

R² ranges from negative infinity to 1:

• R² = 1: Perfect predictions.
• R² = 0.7–0.9: Strong model; often acceptable for production.
• R² = 0.5–0.7: Moderate model; depends on use case.
• R² < 0: Model is worse than simply predicting the mean. Something is seriously wrong.

Why R² alone is dangerous

R² can be misleading without context. Two models might have the same R² but very different error distributions:

# Model A: Slightly off everywhere
y_true = np.arange(1, 101)
y_pred_a = y_true + np.random.normal(0, 1, 100)

# Model B: Perfect on most, terrible on a few
y_pred_b = y_true.copy().astype(float)
y_pred_b[np.random.choice(100, 10)] += 50  # 10 random large errors

r2_a = r2_score(y_true, y_pred_a)
r2_b = r2_score(y_true, y_pred_b)
mae_a = mean_absolute_error(y_true, y_pred_a)
mae_b = mean_absolute_error(y_true, y_pred_b)

print(f"Model A - R²: {r2_a:.3f}, MAE: {mae_a:.1f}")
print(f"Model B - R²: {r2_b:.3f}, MAE: {mae_b:.1f}")
# Likely output: Model A - R²: 0.999, MAE: 1.0
#                Model B - R²: 0.975, MAE: 5.0

Always report R² alongside MAE or RMSE to get the full picture.

Report errors in business units

The most common mistake is reporting metrics without translating them to business language. Here's how to do it right:

Define what "good" looks like first

Before building any model, sit down with the product and business teams to define acceptable error thresholds:

• For pricing models: "Errors larger than 5% of item value are unacceptable."
• For delivery estimates: "95% of predictions should be within ±10 minutes."
• For demand forecasting: "We can tolerate a 15% MAE on quarterly forecasts."

Example: SaaS churn prediction

Say you're predicting monthly revenue retention. You build a model and get these metrics:

# Monthly revenue in thousands
y_true = np.array([50, 75, 120, 45, 90, 200, 65, 110])
y_pred = np.array([52, 72, 125, 43, 92, 195, 68, 108])

mae = mean_absolute_error(y_true, y_pred)
rmse = rmse(y_true, y_pred)
r2 = r2_score(y_true, y_pred)

print(f"MAE: ${mae * 1000:.0f} per customer cohort")
print(f"RMSE: ${rmse * 1000:.0f} per customer cohort")
print(f"R²: {r2:.3f} variance explained")

# Translate to business impact
mean_revenue = y_true.mean()
mae_pct = (mae / mean_revenue) * 100
print(f"Average error: {mae_pct:.1f}% of mean cohort value")

Now your report reads: "Our retention model predicts revenue within ±$2,400 per cohort on average, about 2.8% of typical cohort value. The model explains 99.4% of variance in historical retention." That's actionable.

Create a summary table for stakeholders

This format is clear, concrete, and immediately tells leadership whether the model is production-ready.

Common mistakes

Using only R² without absolute error metrics

R² is relative; it doesn't tell you if errors are $100 or $100,000. Always include MAE or RMSE to ground performance in real units.

Training on MSE but evaluating with MAE

Your optimization target and evaluation metric should align. If you train with mean squared error loss, report RMSE for consistency. If you train with L1 loss, evaluate with MAE.

Ignoring error distribution

A single metric hides outliers. Check a histogram of residuals or percentile errors:

import numpy as np

residuals = y_true - y_pred
print(f"50th percentile (median) error: {np.percentile(np.abs(residuals), 50):.1f}")
print(f"95th percentile error: {np.percentile(np.abs(residuals), 95):.1f}")
print(f"99th percentile error: {np.percentile(np.abs(residuals), 99):.1f}")

If median error is $1k but 95th percentile is $50k, your model has a reliability problem that MAE alone won't reveal.

Not stratifying by data subsets

A model might have great overall metrics but fail on specific segments. Always compute metrics separately for:

• Different time periods (train vs. test vs. future data)
• Customer segments or product categories
• Edge cases or high-value instances

Key takeaways

• MAE is your primary metric for communicating with non-technical stakeholders; it's in business units and treats all errors equally.
• RMSE penalizes large errors; use it when outliers matter or when your training loss was MSE.
• R² measures relative performance against a naive baseline, but never use it alone—always pair it with absolute error metrics.
• Define acceptable error thresholds with business teams before you build the model, not after evaluation.
• Report metrics in business units (dollars, minutes, percentages) rather than abstract numbers.
• Always examine error distribution, not just the mean; check percentiles to catch reliability issues.
• Your evaluation metrics should match your training loss function for consistency.