F1 score, ROC and AUC in Practical AI & Machine Learning

Beyond accuracy: why you need F1 score, ROC and AUC in practical AI and machine learning

In production systems, accuracy alone is a dangerous metric. When you're building a fraud detector, a medical screening tool, or a recommendation system, a single number hides critical trade-offs that will hurt your users or your business. In this article we'll work through F1 score, ROC and AUC—three powerful ways to measure classifiers—along with practical code examples you can run today against your own datasets.

Why accuracy isn't enough

Imagine you're building a fraud detection system. Your dataset has 10,000 transactions: 50 are fraudulent, 9,950 are legitimate. If your model predicts "legitimate" for everything, it achieves 99.5% accuracy. Perfect score. Useless product.

Accuracy treats all mistakes equally. In fraud detection, missing a fraud case (false negative) and wrongly flagging a legitimate transaction (false positive) have very different consequences. You need metrics that surface these asymmetries.

Enter precision, recall, and the F1 score that binds them together.

F1 score: the harmonic mean of precision and recall

Before we can talk about F1, we need precision and recall:

• Precision = TP / (TP + FP). Of the cases we flagged positive, how many were actually positive? High precision means fewer false alarms.
• Recall = TP / (TP + FN). Of all the actual positives, how many did we catch? High recall means we don't miss many real cases.

These two metrics pull in opposite directions. Stricter thresholds improve precision but tank recall. Loose thresholds catch more positives but drown you in false alarms. The F1 score is the harmonic mean of both, penalizing extreme imbalance:

F1 = 2 * (precision * recall) / (precision + recall)

It ranges from 0 to 1. Higher is better. F1 is especially useful when you care about both false positives and false negatives roughly equally, or when your positive class is rare.

Practical example: email spam classifier

You're training a spam filter. Your test set has 1,000 emails: 200 spam, 800 legitimate.

from sklearn.metrics import precision_recall_fscore_support, f1_score

# Predictions from your model
y_true = [1, 0, 1, 0, 1, 0, 0, 1, 1, 0]  # 1 = spam, 0 = ham
y_pred = [1, 0, 1, 0, 0, 0, 0, 1, 1, 0]

precision, recall, f1, support = precision_recall_fscore_support(
    y_true, y_pred, average='binary'
)

print(f"Precision: {precision:.3f}")
print(f"Recall: {recall:.3f}")
print(f"F1 Score: {f1:.3f}")

# Or compute F1 directly
f1_direct = f1_score(y_true, y_pred)
print(f"F1 (direct): {f1_direct:.3f}")

A model with precision=0.90 and recall=0.70 gets F1 ≈ 0.79. A model with precision=0.70 and recall=0.90 also gets F1 ≈ 0.79. F1 rewards balance. If precision and recall diverge sharply, F1 will be lower than either individual metric, pushing you toward more robust classifiers.

In production, you often want to tune your decision threshold to hit a target F1 or balance point specific to your use case. Logging F1 alongside precision and recall in your monitoring dashboards gives you real insight into classifier behavior over time.

ROC curve: true positive rate vs false positive rate

The ROC curve (Receiver Operating Characteristic) is a staple in medical testing and finance. It plots how your classifier behaves across all possible decision thresholds, showing the trade-off between catching positives and avoiding false alarms.

• True Positive Rate (Sensitivity) = TP / (TP + FN). Of the actual positives, how many did we flag?
• False Positive Rate = FP / (FP + TN). Of the actual negatives, how many did we wrongly flag?

As you lower your decision threshold, both TPR and FPR increase. The ROC curve traces this journey. A random classifier produces a diagonal line from (0,0) to (1,1). A perfect classifier hugs the top-left corner.

Practical example: medical screening

You're building a model to screen for a rare disease. Your predictions come as probabilities. By varying the threshold at which you call something "positive," you move along the ROC curve:

from sklearn.metrics import roc_curve, auc
import numpy as np

# True labels
y_true = [0, 0, 1, 1, 0, 1, 0, 1, 1, 0]

# Predicted probabilities (raw model output, before thresholding)
y_prob = [0.1, 0.2, 0.7, 0.8, 0.3, 0.6, 0.15, 0.85, 0.9, 0.25]

# Compute ROC curve
fpr, tpr, thresholds = roc_curve(y_true, y_prob)

print("FPR:", fpr)
print("TPR:", tpr)
print("Thresholds:", thresholds)

# Plot (requires matplotlib)
import matplotlib.pyplot as plt
plt.plot(fpr, tpr, label='ROC Curve')
plt.plot([0, 1], [0, 1], 'k--', label='Random Classifier')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.legend()
plt.show()

The ROC curve is threshold-independent. It shows your classifier's ranking quality: is it better at ranking positive examples higher than negative ones? This is valuable for model selection before you've even decided on a deployment threshold.

In medical screening, you might tolerate higher false positives to catch disease cases early. In fraud prevention, you might accept higher false negatives to avoid blocking legitimate users. The ROC curve shows you the menu of trade-offs available.

AUC: threshold-independent ranking quality

The Area Under the ROC Curve (AUC) is a single number summarizing the entire ROC curve. It ranges from 0.5 (random classifier) to 1.0 (perfect classifier). Interpret it as: the probability that your model ranks a random positive example higher than a random negative example.

from sklearn.metrics import roc_auc_score

y_true = [0, 0, 1, 1, 0, 1, 0, 1, 1, 0]
y_prob = [0.1, 0.2, 0.7, 0.8, 0.3, 0.6, 0.15, 0.85, 0.9, 0.25]

auc_score = roc_auc_score(y_true, y_prob)
print(f"AUC: {auc_score:.3f}")

AUC is robust to class imbalance (unlike accuracy) and threshold-independent (unlike F1). It's ideal for model comparison when you haven't yet committed to a specific operating point. Many teams use AUC to select between models, then tune F1 or precision/recall for production deployment.

In our medical screening example, an AUC of 0.92 tells you the model is discriminative—it's generally good at ranking sick people higher than healthy people. From there, your clinical team chooses a threshold based on the cost of false positives vs false negatives.

PR curves for heavily imbalanced data

ROC curves are misleading when your positive class is extremely rare. With 99% negatives and 1% positives, the FPR axis barely moves even when false positives increase substantially. Precision-Recall (PR) curves zoom in on the class you care about.

A PR curve plots precision (y-axis) against recall (x-axis) as you vary the threshold. In severely imbalanced datasets, this surface reveals performance more honestly than ROC.

Practical example: anomaly detection

You're detecting anomalies in server logs. Out of 100,000 logs, only 50 are true anomalies. Accuracy is useless; AUC can be misleading. PR curves and AUC-PR (area under the PR curve) are more informative:

from sklearn.metrics import precision_recall_curve, auc

y_true = [0] * 1000 + [1] * 50  # 98% negatives
# Simulate model predictions
y_prob = np.random.rand(1050)
# Make positives score higher on average
y_prob[1000:] += 0.3

precision, recall, thresholds = precision_recall_curve(y_true, y_prob)
pr_auc = auc(recall, precision)

print(f"PR AUC: {pr_auc:.3f}")

# Plot PR curve
plt.plot(recall, precision, label=f'PR Curve (AUC={pr_auc:.3f})')
plt.xlabel('Recall')
plt.ylabel('Precision')
plt.legend()
plt.show()

PR AUC is more sensitive to rare positive classes. A PR AUC of 0.65 on an imbalanced dataset often reflects stronger performance than the same ROC AUC would suggest. Always visualize both curves when class distribution is skewed.

Common mistakes

• Reporting F1 without precision and recall. F1 hides the composition of your error. A model with F1=0.80 could be (precision=0.95, recall=0.70) or (precision=0.70, recall=0.95). Your stakeholders need both.
• Using ROC AUC for imbalanced datasets without checking PR AUC. A model with ROC AUC=0.88 and PR AUC=0.45 has a major blind spot on the positive class. Always cross-check.
• Tuning thresholds on test data. If you search for the threshold that maximizes F1 on your test set, you overfit. Tune thresholds on a validation set; evaluate on held-out test data.
• Forgetting that AUC requires probability outputs. AUC needs the raw confidence scores from your model, not hard predictions. Don't threshold first.
• Treating F1 as universal. F1 assumes equal cost for false positives and false negatives. If they're not equal (and in production, they rarely are), define a custom threshold or metric aligned with your business goal.

Key takeaways

• Accuracy is insufficient for real-world classifiers. Precision and recall measure different failure modes; F1 balances them.
• ROC curves and AUC are threshold-independent ways to assess ranking quality. Use AUC for model selection.
• F1 lets you fix a specific operating point once you've chosen a threshold. Log it alongside precision and recall in production.
• PR curves are essential for imbalanced datasets where the positive class is rare. Always check PR AUC alongside ROC AUC.
• In production, your choice of metric should reflect your business costs: the relative harm of false positives and false negatives, not a generic academic standard.