Decision trees and random forests for AI & ML developers

Decision trees and random forests for AI & ML developers

If you've worked with tabular data in production, you've probably reached for a neural network or logistic regression without thinking twice. But there's a simpler, more interpretable class of models that often outperforms them on structured data: tree-based methods. Decision trees and random forests are workhorses in real-world ML pipelines because they handle mixed data types, missing values, and non-linear relationships without extensive preprocessing. Let's explore how to build, debug, and deploy them effectively.

Decision trees — splitting data by thresholds

A decision tree works by recursively splitting your data based on feature thresholds. At each node, the algorithm asks a simple question: "Is feature X greater than value Y?" If yes, go left; if no, go right. This continues until we reach a leaf node, which makes a prediction (classification or regression).

The magic happens during training. The algorithm searches for splits that minimize impurity (in classification, this is typically Gini impurity or entropy; in regression, it's mean squared error). Each split tries to group similar outcomes together.

Here's a practical example: imagine you're building a churn model for an e-commerce platform. Your data has customer lifetime value, account age, recent purchase count, and support tickets filed. A decision tree might first split on lifetime value > $500, then on support tickets > 3, creating interpretable rules that even non-technical stakeholders can understand.

from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=1000, n_features=10, n_informative=5, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

tree = DecisionTreeClassifier(max_depth=5, min_samples_split=10, random_state=42)
tree.fit(X_train, y_train)

print(f"Train accuracy: {tree.score(X_train, y_train):.4f}")
print(f"Test accuracy: {tree.score(X_test, y_test):.4f}")

The max_depth and min_samples_split parameters are crucial. Without them, the tree grows until every sample is perfectly classified, leading to severe overfitting. A tree that's too deep memorizes training noise rather than learning generalizable patterns. Think of it like a team member who can recite every past project but struggles with new problems. You need regularization.

Random forests — wisdom through diversity

Random forests fix the overfitting problem by training many trees on random subsets of your data (bootstrap samples) and averaging their predictions. The randomness comes from two sources: random row samples and random feature subsets at each split.

This ensemble approach works because:

• Individual trees are deep and might overfit, but their errors are uncorrelated
• Averaging uncorrelated errors reduces variance dramatically
• You retain the interpretability of trees while gaining robustness

In a ride-hailing context, you might predict driver rating from trip duration, passenger count, time of day, and route complexity. A single deep tree might overfit to specific time-of-day patterns in your training set. A random forest with 100 trees learns robust patterns across different conditions because each tree sees different slices of your data.

from sklearn.ensemble import RandomForestClassifier

rf = RandomForestClassifier(n_estimators=100, max_depth=10, 
                            min_samples_split=5, random_state=42, n_jobs=-1)
rf.fit(X_train, y_train)

print(f"Train accuracy: {rf.score(X_train, y_train):.4f}")
print(f"Test accuracy: {rf.score(X_test, y_test):.4f}")

Notice the gap between train and test accuracy. With random forests, this gap is typically much smaller than with a single decision tree. The n_jobs=-1 parallelizes training across all cores—important since training multiple trees is embarrassingly parallel.

n_estimators (number of trees) is usually set high; more trees rarely hurt as long as you have compute. Start with 100 and increase if validation performance plateaus. max_depth should be tuned, but random forests are forgiving; values between 5 and 20 work for most tabular datasets.

Feature importance — understanding what drives predictions

A massive advantage over neural networks: you can ask a random forest which features actually matter. Feature importance measures how much each feature reduces impurity across all splits in all trees.

import pandas as pd

feature_importance = pd.DataFrame({
    'feature': [f'feature_{i}' for i in range(X_train.shape[1])],
    'importance': rf.feature_importances_
}).sort_values('importance', ascending=False)

print(feature_importance)

This is invaluable in production. If your fraud detection model says "spending_amount" is 10x more important than "account_age," you immediately know where to invest detection effort. You might also spot unexpected patterns: if a feature you thought was predictive has near-zero importance, it's a signal to investigate data quality or feature engineering.

Feature importance also helps with debugging. In a CRM churn model, if customer support sentiment suddenly becomes the top predictor (when historical data says it's mid-tier), your team should investigate whether support quality recently changed or whether new data leakage crept in.

Good defaults — random forests work well with little tuning

Here's the honest truth: random forests are surprisingly robust out of the box. We've deployed models to production with nearly default hyperparameters and seen solid results. This is rare in machine learning.

Starting configuration:

• n_estimators=100 to 200
• max_depth=10 to 15
• min_samples_split=5 to 10
• min_samples_leaf=2 to 5

If validation metrics are acceptable, ship it. Hyperparameter tuning is often a distraction. Focus instead on feature engineering and data quality.

For regression tasks, use RandomForestRegressor the same way. For imbalanced classification, set class_weight='balanced' to penalize misclassifying the minority class.

from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import classification_report

# Imbalanced data
rf_balanced = RandomForestClassifier(n_estimators=100, max_depth=10,
                                      class_weight='balanced', random_state=42)
rf_balanced.fit(X_train, y_train)

y_pred = rf_balanced.predict(X_test)
print(classification_report(y_test, y_pred))

Common mistakes

Mistake 1: Forgetting to scale features. Unlike neural networks, tree-based models don't need feature scaling. But if you do scale them, it doesn't hurt either. The real mistake is overthinking it.

Mistake 2: Setting max_depth too high or min_samples_split too low. You'll get perfect training accuracy and terrible test accuracy. Your model learned noise. Start conservative and increase complexity gradually.

Mistake 3: Not checking for data leakage. If your feature importance reveals an unexpectedly dominant feature, ask: "Could this be leaking information from the target?" In a loan default model, if "months_since_default" is top-1 important, you've leaked the target into your features.

Mistake 4: Trusting feature importance too much. Feature importance tells you which features the tree used, not which features are causally important. Correlated features can artificially inflate importance. Combine feature importance with domain knowledge and ablation studies.

Mistake 5: Not handling missing values. Some libraries drop rows with NaN. Instead, explicitly impute or let the tree learn to treat NaN as its own category by encoding missing values.

Key takeaways

• Decision trees are interpretable but overfit easily; regularization via max_depth and min_samples_split is essential.
• Random forests reduce overfitting through ensemble averaging, making them robust with minimal tuning.
• Feature importance provides direct insight into model behavior, invaluable for debugging and explainability.
• Random forests work well with little hyperparameter tuning compared to most ML algorithms; good defaults exist.
• For tabular data in production (e-commerce, CRM, trading platforms), tree-based models often outperform neural networks while being faster to train and easier to interpret.
• Always check for data leakage using feature importance; unexpected dominant features are red flags.
• Combine tree models with cross-validation and holdout test sets to catch overfitting early.