Hyperparameter tuning in Practical AI & Machine Learning

Hyperparameter tuning in Practical AI & Machine Learning

We've all been there: you build a model, train it with default settings, and the results are mediocre. The problem usually isn't your architecture or data—it's that you haven't spent time finding the right knobs to turn. Hyperparameter tuning in practical AI & machine learning can feel like black magic, but it's really just disciplined search. Let's walk through how to find better settings without burning through your compute budget.

Start with sensible defaults

Before you touch a single hyperparameter, establish a baseline. Most frameworks and papers publish reasonable starting points. For a neural network, that might be a learning rate of 0.001, batch size of 32, and standard weight initialization. For gradient boosting, maybe max_depth of 5 and learning_rate of 0.1. The goal isn't to find the perfect values yet—it's to confirm your pipeline works end-to-end.

We recommend training a quick model on a subset of your data with these defaults. Does it converge? Does it overfit? Does it underfit badly? These observations tell you which hyperparameters matter most. If your model is severely underfitting, learning rate or model capacity are likely culprits. If it's overfitting within an epoch, regularization or early stopping are your friends.

Keep a simple tracking file or spreadsheet. Record the defaults, the training time, validation loss, and any observations. This becomes your reference point and prevents you from going in circles later.

import xgboost as xgb
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split

X, y = load_breast_cancer(return_X_y=True)
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)

# Sensible defaults for XGBoost
default_params = {
    'max_depth': 5,
    'learning_rate': 0.1,
    'n_estimators': 100,
    'subsample': 0.8,
    'colsample_bytree': 0.8,
    'random_state': 42
}

model = xgb.XGBClassifier(**default_params)
model.fit(X_train, y_train, eval_set=[(X_val, y_val)], verbose=False)
baseline_score = model.score(X_val, y_val)
print(f"Baseline validation score: {baseline_score:.4f}")

Random search beats grid search

Grid search is intuitive: define a set of values for each hyperparameter and try every combination. It's also wasteful. If you have 5 hyperparameters and test 5 values each, that's 3,125 training runs. Most of those are wasted because the value of one hyperparameter doesn't depend equally on another—there are interactions, but they're sparse.

Random search samples the same hyperparameter space uniformly at random. Counter-intuitively, it finds better results faster. Research has shown that random search explores the space more effectively, especially when some hyperparameters are more important than others. With limited compute, random search is your ally.

Define reasonable bounds for each parameter. For learning rate, try a log-uniform distribution between 0.0001 and 0.1. For max_depth, a uniform distribution between 3 and 15. For regularization strength, log-uniform between 1e-5 and 1e-1. Run 50 to 100 random combinations and track the best one.

import numpy as np
from sklearn.model_selection import RandomizedSearchCV

param_dist = {
    'max_depth': np.arange(3, 16),
    'learning_rate': np.logspace(-4, -1, 20),  # 0.0001 to 0.1
    'subsample': np.linspace(0.5, 1.0, 6),
    'colsample_bytree': np.linspace(0.5, 1.0, 6),
    'gamma': np.logspace(-2, 2, 10)  # regularization
}

random_search = RandomizedSearchCV(
    xgb.XGBClassifier(n_estimators=100, random_state=42),
    param_distributions=param_dist,
    n_iter=50,
    cv=5,
    scoring='roc_auc',
    n_jobs=-1,
    random_state=42
)

random_search.fit(X_train, y_train)
print(f"Best parameters: {random_search.best_params_}")
print(f"Best CV score: {random_search.best_score_:.4f}")

Optuna for smarter search

Random search is better than grid search, but it's still somewhat blind. Optuna is a hyperparameter optimization framework that learns as it goes. It uses Bayesian optimization and tree-structured Parzen estimators to intelligently sample the space, pruning unpromising trials early.

Optuna feels like magic because it is—in the best way. You define your search space and an objective function (usually validation loss). Optuna runs trials, observes the results, and biases future samples toward regions that look promising. It also supports early stopping: if a trial looks bad halfway through training, it stops and moves on. On a typical project, Optuna finds better hyperparameters in 20-30% fewer trials than random search.

The workflow is straightforward. Define a study, create an objective function that trains and evaluates your model, then optimize. Optuna's pruner integration with most frameworks makes this seamless.

import optuna
from optuna.pruners import MedianPruner
import xgboost as xgb

def objective(trial):
    params = {
        'max_depth': trial.suggest_int('max_depth', 3, 15),
        'learning_rate': trial.suggest_float('learning_rate', 1e-4, 1e-1, log=True),
        'subsample': trial.suggest_float('subsample', 0.5, 1.0),
        'colsample_bytree': trial.suggest_float('colsample_bytree', 0.5, 1.0),
        'gamma': trial.suggest_float('gamma', 1e-2, 1e2, log=True),
        'n_estimators': 100,
        'random_state': 42
    }
    
    model = xgb.XGBClassifier(**params)
    model.fit(X_train, y_train, eval_set=[(X_val, y_val)], verbose=False)
    score = model.score(X_val, y_val)
    return score

study = optuna.create_study(direction='maximize', pruner=MedianPruner())
study.optimize(objective, n_trials=50, show_progress_bar=True)

print(f"Best parameters: {study.best_params}")
print(f"Best validation score: {study.best_value:.4f}")

Tune on validation, confirm on test

This is the critical discipline that separates good practice from overfitting to your search procedure. Always tune hyperparameters on a validation set, never on your test set. Your test set is sacred—it represents unseen data and must be touched only once, at the very end.

The workflow is: train/validation split for tuning, then final evaluation on a held-out test set. Your validation set can be a simple 80/20 split or cross-validation folds. Either way, hyperparameters are selected based on validation performance. Once you've picked the best hyperparameters, train a fresh model on the combined train+validation data (or just the training fold if using CV) and evaluate exactly once on the test set. That number is your honest estimate of generalization performance.

We've seen teams burn this rule repeatedly. They tune on test data "just to see" and then act surprised when the test score is better than any holdout evaluation. Don't be that team. The test set is inviolate.

from sklearn.model_selection import cross_val_score

# Split: train for tuning, test for final eval
X_tune, X_test, y_tune, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Optuna study optimizes on X_tune (which includes internal CV splits)
# After finding best params, train on full X_tune and eval on X_test once

best_params = study.best_params
best_params['n_estimators'] = 100
best_params['random_state'] = 42

final_model = xgb.XGBClassifier(**best_params)
final_model.fit(X_tune, y_tune, verbose=False)

# Single evaluation on held-out test set
test_score = final_model.score(X_test, y_test)
print(f"Final test score (reported once): {test_score:.4f}")

Common mistakes

• Tuning on test data: We mentioned this, but it's worth repeating. Hyperparameter selection via test set evaluation leaks information and biases your results upward.
• Tuning too many hyperparameters at once: Start with the 2-3 most sensitive ones (learning rate, max_depth, regularization). Once those are stable, tune others if needed. Each hyperparameter multiplies your search space.
• Not setting random seeds: Randomness in training (weight initialization, dropout, batch order) means results vary. Set seeds in your framework, numpy, and Python's random module for reproducibility.
• Ignoring compute cost: A hyperparameter search that takes a week isn't practical. Estimate training time per trial, multiply by the number of trials, and decide if it's worth it. Sometimes 50 trials on a smaller dataset gives insights faster than 200 trials on full data.
• Forgetting to adjust batch size: Batch size affects convergence behavior and is often neglected. A model that works well at batch_size=32 might behave differently at 128. Include it in your search, or at least test a few values.
• Not using a pruner with Optuna: Optuna's pruning capability saves enormous amounts of compute. Enable it and watch the wall-clock time drop.

Key takeaways

• Begin with published defaults and a baseline to understand what matters.
• Use random search over grid search to explore hyperparameter space more efficiently.
• Adopt Optuna or similar Bayesian optimization tools to intelligently guide your search and prune unpromising trials.
• Always tune on validation data, never on test data. Evaluate your final model exactly once on held-out test data.
• Track everything: parameters, validation scores, training time, and observations. You'll learn patterns that apply to future projects.
• Set random seeds and be consistent with train/validation/test splits to ensure reproducibility.
• Remember that hyperparameter tuning has diminishing returns; 80% of the benefit often comes from the first 20-30 trials.