Encoding categorical features: a practical AI & ML guide

Introduction

Raw categorical features—like user country, product category, or payment method—can't be fed directly into most machine learning models. You need a systematic way to convert text labels into numerical representations. This guide on encoding categorical features: a practical AI & ML guide walks through the strategies we use in production systems, when to apply each one, and the trade-offs that matter when your dataset grows.

Over the past two decades, across e-commerce platforms, CRM systems, and trading applications, we've learned that choosing the wrong encoding can silently tank model performance or blow up inference latency. Let's explore the approaches that work.

One-hot encoding for low-cardinality features

One-hot encoding is the go-to technique when you have a small, fixed set of categories. It creates a binary column for each unique value, where exactly one column is 1 and the rest are 0. This works beautifully because it imposes no ordinal relationship between categories—a feature the algorithm can't misinterpret.

Consider a ride-hailing platform tracking vehicle type: sedan, SUV, truck, motorcycle. With one-hot encoding, each trip record gets four binary columns:

import pandas as pd
from sklearn.preprocessing import OneHotEncoder

df = pd.DataFrame({
    'vehicle_type': ['sedan', 'suv', 'truck', 'sedan', 'motorcycle']
})

encoder = OneHotEncoder(sparse_output=False, drop='first')
encoded = encoder.fit_transform(df[['vehicle_type']])
encoded_df = pd.DataFrame(
    encoded,
    columns=encoder.get_feature_names_out(['vehicle_type'])
)
print(encoded_df)

We drop the first category to avoid multicollinearity—if you know the vehicle is not a sedan, SUV, or truck, it must be a motorcycle. The resulting sparse representation feeds cleanly into linear models, neural networks, and tree-based algorithms.

When to use it: Fewer than ~10 unique values, stable category set in production. Drawback: Creates as many new columns as categories. If you have 100 product types, you get 99 new features (after dropping one). For high-dimensional feature spaces, this becomes unwieldy.

Ordinal encoding for ordered categories

Some categories have a natural order: customer satisfaction (poor, fair, good, excellent), education level (high school, bachelor, master, phd), or severity flags. Ordinal encoding maps these to integers 0, 1, 2, 3, preserving the ranking relationship your model can learn from.

In a travel booking system, you might encode customer loyalty tier:

from sklearn.preprocessing import OrdinalEncoder

df = pd.DataFrame({
    'loyalty_tier': ['bronze', 'silver', 'gold', 'platinum', 'silver']
})

# Define the order explicitly
categories = [['bronze', 'silver', 'gold', 'platinum']]
encoder = OrdinalEncoder(categories=categories)
encoded = encoder.fit_transform(df[['loyalty_tier']])

df['tier_encoded'] = encoded
print(df)

Now the model sees 0, 1, 2, 3 and can learn that higher values correlate with higher spend, better retention, or whatever signal matters. This is computationally efficient and prevents the dimension explosion of one-hot encoding.

Key assumption: The distance between levels is meaningful. Silver is "more" than bronze, but is it exactly 1 unit more? That's a model assumption, not always true. Still, tree models and ensemble methods handle this nuance well—they'll learn the actual relationship from data.

When to use it: Clear rank order exists, fewer than 20 categories, you want to keep feature count low.

Target encoding for high-cardinality features

Target encoding (also called mean encoding) is powerful but dangerous. Instead of creating a new column per category, you replace each category with a statistic derived from the training set: the mean of the target variable for records with that category.

Imagine an e-commerce platform with thousands of seller IDs. One-hot encoding is infeasible; ordinal encoding would be arbitrary. Instead, compute the average order value (or conversion rate, or return likelihood) for each seller, then use that value in place of the seller ID:

import pandas as pd
import numpy as np

df = pd.DataFrame({
    'seller_id': [101, 102, 101, 103, 102, 101],
    'order_value': [50, 120, 60, 25, 110, 55]
})

# Compute mean target per category
target_encoding = df.groupby('seller_id')['order_value'].mean().to_dict()
print(target_encoding)
# Output: {101: 55.0, 102: 115.0, 103: 25.0}

df['seller_target_encoded'] = df['seller_id'].map(target_encoding)
print(df)

The model now has a single numerical feature per seller, compact and information-rich. The relationship between seller ID and target is encoded directly.

The critical catch: This is prone to overfitting. Rare categories (few training samples) will have high variance in their target statistic. A seller with only 2 orders worth $200 and $10 gets an average of $105, but that signal is noise. When you encounter that seller again in validation or production, the model may be overconfident.

Mitigation strategies:

• Smoothing: Blend the category mean with the global mean. Categories with fewer samples get pulled toward the global average, stabilizing rare-case estimates.
• Cross-validation: Compute target statistics on one fold, apply to another. Never compute and apply on the same data.
• Holdout set: Reserve a validation set, compute encodings only from training, apply to validation/test.

# Smoothing example: blend category mean with global mean
global_mean = df['order_value'].mean()
category_counts = df.groupby('seller_id').size()
smoothing_factor = 1.0  # Tune based on your data

smoothed_encoding = {}
for seller_id, cat_mean in target_encoding.items():
    count = category_counts[seller_id]
    # Weight by sample size; rare categories shrink toward global mean
    smoothed_mean = (cat_mean * count + global_mean * smoothing_factor) / (count + smoothing_factor)
    smoothed_encoding[seller_id] = smoothed_mean

print(smoothed_encoding)

When to use it: Thousands of unique categories, target is numerical or binary, you have enough data to smooth reliably, and you can enforce proper train-validation splits.

Tree models handle many encodings well

Decision trees, random forests, gradient boosting (XGBoost, LightGBM, CatBoost) and other tree-based models are encoding-agnostic in many ways. Trees split on feature values to minimize loss; they don't care whether a feature is one-hot, ordinal, or target-encoded—they'll find the splits that work.

This flexibility is liberating. You could pass a raw integer category ID directly to XGBoost:

import xgboost as xgb
import pandas as pd
import numpy as np

X = pd.DataFrame({
    'seller_id': [101, 102, 101, 103, 102, 101],
    'product_category_id': [5, 12, 5, 8, 12, 5],
    'days_in_catalog': [30, 90, 30, 60, 90, 30]
})

y = np.array([0.5, 1.2, 0.6, 0.3, 1.1, 0.55])

model = xgb.XGBRegressor(max_depth=5, n_estimators=100)
model.fit(X, y)

print(model.predict(X[:2]))

The tree will learn that seller 101 and 102 produce different target values; it treats the ID as a categorical split point. No encoding needed. This is why CatBoost has a categorical_features parameter—tell it which columns are categorical, and it handles the logic internally, often outperforming pre-encoded approaches.

The limitation: Neural networks and linear models (logistic regression, linear SVM) need numerical inputs with stable scales. They can't split on raw category IDs. So the encoding choice depends on your modeling toolkit.

For production systems, this matters: if you train on CatBoost and switch to a linear model for inference speed, you suddenly need to encode everything up front, cache those mappings, and manage them carefully as new categories appear.

Common mistakes

1. Fitting the encoder on train + test data together. Your validation scores will inflate. Always fit encoders (especially target encoders) on training data only, then apply to validation and test sets. New categories at inference time need a fallback: a default value, the global mean, or rejection logic.

2. Ignoring class imbalance in target encoding. If your training set has 10,000 users from country A and 5 from country B, the target statistic for B is unreliable. Use smoothing, stratified sampling, or move to a different encoding for rare categories.

3. Assuming ordinal encoding preserves distance. An ordinal encoding of 0, 1, 2 looks like linear intervals, but the actual semantic distance might not be uniform. Let the model learn the true relationship from data; don't hand-craft integer assignments.

4. One-hot encoding everything by default. It's safe but creates sparse, high-dimensional feature spaces. For a dataset with 50 categorical columns each with 20 categories, one-hot encoding explodes to ~950 features. Start with tree models or target encoding; reserve one-hot for linear models and neural networks.

5. Forgetting to handle unseen categories. In production, you'll encounter category values not present in training. Have a strategy: map to a special "unknown" value, use a default encoding, or reject the record. The choice depends on your use case and tolerance for missing predictions.

Key takeaways

• One-hot encoding is safe, interpretable, and suitable for low-cardinality features (<10 categories), but creates sparse, high-dimensional outputs.
• Ordinal encoding works when categories have a natural rank; it's memory-efficient and works with any model type.
• Target encoding is compact and powerful for high-cardinality features, but requires careful smoothing and train-test separation to avoid overfitting.
• Tree-based models (XGBoost, LightGBM, CatBoost) can handle raw category IDs or any encoding; they find their own splits. Linear models and neural networks need explicit numerical encodings.
• Choose your encoding based on cardinality, model type, and data volume. Measure the impact on validation performance, not on intuition.
• Always fit encoders on training data only; handle unseen categories gracefully in production.