Clustering with k-means: AI & ML lessons from production

The reality of unlabeled data in production

Most machine learning tutorials focus on supervised learning—you have labels, you train, you predict. But in the real world, you often inherit messy datasets with no labels at all. You might be staring at customer transaction records, sensor readings, or user behavior logs and wondering: what natural groups exist in this data? Clustering with k-means is one of the first tools we reach for when facing that question. This article covers clustering with k-means: AI & ML lessons from production—the practical realities, the gotchas, and when to use something else entirely.

What k-means does and why it matters

K-means is an unsupervised algorithm that partitions your data into k clusters by iteratively assigning points to the nearest cluster center (centroid) and then recalculating those centers. It's simple, fast, and often "good enough" for exploratory work. We've used it to segment customers into spending tiers, group similar API errors, and even bucket server response times into performance categories.

The algorithm converges quickly on datasets under a few million rows, making it practical for many real-world scenarios. But simplicity comes with constraints—constraints that trip up teams who skip the setup phase.

Scaling features first: non-negotiable

This is lesson zero, and we learned it the hard way. K-means uses Euclidean distance to measure how far a point is from a centroid. If one feature spans 0–1000 and another spans 0–1, the large-range feature will dominate the distance calculation. Your clustering will essentially ignore the smaller feature.

We once clustered user engagement metrics without scaling. Revenue (in dollars) ranged from 0–50,000, while session count ranged from 1–100. The result? Every segment was defined almost entirely by revenue. Session frequency barely mattered, even though it was equally important for business logic.

Always scale your features before clustering. StandardScaler (zero mean, unit variance) is the standard choice:

from sklearn.preprocessing import StandardScaler
from sklearn.cluster import KMeans
import pandas as pd

# Load data
df = pd.read_csv('user_metrics.csv')
features = df[['revenue', 'session_count', 'days_active']]

# Scale features
scaler = StandardScaler()
scaled_features = scaler.fit_transform(features)

# Now apply k-means
kmeans = KMeans(n_clusters=3, random_state=42, n_init=10)
clusters = kmeans.fit_predict(scaled_features)

# Add cluster labels back to original data
df['cluster'] = clusters

After scaling, each feature contributes fairly to distance calculations. The improved segmentation was immediately visible—we finally saw meaningful session-count patterns.

Choosing k with the elbow method

The biggest question: how many clusters do you actually want? You have to decide k before you run the algorithm. There's no magic answer, but the elbow method gives you a data-driven starting point.

The elbow method plots the inertia (sum of squared distances from points to their cluster centers) against different values of k. As k increases, inertia decreases—more clusters mean points are closer to their assigned center. But the decrease gets smaller and smaller. You're looking for the "elbow": the point where the curve flattens, suggesting that adding more clusters doesn't help much.

Here's how to compute and visualize it:

inertias = []
silhouette_scores = []
k_range = range(1, 11)

from sklearn.metrics import silhouette_score

for k in k_range:
    km = KMeans(n_clusters=k, random_state=42, n_init=10)
    km.fit(scaled_features)
    inertias.append(km.inertia_)
    if k > 1:
        score = silhouette_score(scaled_features, km.labels_)
        silhouette_scores.append(score)

# Print results for inspection
for k, inertia in zip(k_range, inertias):
    print(f"k={k}: inertia={inertia:.2f}")

In one e-commerce clustering project, we tested k=2 through k=10 on customer purchase behavior. The inertia dropped sharply from k=1 to k=4, then leveled off. We chose k=4, which aligned with the business's intuition about customer segments: bargain hunters, regular shoppers, loyal bulk buyers, and seasonal buyers.

The elbow method is heuristic—it works well most of the time but isn't foolproof. That's why we pair it with another metric.

Validating with the silhouette method

The silhouette score measures how well each point fits within its cluster compared to other clusters. For each point, it calculates:

• a: average distance to other points in the same cluster (cohesion)
• b: average distance to points in the nearest other cluster (separation)

The silhouette score is (b - a) / max(a, b), ranging from -1 to 1. Values closer to 1 mean the point is well-clustered. Values near 0 mean the point is between clusters. Negative values are problems—the point is closer to another cluster than its own.

A high average silhouette score across all points indicates good cluster separation. It's more rigorous than the elbow method and often contradicts it, so we use both:

import matplotlib.pyplot as plt

fig, axes = plt.subplots(1, 2, figsize=(14, 5))

# Elbow plot
axes[0].plot(k_range, inertias, 'bo-')
axes[0].set_xlabel('k')
axes[0].set_ylabel('Inertia')
axes[0].set_title('Elbow Method')
axes[0].grid()

# Silhouette plot
axes[1].plot(range(2, 11), silhouette_scores, 'go-')
axes[1].set_xlabel('k')
axes[1].set_ylabel('Silhouette Score')
axes[1].set_title('Silhouette Method')
axes[1].grid()

plt.tight_layout()
plt.savefig('cluster_metrics.png')
plt.show()

In practice, we pick k where both metrics agree or where business intuition aligns with the data. If they disagree sharply, that's a signal to reconsider your feature set or explore the data more carefully.

When k-means works best: round, similar-size clusters

K-means assumes clusters are roughly spherical and similarly sized. If your true clusters are elongated, crescent-shaped, or vastly different in size, k-means will struggle.

We encountered this in a sensor anomaly detection project. We were clustering IoT device readings to find normal vs. abnormal patterns. Some normal clusters (devices operating in typical ranges) were large and dense. Abnormal patterns were rare, sparse, and scattered. K-means split the large normal cluster into fragments and merged some abnormal points into it, creating a false sense of confidence in the segmentation.

K-means works beautifully when:

• Clusters are roughly spherical (equidistant points from the center)
• Clusters are similar in size and density
• You've normalized and scaled the features properly
• You're clustering under a few million rows on a single machine

It falters when clusters have irregular shapes or vastly different sizes. That's when you need to switch tools.

Irregular shapes and density variations: use DBSCAN instead

DBSCAN (Density-Based Spatial Clustering of Applications with Noise) doesn't assume spherical clusters. It finds clusters by identifying regions of high point density, separated by regions of low density. It also labels outliers as noise, which is valuable for anomaly detection.

DBSCAN has two main parameters:

• eps: the maximum distance between two points for them to be neighbors
• min_samples: the minimum number of points needed to form a core point

Here's how to apply it to the same sensor data:

from sklearn.cluster import DBSCAN
from sklearn.neighbors import NearestNeighbors

# Find a good eps value using k-distance graph
neighbors = NearestNeighbors(n_neighbors=5)
neighbors_fit = neighbors.fit(scaled_features)
distances, indices = neighbors_fit.kneighbors(scaled_features)
distances = np.sort(distances[:, -1], axis=0)

# Plot to find the "knee" in the distance curve
plt.figure(figsize=(10, 6))
plt.plot(distances)
plt.xlabel('Data Points sorted by distance')
plt.ylabel('5th Nearest Neighbor Distance')
plt.title('K-distance Graph for eps Selection')
plt.grid()
plt.savefig('k_distance_graph.png')
plt.show()

# Apply DBSCAN with selected eps (e.g., 0.5)
dbscan = DBSCAN(eps=0.5, min_samples=5)
clusters_dbscan = dbscan.fit_predict(scaled_features)

# clusters_dbscan == -1 indicates noise points
n_clusters = len(set(clusters_dbscan)) - (1 if -1 in clusters_dbscan else 0)
n_noise = list(clusters_dbscan).count(-1)

print(f"Number of clusters: {n_clusters}")
print(f"Number of noise points: {n_noise}")

With DBSCAN on those sensor readings, abnormal patterns finally formed coherent clusters, and normal operations—despite their large size—remained intact as single or few large clusters. The noise points were genuinely anomalous readings that deserved attention.

The trade-off: DBSCAN is slower than k-means on large datasets and requires careful tuning of eps and min_samples. But when your data has irregular cluster shapes or widely varying densities, the extra effort pays off.

Common mistakes we've made and seen

• Forgetting to scale: We've done this. Features with larger ranges dominate, and you get meaningless clusters. Always scale first, every time.
• Picking k based on tradition or gut feel: "Let's use 5 clusters because that feels right." Run the elbow and silhouette methods. Data beats intuition.
• Ignoring the shape assumption: K-means assumes round clusters. If your data is crescent-shaped or has density variations, it will fail. Inspect your clusters visually (via PCA or t-SNE) to spot shape problems early.
• Using arbitrary values for eps and min_samples in DBSCAN: The k-distance graph is your friend. Don't guess.
• Not validating with domain experts: The algorithm is objective, but whether the clusters make business sense is not. Always review results with someone who knows the data domain.
• Treating clustering as a one-shot task: Real datasets change. Retrain periodically. Cluster assignments from months ago may not hold today.

A production checklist

Before deploying a k-means pipeline, we run through this:

1. Data exploration: Check distributions, ranges, and missing values. Decide whether to drop, impute, or engineer features.
2. Feature scaling: Apply StandardScaler or MinMaxScaler consistently. Save the scaler object; you'll need it for new data.
3. Elbow and silhouette analysis: Plot both metrics across a reasonable range of k (usually 2–10).
4. Fit the model: Use n_init=10 or higher to avoid local minima (k-means is non-deterministic without careful seeding).
5. Visual inspection: Reduce to 2D (PCA, t-SNE) and plot clusters. Do they look sensible?
6. Business validation: Show results to a domain expert. Do the segments align with known patterns or business goals?
7. Drift monitoring: Log cluster assignments over time. If the distribution of cluster sizes shifts dramatically, retraining is needed.

Key takeaways

• K-means is fast and practical, but scale your features first—mismatched ranges will ruin your segmentation.
• Use the elbow method and silhouette score together to choose k; neither is perfect alone.
• K-means assumes round, similarly-sized clusters; inspect results visually to verify this assumption holds.
• If clusters are irregular or densities vary widely, use DBSCAN instead; it's slower but more flexible.
• Always validate clusters with domain experts—mathematical quality doesn't guarantee business value.
• Treat clustering as an ongoing process, not a one-time task; retrain and monitor for drift in production.