Recommendation systems basics: a practical AI & ML guide

Introduction

By day 53 of building AI and ML systems, you've likely worked with classification, regression, and clustering. Now comes one of the most commercially valuable problems: how do we suggest the right product, article, or user connection to each person? This guide covers recommendation systems basics: a practical AI & ML guide that scales from a startup's first feature to enterprise personalization pipelines. We'll walk through four proven patterns you can implement today, from the humble popularity baseline through collaborative filtering, and show you how to measure what actually moves the needle.

Why recommendation systems matter

Recommendation systems are everywhere in production. E-commerce sites show "customers who bought this also bought that." Ride-hailing platforms match riders with drivers. Travel platforms surface hotel stays. SaaS products recommend features or content to reduce churn. The business impact is direct: better recommendations lift engagement, conversion, and retention. Yet many teams jump to complex approaches before validating that a simple baseline works well enough.

We'll start simple and build up, because the best recommendation system is one that ships, measures impact, and iterates—not one that's theoretically perfect but never reaches users.

Popularity baseline — Most popular items are hard to beat at first

Before you build anything sophisticated, establish a popularity baseline. This is the simplest possible recommender: rank items by how often they're purchased, clicked, or engaged with across your entire user base, then show the top items to everyone.

Why start here? First, it's fast to implement and requires minimal infrastructure. Second, it often performs better than you'd expect, especially for new users or cold-start scenarios where you have no history. Third, it gives you a benchmark—any fancier approach must beat this or it's waste.

In a real e-commerce platform, your popularity baseline might look like this:

import pandas as pd
from datetime import datetime, timedelta

# Load transaction history
transactions = pd.read_csv('transactions.csv')

# Count purchases in last 30 days
thirty_days_ago = datetime.now() - timedelta(days=30)
recent_transactions = transactions[
    pd.to_datetime(transactions['timestamp']) > thirty_days_ago
]

# Group by product and count
popularity = recent_transactions.groupby('product_id').size().reset_index(name='count')
popularity_sorted = popularity.sort_values('count', ascending=False)

# Top 20 products
top_products = popularity_sorted.head(20)['product_id'].tolist()

# For any user, return top products
def get_recommendations(user_id, num_recommendations=5):
    return top_products[:num_recommendations]

print(get_recommendations('user_123'))

Notice we use a time window (30 days) rather than all-time popularity. Seasonal trends, product launches, and inventory changes matter. A winter coat might have been popular in January but is dead stock by July.

When to use popularity baseline: new user onboarding, cold-start problems, or as a fallback when personalized recommendations fail. When not to use it: if your goal is differentiation or if you want to surface niche items. A purely popularity-driven algorithm will concentrate recommendations on the same bestsellers for everyone, which feels generic and hurts discoverability for long-tail products.

Content-based — Recommend items similar to what the user liked

Content-based recommendation systems work on a simple principle: if a user liked item A, show them items similar to A. Similarity is computed using features—title, category, price range, embeddings from descriptions, or even embeddings trained on click behavior.

Imagine a travel platform. A user searched for "beachfront hotels in Bali under $150." Rather than just showing popularity, we can find other hotels with the same tags: beachfront location, tropical region, budget-friendly. If we've encoded hotel descriptions as embeddings (using a pre-trained model like sentence-transformers), we can compute cosine similarity and rank candidates by relevance.

from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity
import numpy as np

# Load a pre-trained embedding model
model = SentenceTransformer('all-MiniLM-L6-v2')

# Hotel descriptions (from database)
hotels = {
    'h1': 'Beachfront bungalow in Bali with ocean views, budget-friendly',
    'h2': 'Luxury mountain resort in Swiss Alps, expensive',
    'h3': 'Seaside villa Bali Indonesia beach sunset budget',
    'h4': 'Penthouse Tokyo downtown expensive modern'
}

# Embed all hotel descriptions
embeddings = {}
for hotel_id, description in hotels.items():
    embeddings[hotel_id] = model.encode(description)

# User liked h1 (beachfront Bali budget)
liked_hotel = 'h1'
liked_embedding = embeddings[liked_hotel]

# Compute similarity to all other hotels
similarities = {}
for hotel_id, emb in embeddings.items():
    if hotel_id != liked_hotel:
        sim = cosine_similarity([liked_embedding], [emb])[0][0]
        similarities[hotel_id] = sim

# Sort by similarity
recommendations = sorted(similarities.items(), key=lambda x: x[1], reverse=True)
print("Recommended hotels:", [h[0] for h in recommendations])
# Output: [('h3', 0.92), ('h2', 0.15), ('h4', 0.10)]

In practice, you'll combine multiple signals: category match, price range, location distance, and embedding similarity. You might weight them differently based on business rules or learned weights.

Advantages: interpretable (you can explain why), doesn't require user-user data, handles new items well. Disadvantages: misses cross-category discovery (if someone likes sci-fi novels, content-based won't suggest cooking books even if they might), and requires good features or embeddings.

Collaborative filtering — Users who liked X also liked Y

Collaborative filtering operates on a different principle: ignore item features entirely. Instead, use the pattern that "users similar to you liked these items" or "users who bought X also bought Y."

There are two flavors. User-based: find users with similar purchase history, then recommend what they liked. Item-based: find items that co-occur in user baskets, then recommend them together. Item-based is often more practical at scale because item relationships are more stable than user similarity.

Here's a simplified item-based approach for a CRM platform:

from collections import defaultdict
import pandas as pd

# Transaction data: user -> items purchased
transactions_df = pd.read_csv('transactions.csv')

# Build item co-occurrence matrix
co_occurrence = defaultdict(lambda: defaultdict(int))

for user_id, group in transactions_df.groupby('user_id'):
    items = group['item_id'].tolist()
    # For each pair of items this user bought, increment counter
    for i, item_a in enumerate(items):
        for item_b in items[i+1:]:
            co_occurrence[item_a][item_b] += 1
            co_occurrence[item_b][item_a] += 1

# User viewed/purchased item_5, recommend similar items
viewed_item = 'item_5'
related_items = co_occurrence.get(viewed_item, {})
recommendations = sorted(
    related_items.items(), 
    key=lambda x: x[1], 
    reverse=True
)[:5]

print("Users who bought item_5 also bought:", recommendations)
# Output: [('item_12', 45), ('item_7', 38), ...]

This is fast, scalable, and powerful. But it has a cold-start problem: new items with no purchase history get zero recommendations, and new users appear in no co-occurrence patterns.

Hybrid approaches: in production, we combine all three. For a new user, fall back to popularity or content-based. For returning users with rich history, use collaborative filtering. As the user and items age, blend in content similarity to handle drift.

Measure online — A/B test click-through and conversions

Everything we've built so far is offline. You can evaluate recommendations on historical data (precision, recall, NDCG), but those metrics don't guarantee business impact. The gold standard is online A/B testing.

Setup: split users randomly into control (popularity baseline) and treatment (your new approach). Run for at least one week (or 10,000 users, whichever is larger) to avoid novelty effects. Measure:

• Click-through rate (CTR): how many users clicked a recommendation?
• Conversion rate: how many clicks led to purchase or signup?
• Revenue per user: average order value of recommended items.
• Engagement: time on page, repeat visits.
• Diversity: percentage of users who saw at least one item outside their typical category.

Here's a simple A/B test harness:

import random
from datetime import datetime

def assign_variant(user_id, test_name, variant_ratio=0.5):
    """Assign user to control or treatment consistently."""
    seed = int(user_id) + hash(test_name)
    return 'treatment' if random.Random(seed).random() < variant_ratio else 'control'

def log_event(user_id, event_type, variant, details):
    """Log to analytics platform (Segment, Mixpanel, custom DB)."""
    event = {
        'timestamp': datetime.now().isoformat(),
        'user_id': user_id,
        'event_type': event_type,
        'variant': variant,
        'details': details
    }
    # Send to data warehouse
    print(f"Event logged: {event}")

# In your web handler
def show_recommendations(user_id):
    variant = assign_variant(user_id, 'recommendation_v2')
    
    if variant == 'control':
        recs = get_popularity_baseline(num=5)
    else:
        recs = get_collaborative_filtering(user_id, num=5)
    
    log_event(user_id, 'recommendations_shown', variant, {'recs': recs})
    return recs

def handle_recommendation_click(user_id, item_id):
    variant = assign_variant(user_id, 'recommendation_v2')
    log_event(user_id, 'recommendation_clicked', variant, {'item_id': item_id})

def handle_purchase(user_id, item_id, amount):
    variant = assign_variant(user_id, 'recommendation_v2')
    log_event(user_id, 'purchase', variant, {'item_id': item_id, 'amount': amount})

After the test, run stats. If treatment CTR is 12% vs control 10%, with p-value < 0.05, it's statistically significant. If it's significant but revenue per user is the same (clicks don't convert), the recommendation isn't valuable—iterate.

Common pitfalls: running tests too short (variance swamps signal), confusing correlation with causation (new users might click more regardless of recommendation), and optimizing for the wrong metric (clicks don't matter if they don't lead to purchase).

Common mistakes

• Skipping the baseline. You'll overestimate how much your fancy model helps. Always run A/B tests against simplicity, not just offline metrics.
• Optimizing for the wrong signal. High CTR on recommendations is nice, but does it increase revenue? LTV? Retention? Define success before you build.
• Ignoring cold-start. Your algorithm may work beautifully for power users, but new users see nothing useful. Have a graceful fallback.
• Recommending too much of the same category. If everyone gets recommendations from category X, your platform becomes boring. Inject diversity or serendipity intentionally.
• Not monitoring drift. User preferences shift seasonally. Embeddings degrade as language evolves. Retrain and re-evaluate every quarter at minimum.
• Assuming correlation in embeddings. Two products with similar descriptions might not be substitutes (a tent and a sleeping bag are related but serve different needs). Validate with real user behavior.

Key takeaways

• Start with a popularity baseline. It's simple, fast, and often good enough. Any new approach must beat it in A/B tests to ship.
• Content-based systems use item features or embeddings to find similar products. Great for interpretability and new item cold-start, but misses cross-category discovery.
• Collaborative filtering learns from user-item interactions. Powerful for personalization but struggles with new users and new items. Item-based is more scalable than user-based.
• Hybrid systems combine all three. Use popularity for cold-start, collaborative filtering for warm users, and content-based as a fallback and diversity mechanism.
• Online A/B testing is mandatory. Offline metrics (precision, recall) don't guarantee business impact. Measure CTR, conversion, revenue, and diversity.
• Monitor for drift and cold-start failures. Retrain embeddings quarterly. Have fallbacks for new users and new items or your platform will feel stale.
• Optimize for business outcomes, not clicks. A recommendation that gets clicked but doesn't convert or increase LTV is noise, not value.