AI-powered search for products: best practices explained

Search is often the first interaction a customer has with your product catalog, and it's also where most e-commerce conversions either start or stall. AI-powered search for products: best practices explained moves beyond simple keyword matching to understand what users actually want—their intent—and serve results that drive both engagement and revenue. In this guide, we'll walk through the production techniques we've used across multiple platforms to build search systems that understand context, respect business constraints, and measurably improve outcomes.

Understanding the problem with traditional keyword search

Before we discuss AI-powered approaches, let's be honest about why basic keyword search fails. A customer searching for "running shoe" might be looking for shoes for marathons, casual jogging, trail running, or fashion. A simple substring or BM25 match doesn't capture this nuance. You end up with results ordered by document frequency, not by what actually matches intent. Stock-outs and low-margin items float to the top. Conversion rates plateau.

The real cost compounds: poor search drives traffic to category pages and filters instead of product detail pages, which extends user sessions, increases bounce rates, and skews analytics. We've seen this firsthand in e-commerce platforms where search improvement alone moved conversion rates by 2–4 percentage points—that's often a seven-figure annual impact.

Embed product titles and descriptions

The first step toward intent-aware search is converting your product catalog into dense vector embeddings. An embedding is a numerical representation—usually 384 to 1536 dimensions—that captures semantic meaning. Products with similar meaning cluster together in this space, so a search query and a product can be compared for relevance independently of exact keyword overlap.

Here's how we typically approach this in production:

1. Combine your product title, description, category, and tags into a single text document per product.
2. Use a pre-trained embedding model—we often choose all-MiniLM-L6-v2, all-mpnet-base-v2, or proprietary models if you have the scale—to generate embeddings.
3. Store embeddings in a vector database like Pinecone, Weaviate, or Milvus, alongside the original product metadata.
4. At query time, embed the user's search string with the same model and find nearest neighbors.

A practical example: a customer types "lightweight marathon shoe." Your embedding model maps this to a high-dimensional vector. Products genuinely related to marathon running—even if they use different vocabulary like "distance running" or "endurance footwear"—end up close in vector space, so they rank higher.

Here's a minimal Python example using a popular open-source library:

from sentence_transformers import SentenceTransformer
import numpy as np

# Load a pre-trained embedding model
model = SentenceTransformer('all-MiniLM-L6-v2')

# Prepare product documents
products = [
    {
        'id': 'prod_001',
        'doc': 'Nike Pegasus 40 lightweight marathon running shoe responsive cushioning'
    },
    {
        'id': 'prod_002',
        'doc': 'Brooks Ghost heavy-duty motion control orthopedic support walking'
    },
    {
        'id': 'prod_003',
        'doc': 'Salomon Speedcross trail running trail shoe aggressive tread'
    }
]

# Embed all products
product_embeddings = {}
for product in products:
    embedding = model.encode(product['doc'])
    product_embeddings[product['id']] = embedding

# Embed user query
query = "lightweight marathon shoe"
query_embedding = model.encode(query)

# Compute similarities
similarities = {}
for prod_id, prod_embedding in product_embeddings.items():
    score = np.dot(query_embedding, prod_embedding) / (
        np.linalg.norm(query_embedding) * np.linalg.norm(prod_embedding)
    )
    similarities[prod_id] = score

# Sort by relevance
ranked = sorted(similarities.items(), key=lambda x: x[1], reverse=True)
print(ranked)
# Output: [('prod_001', 0.87), ('prod_003', 0.62), ('prod_002', 0.41)]

Notice that the Nike Pegasus ranks first for "lightweight marathon shoe" even though it doesn't contain the exact phrase. The model understands semantic relationships. This is the foundation of intent-aware search.

Combine with keyword search and filters

Semantic search alone isn't enough in production. You need a hybrid approach: vector similarity handles intent, but keyword matching and filtering handle precision and business logic.

Consider a catalog with millions of products. Pure semantic search can be slow and expensive. Also, sometimes users want exact matches—they know the brand, the model number, they just want confirmation. A hybrid strategy retrieves top candidates from both a keyword index (Elasticsearch, or your database's full-text search) and a vector store, then merges and reranks them.

Here's a typical flow:

1. Keyword retrieval: Use Elasticsearch (or PostgreSQL with tsvector, or any full-text engine) to find products matching exact or fuzzy terms. This is fast and handles typos via built-in analyzers.
2. Vector retrieval: In parallel, query your vector database for semantic matches, limiting to the top 100–500 results for speed.
3. Merge candidate sets: Combine results from both, removing duplicates. You now have a smaller, higher-quality candidate pool.
4. Apply filters: Respect user selections: size, color, price range, brand, stock status. Filter the candidate pool.
5. Rerank: Pass remaining candidates through a learned-to-rank model (discussed next).

In our experience, hybrid search improves recall (you find more relevant products) by 15–25% compared to vector-only or keyword-only approaches, with minimal latency overhead if you parallelize retrieval.

A production example (pseudocode in Node.js/TypeScript):

async function hybridSearch(query, filters) {
  // Parallel keyword and vector retrieval
  const [keywordResults, vectorResults] = await Promise.all([
    elasticsearch.search({
      index: 'products',
      body: {
        query: {
          multi_match: {
            query: query,
            fields: ['title^2', 'description', 'category']
          }
        },
        size: 200
      }
    }),
    vectorDb.query({
      vector: await embedModel.encode(query),
      topK: 200,
      namespace: 'products'
    })
  ]);

  // Merge and deduplicate
  const candidateMap = new Map();
  keywordResults.hits.forEach(hit => {
    candidateMap.set(hit._id, { 
      id: hit._id, 
      keywordScore: hit._score, 
      ...hit._source 
    });
  });
  vectorResults.matches.forEach(match => {
    const existing = candidateMap.get(match.id);
    if (existing) {
      existing.vectorScore = match.score;
    } else {
      candidateMap.set(match.id, { 
        id: match.id, 
        vectorScore: match.score, 
        ...match.metadata 
      });
    }
  });

  // Apply user filters
  let candidates = Array.from(candidateMap.values());
  if (filters.price) {
    candidates = candidates.filter(p => 
      p.price >= filters.price.min && p.price <= filters.price.max
    );
  }
  if (filters.brand) {
    candidates = candidates.filter(p => 
      filters.brand.includes(p.brand)
    );
  }

  return candidates; // Pass to reranking next
}

Rerank by business signals like stock

Now you have a high-quality candidate pool, but raw semantic or keyword scores don't account for business reality. A product might be semantically perfect but out of stock, low margin, or bundled with something more profitable. Reranking bridges this gap.

We use a learned-to-rank (LTR) approach: train a gradient boosting model (XGBoost, LightGBM) on your historical search data—clicks, conversions, dwell time—to predict which products users actually prefer. Features include:

• Semantic and keyword scores (from retrieval)
• Stock level and fulfillment speed
• Price and margin
• Historical conversion rate for this query or similar queries
• Freshness (new products often perform better)
• Review rating and review count
• Category affinity (if the user browsed running shoes, prioritize running shoes in results)
• Personalization signals (user's past purchases, saved items)

The model learns a weighting over these features that maximizes your business metric—typically conversion rate or revenue per search. In practice, reranking lifts conversion by 5–15% because it deprioritizes "technically correct but unprofitable" results.

A simplified example with Python and LightGBM:

import lightgbm as lgb
import numpy as np

# Training data: rows are (query, product) pairs with labels
# Label = 1 if the user clicked/converted, 0 otherwise
X_train = np.array([
    [0.85, 0.92, 100, 0.28, 4.7, 245],  # semantic score, keyword score, stock, margin, rating, price
    [0.72, 0.88, 5, 0.32, 4.2, 180],    # another product
    # ... many more
])
y_train = np.array([1, 0, 1, 0, 1, ...])  # click/conversion labels

# Train LTR model
model = lgb.LGBMRanker(objective='rank_xendcg')
model.fit(X_train, y_train)

# At query time, score candidates
candidates = [
    {'semantic_score': 0.80, 'keyword_score': 0.85, 'stock': 150, 'margin': 0.30, 'rating': 4.6, 'price': 200},
    {'semantic_score': 0.78, 'keyword_score': 0.81, 'stock': 0, 'margin': 0.35, 'rating': 4.8, 'price': 190},
    # ...
]

features = np.array([
    [c['semantic_score'], c['keyword_score'], c['stock'], c['margin'], c['rating'], c['price']]
    for c in candidates
])

scores = model.predict(features)
ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
print(ranked)  # Reranked results

The out-of-stock product drops in rank despite its better rating, because the model learned that in-stock products convert better. This is learned, not hardcoded—your model adapts as your business priorities change.

Measure with click-through and conversion

Without measurement, you're flying blind. We instrument every search interaction to compute:

• Click-through rate (CTR): percentage of impressions that result in a click. Higher CTR usually means better ranking.
• Conversion rate: percentage of searches that result in a purchase. This is your north star.
• Mean reciprocal rank (MRR): average position of the first click. MRR closer to 1 is better.
• Normalized discounted cumulative gain (NDCG): measures ranking quality accounting for position decay. Higher is better.

Set up logging to capture:

# When a user performs a search
search_event = {
    'timestamp': datetime.utcnow(),
    'user_id': 'user_123',
    'query': 'lightweight marathon shoe',
    'results': [
        {'product_id': 'prod_001', 'rank': 1, 'semantic_score': 0.87},
        {'product_id': 'prod_003', 'rank': 2, 'semantic_score': 0.62},
        # ...
    ]
}
log_search(search_event)

# When a user clicks a result
click_event = {
    'timestamp': datetime.utcnow(),
    'user_id': 'user_123',
    'search_id': 'search_event.id',
    'product_id': 'prod_001',
    'rank_clicked': 1,
    'dwell_time_seconds': 45
}
log_click(click_event)

# When a user converts (purchases)
conversion_event = {
    'timestamp': datetime.utcnow(),
    'user_id': 'user_123',
    'search_id': 'search_event.id',
    'product_id': 'prod_001',
    'revenue': 129.99
}
log_conversion(conversion_event)

Aggregate these logs daily and compute metrics per query or per ranking algorithm variant. Run A/B tests: expose 10% of traffic to a new reranking model, compare conversion rate against control. If it wins, roll it out to 100%.

We've seen teams make the mistake of optimizing for CTR alone (higher CTR sounds good) without checking conversion. Sometimes showing a flashy but low-quality product first drives clicks but hurts revenue. Conversion rate is what matters.

Common mistakes

Over-optimizing for semantic relevance: A beautiful semantic match doesn't guarantee conversion if the product is overpriced, out of stock, or has poor reviews. Always include business signals in reranking.

Ignoring latency: Searching a billion-product catalog with vector similarity alone takes seconds. Hybrid retrieval (keyword + vector in parallel, then rerank a smaller set) keeps latency under 200ms in production.

Stale embeddings: If you embed your catalog once and never update, new products and product changes won't be reflected. Regenerate embeddings on a schedule (daily or weekly depending on catalog churn).

Not accounting for typos and misspellings: Embeddings handle synonyms but not "niek" instead of "nike." Pair semantic search with fuzzy matching or a spell-corrector upstream.

Treating all users the same: Personalization—incorporating user history, past purchases, browsing behavior—can lift conversion by 10–20%. Don't skip it.

Key takeaways

• Embed your product titles and descriptions with a pre-trained semantic model to capture intent beyond keywords.
• Combine vector search with keyword search and filters in a hybrid pipeline for both recall and precision.
• Train a learned-to-rank model on your historical click and conversion data to rerank candidates by actual user preference, incorporating business signals like stock and margin.
• Measure everything: click-through rate, conversion rate, MRR, and NDCG. Run A/B tests before rolling out new ranking algorithms.
• Keep latency below 200ms by parallelizing retrieval and reranking only a smaller candidate pool.
• Update embeddings and LTR models regularly as your catalog and user behavior evolve.
• Personalize based on user history and context—it compounds the gains from intent-aware ranking.