100 days of Elasticsearch and search engineering: the recap

Introduction

A hundred days ago, we started this journey into Elasticsearch and search engineering with the conviction that search is not an afterthought bolted onto the side of your application—it's a product in itself. We've covered mappings, analyzers, relevance tuning, operational patterns, security, performance optimization, and the emerging frontier of hybrid and vector search. This is the recap: 100 days of Elasticsearch and search engineering: the recap distills the hard-won lessons that separate production-grade search from something that merely returns results.

The knowledge we've built together applies whether you're running search for an e-commerce catalog, a SaaS knowledge base, a trading platform feed, or an internal document repository. What works in theory often breaks in practice—and we've learned why.

Mappings Decide Quality — Text, Keyword and Analyzers Matter

Your mapping is the contract between your data and Elasticsearch. Get it wrong and no amount of tuning later will fix the core problem. We learned that fields are not generic buckets; they must be intentional.

Text fields and keyword fields serve different purposes. A product name should be indexed as both: as text for fuzzy matching and relevance, and as keyword for aggregations and exact filters. An email address should be keyword-only—there's nothing to analyze. A product description needs a full analyzer pipeline; a SKU does not.

Consider this real scenario: a travel platform indexed hotel locations as text, which meant "New York" was stemmed and tokenized. Faceted filtering then broke because the aggregation saw fragments instead of whole values. The fix was simple—add a keyword subfield—but it required a reindex. That's expensive at scale.

Here's the kind of mapping we recommend now:

{
  "mappings": {
    "properties": {
      "name": {
        "type": "text",
        "analyzer": "standard",
        "fields": {
          "keyword": {
            "type": "keyword"
          }
        }
      },
      "description": {
        "type": "text",
        "analyzer": "english"
      },
      "location": {
        "type": "keyword"
      },
      "price": {
        "type": "float"
      },
      "created_at": {
        "type": "date"
      }
    }
  }
}

Custom analyzers matter too. English analyzers stem words—"running" becomes "run"—which improves recall but can hurt precision. A ride-hailing platform discovered that searching for "driver" was matching "drive," producing false positives. They built a custom analyzer with a synonym filter and a stop-word list tuned to their domain.

The lesson: invest time in your mapping at day one. Understand your queries and your data. Test analyzers against real search terms before going to production.

Relevance Is a Product — Boosts, Synonyms and Measurement

Relevance tuning is where search engineering becomes a discipline. You can't improve what you don't measure. We've learned that relevance requires three things: a scoring strategy, feedback loops, and continuous iteration.

Boosts are your primary lever. You might boost recently updated documents, bestsellers, or verified sellers. A CRM platform boosted results from accounts with longer engagement history, creating a virtuous cycle where power users got better results and engagement went up. They monitored click-through rate and conversion as their north star.

Synonyms are underrated. Users search for "broken" but you stock "defective." Users search for "running shoes" but your catalog says "athletic footwear." Synonyms bridge that gap. The catch: test them. Overly aggressive synonym rules can muddy precision. One e-commerce team added "phone" as a synonym for "mobile" and suddenly searched for "landline phones" returned mobile phones.

Here's a practical synonym filter configuration:

{
  "settings": {
    "analysis": {
      "filter": {
        "my_synonyms": {
          "type": "synonym",
          "synonyms": [
            "quick,fast,rapid",
            "jumps,leaps,hops",
            "broken => defective",
            "running shoes => athletic footwear"
          ]
        }
      },
      "analyzer": {
        "my_analyzer": {
          "tokenizer": "standard",
          "filter": ["lowercase", "my_synonyms"]
        }
      }
    }
  }
}

Measurement is mandatory. Track queries that return zero results. Monitor mean reciprocal rank and precision at k. A/B test ranking changes. One trading platform discovered that their default sort (by price) was actually hurting fill rates; sorting by recency with a price boost worked better. Numbers proved it.

Relevance is a product because it changes what users see, influences behavior, and drives business outcomes. Treat it with the same rigor you'd apply to your checkout flow.

Operate With Care — Shards, Aliases, ILM and Security

We've seen production outages caused by poor shard strategy, unplanned downtime from missing aliases, data loss from ignored retention policies, and breaches from unsecured clusters. Operations matter as much as features.

Shard count is a common mistake. Too many shards and overhead becomes substantial; too few and you can't parallelize queries. A rule of thumb: aim for shards between 10GB and 50GB each. Monitor shard count over time. One ride-hailing platform had shards that drifted to 200GB—query latency climbed and they had to do an emergency reindex.

Aliases decouple your application from index names. When you need to reindex, you create a new index and switch the alias. Zero downtime. We've seen teams skip aliases and get trapped: they can't add fields, can't tune analyzers, can't upgrade without downtime. Aliases cost nothing and save everything.

PUT /_aliases
{
  "actions": [
    {
      "add": {
        "index": "products-v2",
        "alias": "products"
      }
    },
    {
      "remove": {
        "index": "products-v1",
        "alias": "products"
      }
    }
  ]
}

Index Lifecycle Management (ILM) automates what used to be manual busywork. Define policies that roll over indices, delete old data, optimize for read-only phases. A SaaS platform running logs through Elasticsearch used ILM to keep only the last 90 days, saving 60% on storage and reducing query overhead at the same time.

Security is not optional. Encrypt in transit, use strong authentication, restrict network access, audit who queries what, encrypt sensitive fields. We've seen teams skip TLS "just for dev" and it's still there in prod a year later. One financial trading platform discovered a data breach because search queries weren't logged. Now every search request is audited.

Search Is Evolving — Hybrid and Vector Search for AI Features

Keyword and full-text search have dominated for two decades. They're still the foundation. But vector search and hybrid approaches are now table stakes for new features.

Vector search lets you embed user intent, product features, or document content as dense numerical representations. A travel platform embedded hotel amenity descriptions and guest preferences as vectors, then found similar properties using cosine similarity. Results became more nuanced. A customer searching for "quiet place near beach" got recommendations that matched semantic meaning, not just keyword presence.

Hybrid search combines keyword precision with semantic understanding. Query a full-text index for exact matches, simultaneously query a vector index for semantic neighbors, and merge results. A CRM platform hybrid-searches accounts by company name (keyword) and by business similarity (vector), giving users both exact hits and intelligent alternatives when exact matches are sparse.

The challenge with vectors: they're dense, require external embedding models, and add complexity. You need to embed new documents on ingestion, embed queries at search time, and choose the right model for your domain. A team built semantic search for their knowledge base using a general-purpose embedding model, only to find it performed poorly on domain-specific jargon. They retrained with domain data and results improved dramatically.

Vector search in Elasticsearch looks like this:

{
  "mappings": {
    "properties": {
      "title": {
        "type": "text"
      },
      "embedding": {
        "type": "dense_vector",
        "dims": 384,
        "index": true,
        "similarity": "cosine"
      }
    }
  }
}

The future of search is not vectors or keywords—it's both. Combine them intelligently and you unlock new product capabilities.

Common Mistakes

• Ignoring the mapping: Changing analyzers or field types after data is indexed requires a reindex. Plan your mapping. Get feedback. Move slowly the first time.
• No replication or backups: We've seen teams lose data because they ran a single node or never tested recovery. Use at least one replica in production. Test backup and restore procedures.
• Treating relevance as set-and-forget: Relevance decays. User behavior changes. Queries evolve. Monitor. Iterate. A/B test ranking changes.
• Scaling horizontally without understanding shards: Adding nodes doesn't help if your shard strategy is wrong. Know your data volume and access patterns first.
• Embedding confidential data in vectors: Vector models are trained on text. If you embed something sensitive, it's exposed. Mask PII before embedding.

Key Takeaways

• Mappings are foundational. Text, keyword, and analyzer choices drive quality and must be intentional. Plan them carefully and version your indices.
• Relevance is a product, not a feature. Measure click-through rate, precision, and user behavior. Boost strategically. Test synonyms. Iterate.
• Operationalize with aliases, ILM, proper shard sizing, and security from day one. Design for reindexing and avoid downtime.
• Vector search and hybrid approaches are now viable. Embed semantic understanding alongside keyword matching for better user experiences.
• Search engineering is a discipline. It requires domain knowledge, measurement, iteration, and a commitment to solving for the user's intent, not just matching strings.