Custom analyzers: Elasticsearch best practices explained

Introduction

Text analysis is where many search implementations stumble—you index documents one way, query another, and wonder why "colour" doesn't match "color" or why acronyms get mangled. Custom analyzers: Elasticsearch best practices explained shows you how to take control of the entire text processing pipeline, from raw input to searchable tokens. In production systems handling user searches across product catalogs, support tickets, or user-generated content, getting this right is the difference between a search experience users trust and one that frustrates them.

By day nine of serious Elasticsearch work, you've likely seen the default standard analyzer and its limitations. Today we'll show you how to build analyzers that understand your data's quirks, handle multiple languages, and give you the precision your business needs.

Define custom analyzers in index settings

Every index in Elasticsearch ships with a default standard analyzer, but real-world data rarely fits neatly into defaults. Custom analyzers live in your index settings—think of them as configuration blueprints that define exactly how text gets broken down and transformed.

When you create an index, you specify analyzers under settings.analysis.analyzer. Each analyzer gets a name, and you can reference that name in your field mappings. Let's say we're building search for an e-commerce platform selling clothing. We want to handle product titles with special characters, size codes, and brand names intelligently.

{
  "settings": {
    "analysis": {
      "analyzer": {
        "product_title_analyzer": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": ["lowercase", "stop", "snowball"]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "title": {
        "type": "text",
        "analyzer": "product_title_analyzer"
      }
    }
  }
}

This example defines a custom analyzer called product_title_analyzer that will be used whenever text is indexed into the title field. The analyzer has three pieces: a tokenizer (how to split text) and two filters (how to transform those tokens). We'll break down these components in the next section, but notice that defining the analyzer in settings and then referencing it in mappings keeps your configuration clean and reusable across multiple fields.

One practical tip from teams we've worked with: version your analyzer names. As your search requirements evolve, you might create product_title_analyzer_v2 and reindex gradually, avoiding the disruption of updating live indices.

Combine tokenizers and filters strategically

A tokenizer splits text into individual tokens. A filter processes each token, optionally removing it or transforming it. Think of the tokenizer as a knife and filters as refinements—the knife cuts, then each filter polishes or discards what the knife created.

Common tokenizers include:

• standard – splits on word boundaries and punctuation; the safe default.
• keyword – treats the entire input as a single token; useful for IDs or fixed formats.
• whitespace – splits only on whitespace; useful when you control the input format.
• ngram – creates substrings; essential for prefix matching and typo tolerance.
• path_hierarchy – tokenizes hierarchical paths like URLs or file systems.

Filters process tokens after tokenization. Some common ones:

• lowercase – converts to lowercase for case-insensitive matching.
• stop – removes common words like "the" and "a"; reduces noise.
• snowball – stemming filter that reduces words to roots (e.g., "running" → "run").
• synonym – expands or replaces tokens based on a synonym list.
• asciifolding – converts accented characters to ASCII equivalents.
• length – removes tokens below or above a certain character count.

The order of filters matters. Lowercase should generally come early so subsequent filters work on consistent input. Here's a more sophisticated analyzer for a travel booking platform that handles hotel and destination searches:

{
  "settings": {
    "analysis": {
      "filter": {
        "destination_stop": {
          "type": "stop",
          "stopwords": ["_english_", "hotel", "room", "book"]
        }
      },
      "analyzer": {
        "destination_search": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": ["lowercase", "destination_stop", "snowball", "asciifolding"]
        }
      }
    }
  }
}

Notice we defined a custom stop filter called destination_stop that includes domain-specific words ("hotel", "room") alongside English stopwords. Then our analyzer uses it. This prevents noise and improves recall—when someone searches for hotels in "São Paulo", the asciifolding filter converts it to "sao paulo", matching records regardless of accent encoding.

Leverage synonyms and ASCII folding

Synonyms are a powerful tool for bridging the language gap between how users search and how your content is written. If your e-commerce platform sells both "trainers" and "sneakers", without synonyms a UK customer searching for "trainers" won't find US-stocked inventory labeled "sneakers".

Synonyms are defined as a filter and can be specified inline or loaded from a file:

{
  "settings": {
    "analysis": {
      "filter": {
        "synonyms_filter": {
          "type": "synonym",
          "synonyms": [
            "trainers, sneakers, running shoes",
            "couch => sofa",
            "jumper, sweater, pullover"
          ]
        }
      },
      "analyzer": {
        "product_analyzer": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": ["lowercase", "synonyms_filter"]
        }
      }
    }
  }
}

When a document contains "trainers", the synonym filter expands it to include "sneakers" and "running shoes" during indexing. The => notation is a one-way replacement: "couch" becomes "sofa" (useful when you want to normalize inconsistent terminology). Without the arrow, all terms are treated as equivalent in both directions.

For production use, load synonyms from a file so you can update them without reindexing:

{
  "filter": {
    "synonyms_filter": {
      "type": "synonym",
      "synonyms_path": "synonyms.txt"
    }
  }
}

Place synonyms.txt in the Elasticsearch config directory or use a mounted volume in containerized deployments. Each line is one synonym rule.

ASCII folding solves a complementary problem: international characters. The asciifolding filter converts accented characters to their base ASCII form. "Müller" becomes "muller", "Ñoño" becomes "nono". This is essential for platforms serving international customers. Add it near the end of your filter chain:

{
  "analyzer": {
    "international_search": {
      "type": "custom",
      "tokenizer": "standard",
      "filter": ["lowercase", "stop", "snowball", "asciifolding"]
    }
  }
}

Use different analyzers for indexing versus searching

This is where many teams miss optimization opportunities. You don't always want the same analyzer during indexing and querying. Consider a CRM system with customer company names. During indexing, you might apply aggressive stemming to improve recall. During search, you want the query to match more precisely.

Set an analyzer for indexing and a different one for searching using analyzer and search_analyzer in your field mapping:

{
  "mappings": {
    "properties": {
      "company_name": {
        "type": "text",
        "analyzer": "company_index_analyzer",
        "search_analyzer": "company_search_analyzer"
      }
    }
  },
  "settings": {
    "analysis": {
      "analyzer": {
        "company_index_analyzer": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": ["lowercase", "snowball", "asciifolding"]
        },
        "company_search_analyzer": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": ["lowercase", "asciifolding"]
        }
      }
    }
  }
}

The indexing analyzer uses snowball stemming, so "running", "runs", and "run" all become "run". The search analyzer skips stemming, so a user's query for "running" is not stemmed—it matches the stemmed "run" in the index anyway. This approach gives you better recall (finding more results) without introducing false positives that aggressive stemming can cause at query time.

Another practical scenario: e-commerce platforms often use edge_ngram tokenizer during indexing for autocomplete, but regular tokenization during search. The index contains substrings like "app", "appl", "apple", while the query "apple" matches cleanly without substring noise.

{
  "analyzer": {
    "autocomplete_index": {
      "type": "custom",
      "tokenizer": {
        "type": "edge_ngram",
        "min_gram": 2,
        "max_gram": 20,
        "token_chars": ["letter", "digit"]
      },
      "filter": ["lowercase"]
    },
    "autocomplete_search": {
      "type": "custom",
      "tokenizer": "standard",
      "filter": ["lowercase"]
    }
  }
}

Common mistakes

Mistake 1: Applying stemming uniformly without understanding domain impact. Stemming "policies" to "polic" might cause false matches in insurance platforms. Test aggressively against real queries before deploying.

Mistake 2: Reindexing without an alias strategy. When you need to update an analyzer, you must reindex. If you update the analyzer in-place and then start reindexing, old data still uses the old analyzer. Use aliases: create a new index with the updated analyzer, reindex, then swap the alias. Zero downtime.

Mistake 3: Ignoring the order of filters. Lowercase must come before synonym matching (otherwise "The" won't match synonyms for "the"). Stemming should come before or after synonyms depending on whether your synonym list contains root forms or inflected forms.

Mistake 4: Not testing search_analyzer separately. It's easy to assume the search analyzer will work because indexing seems fine. Write explicit queries using the analyze API to debug:

POST /my_index/_analyze
{
  "analyzer": "company_search_analyzer",
  "text": "running companies"
}

Mistake 5: Hardcoding stopwords or synonyms instead of externalizing them. Your business requirements change. Load from files or configuration services, not from index settings baked into your code.

Key takeaways

• Custom analyzers are defined in index settings under analysis.analyzer and referenced in field mappings. They give you precise control over text processing.
• Every analyzer combines a tokenizer and one or more filters. Order matters—lowercase early, stemming later, asciifolding near the end.
• Synonyms expand coverage across language variants and regional terminology. Load them from external files for easy updates without reindexing.
• ASCII folding normalizes accented characters, essential for international search. Pair it with lowercase for robust multilingual support.
• Separate indexing and search analyzers optimize for both recall and precision. Index aggressively, search conservatively.
• Use the _analyze API to debug tokenization and filter behavior. Never deploy an analyzer to production without testing it against real data.
• Plan for analyzer evolution—version names, use aliases for zero-downtime reindexing, and externalize synonyms and stopwords.