Multilingual search in Elasticsearch & Search Engineering

Multilingual search in Elasticsearch & Search Engineering

When you're building search across a global user base, a single analyzer doesn't cut it. We've all shipped search features that work beautifully for English but fall apart the moment a user in Japan or Brazil tries to search in their native language. Multilingual search in Elasticsearch & Search Engineering requires deliberate architecture decisions at index time, thoughtful analyzer selection, and sometimes separate data structures entirely.

The challenge of language diversity

Search fundamentals like stemming, stop words, and tokenization are deeply language-specific. The English word "running" stems to "run," but a French stemmer won't recognize it. A German analyzer needs to handle compound words like "Donaudampfschifffahrtsgesellschaftskapitän," while Japanese requires character-level segmentation to even identify word boundaries. When you index content without considering these differences, you lose precision, recall, or both.

We typically encounter this problem in three scenarios: mixed-language corpora where documents might be in multiple languages, language-specific indexes serving regional products, and user search queries that must respect the user's language preference. Each scenario requires slightly different handling.

Language-specific fields and analyzers

The most practical approach for mixed-content is to store language variants in separate fields within the same document. This lets you index and query by language without maintaining separate indexes, and it keeps related content together for sorting and retrieval context.

Here's how we structure a product document serving English, French, and German markets:

{
  "product_id": "SKU-12345",
  "category": "Electronics",
  "description_en": "Wireless noise-cancelling headphones",
  "description_fr": "Casques antibruit sans fil",
  "description_de": "Kabellose geräuschunterdrückende Kopfhörer",
  "price_usd": 199.99,
  "price_eur": 189.99,
  "language": "multi"
}

Now we define language-specific analyzers in our index settings. Each analyzer handles the particular morphology and tokenization rules of its language:

{
  "settings": {
    "analysis": {
      "analyzer": {
        "english_analyzer": {
          "type": "standard",
          "stopwords": "_english_"
        },
        "french_analyzer": {
          "type": "standard",
          "stopwords": "_french_"
        },
        "german_analyzer": {
          "type": "standard",
          "stopwords": "_german_"
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "description_en": {
        "type": "text",
        "analyzer": "english_analyzer"
      },
      "description_fr": {
        "type": "text",
        "analyzer": "french_analyzer"
      },
      "description_de": {
        "type": "text",
        "analyzer": "german_analyzer"
      }
    }
  }
}

At query time, we target the appropriate field based on the user's language preference. If a user searches in French, we query only description_fr. This ensures the French analyzer tokenizes and stems the query the same way the French content was indexed, maximizing relevance. Elasticsearch's built-in language analyzers handle stop words (common words that don't affect meaning), stemming rules, and basic morphology for around 30 languages out of the box.

Detect language at index time

For a platform where content authors don't specify language, we need automatic detection. Elasticsearch doesn't have built-in language detection, but we can add it upstream in our indexing pipeline using libraries like the fasttext or langdetect in Python, or textcat in Java.

A typical flow: content arrives in our ingestion queue, we detect its language, then we route it to the appropriate field before indexing. Here's a simplified example using a Lambda or container function:

{
  "pipeline": "detect_language_pipeline",
  "body": {
    "doc": {
      "content": "Der Himmel ist blau",
      "language_code": "de",
      "description_de": "Der Himmel ist blau"
    }
  }
}

In practice, we implement this in our application layer. When a piece of content comes in, we detect its language using a third-party service or library, extract a language code, and then populate the corresponding description_XX field. If detection is uncertain (which happens with short strings or mixed-language content), we either fall back to a default language, store in multiple fields, or flag for manual review.

For higher accuracy with production volumes, consider running language detection separately from indexing—perhaps via a background job that processes documents in batches. This avoids blocking the indexing pipeline and lets you tune detection models independently.

ICU plugin for Asian languages

Standard analyzers struggle with languages that don't use whitespace as word delimiters. Japanese, Chinese, and Korean require character-level analysis and specialized tokenization. Elasticsearch's ICU (International Components for Unicode) plugin provides analyzers that handle these languages correctly.

First, install the plugin:

bin/elasticsearch-plugin install analysis-icu

Then configure your index to use ICU analyzers. For Japanese with morphological analysis, we use the icu_tokenizer combined with optional stemming:

{
  "settings": {
    "analysis": {
      "tokenizer": {
        "icu_tokenizer": {
          "type": "icu_tokenizer"
        }
      },
      "analyzer": {
        "japanese_analyzer": {
          "type": "custom",
          "tokenizer": "icu_tokenizer",
          "filter": ["lowercase"]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "title_ja": {
        "type": "text",
        "analyzer": "japanese_analyzer"
      },
      "description_ja": {
        "type": "text",
        "analyzer": "japanese_analyzer"
      }
    }
  }
}

The ICU tokenizer recognizes Japanese hiragana, katakana, and kanji, automatically segmenting them into logical units. Without this, a Japanese string like "ノイズキャンセリング" (noise cancelling) would be indexed as a single token, making partial matches impossible.

For Chinese, ICU also provides better segmentation than standard analyzers. Korean benefits similarly, especially with the ICU tokenizer's understanding of Hangul decomposition.

A real-world note: some teams use dedicated plugins like elasticsearch-kuromoji for Japanese if they need more sophisticated morphological analysis. These trade complexity for higher relevance in Japanese-heavy environments. For most e-commerce and SaaS use cases, ICU's out-of-the-box behavior is sufficient.

Separate indexes per language if needed

As your multilingual corpus grows, you may reach a point where mixed-field indexes become unwieldy. If you're serving a dedicated Japanese site with millions of products, or a French marketplace with its own team, separate indexes can offer clearer data ownership, easier schema evolution, and potential performance isolation.

The tradeoff: you'll manage more infrastructure. A separate index per language means n times the shards, replicas, and operational complexity. Use this approach when you have:

• Regional teams maintaining language-specific catalogs
• Different schema requirements per language (e.g., French uses different metadata than English)
• Traffic patterns that make independent scaling valuable
• Compliance or data residency requirements by region

If you do separate indexes, your application code routes requests to the appropriate index at query time:

{
  "index": "products_en",
  "query": {
    "multi_match": {
      "query": "wireless headphones",
      "fields": ["title", "description"]
    }
  }
}

Or for French:

{
  "index": "products_fr",
  "query": {
    "multi_match": {
      "query": "casques sans fil",
      "fields": ["title", "description"]
    }
  }
}

A hybrid approach many teams find effective: maintain one "universal" index with language-specific fields for fallback and analytics, plus dedicated indexes for high-traffic languages. This gives you best of both worlds—operational simplicity for most languages, specialized tuning where it matters.

Common mistakes

Using the wrong analyzer at query time. Indexing French content with the French analyzer but querying it with the English analyzer is a recipe for low recall. Your query gets tokenized and stemmed differently than your indexed terms. Always match the analyzer to the language, or use language-agnostic analyzers (like standard) consistently everywhere—though this hurts precision in morphologically rich languages.

Forgetting that stop words vary by language. English stop words like "a" and "the" are indexed by default. But in French, "le," "la," and "les" are critical stop words. If you index a French query with English stop words rules, you'll index every occurrence of "le" and search will perform poorly. Use language-appropriate stop word lists.

Mixing language detection with query parsing. Don't detect the language of a user's query using textcat if the query is three characters long. Detection works reliably on longer text. For short queries, ask the user's preference, infer from their browser locale, or default to their most recent search language.

Ignoring diacritics and normalization. "Côté" and "Cote" might refer to the same concept to a French speaker, but as byte strings they're different. Use ICU's ICUFoldingFilter to normalize accented characters if your domain allows it, or handle it explicitly in your query parsing logic.

Key takeaways

• Language-specific fields and analyzers within a single index are the simplest starting point for most multilingual search projects. Index content in separate fields (description_en, description_fr) and query the appropriate field based on user language.
• Detect language at index time using upstream libraries or services, not within Elasticsearch. Store the detected language or route content to the right field early in your pipeline.
• Use the ICU plugin for Asian languages (Japanese, Chinese, Korean) to handle character-level tokenization. Standard analyzers won't segment these languages correctly.
• Separate indexes per language make sense only at scale or when regional teams need independent schema control. For most startups and mid-sized platforms, mixed-field indexes are easier to operate.
• Test with real queries in each language. Relevance tuning is language-specific—what works for English ranking may feel wrong for French or German stemming. Use A/B testing with actual users to validate changes.