Index templates: Elasticsearch best practices explained

Index templates: Elasticsearch best practices explained

We've all been there: you're running a time-series application with hundreds of indexes being created daily, and suddenly you notice that half of them are missing the same custom analyzer or have the wrong shard count. Index templates solve this problem by letting you define consistent settings and mappings that automatically apply to new indexes matching a pattern. This is Day 34 of our Elasticsearch series, and understanding templates is critical for anyone managing production search infrastructure at scale.

Why templates matter in production

Without index templates, every new index you create relies on your team remembering to apply the right configuration manually. This introduces two serious risks: configuration drift and operational overhead. Imagine onboarding a new engineer who accidentally creates an index without your custom analyzers, or forgetting to set the correct number of replicas during a scaling operation. Templates eliminate this category of mistakes entirely by enforcing your infrastructure decisions consistently.

Templates become especially valuable when you're generating indexes programmatically. If your logging pipeline creates a new index every day, or your multi-tenant SaaS spins up indexes on-demand per customer, you need a single source of truth for how those indexes should be configured. Templates give you that control without cluttering your application code.

Composable templates: the modern approach

Elasticsearch evolved its template system over time. If you're working with Elasticsearch 7.10 or later, you should be using composable templates (also called index templates in the new API), not the older "legacy" format. Composable templates are more flexible and allow you to break your configuration into reusable component templates.

Here's how composable templates work: you create one or more component templates that define specific aspects of your configuration—say, custom analyzers, or index lifecycle policies, or field mappings—then you compose them into an index template that applies to indexes matching a name pattern.

PUT _component_template/my-analyzers
{
  "template": {
    "settings": {
      "analysis": {
        "analyzer": {
          "my_custom_analyzer": {
            "type": "custom",
            "tokenizer": "standard",
            "filter": ["lowercase", "stop"]
          }
        }
      }
    }
  }
}

PUT _component_template/my-mappings
{
  "template": {
    "mappings": {
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "level": {
          "type": "keyword"
        },
        "message": {
          "type": "text",
          "analyzer": "my_custom_analyzer"
        }
      }
    }
  }
}

Now compose these into an index template:

PUT _index_template/logs-template
{
  "index_patterns": ["logs-*"],
  "composed_of": ["my-analyzers", "my-mappings"],
  "priority": 100,
  "template": {
    "settings": {
      "number_of_shards": 2,
      "number_of_replicas": 1
    }
  }
}

PUT _index_template/metrics-template
{
  "index_patterns": ["metrics-*"],
  "composed_of": ["my-analyzers"],
  "priority": 101,
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 0
    }
  }
}

The advantage here is clear: your my-analyzers component is defined once and reused across multiple templates. When you need to update the analyzer logic, you update the component template, and it automatically affects all indexes using it going forward.

Apply mappings and settings by pattern

Templates match new indexes by their name pattern. The index_patterns field accepts wildcards, so logs-* matches logs-2024-01-15, logs-app-errors, and anything else starting with logs-.

When you create a new index that matches multiple template patterns, Elasticsearch uses the one with the highest priority value. This lets you create general-purpose templates with low priority that serve as defaults, then override them with specific templates for particular use cases.

Consider a real scenario: you have a general logs template, but your authentication logs need different shard counts and an additional field mapping for API keys. Instead of duplicating the entire configuration, you'd use a higher-priority template just for auth logs:

PUT _index_template/logs-generic
{
  "index_patterns": ["logs-*"],
  "priority": 50,
  "template": {
    "settings": {
      "number_of_shards": 3,
      "number_of_replicas": 1
    },
    "mappings": {
      "properties": {
        "@timestamp": {"type": "date"},
        "message": {"type": "text"},
        "level": {"type": "keyword"}
      }
    }
  }
}

PUT _index_template/logs-auth
{
  "index_patterns": ["logs-auth-*"],
  "priority": 100,
  "template": {
    "settings": {
      "number_of_shards": 5,
      "number_of_replicas": 2,
      "index.refresh_interval": "30s"
    },
    "mappings": {
      "properties": {
        "@timestamp": {"type": "date"},
        "message": {"type": "text"},
        "level": {"type": "keyword"},
        "api_key": {"type": "keyword"},
        "user_id": {"type": "keyword"}
      }
    }
  }
}

An index named logs-auth-2024-01-15 will match both patterns, but the logs-auth template wins due to higher priority, so it gets 5 shards, 2 replicas, and the additional fields.

Templates apply at index creation time. If you modify a template, existing indexes keep their original settings and mappings. Only newly created indexes get the updated configuration. This is actually a feature, not a bug—it prevents surprises on live data—but it means you need to plan your template changes carefully.

Great for time-based indexes

Time-based index naming is standard practice in Elasticsearch, and templates shine here. Your application logs, metrics, or events often live in indexes like logs-2024-01-15, metrics-2024-01-16, etc. Every single day, a new index is created automatically. Without templates, you'd need manual intervention or cron jobs to ensure each one has the right configuration.

With templates, you define once and forget: every index matching the pattern gets the settings you specified. This is how major products like the Elastic Stack's built-in observability features work—they rely entirely on templates to manage hundreds of indexes without manual setup.

A practical example: you're storing application events with a 7-day retention policy. You want old indexes deleted automatically, so you attach an index lifecycle management (ILM) policy through your template:

PUT _component_template/events-ilm
{
  "template": {
    "settings": {
      "index.lifecycle.name": "events-policy",
      "index.lifecycle.rollover_alias": "events-write"
    }
  }
}

PUT _ilm/policy/events-policy
{
  "policy": {
    "phases": {
      "delete": {
        "min_age": "7d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

PUT _index_template/events-template
{
  "index_patterns": ["events-*"],
  "composed_of": ["events-ilm"],
  "template": {
    "settings": {
      "number_of_shards": 2,
      "number_of_replicas": 1
    },
    "mappings": {
      "properties": {
        "@timestamp": {"type": "date"},
        "event_type": {"type": "keyword"},
        "payload": {"type": "object", "enabled": false}
      }
    }
  }
}

Now every new index created with the events-* pattern automatically gets the ILM policy attached. After 7 days, Elasticsearch deletes it without you writing a single line of cleanup code.

Version your templates

Templates should be versioned, both in your infrastructure-as-code repository and potentially in Elasticsearch itself. Treat your template definitions like application code: store them in version control, review changes before deployment, and track which version is active.

Elasticsearch lets you include a version field in your template, which has no functional effect but serves as documentation:

PUT _index_template/logs-template
{
  "version": 3,
  "index_patterns": ["logs-*"],
  "priority": 100,
  "template": {
    "settings": {
      "number_of_shards": 2,
      "number_of_replicas": 1
    },
    "mappings": {
      "properties": {
        "@timestamp": {"type": "date"},
        "message": {"type": "text"},
        "level": {"type": "keyword"}
      }
    }
  }
}

The real versioning happens in your deployment pipeline. Store template definitions in your repository with clear naming: templates/logs-v1.json, templates/logs-v2.json, etc. When you need to update a template, create a new version and deploy it. If a new index should use the updated settings, use a new template with a higher priority, leaving the old one in place. This way, existing indexes aren't affected, but new ones get the improvements.

For significant mapping changes—like adding a new field type or changing analysis behavior—you often need to reindex data anyway. A versioned template approach makes it clear what changed and when, simplifying troubleshooting and rollbacks.

Common mistakes

• Forgetting that templates apply at creation time only: If you update a template, existing indexes don't change. Many teams discover this in production when an old index is missing a field added to the template. Use consistent naming and plan migrations upfront.
• Not considering priority: When multiple templates match an index name, the highest priority wins. Accidentally setting a low priority on a specific template, then wondering why the generic one applied, is a classic mistake. Document your priority strategy clearly.
• Over-composing templates: Composable templates are powerful, but creating too many tiny components makes management harder. Find a balance between reusability and maintainability. Usually 2-4 well-designed components is ideal.
• Ignoring template inheritance: When you have overlapping patterns, explicit mappings and settings in higher-priority templates don't override those in lower-priority ones—they merge. Unexpected field conflicts can arise. Test your multi-template scenarios.
• Skipping ILM integration: Templates are the perfect place to attach lifecycle policies for time-based indexes. Not using this automation means manual cleanup, which fails. Always define retention policy in your template if applicable.

Key takeaways

• Use composable templates (not legacy ones) to define consistent settings and mappings for new indexes matching name patterns.
• Break configuration into reusable component templates for analyzers, ILM policies, and common field mappings, then compose them into index templates.
• Templates apply only at index creation time, so plan migrations and versioning carefully.
• For time-based indexes, templates eliminate the need for manual configuration on every new index and integrate seamlessly with index lifecycle management.
• Version your templates in code, use priority to resolve conflicts between overlapping patterns, and test multi-template scenarios before production deployment.