Elasticsearch core concepts: what developers should know

If you're building a search feature or working with large datasets, you'll eventually reach for Elasticsearch. But before you start indexing documents and running queries, you need to understand how Elasticsearch organizes and stores your data. The Elasticsearch core concepts: what developers should know are foundational—get these right, and the rest becomes intuitive; miss them, and you'll spend weeks debugging mysterious performance issues or data inconsistencies.

Today we'll walk through the four pillars that hold up every Elasticsearch cluster: indexes, documents, shards, and replicas. These aren't just theory—they directly affect how fast your searches run, how much data you can store, and what happens when a server fails.

Index — A collection of documents, like a table

Think of an index as a database table, or more precisely, as a collection of related documents. If you're building an e-commerce platform, you might have a products index, a orders index, and a customers index. Each index has its own mapping—essentially a schema that defines which fields exist, their data types, and how they should be analyzed for search.

Creating an index in Elasticsearch is straightforward. You define a mapping that describes your documents:

PUT /products
{
  "settings": {
    "number_of_shards": 3,
    "number_of_replicas": 1
  },
  "mappings": {
    "properties": {
      "name": {
        "type": "text",
        "analyzer": "standard"
      },
      "price": {
        "type": "float"
      },
      "sku": {
        "type": "keyword"
      },
      "created_at": {
        "type": "date"
      }
    }
  }
}

Here's what's happening: the products index is being created with specific settings for shards and replicas (more on those in a moment). The mappings define four fields. The name field is text, which means Elasticsearch will tokenize and analyze it for full-text search. The sku is a keyword, meaning it's stored as-is without analysis—perfect for exact matching. The price is a float for range queries, and created_at is a date for time-based filtering.

In practice, you'll often create indexes programmatically when your application starts up, or manage them through a dedicated indexing pipeline. Each index is independent; queries against the products index won't touch the orders index. This separation lets you tune each index's performance and storage characteristics separately.

Document — A JSON object with fields

A document is the unit of data you store and search in Elasticsearch. It's a JSON object with fields that match your mapping. If an index is like a table, a document is like a row.

Here's what a document might look like when you index it into the products index:

POST /products/_doc
{
  "name": "Wireless Headphones Pro",
  "price": 149.99,
  "sku": "WH-PRO-001",
  "created_at": "2024-01-15T10:30:00Z"
}

Elasticsearch automatically assigns this document a unique ID (or you can provide one). When you retrieve it, you get back the exact JSON plus metadata:

{
  "_index": "products",
  "_id": "1",
  "_version": 1,
  "_seq_no": 0,
  "_primary_term": 1,
  "found": true,
  "_source": {
    "name": "Wireless Headphones Pro",
    "price": 149.99,
    "sku": "WH-PRO-001",
    "created_at": "2024-01-15T10:30:00Z"
  }
}

The underscore-prefixed fields are metadata that Elasticsearch maintains for every document. The _version increments each time the document is modified, which is useful for optimistic locking. The _seq_no and _primary_term are used internally for replication and consistency.

Documents are schemaless in the sense that you don't have to pre-define every field, but your mapping defines how existing fields are indexed and searched. If you add a field that isn't in the mapping, Elasticsearch will dynamically create it based on the value type. This flexibility is powerful, but in production systems we typically set "dynamic": false to prevent accidental field creep.

Shards — An index is split into shards spread across nodes

Here's where things get interesting. An index doesn't live on a single machine. Elasticsearch divides each index into multiple shards, and distributes those shards across nodes in your cluster. A shard is like a mini-index—it's a self-contained Lucene instance that holds a subset of your documents.

Why shard? Two reasons: scale and parallelism. If your products index has 100 million documents, putting them all on one machine is impractical. Sharding lets you spread the load. If you have 3 shards, each holds roughly 33 million documents. When you search, Elasticsearch queries all shards in parallel and combines results. This dramatically speeds up search on large datasets.

When you created the products index earlier, you specified "number_of_shards": 3. This means the index will be divided into three primary shards. You can visualize it like this:

When you index a document, Elasticsearch uses a hash function on the document ID to determine which shard it belongs to. All documents with the same parent ID (or a routing parameter) will land in the same shard. This is deterministic—the same document always goes to the same shard.

The tradeoff is that you must decide the number of shards at index creation time. Changing it later requires reindexing, which is expensive. So plan ahead: consider how much data you'll have in a year, and how many queries per second. A rough heuristic from teams in production is that a shard should hold between 10-50 GB of data, depending on your query patterns. Too few shards and you're bottlenecked; too many and coordination overhead hurts.

Replicas — Copies of shards for availability and read throughput

Replication is your safety net. For every primary shard, you can have one or more replica shards—exact copies stored on different nodes. If a node fails, your data isn't lost. Additionally, replicas absorb read traffic, increasing throughput.

When you set "number_of_replicas": 1, you're saying each primary shard should have one copy. With 3 primary shards and 1 replica, you need a minimum of 2 nodes (ideally more). The cluster distributes replicas intelligently—replica 0 won't be on the same node as primary 0.

When you index a document, Elasticsearch writes to the primary shard first. Once acknowledged, it replicates to all replica shards. For reads, Elasticsearch can use either the primary or a replica. This flexibility is crucial in production: you can tune replica count based on load. High read volume? Add replicas. Write-heavy workload? Reduce replicas to cut coordination overhead.

In a real scenario: imagine a ride-hailing company indexing driver locations. They might use 5 shards with 2 replicas, spread across a 10-node cluster in multiple availability zones. This ensures that even if 3 nodes fail simultaneously, the cluster remains operational. For a lower-stakes index like analytics logs, they might use 10 shards with no replicas to save storage, accepting that a node failure means data loss for that shard (though they can rebuild from the source system).

You can change the replica count dynamically without reindexing:

PUT /products/_settings
{
  "number_of_replicas": 2
}

Elasticsearch will create new replica shards on available nodes. Useful for ramping up read capacity or scaling down to save costs.

Common mistakes

• Setting too many shards early on. Teams often create 20 shards for an index that will never grow beyond a few GB. This fragments data unnecessarily, increases merge overhead, and slows searches. Start conservative; you can always reindex if needed.
• Ignoring replica settings in production. A single-replica setup might seem redundant, but it's the bare minimum for high availability. With zero replicas, a node failure is a data loss event.
• Not planning shard strategy upfront. Elasticsearch shard decisions are architectural. Changing them requires careful planning and downtime. Involve your infrastructure team early.
• Confusing document IDs with shard assignment. The shard a document lands in is determined by hash(document_id). If you're not thinking about ID distribution, you can end up with hot shards that receive disproportionate traffic.
• Misunderstanding mapping immutability. Once a field is indexed with a certain analyzer or type, you can't change it. You'd have to reindex. Plan your mappings carefully.

Key takeaways

• An index is a collection of related documents with a shared mapping, similar to a database table.
• A document is a JSON object with fields; it's the fundamental unit of storage and retrieval.
• Shards split an index across cluster nodes, enabling parallelism and allowing you to scale beyond a single machine. Shard count is set at creation and cannot be changed without reindexing.
• Replicas are copies of primary shards distributed across nodes. They provide fault tolerance and increase read throughput. Replica count can be adjusted dynamically.
• Together, shards and replicas form the backbone of Elasticsearch's distributed architecture. Understanding them is essential for tuning performance and ensuring reliability.