Elasticsearch on AWS: a practical guide for developers

Introduction

Running Elasticsearch on AWS forces you to make a choice: managed services that handle operations for you, or self-hosted clusters where you control everything. Elasticsearch on AWS: a practical guide for developers walks through the real trade-offs, sizing patterns, and security decisions we've made across production systems handling millions of events daily. Whether you're migrating from on-premises infrastructure or building search from scratch, understanding your options—and their hidden costs—saves months of firefighting later.

Options: Managed vs Self-Hosted

Your first decision shapes everything downstream: operational burden, feature velocity, and monthly spend. Let's look at three realistic paths.

Elastic Cloud on AWS

Elastic Cloud is Elastic's official managed service, deployed into your AWS account or Elastic's infrastructure. You get the latest features immediately—new vector database capabilities, security updates, advanced machine learning—without waiting for patch releases.

When to choose it: Your team lacks Elasticsearch depth, you need bleeding-edge features, or observability matters more than cost. You pay per GB of data and per compute unit.

Real scenario: A SaaS platform indexing customer data across multiple tenants. They chose Elastic Cloud because their security model required encryption-in-transit and at-rest by default, plus they needed Elasticsearch's built-in ML features for anomaly detection in logs. Operational overhead dropped from two engineers to nearly zero; they paid roughly 40% more per month but reclaimed engineering hours.

Tradeoffs: Less control over node configuration, network isolation requires VPC peering, and you can't tune JVM settings directly. Useful for teams prioritizing velocity and stability over cost optimization.

Amazon OpenSearch Service

AWS's fork of Elasticsearch (from the v7.10 codebase). Fully managed, deeply integrated with AWS services like CloudWatch, S3, and VPC. No licensing fees—you pay only for compute and storage.

When to choose it: You're AWS-native, budget-conscious, and your use case fits the OpenSearch feature set. Integration with AWS Identity and Access Management (IAM) is seamless.

Real scenario: A fintech firm ingesting market data and trading logs. They chose OpenSearch because it integrates natively with CloudWatch for alarms and dashboards, and IAM policies controlled access without a separate authentication layer. Cost savings were 35–45% versus Elastic Cloud for identical workloads.

Tradeoffs: Feature lag behind open-source Elasticsearch—OpenSearch is typically 1–2 years behind on functionality. Custom plugins require extra work. The ecosystem is smaller; fewer third-party tools integrate seamlessly.

Self-Managed on EC2

You provision EC2 instances, manage clustering, handle upgrades, and monitor everything yourself. Maximum control, maximum overhead.

When to choose it: You have mature Elasticsearch expertise, strict data residency requirements, or highly custom JVM tuning needs. Rare in teams smaller than 20 engineers.

Real scenario: A trading firm with microsecond-latency requirements for market data search. They self-hosted because they needed to co-locate Elasticsearch on specific EC2 instance types, tune GC pause times to sub-millisecond windows, and run custom ingest pipelines that required Java plugin development. The operational cost was high, but latency SLA was non-negotiable.

Tradeoffs: You own availability, upgrades, security patches, and node failures. Expect on-call burden. Requires deep Elasticsearch knowledge and AWS infrastructure tooling (Terraform, Ansible, or CloudFormation).

Sizing: Start with Three Nodes Across Zones

A common mistake: spinning up a single-node cluster to "test" and then running production on it. Single nodes have no failover; a restart means downtime. Three is the minimum for any production cluster.

The Three-Node Pattern

Deploy three nodes across three availability zones in your region. Each node runs on a separate instance, in a separate subnet, in a separate zone. This buys you:

• Quorum: two nodes can elect a master if one fails.
• Zone independence: an AZ outage doesn't take down the cluster.
• Rolling upgrades: take one node at a time without data loss.

Instance sizing depends on your workload:

Set heap to 50% of instance RAM, capped at 31 GB. Never exceed 31 GB because Java's pointer compression stops working, and memory jumps dramatically. The remaining RAM becomes the OS page cache, which Elasticsearch uses for fast segment access.

Scaling Strategy

Start with three m6i.2xlarge nodes. Measure CPU and memory over 4–6 weeks. If CPU tops 60–70% regularly, add a fourth node (or upgrade to larger instances). If memory is the bottleneck and you're storing more than 1.5 TB per node, split into more nodes rather than bigger ones—Elasticsearch clusters recover faster from smaller node failures.

Storage: Use gp3 Volumes or Managed Tiers

Elasticsearch is I/O-hungry. Indexing, searching, and merging all hit disk constantly. Your storage choice directly impacts latency and cost.

GP3 Volumes

AWS gp3 gives you control over throughput (up to 1,000 MB/s) and IOPS (up to 64,000) independently of size. For Elasticsearch:

• Allocate at least 1,000 provisioned IOPS per node.
• 250 MB/s throughput is usually sufficient for indexing-heavy workloads.
• Size the volume to hold your index data plus 50% headroom for merge operations.

If you're indexing 50 GB per day and retaining 30 days, size the volume for 1.5 TB minimum.

Example Terraform:

resource "aws_ebs_volume" "elasticsearch_node" {
  availability_zone = "us-east-1a"
  size              = 1500  # GB
  type              = "gp3"
  iops              = 5000
  throughput        = 250
  encrypted         = true
  tags = {
    Name = "es-node-1-storage"
  }
}

resource "aws_volume_attachment" "es_attach" {
  device_name = "/dev/sdf"
  instance_id = aws_instance.elasticsearch_node.id
  volume_id   = aws_ebs_volume.elasticsearch_node.id
}

Managed Tiers on Elastic Cloud and OpenSearch

Both Elastic Cloud and OpenSearch offer managed storage tiers. You can keep "hot" recent data on fast gp3-backed storage, and automatically move older indices to cheaper "warm" or "cold" storage (often backed by S3). This reduces your monthly bill significantly for time-series data like logs or metrics.

Real example: A ride-hailing platform indexes trip logs (20 GB per day). Using a 7-day hot tier and 90-day warm tier cut storage costs from $2,800/month to $1,200/month—same query performance for recent queries, acceptable latency for historical analytics.

Access: IAM or Fine-Grained Access Control

Controlling who can index, search, or delete is critical. Your options depend on your deployment model.

Elastic Cloud Security

Elastic Cloud uses role-based access control (RBAC) with users defined in Elastic's own system or integrated via SAML/OIDC. You define roles that grant access to specific indices, APIs, and actions.

Example role definition:

{
  "role": "app_developer",
  "indices": [
    {
      "names": ["logs-app-*"],
      "privileges": ["read", "view_index_metadata"]
    }
  ],
  "cluster": ["monitor", "cluster:monitor/nodes/info"],
  "applications": [
    {
      "application": "kibana-.kibana",
      "privileges": ["feature_discover.read"],
      "resources": ["*"]
    }
  ]
}

This role can search logs-app-* indices and view Kibana dashboards, but cannot delete or reindex.

OpenSearch and IAM Integration

Amazon OpenSearch integrates with AWS IAM. You attach IAM policies to users or roles, and OpenSearch honors them. This is powerful if your team already uses IAM for EC2, RDS, and S3 access—one source of truth.

Example IAM policy for an application:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:role/app-lambda-role"
      },
      "Action": [
        "es:ESHttpGet",
        "es:ESHttpPost"
      ],
      "Resource": "arn:aws:es:us-east-1:123456789012:domain/search-domain/*"
    }
  ]
}

Combined with VPC security groups (for network isolation) and encryption in transit, this locks down access without additional credential management.

Self-Managed Authentication

On EC2, use X-Pack (free tier) or elastic-certified plugins. X-Pack's basic tier provides authentication and per-index authorization. For more sophisticated scenarios—user audit trails, attribute-based access—implement an API gateway (Kong, AWS API Gateway) in front of Elasticsearch that handles authentication and logs access.

Common Mistakes

• Single-node clusters in production. They fail; there's no backup. Always three nodes minimum.
• Heap over 31 GB. Pointer compression breaks; memory bloats. Split into more nodes instead.
• No cross-AZ deployment. A single availability zone outage takes you offline. Always spread nodes.
• Mixing managed tiers without a retention policy. Cold data still costs money to store; define and enforce TTLs.
• Enabling fine-grained access control retroactively. Audit your access patterns first; adding auth later often breaks integrations.
• Forgetting encryption at rest on self-managed clusters. EC2 instances default to unencrypted EBS. Enable it in your Terraform or CloudFormation.

Key Takeaways

• Choose managed services (Elastic Cloud or OpenSearch) if operational simplicity outweighs cost; choose self-hosted only if you have deep Elasticsearch expertise and strict non-functional requirements.
• Always deploy three nodes across three availability zones. It's the minimum for production; scale up by adding nodes, not bigger nodes.
• Use gp3 volumes with 1,000+ IOPS and appropriate sizing; for time-series data, use managed tiers to move cold data off expensive storage.
• Secure access through IAM (OpenSearch), Elastic's RBAC (Elastic Cloud), or API gateway patterns (self-hosted). Don't skip authentication.
• Size heap to 50% of instance RAM, capped at 31 GB. Monitor CPU and memory weekly; adjust instance types or node count based on sustained load, not spikes.