Snapshots and restore: a practical guide for developers

Introduction

Data loss in Elasticsearch isn't a matter of if, but when. Whether you're dealing with a corrupted index, accidental deletion, hardware failure, or a botched upgrade, having a robust backup strategy keeps your search engine online and your team sleeping at night. Snapshots and restore: a practical guide for developers walks you through the mechanics of protecting your Elasticsearch cluster, automating backup routines, and recovering data with confidence.

In this guide, we'll cover snapshot repositories on S3, lifecycle policies that automate retention, testing your restore procedures, and recovering individual indexes without touching the rest of your cluster. These aren't theoretical concepts—they're the patterns we've used in production across e-commerce platforms, SaaS applications, and real-time analytics systems handling billions of documents.

Snapshot repositories on S3

A snapshot repository is Elasticsearch's connection point to external storage. S3 is the most common choice because it's durable, scalable, and integrates cleanly with AWS infrastructure. Before you create a snapshot, you need to tell Elasticsearch where to store it.

First, ensure your Elasticsearch cluster has the S3 plugin installed. Modern versions often include it by default, but verify by checking the installed plugins:

GET /_cat/plugins

Next, configure your S3 credentials. We recommend using IAM roles rather than hardcoded keys. Attach this policy to your Elasticsearch instance role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::your-backup-bucket",
        "arn:aws:s3:::your-backup-bucket/*"
      ]
    }
  ]
}

Now create the repository:

PUT /_snapshot/my_s3_repo
{
  "type": "s3",
  "settings": {
    "bucket": "your-backup-bucket",
    "region": "us-east-1",
    "base_path": "elasticsearch-backups",
    "compress": true
  }
}

The base_path organizes snapshots within your bucket, and compress reduces storage costs. Verify the repository is healthy:

GET /_snapshot/my_s3_repo/_verify

A successful response returns a list of nodes that can access the repository. If any node fails, check IAM permissions and bucket access. Real-world scenario: we once debugged snapshot failures for 20 minutes only to realize the bucket name was misspelled in the repository config. Always verify.

Snapshot lifecycle policies

Manual snapshots are better than no snapshots, but automation is better still. Snapshot Lifecycle Management (SLM) policies let you define when snapshots run, how many to keep, and when to delete old ones. This removes the human element and guarantees backups happen on schedule.

Create a policy that snapshots daily and retains 30 days of backups:

PUT /_slm/policy/daily_backups
{
  "schedule": "0 2 * * ?",
  "name": "",
  "repository": "my_s3_repo",
  "indices": ["*"],
  "retention": {
    "expire_after": "30d",
    "min_count": 3,
    "max_count": 50
  }
}

This policy runs at 2 AM UTC every day (cron syntax), snapshots all indices, and keeps between 3 and 50 snapshots. Old snapshots beyond 30 days get deleted automatically unless doing so would drop you below the minimum count. The name includes the date, so each snapshot is timestamped.

You can also create policies per index pattern. For high-churn logs, snapshot every 6 hours:

PUT /_slm/policy/logs_frequent
{
  "schedule": "0 */6 * * ?",
  "name": "",
  "repository": "my_s3_repo",
  "indices": ["logs-*"],
  "retention": {
    "expire_after": "7d",
    "min_count": 2,
    "max_count": 20
  }
}

Monitor policy execution to catch failures early:

GET /_slm/policy/daily_backups/_stats

Check the response for last_execution, last_success, and last_failure. If a policy is failing repeatedly, investigate the cluster logs and verify repository connectivity. In production, wire these stats into your monitoring system—Datadog, New Relic, or Prometheus—so you're alerted if backups stop.

Test restores

A snapshot you've never restored is a snapshot you don't trust. Testing recovery procedures isn't optional—it's insurance. We recommend running a monthly restore test on a staging cluster to catch problems before you're under pressure.

First, list available snapshots:

GET /_snapshot/my_s3_repo/_all

This returns metadata for every snapshot in the repository, including creation time, status, and size. Pick a recent one and check its details:

GET /_snapshot/my_s3_repo/daily-2024-01-15

Now restore it. On a staging cluster, do a full restore to validate integrity:

POST /_restore
{
  "snapshot": "daily-2024-01-15",
  "repository": "my_s3_repo",
  "include_global_state": false
}

Omit include_global_state or set it to false to skip cluster settings and aliases—useful when restoring to a different cluster. Watch the restore progress:

GET /_recovery?active=true

On large indexes, restores can take hours. Run a spot check to ensure document counts match expectations:

GET /restored_index/_count

Compare this to your snapshot metadata. If counts diverge significantly, investigate whether the snapshot was incomplete. Real-world example: after restoring a 100GB index, we noticed shard counts were off. Turned out the source cluster had replicas we'd forgotten about. The snapshot was fine, but our mental model wasn't.

Restore single indexes

Full cluster restores are rare in production. More often, you need to recover a single corrupted index or roll back a specific dataset after a bad deployment. Elasticsearch lets you restore individual indexes without touching the rest of your cluster.

Restore a single index with a new name to avoid conflicts:

POST /_restore
{
  "snapshot": "daily-2024-01-15",
  "repository": "my_s3_repo",
  "indices": "products",
  "rename_pattern": "(.+)",
  "rename_replacement": "restored_$1"
}

This restores only the products index and renames it to restored_products. You can now validate the data, run comparisons, and decide whether to promote it or discard it. Once validated, swap aliases:

POST /_aliases
{
  "actions": [
    { "remove": { "index": "products", "alias": "products_live" } },
    { "add": { "index": "restored_products", "alias": "products_live" } }
  ]
}

Applications reading from products_live now see the restored data. The old index is still there, so you can keep it for a rollback window if needed, then delete it later:

DELETE /products

Restore multiple indexes at once by listing them:

POST /_restore
{
  "snapshot": "daily-2024-01-15",
  "repository": "my_s3_repo",
  "indices": "orders,invoices,transactions",
  "rename_pattern": "(.+)",
  "rename_replacement": "restored_$1"
}

This approach—restore to a temporary index, validate, then swap—gives you a safety window and minimizes downtime. We've used it dozens of times for incident response.

Common mistakes

• Forgetting to verify the repository. A repository that passes creation doesn't guarantee all nodes can write to it. Always run _verify after setup and periodically in production.
• Excluding critical indexes from SLM policies. Don't snapshot "*" and then realize later you forgot to include your business-critical index because it has a slightly different naming pattern. Be explicit about what matters.
• Never testing restores. Restore procedures are easy to get wrong. Test them in staging under realistic conditions before you need them in anger.
• Ignoring snapshot status and retention metrics. Snapshots that fail silently or retention policies that never delete anything (accumulating storage costs) are silent killers. Monitor both.
• Restoring to the original index without a safety step. Always rename or use aliases during restore so you can validate before swapping live traffic.
• Assuming IAM permissions are correct without testing. Permissions errors surface only when you try to restore under pressure. Test them upfront.

Key takeaways

• Set up S3 repositories with IAM roles and verify connectivity before creating snapshots.
• Automate snapshots with SLM policies, defining schedule, retention windows, and minimum/maximum snapshot counts.
• Test restores monthly on staging to catch failures early and validate your recovery time objectives.
• Restore single indexes to temporary names, validate data, then swap using aliases to minimize risk and downtime.
• Monitor snapshot execution and retention metrics continuously—integrate with your alerting system.
• Always rename restored indexes or use aliases to avoid accidentally overwriting live data.