Elasticsearch for observability: lessons from production

Elasticsearch for Observability: Lessons from Production

When a critical service goes down at 3 AM, your team needs answers fast—and that's where observability becomes your lifeline. We've learned that Elasticsearch for observability: lessons from production means treating logs, metrics, and traces as a unified system rather than three separate tools scattered across your infrastructure. In this article, we'll walk through how to build a production-grade observability stack using Elasticsearch, drawing from real incidents where visibility made the difference between a five-minute fix and a five-hour outage.

Why Unified Observability Matters

Most teams start observability piecemeal: logs go to one place, metrics to another, traces to a third. When production breaks, you're context-switching between dashboards, losing precious time correlating events. A unified Elasticsearch-based observability platform lets you jump from a slow trace to the logs that explain it, all within seconds. We've found that the teams with the fastest MTTR (Mean Time To Resolution) are those who made this investment early.

Logs—Structured Logs in Data Streams

Structured logging is the foundation. Rather than free-form text messages, we structure every log entry as JSON with consistent fields. Elasticsearch 8+ uses data streams for logs, which automatically roll over indices based on time and manage retention for you.

Here's what structured logs look like in practice:

{
  "timestamp": "2024-01-15T03:42:17.123Z",
  "level": "error",
  "logger": "payment-service",
  "message": "Payment processing failed",
  "service": "payment-api",
  "environment": "production",
  "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736",
  "span_id": "00f067aa0ba902b7",
  "user_id": "user_12345",
  "request_id": "req_98765",
  "error": {
    "type": "PaymentGatewayException",
    "message": "Connection timeout after 30s",
    "stacktrace": "..."
  },
  "context": {
    "amount": 99.99,
    "currency": "USD",
    "gateway": "stripe",
    "retry_count": 2
  },
  "duration_ms": 30042
}

The critical fields here are trace_id and span_id—more on those later. Notice we're logging the business context (user_id, amount) alongside technical details. When a customer reports a failed transaction, you can search by user_id and immediately see what went wrong.

To get logs into Elasticsearch as data streams, use Filebeat or Fluentd. Here's a minimal Filebeat configuration:

{
  "filebeat.inputs": [
    {
      "type": "log",
      "enabled": true,
      "paths": ["/var/log/app/*.log"],
      "fields": {
        "service": "payment-api",
        "environment": "production"
      }
    }
  ],
  "output.elasticsearch": {
    "hosts": ["elasticsearch:9200"],
    "index": "logs-%{[agent.version]}-%{+yyyy.MM.dd}"
  }
}

Elasticsearch will automatically convert this to a data stream. Query your logs with:

{
  "query": {
    "bool": {
      "must": [
        { "match": { "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736" } },
        { "range": { "timestamp": { "gte": "2024-01-15T03:40:00Z" } } }
      ]
    }
  }
}

We've found that teams often skip structured logging because it requires discipline—every service needs to emit the same schema. The payoff, however, is enormous. A poorly structured log (a wall of text) tells you almost nothing in a production emergency. A well-structured log, searchable and filterable in seconds, can point you straight to the bug.

APM—Traces with Elastic APM Agents

Traces show you the journey of a request through your system. Elastic APM agents instrument your application automatically, capturing every database query, HTTP call, and custom timing. Unlike logs (which you write explicitly), traces are mostly automatic—the agent just watches.

To start, install the APM agent for your language. For Node.js:

npm install elastic-apm-node

Initialize it at the very top of your application:

const apm = require('elastic-apm-node');
apm.start({
  serviceName: 'payment-api',
  serverUrl: 'http://apm-server:8200',
  environment: 'production',
  centralConfig: true
});

// Rest of your app follows...
const express = require('express');
const app = express();

The agent now captures:

• Transactions: High-level requests (e.g., POST /checkout)
• Spans: Lower-level operations (e.g., SELECT * FROM orders, call to Stripe API)
• Errors: Exceptions with full context

Each trace gets a trace_id. This is the magic thread that ties everything together. In Kibana, you can drill into a trace and see exactly what happened—millisecond by millisecond. We once tracked down a race condition that only manifested under high load by examining traces; without Elastic APM, we would have been blind.

For custom business logic, add spans manually:

const span = apm.currentTransaction.startSpan('validate_address');
span.setLabel('address_type', 'shipping');
// ... your validation logic ...
span.end();

Metrics—Elastic Agent or Prometheus Integrations

Metrics are the long view: CPU usage, memory, request rates, error rates, latency percentiles. Elasticsearch can ingest metrics from Elastic Agent, Prometheus, or Metricbeat. Most teams use one of two approaches:

Option 1: Elastic Agent (simpler, recommended for new projects)

Elastic Agent is a single binary that collects logs, metrics, and traces. Deploy it once, configure integrations in Kibana, done. It supports hundreds of integrations out of the box: Nginx, PostgreSQL, Redis, Kubernetes, etc.

Option 2: Prometheus + Elasticsearch

If you're already running Prometheus, Elastic can scrape it. Add this to your elasticsearch.yml:

monitoring:
  collection:
    enabled: true

Common metrics to track in production:

• Request latency (p50, p95, p99)
• Error rate
• Database connection pool utilization
• Queue depth
• Memory and CPU per service
• Elasticsearch index size and search latency

Once metrics are flowing, create alerts. For instance, if error rate spikes above 5%:

{
  "trigger": {
    "schedule": {
      "interval": "1m"
    }
  },
  "input": {
    "search": {
      "request": {
        "indices": ["metrics-*"],
        "body": {
          "query": {
            "range": { "timestamp": { "gte": "now-5m" } }
          },
          "aggs": {
            "error_rate": {
              "avg": { "field": "error_rate" }
            }
          }
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.aggregations.error_rate.value": {
        "gt": 0.05
      }
    }
  }
}

Correlation—Trace IDs Link Logs and Traces

The secret ingredient: trace_id. When a customer reports an issue, you should be able to:

1. Find their request's trace_id from logs or APM
2. Jump to the full trace in Kibana to see every span
3. See all logs emitted during that request by filtering on the same trace_id

To make this work, every log statement must include the trace ID. In Express with Elastic APM:

app.use((req, res, next) => {
  const traceId = req.headers['x-trace-id'] || 
                  apm.currentTransaction?.trace_id || 
                  generateId();
  req.traceId = traceId;
  
  // Inject into logs
  logger.info('Request received', { 
    trace_id: traceId,
    user_id: req.user?.id,
    path: req.path
  });
  
  next();
});

app.post('/checkout', (req, res) => {
  logger.info('Processing checkout', { trace_id: req.traceId });
  // ... business logic ...
  logger.info('Checkout complete', { trace_id: req.traceId });
});

Elasticsearch APM agent automatically adds the trace ID to logs if you use the agent's built-in logger wrapper. But if you use Winston or Bunyan, you need to manually inject it.

In Kibana, Discover, add a filter for your trace ID and you'll see every event from every service in order. This is where the light bulb moment happens for most teams—suddenly the chaos of production becomes traceable and understandable.

Common Mistakes

Incomplete logging. Teams log happy paths but skip error details. When production breaks, you have no error logs. Log at ERROR level whenever something goes wrong, with full context (user, request ID, stack trace).

Inconsistent trace ID propagation. If microservice A calls service B, the trace ID must travel with the request. Many teams forget this, resulting in "broken traces" where you see the transaction but not its downstream calls. Use HTTP headers like X-Trace-ID or Traceparent (W3C standard).

Over-retention without cost management. Elasticsearch storage is not cheap. We've seen teams ingest gigabytes of logs daily without any retention policy. Set a retention policy (e.g., 30 days for logs, 14 days for traces) and use ILM (Index Lifecycle Management) to roll cold indices to cheaper storage.

Ignoring cardinality. High-cardinality fields (like request IDs with millions of unique values) make aggregations slow and expensive. Be thoughtful about which fields you index as keywords vs. text.

APM sampling too aggressive. Some teams sample 1% of traces to save costs. That's fine for baseline, but when an incident hits, you need full fidelity. Use dynamic sampling or raise it to 10% minimum in production.

Key Takeaways

• Structured logs are non-negotiable. JSON with consistent schema lets you query logs in seconds instead of grepping log files for hours.
• Elastic APM provides traces automatically with minimal code changes, giving you microsecond-level visibility into request flows.
• Metrics (CPU, latency, error rate) complete the picture and trigger alerts before users notice problems.
• Trace IDs are the thread that ties logs, traces, and metrics together. Propagate them across all services and log statements.
• Data streams and ILM simplify operations. Let Elasticsearch manage index rollover and retention so you focus on fixing bugs, not managing indices.
• Start with one service, get the logs and traces flowing, then expand. Full-stack observability is a journey, not a big bang.
• Tune retention policies and sampling rates based on your incident volume and storage budget. You don't need to keep everything forever, but you need enough data to investigate incidents.