Centralized logging with Elasticsearch, explained simply

The logging problem at scale

When your application runs on a single server, grep works fine. The moment you scale to multiple services, containers, and regions, logs scatter everywhere. You're SSH-ing into different machines, tailing files, losing context, and missing the patterns that matter. Centralized logging with Elasticsearch, explained simply, is the practice of collecting all logs from all your services into one searchable system so you can debug faster and understand your production environment better.

We've all been there. A customer reports an issue. You check three different services, find logs in different formats, lose the request trail halfway through, and spend an hour reconstructing what happened. A centralized logging setup with Elasticsearch eliminates that friction. You get one place to search, filter, and correlate events across your entire stack.

Structured JSON logs: the foundation

The first rule of centralized logging is simple: logs must be structured. Plain text logs are noise. You can't index them effectively, can't filter them precisely, and can't aggregate them reliably.

Structured logging means every log entry is valid JSON with consistent fields. Instead of writing:

"User login attempt from 192.168.1.1"

You write:

{
  "timestamp": "2025-01-15T09:23:45.123Z",
  "level": "INFO",
  "message": "User login attempt",
  "service": "auth-service",
  "user_id": 12847,
  "ip_address": "192.168.1.1",
  "attempt_number": 1,
  "success": false
}

This structure does three things. First, it makes parsing reliable. Your log shipping tool knows exactly what fields exist. Second, it enables precise filtering. You can query "all failed login attempts from a specific IP in the last hour" instead of grepping for strings. Third, it creates a queryable audit trail that supports debugging, compliance, and analytics.

Most teams use a logging library that handles JSON formatting automatically. In Node.js, Pino or Winston with a JSON transport. In .NET, Serilog with structured property support. In Python, the structlog library. The pattern is consistent: pass fields as objects, not as string concatenation.

Set a standard for your team. Decide which fields every log entry needs. At minimum: timestamp, level, message, service name, environment. Then decide which fields are context-specific. User IDs for auth logs. Request IDs for API calls. Transaction amounts for payment logs. Document it, enforce it in code review, and your logs become an asset instead of noise.

Common fields: service and trace ID

In a distributed system, a single user action touches multiple services. A checkout request hits the API gateway, then the orders service, then payments, then inventory. If each service logs independently, you have no way to correlate the logs from that one request.

This is why two fields matter more than any others: service name and trace ID.

The service name tells you which application wrote the log. Every log entry should include it:

{
  "timestamp": "2025-01-15T09:24:12.456Z",
  "level": "INFO",
  "message": "Order created",
  "service": "orders-service",
  "order_id": "ORD-98765",
  "trace_id": "abc123def456"
}

The trace ID is the breakthrough. It's a unique identifier generated at the entry point (usually the API gateway or message queue consumer) and passed through every downstream service call via headers or context. The same trace ID appears in logs from the API gateway, orders service, payments service, and inventory service. One user action, one trace ID, logs you can follow end-to-end.

Implementing this takes discipline. Your API gateway or middleware must generate a trace ID if one doesn't exist and extract it if it does. Your HTTP clients must add it to outgoing request headers. Your gRPC or message queue handlers must extract it and pass it to child services. Tools like OpenTelemetry handle this automatically, but even a manual implementation isn't complex:

{
  "implementation_example": "Pass trace ID in request context",
  "pseudo_code": "const traceId = req.headers['x-trace-id'] || generateId(); context.set('traceId', traceId);"
}

Once trace IDs are in place, Elasticsearch queries become powerful. Search for a trace ID and you see the complete story of that request across all services. Payment failed? Trace the request from API to orders service to payments service and see exactly where it broke and why.

Retention with ILM: keeping costs down

Elasticsearch stores a lot of data. Every log entry from every service, all day. If you log 10,000 events per second and keep everything forever, you'll spend a fortune on storage and your queries will slow down.

Index Lifecycle Management (ILM) is Elasticsearch's built-in mechanism for managing data retention cost-effectively. Instead of storing all logs in one index, you create new indices daily or hourly. Then you define a lifecycle policy that automatically moves old indices through phases: hot (current, searchable), warm (older, less frequently searched), cold (archived), and delete (truly old).

A typical policy might look like this:

• Hot phase (days 0-7): Searchable, fully indexed. This is where recent logs live.
• Warm phase (days 7-30): Still searchable but stored on slower hardware. Used for forensics and analysis.
• Cold phase (days 30-90): Archived with search capability but slower performance. Used for compliance lookups.
• Delete phase (day 91): Index deleted permanently.

This approach saves money. Hot indices use fast SSD storage. Warm and cold indices use cheaper hardware. Deleted indices cost nothing. You keep recent logs fast and searchable while archiving older logs cost-effectively.

Configuring ILM requires one template for your indices:

{
  "index_patterns": ["logs-*"],
  "settings": {
    "index.lifecycle.name": "production-logs",
    "index.lifecycle.rollover_alias": "logs-write"
  }
}

And one policy definition that defines the phases. Elasticsearch handles the rest automatically. Old indices roll into warm storage without intervention. Cold archives sit dormant without querying. After 90 days, they're gone.

Adjust the timeline based on your compliance needs and budget. Some teams keep 30 days hot, others 90. The point is: ILM removes the manual labor of managing indices and makes centralized logging economical at scale.

Alert on error spikes: catching problems early

Centralized logs solve the search problem. But you don't want to be reactive, searching for problems after customers complain. You want to be proactive, detecting anomalies before they matter.

Error spikes are the most obvious pattern to catch. If your error rate jumps from 0.1% to 5% in five minutes, something broke. You want to know immediately, not when the Slack channel explodes.

Elasticsearch alerting (via the Alerting plugin or a complementary tool like Opsgenie) lets you define rules. For example: "If the count of logs with level=ERROR in the orders-service exceeds 100 in any five-minute window, trigger an alert."

The alert rule might look conceptually like this:

{
  "alert_name": "orders_service_error_spike",
  "query": {
    "bool": {
      "must": [
        { "match": { "service": "orders-service" } },
        { "match": { "level": "ERROR" } }
      ]
    }
  },
  "condition": "count > 100",
  "time_window": "5 minutes",
  "action": "send_to_pagerduty"
}

Once triggered, the alert goes to PagerDuty, Slack, or your on-call system. The engineer on call gets paged, opens Kibana, and searches for the trace IDs of those errors. They find the root cause in minutes instead of hours.

Start with simple alerts. Error count spikes. Payment transaction failures. Database query timeouts. As you grow, add more sophisticated rules: anomaly detection, missing expected logs from a service, unusual latency patterns. But begin with the obvious problems.

Common mistakes

We've seen teams stumble in predictable ways:

• Unstructured logs: Shipping plain text to Elasticsearch defeats the purpose. If you're not logging JSON, fix it first.
• Missing trace IDs: Services log independently without correlation. The first step is adding trace ID propagation.
• No retention strategy: Logs grow unbounded, storage bills skyrocket, queries slow down. Set ILM policy on day one, not month six.
• Too many alerts: Creating alerts for every possible condition leads to alert fatigue. Your team ignores them. Start with two or three that matter and tune them.
• No log parser configuration: Elasticsearch doesn't automatically parse logs that arrive as raw strings. You need a Logstash pipeline or similar to parse and enrich before ingestion.

Key takeaways

• Structured JSON logs are the foundation. Every log is an object with consistent fields, not a string.
• Service name and trace ID are mandatory fields. They enable correlation across your distributed system.
• ILM policies keep centralized logging economical by automatically moving old indices to cheap storage and deleting very old data.
• Alerting on error spikes catches problems early. Start with simple rules: error count, failed transactions, missing heartbeats.
• Centralized logging is not just a debugging tool. It's a foundation for observability, compliance, and understanding your production environment.