The ELK stack explained: a practical Elasticsearch guide

Introduction

If you've spent time managing logs across microservices, you know the pain: logs scattered across multiple servers, no central way to search them, and dashboards that take weeks to build. The ELK stack explained is straightforward—it's your central nervous system for observability. Elasticsearch, Logstash, and Kibana work together to collect, transform, store, and visualize logs at scale, giving your team the visibility needed to debug issues in minutes rather than hours.

Over this 100-day Elasticsearch & Search Engineering series, we've built up from basic queries to complex aggregations. Today we zoom out and see how Elasticsearch fits into the broader picture of log management and observability. Whether you're running three services or three hundred, understanding the ELK stack architecture will help you make the right choices about ingestion, transformation, and visualization.

Components: the data pipeline from collection to insight

The ELK stack is fundamentally a pipeline. Data flows in one end and insights come out the other. Let's walk through each stage with a concrete scenario: imagine a payment processing service that logs every transaction, error, and state change across multiple instances.

Stage 1: Collection (Beats or Agents)

Your application logs land on disk or stream to stdout. You need something to pick them up and forward them to the next stage. This is where Beats come in—lightweight, single-purpose data shippers. Filebeat watches log files, Metricbeat collects system metrics, Packetbeat captures network traffic. Each Beat is small (a few MB) and designed to run on the same host as your application.

In our payment service example, you'd deploy Filebeat on each payment processor instance, configured to watch /var/log/payment-service.log. Filebeat reads new lines, parses them lightly, and ships them to the next stage.

Stage 2: Transformation (Logstash or Ingest Pipelines)

Raw logs are often messy. A single log line might contain a timestamp, transaction ID, merchant ID, amount, status, and error message all concatenated together. You need to parse, enrich, and standardize that data before it lands in Elasticsearch.

Logstash is the traditional choice for heavy lifting. It runs as a separate service, listens for incoming events (from Beats or many other sources), applies filters to transform them, and outputs them to Elasticsearch or elsewhere.

For lighter transformations, you can use Elasticsearch ingest pipelines instead. These run directly on the Elasticsearch nodes and are cheaper operationally since you don't need a separate cluster.

Stage 3: Storage and Indexing (Elasticsearch)

Once parsed and enriched, events land in Elasticsearch indices. Elasticsearch indexes every field, builds inverted indices for full-text search, and stores the raw document. For the payment service, you'd structure each log event as a JSON document with fields like timestamp, transaction_id, merchant_id, amount, status, and error_message.

Stage 4: Search and Visualization (Kibana)

Now your logs are searchable and analyzable. Kibana connects to Elasticsearch and lets your team search, filter, and visualize. A single dashboard can show transaction success rates, error distributions by merchant, and latency percentiles—all updated in real time or near-real time.

The beauty of this pipeline is that each component has a single responsibility. Beats don't transform; Logstash doesn't store; Kibana doesn't ingest. This separation makes the system modular and replaceable.

Elastic Agent: a unified approach

As the Elastic stack matured, the company introduced Elastic Agent as a newer alternative to scattered Beats. Instead of deploying Filebeat for logs, Metricbeat for metrics, and Packetbeat for network data, you deploy a single Elastic Agent that can handle all three—plus security events and more.

Elastic Agent is policy-driven. You define integrations in a central location (Kibana or the Fleet UI), and agents automatically pull down their configuration. This scales much better than manually configuring dozens of Beats.

For a mid-sized platform, you might define an agent policy that says: collect logs from /var/log/, collect CPU and memory metrics every 10 seconds, and send everything to an Elasticsearch cluster in AWS. Then you deploy the agent to 50 hosts and it just works. When you need to adjust the log path or add a new integration, you update the policy once and it rolls out to all agents.

The tradeoff is complexity. Elastic Agent requires infrastructure to manage policies (Fleet Server). For small deployments or development, individual Beats might be simpler. For production at scale, Elastic Agent is often the right call.

Logstash: handling complex transformations

While ingest pipelines are lightweight and convenient, Logstash excels when your transformation logic is complex or you need to accept data from many different sources.

Consider this scenario: your payment service logs are being collected, but you also have logs from a legacy billing system that writes to Kafka, metrics from a Prometheus server, and security events from your cloud provider's audit logs. You need to normalize all of these into a consistent schema before indexing.

Here's a simplified Logstash pipeline that collects payment logs and enriches them:

input {
  beats {
    port => 5044
  }
}

filter {
  if [type] == "payment-log" {
    grok {
      match => {
        "message" => "%{TIMESTAMP_ISO8601:timestamp} %{DATA:service} transaction_id=%{INT:transaction_id} merchant_id=%{INT:merchant_id} amount=%{NUMBER:amount} status=%{WORD:status}"
      }
    }
    
    if [status] == "failed" {
      mutate {
        add_field => { "severity" => "warning" }
      }
    } else if [status] == "error" {
      mutate {
        add_field => { "severity" => "critical" }
      }
    }
    
    ruby {
      code => "event.set('amount_usd_cents', (event.get('amount').to_f * 100).to_i)"
    }
  }
}

output {
  elasticsearch {
    hosts => ["elasticsearch:9200"]
    index => "payments-%{+YYYY.MM.dd}"
  }
}

This pipeline uses the grok filter to parse unstructured logs into fields, applies conditional logic to assign severity levels, and even runs Ruby code to normalize amounts. After Logstash, every payment event arrives in Elasticsearch with a consistent schema, making queries and visualizations predictable.

Logstash runs as a service and is stateless—you can scale it horizontally by running multiple instances behind a load balancer. This is important in high-volume environments where a single Logstash process can't keep up.

Kibana: searching, dashboarding, and alerting

Once logs are in Elasticsearch, Kibana is your window into them. We'll cover Kibana in detail later in this series, but here's the practical overview.

Discover is your ad-hoc search interface. You point it at an index pattern (e.g., payments-*) and start exploring. You can filter by any field, zoom into time ranges, and see individual documents. This is where you go when something breaks and you need answers fast.

Dashboards are curated collections of visualizations. Your team builds a payments dashboard that shows transaction volume, success rate, average latency, and the top 10 merchants by transaction count. These dashboards are saved and shared—everyone on the team sees the same view of system health.

Alerts watch your data continuously. When transaction success rate drops below 95% for five minutes, or when you see more than 100 errors in 60 seconds, Kibana can trigger an alert—sending a Slack message, creating a PagerDuty incident, or hitting a webhook.

A typical workflow: a customer reports slow transactions. On-call engineer opens the payments dashboard and immediately sees the latency spike started 10 minutes ago. They drill down to Discover, filter by the affected merchant, and find that a particular payment gateway started timing out. Armed with that insight, they contact the gateway provider. Meanwhile, Kibana already triggered an alert that woke up the platform team.

Common mistakes

• Logging everything without a retention strategy. Logs grow fast. If you're indexing every debug-level message from every service, your Elasticsearch cluster will bloat. Index only what you need; filter noise at the Beat or Logstash level.
• Parsing the same log format multiple times. Parse once at ingestion (Logstash or ingest pipeline), never in Kibana. Repeating transformation work wastes query time and introduces inconsistency.
• Building dashboards without understanding your questions. Dashboards should answer specific business and technical questions: "Is the system healthy?", "Why are transactions slow?", "Which services are erroring?" If a dashboard doesn't answer a question, it's noise.
• Ignoring data cardinality. A field like request_id might have millions of unique values. Aggregating on high-cardinality fields is slow and expensive. Use fixed dimensions like service, status_code, and region instead.
• Not planning for cost. Elasticsearch storage and ingestion can be your biggest cloud expense. Use index lifecycle management (ILM) to roll old data to cheaper storage tiers, and set aggressive retention windows.

Key takeaways

• The ELK stack is a data pipeline: Beats collect → Logstash or ingest pipelines transform → Elasticsearch stores and indexes → Kibana searches and visualizes.
• Use Beats for simple log collection; use Elastic Agent when managing many hosts and integrations at scale.
• Logstash shines for complex transformations and multi-source ingestion; ingest pipelines work for simpler cases and reduce operational overhead.
• Parse and normalize data at ingestion time, never in your queries or dashboards.
• Kibana's Discover, Dashboards, and Alerts are your three main tools; use Discover for investigation, Dashboards for team visibility, and Alerts for on-call automation.
• Plan for retention and cost from day one; uncontrolled logging growth will drain your budget.