Logs with Loki: what every Monitoring developer should know

You've mastered Prometheus metrics, tuned your scrape intervals, and built dashboards that show the state of your system in real time. But when something goes wrong, metrics alone won't tell you why. That's where Logs with Loki: what every Monitoring developer should know becomes essential. Unlike traditional log aggregation systems that index everything, Loki brings the Prometheus philosophy to logs—ship only the labels you need, and query them with a Prometheus-like syntax.

Why Loki, and why now

Traditional log aggregation systems like ELK or Splunk work by indexing every word in every log line. This approach works, but it's expensive. Your logs grow, your index grows, and your costs grow with them. We've seen teams spend thousands of pounds per month on log storage that could have been cut in half with a smarter strategy.

Loki takes a different approach. It indexes only the labels you attach to logs—like job, instance, environment, or service—and stores the raw log lines themselves in cheap object storage. When you query Loki, you filter by labels first (fast), then search log content if needed (optional). This model mirrors Prometheus exactly, which is why it feels natural to teams already running Prometheus.

In a production system handling thousands of requests per second, we've seen Loki reduce log storage costs by 60–70% compared to full-text indexing, while keeping query latency under 200ms for most dashboards.

Index labels, not full text

This is the core principle. When you send logs to Loki, you attach labels. These labels are indexed. The log line itself is compressed and stored, but not indexed word-by-word. This is different from what you might expect if you're coming from Elasticsearch.

Let's say you're running a payment service. Your application logs transactions. Instead of shipping this:

2025-01-15T10:34:22Z user_id=12345 transaction_id=txn_abc123 amount=99.99 currency=GBP status=completed processing_time_ms=342 service=payment-api

You'd send it with labels like this:

labels:
  job: payment-api
  instance: payment-api-prod-1
  environment: production
  service: payment-api
message: 'user_id=12345 transaction_id=txn_abc123 amount=99.99 currency=GBP status=completed processing_time_ms=342'

Now Loki indexes job, instance, environment, and service. When you query for all logs from payment-api in production, that's a simple label lookup—very fast. If you then want to find the word "completed" within that result set, Loki will search the message content, but only within the already-filtered logs.

The key practice: keep your labels small and high-cardinality-resistant. Labels like user_id or request_id should go in the message, not as labels. High-cardinality labels (ones with thousands or millions of unique values) will blow out your index and defeat the purpose of Loki. Stick to labels like job, environment, region, service, and pod_name.

LogQL queries

Once your logs are in Loki, you query them with LogQL. If you know PromQL, LogQL will feel familiar, because it borrows the same syntax and philosophy.

A basic LogQL query filters by labels:

{job="payment-api", environment="production"}

This returns all logs from the payment-api job in production. You can add operators:

{job="payment-api", environment="production"} |= "error"

The |= operator means "keep lines that contain this string". Other operators include:

• != – exclude lines matching the pattern
• |~ – regex match
• !~ – regex non-match

For example, to find slow transactions (processing time over 1 second) that didn't error:

{job="payment-api", environment="production"} |= "processing_time_ms" != "error" |~ "processing_time_ms=([1-9][0-9]{3,})"

You can also extract values from logs and aggregate them. If your logs contain JSON, you can parse and label them inline:

{job="payment-api"} | json | status="completed" | rate(1s)

This parses JSON, filters for status=completed, and returns the rate of matching logs per second. Powerful, and it works in Grafana dashboards too.

Promtail or Alloy collect logs

Your application logs need to reach Loki somehow. The most common way is via Promtail (the Grafana-maintained log shipper) or Alloy (the newer, more flexible agent that can also scrape metrics).

Promtail is simple: it runs as a sidecar or DaemonSet on your Kubernetes cluster, reads logs from files or systemd, and ships them to Loki with labels you define.

Here's a basic Promtail config that reads from a Docker container log file and adds labels:

global:
  clients:
    - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: payment-api
    static_configs:
      - targets:
          - localhost
        labels:
          job: payment-api
          environment: production
          service: payment-api
    file_sd_configs:
      - files:
          - /etc/promtail/targets.yml
    relabel_configs:
      - source_labels: [__meta_docker_container_name]
        target_label: container
      - source_labels: [__meta_docker_container_log_stream]
        target_label: stream

Promtail supports reading from files, syslog, journal, and container logs. In Kubernetes, it typically runs as a DaemonSet, with each node's Promtail instance reading pod logs and shipping them to Loki with pod name, namespace, and image labels pre-attached.

Alloy is newer and more powerful. It's a single agent that can collect logs, scrape metrics, send to Loki, Prometheus, and other backends, all in one process. If you're building a monitoring stack from scratch, Alloy often makes sense because it replaces both Promtail and Prometheus agents.

A basic Alloy config looks like this:

loki.write "default" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
  }
}

loki.source.file "payment_logs" {
  targets    = [{__path__ = "/var/log/payment-api/*.log"}]
  forward_to = [loki.process.add_labels.receiver]
}

loki.process.add_labels "add_labels" {
  values = {
    job         = "payment-api"
    environment = "production"
  }
  forward_to = [loki.write.default.receiver]
}

Alloy's declarative syntax is cleaner, and it scales better in large deployments because one agent handles metrics and logs together.

Correlate with metrics in Grafana

The real power emerges when you use Loki logs alongside your Prometheus metrics in Grafana. You see a spike in request latency on your dashboard; you drill down to the logs for that time window and spot the root cause—a database lock, a cache miss, or a third-party API timeout.

In Grafana, you can link metrics and logs by building dashboards that show both. For example, a dashboard for your payment service might have:

• A time-series chart showing transaction success rate (Prometheus metric)
• A table of recent transaction errors filtered from Loki logs
• A heatmap of processing time distribution (Prometheus histogram)
• A log panel showing raw logs for the selected time range

More importantly, you can use trace IDs to pivot between logs and metrics. If a transaction fails, it has a trace ID. Your metrics can include that trace ID as a label, and your logs include it in the message. Click the trace ID in the log line, and jump to the metrics and traces for that request.

Here's a practical example: you're on-call, and a Prometheus alert fires: "Payment API latency p95 > 2s". You click to the dashboard, see the spike starting at 14:32 UTC, and add a Loki log panel:

{job="payment-api", environment="production"} |= "processing_time_ms" |~ "processing_time_ms=[2-9][0-9]{3,}"

You can see which endpoints or user cohorts experienced the slowness. You spot a pattern: all slow requests hit a specific payment gateway. You check the gateway's status page, confirm it had an outage, and resolve the alert with context, not guesswork.

Common mistakes

High-cardinality labels: The most common mistake. Adding user_id, request_id, or account_id as labels will cause your Loki index to explode and cost to skyrocket. These belong in the log message, not the labels.

Forgetting to parse JSON: If your app logs JSON, use LogQL's json parser to extract fields. Without it, you're searching raw text, which is slow and error-prone.

Shipping too many logs: It's tempting to log everything. In production, you'll regret it. Use sampling for verbose application logs (log 1 in 100 requests), or use log levels and only ship ERROR and WARN by default.

Not setting a retention policy: Loki will keep logs forever unless you tell it not to. Set a retention period in your Loki config (e.g., 7 days, 30 days) based on your compliance and debug needs. Object storage fills up fast.

Ignoring the Loki documentation on scaling: Loki works well at small scale, but at high volume (millions of logs per second), you need to understand distributed Loki, write-ahead logs, and index caching. Start simple, monitor your Loki instance's own metrics, and scale when needed.

Key takeaways

• Loki indexes labels, not full text, making it cheap and fast for production logging.
• Keep labels low-cardinality: job, environment, service, pod_name. Put everything else in the message.
• LogQL syntax mirrors PromQL, so if you know Prometheus, you'll be productive with Loki quickly.
• Promtail or Alloy ship logs to Loki; choose Alloy if you also need metrics collection in a single agent.
• Correlate logs and metrics in Grafana dashboards to move from alerting to understanding. Trace IDs are your friend.
• Watch out for high-cardinality labels and over-logging; they'll waste storage and slow your queries.
• Set a retention policy and monitor Loki's own metrics to stay ahead of scaling issues.