Kibana Discover and dashboards: best practices explained

Kibana Discover and dashboards: best practices explained

When your application generates millions of log events daily, finding the right needle in the haystack becomes your biggest operational challenge. Kibana's Discover interface and dashboards are built to solve exactly this—giving your team a way to explore, filter, and visualize logs at scale. We'll walk through the core techniques that let you move from "I need to find something" to "here's the insight" in seconds rather than hours.

Data views: the foundation of your searches

Before you can search anything in Kibana, you need to define what you're searching. That's where data views come in. A data view is essentially a configuration that tells Kibana which Elasticsearch indices to query and how to interpret the fields inside them. Think of it as a lens through which you view your data.

In our production systems, we typically create a data view per application or service. For example, if you have an order processing service shipping logs to indices named orders-2024-01-*, orders-2024-02-*, and so on, you'd create a data view with the index pattern orders-*. This single pattern matches all monthly indices without needing manual updates.

To create a data view in Kibana:

1. Navigate to Stack Management → Data Views
2. Click "Create data view"
3. Enter your index pattern (e.g., app-logs-*)
4. Select the timestamp field (usually @timestamp)
5. Configure field settings—mark sensitive fields as filtered or hidden if needed

One critical step many teams skip: set runtime fields and field formatting at the data view level. If you have a duration_ms field that represents milliseconds, format it as a number with 2 decimal places. If you have a user_id that's just a long integer, consider adding a runtime field that joins it with a user lookup index to show the actual username. This work upfront saves your team from doing it repeatedly in each dashboard.

{
  "name": "production-api-logs",
  "title": "Production API Logs",
  "timeFieldName": "@timestamp",
  "sourceFilters": [],
  "fields": [
    {
      "name": "response_time_ms",
      "type": "number",
      "format": {
        "id": "number",
        "params": {
          "pattern": "0,0.00"
        }
      }
    }
  ]
}

KQL for fast filtering

Kibana Query Language (KQL) is your day-to-day tool for narrowing down millions of events to the ones you care about. It's simpler than Query DSL but powerful enough for real operational work.

Let's say you're investigating a spike in 5xx errors. In Discover, you open the data view for your API logs and type:

status: 500 AND service: "payment-api" AND @timestamp >= now-1h

Kibana instantly filters to the last hour of errors from your payment service. You can chain conditions with AND, OR, and NOT. Parentheses work for grouping: (status: 500 OR status: 503) AND environment: prod.

Field syntax matters. For exact matches on text fields, Kibana performs keyword matching by default. For partial matches, use wildcards:

• error_message: "*timeout*" finds any message containing "timeout"
• request_path: "/api/orders/*" finds all order endpoints
• user_tier: (premium OR enterprise) searches multiple values
• response_time_ms: [100 TO 500] ranges work on numeric fields
• NOT status: 2* excludes all 2xx responses

In a real incident, we've used KQL to spot patterns fast. When tracking down a customer-reported issue, we filtered by customer_id: "acme-corp" AND @timestamp >= now-30m to see their recent activity in seconds. Without KQL, we'd have been manually scrolling or writing custom ES queries.

One pro tip: build complex filters incrementally. Start with service: "auth", then add AND status: 401, then AND user_agent: "*mobile*". Watch the event count drop as you narrow focus. This iterative approach often reveals the actual problem faster than trying to write the perfect filter on the first try.

Saving searches for the whole team

Once you've built a filter that reveals something useful, save it. A saved search in Kibana captures your data view, time range, KQL filter, and column layout. Your teammates can then rerun the exact same search without recreating it.

We organize our saved searches by use case: incident response, performance analysis, security audits. A typical workflow is:

1. Discover finds the problem in the logs
2. Refine the KQL filter and columns until the view is clear
3. Click "Save" and give it a descriptive name like "5xx errors by service"
4. Add a description explaining when to use it
5. Share it with the team by making it visible in the saved searches list

Saved searches are also the building blocks for dashboards. When you visualize a saved search, you're capturing a reusable analytical pattern. If your oncall engineer always needs to check "Errors in last 15 minutes by severity," that's a candidate for a saved search and dashboard panel.

In our trading platform, we maintain about 40 saved searches covering common operational queries. New team members can onboard in days instead of weeks because the hard work of learning our log structure is already done and shared.

Building dashboards from visualizations

A dashboard combines visualizations—charts, tables, metrics—into a single view. We build dashboards to monitor what matters: throughput, latency, error rates, and user impact.

The typical workflow is create-visualize-compose. First, you build individual visualizations in Discover or the Visualize app. A visualization takes a saved search (or creates one implicitly) and renders it as a bar chart, time series, heatmap, or table.

For example, to visualize error rates over time:

1. Start in Discover with a KQL filter for errors: level: ERROR
2. Go to Visualize → Create visualization
3. Choose "Line" chart type
4. Set X-axis to "Date histogram" on @timestamp (bucketed by 5 minutes)
5. Set Y-axis to "Count" of documents
6. Add a filter for your production environment
7. Save the visualization as "Prod errors per 5 min"

Once you have 5–10 visualizations, assemble them into a dashboard. Click "Create dashboard," add panels (your saved visualizations), resize them, and arrange for readability. Top row might show key metrics: overall error count, average response time, and active users. Rows below drill deeper into specific services or endpoints.

A real production example: our e-commerce checkout service has a dashboard with six panels:

• Total transactions (large metric)
• Success vs failure rate (pie chart)
• Errors by type (table, sortable)
• Response time over time (line chart with thresholds)
• Database query latency (histogram)
• Recent high-impact errors (scrollable list)

The dashboard auto-refreshes every 10 seconds and uses time-range controls so oncall can zoom into incidents. A developer can glance at it during a deploy or check it first thing when paged.

Pro tip: use dashboard-level filters. Add a filter control for environment or region at the top, and all panels below inherit that filter. This lets you slice the same dashboard by different dimensions without duplicating panels.

Common mistakes

Overcomplicating data views: We've seen teams create 20 separate data views when 3–4 would suffice. Each extra data view adds cognitive load. Stick to one per application or microservice.

Ignoring data view field settings: If you don't set numeric formatting, date parsing, or field type hints at the data view level, every dashboard and search has to work around it. Spend an hour configuring fields once; save days of friction later.

Writing KQL without understanding boolean logic: status: 500 OR 503 doesn't mean what you think; it's parsed as status: 500 OR (implicit field name: 503). Use status: (500 OR 503) or status >= 500 AND status < 600 for clarity.

Dashboards as a dumping ground: Adding every visualization you create to a dashboard defeats the purpose. Ruthlessly remove panels that don't answer a real operational question. A tight dashboard your team actually uses beats a bloated one gathering dust.

Not versioning saved searches: If a saved search changes and breaks downstream dashboards, you have no way to revert. Consider prefixing versions in names: "Errors by service v2" or keeping old searches with a "_archived" suffix.

Key takeaways

• Data views are your first layer of abstraction—configure them once with proper field formatting and runtime fields to save repetitive work across dashboards and searches.
• KQL is approachable and powerful for filtering; learn wildcards, ranges, and boolean grouping to unlock fast incident response.
• Saved searches capture reusable analytical patterns and serve as the atomic unit for team knowledge sharing.
• Dashboards multiply value when built with intent: each panel should answer a specific question your team actually asks during operations or troubleshooting.
• Treat Discover and dashboards as iterative tools; start with quick exploration in Discover, then stabilize useful patterns into saved searches and dashboard panels for long-term team use.