Query logs with Logs Insights: a AWS guide for developers

The problem: logs without answers

You're on call at 2 AM. Your monitoring alerts are firing. Users report slow transactions. You log into CloudWatch and find 50 million log entries from the past hour. Grep doesn't help. Exporting to S3 and querying takes 20 minutes you don't have. This is where query logs with Logs Insights: a AWS guide for developers becomes your lifeline. Instead of drowning in raw text, you get structured answers in seconds.

Why structured JSON logs matter first

Before you can query effectively, your logs need shape. A raw text log like "Request took 234ms" is almost useless when you have millions of them. A structured JSON log entry with fields attached—timestamp, path, duration, error level, user ID, request ID—gives you something to grab onto.

Here's the difference:

Unstructured:

ERROR: Payment service timeout after 5000ms calling /api/v1/checkout

Structured:

{
  "timestamp": "2024-01-15T02:34:12.456Z",
  "level": "error",
  "service": "payment",
  "path": "/api/v1/checkout",
  "durationMs": 5000,
  "errorCode": "TIMEOUT",
  "userId": "user_8d42f",
  "requestId": "req_abc123"
}

The second version gives Logs Insights something to work with. Each field becomes a column you can filter, aggregate, and sort. Most modern frameworks (NestJS, Express, Next.js APIs) can output JSON with the right logger. Use something like Pino or Winston. The setup takes an hour. The time savings on incident response pays for itself in a week.

Filter, aggregate, and sort: the core queries

Logs Insights uses a query language similar to SQL but built for log analysis. The three operations you'll do 90% of the time are filtering, grouping (aggregating), and sorting.

Filter: Narrow down to the logs that matter.

fields @timestamp, level, message, durationMs
| filter level = "error"

This shows you only errors. Add more conditions:

fields @timestamp, level, message, durationMs, path
| filter level = "error" and durationMs > 5000

Now you're looking at errors that took longer than 5 seconds. You can also filter with pattern matching:

fields @timestamp, errorMessage
| filter errorMessage like /timeout/i

Aggregate: Summarize patterns across thousands of logs.

fields @timestamp, path, durationMs
| filter level = "error"
| stats count() by path
| sort count() desc

This query does real work. It answers: "Which endpoint is broken the most?" You get:

Now you know to look at checkout first. Real incident response.

Sort: Order results by whatever matters—count, duration, timestamp.

fields @timestamp, userId, durationMs
| stats avg(durationMs) as avgDuration by userId
| sort avgDuration desc
| limit 10

This finds the 10 slowest users (or integrations, or accounts). Sometimes a single customer's requests are timing out because they're sending malformed data or hammering your API.

Real example: debugging a production incident

Last month, a ride-hailing platform saw a spike in failed booking requests at 3 PM. On-call engineer opens Logs Insights and runs:

fields @timestamp, bookingId, status, durationMs, errorCode
| filter status = "failed"
| stats count() as failures, pct(durationMs, 95) as p95Duration by errorCode
| sort failures desc

Result: "PAYMENT_DECLINED" had 3,200 failures. The payment gateway was rejecting transactions above $50. A pricing change had rolled out that morning but payment limits weren't updated. The query took 8 seconds. A manual log search would have taken an hour. Fix deployed in 15 minutes.

Save common queries for your team

The best queries live in your incident playbooks. CloudWatch Logs Insights lets you save queries as favorites. Better: store them in your repo's wiki or runbook alongside the context for when to use them.

A typical set of saved queries for a backend team might look like:

• Errors by endpoint: Find which API is broken
• P95 latency by service: Spot performance degradation
• Database connection pool exhaustion: Watch for "connection timeout" errors
• Failed authentication attempts: Security incidents
• Memory warnings in logs: Early signs of a leak
• External API timeouts: Dependencies failing

When an alert fires, you're not starting from scratch. You're running a proven query that's answered the question before. That's the difference between 5-minute context gathering and 30-minute detective work.

Use Logs Insights during incidents, not after

Many teams treat Logs Insights as a post-mortem tool. Wrong mindset. It's an incident response tool. The moment a alert fires, you query live logs in parallel with your monitoring dashboard. Your dashboard tells you something is wrong. Logs Insights tells you what and where.

Best practice: have a shared Logs Insights tab open during any active incident. The on-call engineer runs a few standard queries every 30 seconds to track changes. Is the error rate rising or falling? Which users are affected? Is the issue spreading to other services?

Connect this to your incident channel (Slack, PagerDuty, etc.). Someone posts a query result, the team aligns on the problem, and the fix gets prioritized. Communication improves. Mean time to resolution drops.

Common mistakes

Mistake 1: Querying without filtering first. Running `fields * | stats count()` across a week of logs from a high-traffic service can time out. Always start with `filter` to narrow the dataset. Filter by time range, log level, and service first.

Mistake 2: Not parsing semi-structured logs. If your logs are JSON embedded in text (like `"message": "User action: {JSON blob}"`), use the `parse` command to extract inner fields before filtering. Otherwise you're treating the whole thing as a string and losing precision.

Mistake 3: Forgetting to set the time range. The default is 5 minutes. When you're investigating a 2-hour incident, you need to expand the range in the UI or use `filter @timestamp > "2024-01-15T00:00:00Z"` in your query. Many teams waste 10 minutes wondering why they're not seeing logs.

Mistake 4: Overloading a single query. If you're joining multiple log groups, filtering by 10 conditions, and running 4 aggregations, the query gets slow and hard to debug. Break it into smaller queries. Run one to get the error codes, then drill into the top one with a second query.

Mistake 5: Not instrumentation your logs with request IDs. A request ID lets you trace one user's flow across multiple services. Without it, you're guessing which logs belong together. Add it to every structured log entry from day one.

Key takeaways

• Structured JSON logging is the foundation. Raw text logs are unsearchable at scale.
• Learn the three core operations: filter, stats (aggregate), and sort. Most production queries use only these.
• Save your most common queries and run them during active incidents, not after.
• Always filter first to reduce dataset size. Narrow before you aggregate.
• Include request IDs and user IDs in every log. They're how you trace problems across services.
• Treat Logs Insights as part of your incident toolkit alongside dashboards and alerts. Use it to answer "what failed and why" in seconds.