Choosing good SLIs: Monitoring best practices explained

Introduction

We've spent the last 40 days building observability systems, collecting metrics, and wiring up dashboards. But here's the uncomfortable truth: without the right Service Level Indicators, you're measuring noise instead of signal. Choosing good SLIs: Monitoring best practices explained means shifting from "what can we measure?" to "what do users actually feel?" This distinction separates teams that sleep well from teams that wake at 3 AM to fix alerts that don't matter.

SLIs sit at the foundation of SLOs and error budgets. They're the bridge between infrastructure metrics and business impact. Get them wrong, and you'll either create alert fatigue or miss real problems. Get them right, and your monitoring becomes a reliable voice for the user experience.

Understanding SLIs vs. metrics

Before we dive into choosing good SLIs, we need to clarify what they are. A Service Level Indicator is a measured value that reflects the quality of your service as experienced by users. It's not the same as a metric like CPU usage or memory allocation. Those are infrastructure metrics—important for operations, but they're not SLIs.

An SLI is always a ratio or percentage. It answers a specific question: "What percentage of requests met the user's expectations for this dimension?" That could be "99% of requests returned before 100ms" or "98.5% of logins succeeded on the first try." Infrastructure metrics answer different questions—usually about resource consumption or internal behavior.

This matters because alerts based on CPU thresholds fire when your system is under load, even if users are experiencing great performance. Alerts based on SLIs fire when users are genuinely hurt. We need both types of monitoring, but SLIs should drive your alerting strategy.

Availability: Successful requests

Availability is the easiest SLI to understand, yet teams often measure it wrong. The simplest definition is: what percentage of requests succeeded?

A successful request should represent a request that served the user's intent. If a user hits your API and gets a 500 error, that failed. If they get a 429 rate-limit response because they're abusing the system, that's different—many teams count 429s as failed, but consider whether a user getting rate-limited after legitimate traffic is actually a service failure, or whether it's intentional protection.

Here's a Prometheus rule for a basic availability SLI:

- record: sli:availability:requests:success_rate
  expr: |
    sum(rate(http_requests_total{status=~"2.."}[5m])) by (service)
    /
    sum(rate(http_requests_total[5m])) by (service)

This calculates the ratio of successful 2xx responses to all requests over a 5-minute window. That's the foundation. But here's where it gets real: measure at the edge when possible. Don't count internal 500s as failures if they're retried transparently to the user. If your load balancer automatically retries failed requests and the user sees success, measure it that way.

A real scenario: your payment service has a flaky database connection that causes 1% of requests to fail initially, but your retry logic succeeds on the second attempt 99% of the time. From the user's perspective, the service is available. But if you measure raw request success without considering retries, you're reporting 99% availability when users experience 99.99% availability. This misalignment means your SLO is more conservative than the actual user experience warrants.

Another consideration: should you measure availability per-request or per-user? For APIs, per-request makes sense. For web applications where a page load triggers 50 requests, you might want to measure whether the page loaded successfully (user perspective) rather than whether all 50 sub-requests succeeded. Choose based on what users care about.

Latency: Requests under a threshold

Latency as an SLI should answer: what percentage of requests completed within an acceptable time? Not "what's the average latency"—that's a useful diagnostic metric, but it's not an SLI. Users don't care about average; they care about whether their request is fast enough.

The threshold matters enormously. A 100ms SLI for a search API makes sense. A 100ms SLI for a long-running data export doesn't. Choose thresholds based on user expectations and business value. For interactive features, aim for sub-second. For batch operations, multi-second thresholds are reasonable.

Here's a Prometheus rule for latency SLI:

- record: sli:latency:requests:under_threshold
  expr: |
    sum(rate(http_request_duration_seconds_bucket{le="0.1",endpoint="/api/search"}[5m]))
    /
    sum(rate(http_request_duration_seconds_count{endpoint="/api/search"}[5m]))

This measures the percentage of search requests completing in under 100ms. The key here is le="0.1"—the histogram bucket boundary. Your application needs to instrument histogram metrics with boundaries that match your SLI thresholds.

A gotcha: percentiles are not SLIs. "p99 latency is 200ms" tells you about distribution, but it's not an SLI. "95% of requests are under 200ms" is an SLI. They're different and serve different purposes. Use percentiles for diagnosis, SLIs for SLOs.

Real scenario: your checkout flow has multiple steps, each with its own SLI threshold. Step 1 (product details load) should be sub-second. Step 2 (payment processing) might be 5 seconds acceptable. Step 3 (order confirmation email) isn't even synchronous—it's background. Measure each independently because they have different user expectations. A single "checkout latency SLI" hides the real performance picture.

Freshness: Data pipelines and eventual consistency

Freshness is the SLI for systems where data changes over time. It measures: what percentage of queries returned data within an acceptable age?

This is critical for event-driven systems, data warehouses, caching layers, and real-time analytics. A cached value that's 5 seconds old might be unacceptable for live trading, but perfectly fine for a business dashboard refreshed hourly.

Here's how to instrument freshness:

- record: sli:freshness:data:within_acceptable_age
  expr: |
    sum(rate(data_freshness_seconds{freshness_seconds=~"(5|10|15|20)"}[5m]))
    /
    sum(rate(data_queries_total[5m]))

But tracking this requires your application to instrument when data was last updated. Here's what that looks like in practice:

- record: sli:analytics:cache_age_seconds
  expr: |
    time() - analytics_cache_updated_timestamp_seconds

Then when responding to a query, your application logs the cache age and whether it's within the SLI threshold (say, 30 seconds). This lets you track freshness from the user's perspective.

Real scenario: your e-commerce platform uses eventual consistency—inventory updates propagate through Kafka to search indices over 10-60 seconds. Users won't see the exact inventory count, but they see "likely in stock" or "limited stock" based on cached data. Your freshness SLI might be "95% of search results reflect inventory from the last minute." This sets clear expectations and guides infrastructure decisions about cache invalidation.

Measure at the edge when possible

This principle deserves its own emphasis because it changes how you think about SLI collection. Measuring at the edge means capturing user experience as close to the user as possible, not just inside your data center.

If you measure availability only at your origin server, you miss failures in the CDN, DNS, or network. If you measure latency only from the application layer, you miss serialization costs, network latency, and browser parsing time. The user experiences the full stack.

Options for edge measurement:

• Real User Monitoring (RUM): Inject timing code into the browser or mobile app. Capture when pages load, when API calls complete, what the user actually sees.
• Synthetic monitoring: Run automated tests from geographies where your users are. Not a replacement for RUM, but catches failures that real traffic might not hit.
• CDN and load balancer logs: These capture first-mile and last-mile timing and success rates.
• Client-side instrumentation: Mobile apps and SPAs can report when requests complete and how long they took from the user's device.

The cost is that edge metrics are noisier and harder to aggregate. A user on a slow network will report high latency that's not your service's fault. Filter thoughtfully—exclude catastrophic network failures, but don't over-filter away real user experience. Most teams measure availability at the origin but latency from the edge.

A concrete example: your API serves 95% of traffic from AWS us-east-1, but 5% from a mobile app in India with satellite backup. Your data-center latency SLI is 95% under 50ms. But users in India experience 200-400ms naturally. Your edge latency SLI from India might be 90% under 400ms. These are different SLIs for the same service, because users have different expectations. Both belong in your monitoring.

Common mistakes

Teams starting with SLIs often fall into these traps:

• Using uptime instead of availability. Uptime is "was the service running?" (yes/no). Availability is "did successful requests happen?" (percentage). The server can be up while serving errors.
• Setting SLI thresholds too high. A 99.99% SLI for latency sounds impressive but might be unrealistic. Set thresholds based on actual user needs and current capability, then improve.
• Forgetting about SLI attribution. When availability drops to 95%, what caused it? You need dimensions—service name, endpoint, region, client type. A single aggregate SLI number is useless for debugging.
• Measuring internal success instead of user success. Your internal cache hit rate isn't an SLI. Whether the user got data in time is.
• Not updating SLIs as the system evolves. SLIs are for today's system. As you add retry logic, caching, or async processing, revisit whether your SLIs still measure user experience accurately.
• Creating too many SLIs. Aim for 3-5 per service: availability, latency, freshness if applicable, and maybe correctness. Beyond that, you're creating noise instead of signal.

Key takeaways

• SLIs measure user experience as a percentage or ratio, not raw infrastructure metrics. They should answer "what did users actually feel?"
• Availability is the percentage of requests that succeeded. Latency is the percentage of requests that completed within a threshold. Freshness is the percentage of data returned within acceptable age.
• Thresholds matter. Choose them based on user expectations and business requirements, not arbitrary round numbers.
• Measure at the edge when feasible—capture user experience from CDNs, mobile clients, and geographic endpoints, not just from your origin.
• Keep SLIs simple and tied directly to user-visible behavior. Too many SLIs dilute focus and create alert fatigue.
• Revisit SLIs when your architecture changes. Retry logic, caching, and async processing change what "user success" means.
• Use SLIs to define SLOs and error budgets. This creates a direct link from monitoring to decision-making about reliability investment.