SLOs explained: what every Monitoring developer should know

Introduction

Most teams measure uptime. Few teams use that measurement to make better decisions. That's the gap SLOs explained: what every Monitoring developer should know addresses. Service level objectives sit at the intersection of reliability engineering and product velocity—they let you quantify acceptable failure, spend that allowance deliberately, and ship features faster without breaking trust with customers.

This isn't theoretical. We've built systems where SLOs transformed how we deploy. Without them, we either shipped too cautiously or too fast. With them, we had a shared language for risk. By day 39 of our monitoring series, you've learned to observe systems. Now learn to govern them.

SLI — A measurement: share of requests under 300 ms

An SLI (Service Level Indicator) is a concrete metric that represents what "good" means for your service. It's not about infrastructure. It's about what users experience.

Consider a payment processing API. You could measure CPU, network latency, or database query times. But none of those directly answer the question: did the user's transaction complete successfully and fast enough? That's what an SLI answers.

For that payment API, your SLI might be: the percentage of requests that complete in under 300 milliseconds with an HTTP 2xx or 3xx response code. That single metric combines latency and availability into one user-centric view.

Good SLIs share three properties:

• User-centric. They measure what customers perceive, not what operators manage. "Requests under 300 ms" beats "CPU below 70%."
• Measurable. You can emit the data reliably from your application or infrastructure. If you can't measure it consistently, it's not an SLI.
• Meaningful over time. A single data point is noise. An SLI aggregates over minutes, hours, or days so trends emerge.

In Prometheus, you'd track this with a histogram:

http_request_duration_seconds_bucket{le="0.3",endpoint="/payments"}

You query it to get the percentage of requests under 300 ms over the last day. That's your SLI for the day.

SLO — A target: 99.9% over 30 days

An SLO (Service Level Objective) is a target you commit to. It's your SLI with a threshold and a time window. If your SLI is "percentage of requests under 300 ms," your SLO might be "99.9% of requests complete in under 300 ms, measured over any rolling 30-day window."

That 99.9% is not arbitrary. Here's why it matters:

• 99% uptime allows ~7 hours of downtime per month.
• 99.9% uptime allows ~43 minutes of downtime per month.
• 99.99% uptime allows ~4 minutes of downtime per month.
• 99.999% uptime allows ~26 seconds of downtime per month.

The difference between 99.9% and 99.99% is the difference between a few maintenance windows and an incident response team on permanent standby. Your SLO must reflect what your customers actually need and what your team can sustain.

Setting an SLO is a business decision, not an engineering one. A mobile app might target 99.5% because network blips are expected. A healthcare system might target 99.99%. A startup's internal admin tool might target 95%. Each is correct in its context.

The 30-day window is deliberate too. It's long enough to smooth out blips but short enough to matter operationally. A weekly SLO is noisy. A yearly SLO is too abstract to guide daily decisions.

You'll track SLO compliance in your monitoring system. In Prometheus + Grafana, you might have a dashboard that shows:

SLO_compliance = (successful_requests / total_requests) * 100
Target: 99.9%
Current: 99.87% (4 minutes remaining in budget this month)

Error budget — The allowed failures: 0.1% of requests

Your error budget is the inverse of your SLO. If your SLO is 99.9%, your error budget is 0.1%. Over a 30-day month, that means you can afford to break 0.1% of your requests—approximately 43 minutes of complete unavailability, or scattered failures across the month.

This flips the conversation. Instead of "can we deploy today?" the question becomes "do we have budget to deploy today?"

Let's walk through a real scenario. Your payment API has a 99.9% SLO over 30 days:

• Month starts. You have 100% of your error budget (0.1% of requests).
• Week 1. A database failover burns 0.03% of your budget. You have 0.07% remaining.
• Week 2. You ship a feature that causes a 15-minute outage, burning another 0.04%. You have 0.03% remaining.
• Week 3. Your team wants to roll out a major refactor. Do you have budget? Yes—0.03% remains, which is about 13 minutes of downtime. You deploy with extra monitoring.
• Week 4. A wild card incident happens. You've exhausted your budget. Now deployment freezes until the month ends. You focus on stability, not new features.

Error budgets make risk explicit. They answer: how aggressive can we be this week?

To track error budget in practice, you'd emit this metric each day:

error_budget_consumed = (1 - sli_actual) / (1 - slo_target) * 100%

Day 1: SLI = 99.92%, SLO = 99.9%
Consumed = (1 - 0.9992) / (1 - 0.999) * 100% = 80%

Remaining for month: 20% of your budget gone, 80% to spend.

Use the budget — Ship faster when healthy, slow down when burning

The magic of error budgets is they connect reliability to velocity. They're the governance framework that answers: when is it safe to be aggressive?

When you have budget: Ship. Run canary deployments on 5% of traffic instead of 1%. Deploy Friday afternoons instead of only Monday mornings. Do the refactor that improves velocity. Spend the budget deliberately on technical bets that pay dividends.

When you're low on budget: Slow down. Reduce deployment frequency. Revert to manual approval gates. Pause feature work and focus on stability. Investigate what burned your budget—was it a cascade, a traffic spike, or a bad deployment?

When you're out of budget: Deployment freeze. The SLO is a hard commitment. You've communicated to users that you'll hit 99.9%—if you blow that, you've broken trust. Teams often implement strict controls here: only critical production fixes allowed.

This creates a feedback loop. Bad deployments cost you budget. Good deployments don't. Over time, teams naturally improve deployment practices because they see the direct cost.

Imagine your team as a captain with a fuel gauge. You could idle the engine (never deploy, perfect reliability, zero velocity). You could burn fuel recklessly (deploy constantly, reliability collapses). Or you could navigate intelligently: deploy at a pace your fuel budget sustains, and invest fuel-saving improvements when the budget runs low.

Most high-performing teams we've worked with spend their error budgets near 100% monthly. They ship fast, burn the budget on bold moves, and when it runs out they pause and stabilize. It's a rhythm that works.

Common mistakes

Mistake 1: SLOs set too high. If you target 99.99% uptime but your team can only achieve 99.8%, your SLO becomes a demoralizing fiction. Set SLOs based on what you can actually deliver. Start conservative; tighten as your platform matures.

Mistake 2: SLOs include the wrong metrics. A common trap: measuring internal latency instead of end-to-end. Your database query might be fast but the network to the client slow. Measure what the user sees, not what you control.

Mistake 3: Ignoring the error budget. The worst outcome is an SLO that sits in a dashboard nobody looks at. For error budgets to work, they must influence decisions. If you never say "we can't deploy because we're out of budget," your SLO isn't driving behavior.

Mistake 4: SLOs with no consequences. If missing an SLO has no impact on your roadmap or deployment process, teams optimize for other things. Tie SLO compliance to decisions your team actually makes.

Mistake 5: Static SLOs forever. Your first SLO guess will be wrong. Review every quarter. If you consistently beat your target by 1%, tighten it. If you're always near failure, loosen it. SLOs should reflect reality plus ambition, not either alone.

Key takeaways

• An SLI is what you measure: "99.92% of requests complete under 300 ms." It's factual, from the past.
• An SLO is what you target: "99.9% over rolling 30 days." It's a promise about the future.
• An error budget is the gap: 0.1% of requests can fail. This budget is your permission to move fast.
• Error budgets drive deployment decisions. When you have budget, ship. When you don't, stabilize. This alignment between reliability and velocity is where SLOs create real value.
• SLOs are not one-time setup. Review them quarterly. Let real performance and customer feedback shape what targets matter.
• Make error budgets visible. Dashboard them alongside your other metrics. When the team sees the budget declining, behavior changes naturally.