SLIs, SLOs and error budgets: what developers should know

SLIs, SLOs and error budgets

You've shipped a feature that works 99% of the time in production, but your CEO is still upset about downtime. Your incident commander wants to know if you can deploy again today. Your on-call engineer is burned out from firefighting. These conflicts exist because engineering and business don't speak the same language about reliability. SLIs, SLOs and error budgets fix that by giving you a shared framework to discuss what "good enough" actually means and how much failure you can afford.

What is an SLI and why it matters

An SLI (Service Level Indicator) is a metric that measures whether your system is working from the user's perspective. It's concrete, measurable, and tied to real behavior. Not "the database was up"—that's infrastructure. We care about what users actually experience.

At a ride-hailing platform I worked on, our main SLI was the percentage of ride requests that completed successfully within 120 seconds of the user tapping "Book Now". We also tracked payment success rate as a separate SLI, because a ride that completes but payment fails is a failed user journey. For an e-commerce checkout, you might measure "add-to-cart → payment successful → order confirmation page delivered", not just "application process exit code 0".

The key is choosing SLIs that align with what your users care about, not what's easiest to measure. Common SLIs include:

• Request success rate (HTTP 2xx responses / total requests)
• Latency (percentage of requests completing under a threshold, e.g., p99 under 500ms)
• Availability (can users reach the service at all)
• Correctness (data processed matches expected output)

You'll typically have 2–4 SLIs per service. More than that dilutes focus and makes decisions harder. At our travel booking platform, we had success rate (most critical), latency (user experience), and payment completion (revenue impact). Everything else was secondary.

Measuring SLIs requires instrumentation. A simple approach:

// Track successful vs. failed requests
const requestGauge = new prometheus.Counter({
  name: 'http_requests_total',
  help: 'Total HTTP requests',
  labelNames: ['method', 'route', 'status'],
});

// In your middleware
app.use((req, res, next) => {
  res.on('finish', () => {
    requestGauge.inc({
      method: req.method,
      route: req.route?.path || 'unknown',
      status: res.statusCode,
    });
  });
  next();
});

// Query: requests with status 200-299 / total requests = SLI
// SELECT rate(http_requests_total{status=~"2.."}[5m]) / 
//        rate(http_requests_total[5m])

What is an SLO and how to set one

An SLO (Service Level Objective) is the target you commit to. It's a business decision wrapped in a number. "We will keep our API at 99.9% success rate measured over any 30-day rolling window" is an SLO. It says: this is acceptable, and breaching it is a problem we need to solve.

SLOs are NOT set by engineers alone. Your product manager needs to weigh the cost of downtime against development velocity. Your operations team needs input on what's realistic. I've seen teams pick 99.99% uptime and then burn out trying to achieve it for a feature that loses them $0.10 per minute of downtime.

Start with your business impact. Ask:

1. How much revenue do we lose per minute of downtime?
2. How many users are blocked?
3. Can users work around it?
4. What infrastructure cost does 99.9% vs. 99.99% actually add?

At the ride-hailing platform, we set 99.5% success rate over 30 days. That sounds low, but it meant we'd tolerate about 3.5 hours of complete outage per month. We calculated that 30 minutes of unexpected downtime cost us roughly $50k in cancellations and refunds. Hitting 99.99% would have required multi-region failover, increasing costs by $200k/year. The math didn't work. 99.5% was the right target.

For a trading platform (much lower tolerance for downtime), we went 99.95%. For an internal admin tool, 95% was acceptable.

The SLO also needs a measurement window. Common choices:

• 30-day rolling window: Most useful. Gives you breathing room after incidents without resetting to zero.
• Calendar month: Simpler for billing and stakeholders, but harsh—one bad day early in the month ruins your budget.
• Weekly: Tighter feedback loop, but creates pressure to do postmortems every week if you miss it.

I prefer 30-day rolling because it smooths out incidents and forces you to think about systematic reliability, not just "did we have an outage this week".

Error budget: the failure you can afford

An error budget is what you calculate from your SLO. If you commit to 99.9% success, you're implicitly saying "we can fail 0.1% of the time". Over 30 days, that's a budget of failures you can afford.

The math is simple:

Once you know your budget, you make decisions with it. Early in the month when you're close to your budget target, you're more conservative: feature deploys go slower, you demand higher test coverage, you delay risky refactors. Late in the month after an incident, you're protecting what remains.

At the CRM platform we built, we tracked this in Datadog:

// Calculate remaining error budget
const sloThreshold = 0.999; // 99.9%
const successRate = successfulRequests / totalRequests;
const errorBudgetRemaining = ((sloThreshold - (1 - successRate)) / (1 - sloThreshold)) * 100;

// If errorBudgetRemaining < 20%, trigger deployment freeze
if (errorBudgetRemaining < 20) {
  console.warn('Error budget critically low. Pausing feature releases.');
  // Notify on-call lead, suspend CI/CD for non-critical features
}

This number tells your story. When error budget is above 50%, you have breathing room—deploy multiple times a day if you want. When it drops below 20%, you're in protect mode: only critical bug fixes, enhanced monitoring, extra reviews. When it hits zero, you've violated your SLO and owe your customers an explanation and a plan to recover.

Spending budget on releases, protecting it when low

The real magic of error budgets is they give you permission to fail strategically. You're not trying to achieve zero downtime—that's impossible and wastes money. You're trying to be predictable about failure.

When error budget is healthy (60%+ remaining in a rolling 30-day window), deploy often. Test in production if you have the controls. Roll out experiments. This is when you move fast. We'd deploy the backend 3–5 times a day during high-budget periods.

When error budget drops below 40%, you shift mode:

• Require explicit sign-off on any non-critical deployment
• Expand testing scope—add more integration tests, run chaos engineering
• Increase canary rollout percentage from 10% to 5% or slower
• Have a senior engineer review deployment runbooks

When error budget is nearly depleted (<10%), you freeze. No deploys except critical security fixes or rollbacks. Your SLO isn't being met—you're in debt. Until you climb back above 10%, you're in stabilization mode.

This aligns engineering incentives with business reality. Your PM wants features fast. Your ops team wants stability. Your SLO and error budget let you negotiate the tradeoff in numbers they both understand: "We can ship this feature, but we're burning 15% of our monthly error budget. That leaves us 5% cushion. Do you want to ship or should we spend the week on hardening?"

Common mistakes

• Setting SLOs too tight: 99.99% sounds professional but often wastes millions on infrastructure for unproven returns. Start at 99% and tighten only when the business proves it matters.
• Measuring the wrong SLI: Tracking uptime when you should track success rate means you miss cascading failures (service is "up" but requests timeout). Test your SLIs against actual user pain.
• Ignoring error budget discipline: Treating error budget as advisory instead of law defeats the purpose. If you violate your SLO and ignore it, you've made a different commitment—just not one you meant to make.
• Not communicating budget to the team: Engineers can't make good decisions if they don't know whether budget is healthy or depleted. Display it in Slack, on dashboards, in standup.
• Confusing SLO with SLA: SLO is internal (what we commit to ourselves). SLA is external (what we commit to customers with penalties). Your internal SLO should be tighter than your external SLA so you hit your promises.

Key takeaways

• An SLI is what you measure: the percentage of user-facing operations that succeed.
• An SLO is what you promise: a target like 99.9% over a 30-day rolling window, set by business and engineering together.
• An error budget is your failure allowance: the percentage of requests you can afford to lose before violating your SLO.
• Spend budget freely when healthy (above 50% remaining). Defend it fiercely when low (below 20%). Freeze deploys when exhausted (below 5%).
• Error budgets make the reliability conversation concrete. Instead of "we should be more stable", you discuss: "we have 8 hours of downtime budget left this month; does that feature ship or not?"
• Track and communicate error budget constantly so the entire team understands the tradeoff between speed and stability.