Benchmarking with Criterion: a Rust guide for developers

The Cost of Guessing

Most performance problems in production start the same way: a developer optimizes code they think is slow, ships it, and discovers the bottleneck was elsewhere. Benchmarking with Criterion: a Rust guide for developers exists to break that cycle. If you've worked on high-throughput systems—whether handling payment processing, serving millions of requests, or processing streaming data—you know that intuition about performance is rarely accurate. Criterion gives you the statistical rigor to measure what actually matters.

Statistical Benchmarks: Understanding the Numbers

Criterion is Rust's de facto standard benchmarking framework because it doesn't just run your code and tell you how fast it is. It runs your code many times, collects statistics, and reports confidence intervals. This approach filters out noise from system interrupts, cache behavior, and OS scheduling.

When you benchmark, you're not looking for a single number. You're looking for a distribution. Criterion samples your function repeatedly, analyzes the results, and estimates the mean and standard deviation. It then reports a confidence interval—typically 95%—which tells you where the true performance likely falls.

Here's a minimal Criterion benchmark:

use criterion::{black_box, criterion_group, criterion_main, Criterion};

fn fibonacci(n: u32) -> u32 {
    match n {
        0 | 1 => 1,
        n => fibonacci(n - 1) + fibonacci(n - 2),
    }
}

fn fib_benchmark(c: &mut Criterion) {
    c.bench_function("fib 20", |b| b.iter(|| fibonacci(black_box(20))));
}

criterion_group!(benches, fib_benchmark);
criterion_main!(benches);

Add this to benches/my_bench.rs in your Cargo project. The black_box function is critical: it prevents the compiler from optimizing away your benchmark by inlining constants or eliminating dead code. Without it, you're not measuring real performance—you're measuring the compiler's ability to eliminate your code.

When you run cargo bench, Criterion performs a warmup phase, then collects samples. After a few runs, you'll see output like:

fib 20                  time:   [49.321 ms 49.405 ms 49.501 ms]

That bracketed range is your confidence interval. If your interval is wide, Criterion ran more samples to narrow it. If it's tight, you have a precise measurement.

Compare Against Baselines: Detecting Regressions

Raw numbers are useful, but their real power emerges when you compare them over time. Did your optimization actually help? Did a refactor introduce a regression? Criterion stores baseline data and automatically compares new runs against it.

On your first cargo bench, Criterion creates target/criterion with baseline results. On subsequent runs, it compares new measurements against those baselines and flags statistically significant changes.

Let's say you're optimizing a string parsing function used in your API handlers:

fn parse_user_id(input: &str) -> Option {
    input.trim().parse().ok()
}

fn parse_benchmark(c: &mut Criterion) {
    c.bench_function("parse_user_id", |b| {
        b.iter(|| parse_user_id(black_box("12345")))
    });
}

After establishing a baseline, you refactor to use a faster parsing strategy. The next benchmark run automatically compares against the stored baseline. If the new implementation is 15% faster with statistical significance, Criterion tells you. If it's slower, you know immediately.

This is where benchmarking in CI becomes invaluable. Run cargo bench --bench my_bench -- --output-format bencher | tee output.txt to generate machine-readable output. Many teams integrate this into pull request checks to catch performance regressions before they reach production.

Avoid Optimizing Without Data: The Empirical Approach

Premature optimization is the root of all evil, but so is ignoring real bottlenecks. The key is having data. Too many teams optimize based on assumptions: "I bet allocations are slow," or "JSON parsing must be the bottleneck." Criterion lets you measure instead of guess.

Consider a typical web service scenario. You have request processing, database queries, and response serialization. A developer might assume JSON serialization is slow and spend two days optimizing it, only to discover the database query was the actual bottleneck.

With Criterion, you benchmark each step in isolation:

use criterion::{black_box, criterion_group, criterion_main, Criterion};
use serde_json::json;

fn serialize_response(c: &mut Criterion) {
    let data = vec![
        json!({"id": 1, "name": "Alice", "email": "alice@example.com"}),
        json!({"id": 2, "name": "Bob", "email": "bob@example.com"}),
        json!({"id": 3, "name": "Charlie", "email": "charlie@example.com"}),
    ];

    c.bench_function("serialize_response", |b| {
        b.iter(|| {
            serde_json::to_string(&black_box(&data)).unwrap()
        })
    });
}

criterion_group!(benches, serialize_response);
criterion_main!(benches);

Now you have concrete numbers. If serialization takes 100 microseconds and database queries take 50 milliseconds, you know where optimization effort should focus. This empirical approach scales: benchmark your critical paths, identify the slowest, and optimize there.

Many teams also use Criterion to set performance budgets. You decide that a particular operation should never exceed 10 milliseconds, and you enshrine that in your benchmarks. If a future change violates that constraint, the benchmark fails and alerts the team before deployment.

Cargo Bench Runs Them: Integration and Workflow

Benchmarks live in your repository just like tests. By convention, they go in the benches/ directory. Configure them in Cargo.toml:

[[bench]]
name = "my_bench"
harness = false

The harness = false line tells Cargo that Criterion itself is the test harness, not Rust's built-in one.

Running benchmarks is simple:

cargo bench

This compiles in release mode, runs all benchmarks, and generates HTML reports in target/criterion/report/index.html. The reports include graphs, statistical analysis, and comparisons to previous runs.

To run a specific benchmark:

cargo bench -- parse_user_id

For local development, you might want faster turnaround. Criterion supports a warm-up phase that can be lengthy. Pass --sample-size 10 to reduce iterations during development:

cargo bench -- --sample-size 10

In CI, always use the default settings for reliable results. The full statistical analysis takes longer but produces trustworthy data.

Common Mistakes

Forgetting black_box: Without it, the compiler optimizes away your benchmark. Always wrap inputs and use the result.

Benchmarking too narrowly: If you benchmark only allocation without the surrounding logic, you might optimize allocations when the real cost is elsewhere. Benchmark realistic workloads.

Running on inconsistent hardware: Benchmarks are sensitive to system state. Run them on the same machine or in an identical CI environment. Comparing a MacBook benchmark to a Linux server benchmark is meaningless.

Ignoring confidence intervals: If your interval is 10 ms ± 5 ms, any optimization smaller than the interval width isn't real. Larger sample sizes narrow intervals.

Benchmarking in debug mode: cargo bench compiles in release mode by default, but verify your Cargo.toml doesn't override it. Debug code performs very differently.

Key Takeaways

• Criterion provides statistical rigor: it runs your code many times, measures variance, and reports confidence intervals instead of single numbers.
• Baseline comparisons detect regressions automatically; integrate into CI to catch performance changes before production.
• Always measure before optimizing; empirical data beats intuition about which code paths matter.
• Use black_box to prevent the compiler from eliminating your benchmark; it's non-negotiable.
• Benchmark realistic workloads and run on consistent hardware; narrow intervals require higher sample counts.
• Store benchmarks alongside your code and run them regularly; performance is a feature that decays without attention.