Benchmarking with BenchmarkDotNet in .NET & ASP.NET Core

Benchmarking with BenchmarkDotNet

You've optimized a critical path in your API, trimmed some allocations, and you're confident it's faster. Then you run it on production and see almost no difference. The problem? You benchmarked it wrong. Benchmarking with BenchmarkDotNet forces you to measure performance properly, accounting for the invisible forces that make naive timing lie: JIT compilation, garbage collection, CPU caching, and branch prediction. In this post, I'll show you how to stop guessing and start measuring.

Why Stopwatch Lies — The Hidden Factors That Break Naive Timing

Every developer has done this at least once:

var sw = Stopwatch.StartNew();
for (int i = 0; i < 1000; i++)
{
    MyExpensiveOperation();
}
sw.Stop();
Console.WriteLine($"Total: {sw.ElapsedMilliseconds}ms");

This tells you almost nothing useful. Here's why:

JIT Warm-up: The first time your method runs, the .NET JIT compiler compiles it from IL to native machine code. This adds overhead that doesn't represent steady-state performance. Run the same code a second time and it's faster—sometimes dramatically. Production will see warm-up overhead on cold start, but your benchmark might only measure the hot path.

Garbage Collection: Full GC pauses are unpredictable. A benchmark that collects 10GB of temporary objects might include one full GC that skews your numbers by 500%. The next run might not trigger a collection. Same code, wildly different results.

CPU Caching and Branch Prediction: Modern CPUs optimize based on what they've seen recently. Loop-unrolling, instruction caching, and branch predictor history all affect speed. Run the benchmark once and the CPU state is hot. Run it fresh and you're measuring something different.

The real cost: You ship "optimized" code that's actually slower in production because you measured under fake conditions.

BenchmarkDotNet Handles the Complexity — A Real Benchmark

BenchmarkDotNet is a free NuGet package that automates everything: JIT warm-up, multiple iterations, statistical analysis, and memory profiling. It runs your code under controlled conditions and gives you reproducible results.

Install it first:

dotnet add package BenchmarkDotNet

Here's a real example. I want to compare JSON serialization strategies for an order object:

using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;
using System.Text.Json;

[MemoryDiagnoser]
[SimpleJob(warmupCount: 3, targetCount: 10)]
public class JsonSerializationBench
{
    private Order _order;

    [GlobalSetup]
    public void Setup()
    {
        _order = new Order
        {
            Id = 12345,
            CustomerId = 67890,
            Items = new[]
            {
                new OrderItem { Sku = "PROD-001", Qty = 2, Price = 29.99m },
                new OrderItem { Sku = "PROD-002", Qty = 1, Price = 99.50m },
            },
            CreatedAt = DateTime.UtcNow,
            Status = "Pending"
        };
    }

    [Benchmark(Baseline = true)]
    public string SystemTextJsonDefault()
    {
        return JsonSerializer.Serialize(_order);
    }

    [Benchmark]
    public string SystemTextJsonSourceGen()
    {
        var options = new JsonSerializerOptions 
        { 
            TypeInfoResolver = OrderJsonContext.Default 
        };
        return JsonSerializer.Serialize(_order, options);
    }
}

class Program
{
    static void Main(string[] args)
    {
        var summary = BenchmarkRunner.Run();
    }
}

Breaking this down:

• [MemoryDiagnoser] — Tracks memory allocations (we'll dig into this next).
• [SimpleJob] — Runs 3 warm-up iterations (discarded) then 10 measured iterations. This ensures the JIT is fully warm.
• [GlobalSetup] — Runs once per benchmark method to initialize the shared state. Your test object is created once and reused.
• [Benchmark(Baseline = true)] — Marks the first method as the baseline. Other results are shown relative to it (1.00x, 1.15x, etc.).

When you run this, BenchmarkDotNet produces output like:

The source-generated version is about 14% faster and allocates less. That's a real, reproducible difference—not a lucky run or measurement noise.

Watch Allocations — Memory Often Matters More Than Speed

Raw execution time can be misleading. Two methods might take the same wall-clock time, but one allocates 100MB and the other allocates 1MB. The high-allocation method triggers GC stops that freeze your entire application. On a high-throughput server, GC pauses are a silent killer.

MemoryDiagnoser shows you bytes allocated per iteration. It's one of the most valuable metrics you'll see.

Consider a LINQ query that builds a list:

[MemoryDiagnoser]
public class AllocationBench
{
    private int[] _numbers;

    [GlobalSetup]
    public void Setup()
    {
        _numbers = Enumerable.Range(1, 1000).ToArray();
    }

    [Benchmark(Baseline = true)]
    public int SumWithLinq()
    {
        return _numbers
            .Where(x => x % 2 == 0)
            .Select(x => x * 2)
            .Sum();
    }

    [Benchmark]
    public int SumWithLoop()
    {
        int sum = 0;
        foreach (var num in _numbers)
        {
            if (num % 2 == 0)
                sum += num * 2;
        }
        return sum;
    }
}

When you run this, the LINQ version allocates for the enumerators and temporary buffers. The loop version allocates almost nothing. In a hot path called millions of times per second, the loop wins decisively not because it's 10% faster, but because it doesn't trigger GC.

This is why allocation often matters more than raw speed. A method that's 5% faster but allocates 10x more is a trap. Use [MemoryDiagnoser] to see it.

Benchmark in Release Mode — Debug Builds Are Meaningless

Always run benchmarks in Release mode. Always. Debug builds disable optimizations, inline expansion, and JIT improvements. You're measuring a false version of your code.

When you run BenchmarkRunner.Run(), it automatically builds your benchmark project in Release mode. Good. But if you run benchmarks from the IDE with the debug configuration, you'll see laughably slow numbers.

Create a dedicated benchmark project separate from your main code:

dotnet new console -n MyApp.Benchmarks
cd MyApp.Benchmarks
dotnet add package BenchmarkDotNet

Then build explicitly before running:

dotnet build -c Release
dotnet run -c Release

The configuration flag matters. I've seen developers measure "improvements" that were just the difference between Debug and Release JIT settings.

Advanced Features Worth Knowing

BenchmarkDotNet has several other attributes that help in real scenarios:

• [Params] — Benchmark the same method with different input sizes. E.g., [Params(10, 100, 1000)] to test with 10, 100, and 1000 items.
• [InvocationCount] — If a single invocation is too fast to measure accurately, run it multiple times per benchmark iteration.
• [ThreadingDiagnoser] — For concurrent code, tracks lock contention and thread switching overhead.
• HtmlExporter — Generates a detailed HTML report with charts and comparisons.

Example with parameters:

[MemoryDiagnoser]
public class SearchBench
{
    private int[] _data;

    [Params(100, 1000, 10000)]
    public int Size { get; set; }

    [GlobalSetup]
    public void Setup()
    {
        _data = Enumerable.Range(0, Size).ToArray();
    }

    [Benchmark]
    public int LinearSearch()
    {
        for (int i = 0; i < _data.Length; i++)
            if (_data[i] == Size / 2) return i;
        return -1;
    }

    [Benchmark]
    public int BinarySearch()
    {
        return Array.BinarySearch(_data, Size / 2);
    }
}

This runs both methods with arrays of 100, 1000, and 10,000 elements. You see how algorithmic complexity plays out in practice.

Common Mistakes

1. Benchmarking Debug builds. Release mode is 5–50x faster. Your numbers are fiction.

2. Not isolating what you're measuring. If you benchmark "serialize and write to disk", you're measuring I/O, not serialization. Measure one thing at a time.

3. Ignoring allocation in favor of speed. A 2% faster method that allocates 10x more is usually worse. Look at both metrics.

4. Running benchmarks on a busy machine. Close your browser, Slack, and VS. Background processes skew results. Dedicated hardware is ideal.

5. Micro-optimizing code that's never called. Benchmark your actual bottlenecks first. Use a profiler (like dotTrace or PerfView) to find them. Don't guess.

Key Takeaways

• Stopwatch alone is unreliable. JIT warm-up, garbage collection, and CPU state distort naive measurements.
• BenchmarkDotNet automates proper benchmarking. It handles warm-up, multiple iterations, and statistical analysis so you don't have to.
• Use [MemoryDiagnoser] to track allocations. Often the difference that matters most in high-throughput code isn't speed—it's GC pressure.
• Always measure in Release mode. Debug builds lie. The JIT in Release mode optimizes differently.
• Measure one thing at a time. Isolate the specific operation you want to optimize. Don't bundle I/O, network, or other variables into your benchmark.
• Profile first, optimize second. Use a profiler to find real bottlenecks. Benchmark your changes to prove they work. Never guess.