Elasticsearch with .NET: real-world lessons and examples

Elasticsearch with .NET: real-world lessons and examples

After fifty days of diving into Elasticsearch fundamentals, we now focus on the practical reality of integrating it with .NET applications. The Elastic.Clients.Elasticsearch NuGet package has matured significantly, and most teams adopting Elasticsearch in production .NET environments rely on it to bridge their domain logic and search infrastructure. In this article, we'll cover what we've learned from shipping search features in e-commerce platforms, SaaS dashboards, and high-volume transaction systems.

Typed requests and responses

One of the strongest advantages of the modern Elastic.Clients library is its full type safety. Unlike the older NEST library or raw REST calls, every query and filter you construct is validated at compile time. This matters enormously when you're refactoring a search feature or adding fields to your domain models.

Consider an e-commerce scenario where you need to search products by category, price range, and availability status. Without types, a typo in a field name or filter condition would silently fail at runtime. With typed requests:

var response = await client.SearchAsync<Product>(s => s
    .Query(q => q
        .Bool(b => b
            .Must(m => m.Term(t => t.Category, "Electronics"))
            .Filter(f => f.Range(r => r.Price.Gte(50).Lte(500)))
            .Filter(f => f.Term(t => t.Status, "in_stock"))
        )
    )
    .Size(20)
);

foreach (var hit in response.Hits.Hits)
{
    var product = hit.Source;
    Console.WriteLine($"{product.Name} - ${product.Price}");
}

The compiler ensures that Category, Price, and Status exist on your Product class. If you rename a field, your code won't compile until you update the query. This safety saves hours of debugging in production.

Responses are equally typed. The SearchResponse<T> object gives you structured access to hits, aggregations, total count, and metadata. No parsing JSON strings or casting dynamic objects.

Fluent query builders

The fluent API in Elastic.Clients makes complex queries readable and maintainable. Instead of constructing query DSL by hand, you build queries declaratively, and the library emits the correct JSON behind the scenes.

Let's imagine a ride-hailing service filtering available drivers. You need drivers within a radius, with a minimum rating, and currently online. A fluent approach looks like:

var nearbyDriversQuery = client.SearchAsync<Driver>(s => s
    .Query(q => q
        .Bool(b => b
            .Must(m => m
                .GeoDistance(g => g
                    .Field(f => f.Location)
                    .DistanceType(GeoDistanceType.Arc)
                    .Distance("5km")
                    .Location(new GeoLocation { Latitude = 40.7128, Longitude = -74.0060 })
                )
            )
            .Filter(f => f.Range(r => r.Rating.Gte(4.5)))
            .Filter(f => f.Term(t => t.Status, "online"))
        )
    )
    .Sort(sort => sort
        .GeoDistance(g => g
            .Field(f => f.Location)
            .Location(new GeoLocation { Latitude = 40.7128, Longitude = -74.0060 })
            .Order(SortOrder.Asc)
        )
    )
    .Size(10)
);

var result = await nearbyDriversQuery;

Each method call is chainable and self-documenting. Anyone reading this code understands immediately what filters are applied and in what order. Contrast this with handwriting JSON DSL as a string—easy to misquote, hard to refactor, and prone to silent bugs.

Register client as singleton

The Elasticsearch client is thread-safe and designed to be reused across your application lifetime. Registering it as a singleton in your dependency injection container is not just a best practice—it's essential for connection pooling, performance, and resource efficiency.

In a typical .NET 6+ application:

// Program.cs
var settings = new ElasticsearchClientSettings(new Uri("https://elasticsearch:9200"))
    .Authentication(new BasicAuthentication("elastic", "your-password"))
    .DefaultIndex("products");

var client = new ElasticsearchClient(settings);

builder.Services.AddSingleton(client);

Now inject ElasticsearchClient wherever you need it:

public class ProductSearchService
{
    private readonly ElasticsearchClient _esClient;

    public ProductSearchService(ElasticsearchClient esClient)
    {
        _esClient = esClient;
    }

    public async Task<List<Product>> SearchAsync(string query, int page = 1)
    {
        var from = (page - 1) * 20;
        var response = await _esClient.SearchAsync<Product>(s => s
            .From(from)
            .Size(20)
            .Query(q => q.Match(m => m.Field(f => f.Name).Query(query)))
        );

        return response.Documents.ToList();
    }
}

Creating a new client per request wastes TCP connections, exhausts ports, and degrades performance under load. A singleton ensures all requests share the same connection pool and HTTP message handler.

Bulk helpers for indexing

When indexing hundreds of thousands of records—common in data migrations, daily refreshes, or real-time streaming scenarios—the bulk API is non-negotiable. Sending one document at a time will crush throughput and latency.

The Elastic.Clients library includes a BulkAll helper that handles batching, retries, and backpressure automatically:

public async Task IndexProductsInBulkAsync(List<Product> products)
{
    var bulkAllObservable = _esClient.BulkAll(
        products,
        b => b
            .Index("products")
            .BackoffTime(TimeSpan.FromSeconds(1))
            .BackoffRetries(3)
            .Size(500)  // batch size
            .MaxDegreeOfParallelism(4)
    );

    var waitHandle = new CountdownEvent(1);
    var exceptionCount = 0;

    bulkAllObservable.Subscribe(
        onNext: response =>
        {
            if (!response.IsValid)
            {
                Interlocked.Increment(ref exceptionCount);
                Console.WriteLine($"Bulk error: {response.DebugInformation}");
            }
        },
        onError: e =>
        {
            Console.WriteLine($"Fatal error: {e.Message}");
            waitHandle.Signal();
        },
        onCompleted: () =>
        {
            Console.WriteLine($"Bulk indexing complete. Exceptions: {exceptionCount}");
            waitHandle.Signal();
        }
    );

    waitHandle.Wait();
}

This approach batches documents into requests of 500 items, parallelizes up to 4 concurrent bulk calls, and automatically backs off and retries on transient failures. For a dataset of 1 million products, you'll see throughput improvements of 10-50x compared to serial indexing.

Common mistakes

Creating a new client per request. We've seen teams instantiate ElasticsearchClient inside service methods or controllers, leading to exhausted connection pools and port starvation under load. Always use dependency injection with a singleton.

Ignoring response validity. Elasticsearch queries can partially succeed—some shards might fail while others succeed. Always check response.IsValid and inspect response.Shards to understand what actually executed.

Not setting index mappings before bulk indexing. If you're indexing nested objects or geolocations, define your index mappings first. Otherwise, Elasticsearch will auto-detect types, which can cause unexpected behavior when the first document for a field arrives.

Querying without pagination limits. A search without a Size() parameter defaults to 10 results, which is fine for most UIs but easy to miss. More importantly, never use From/Size for deep pagination (e.g., page 10,000)—use search-after or scroll for large result sets.

Neglecting error handling in bulk operations. The BulkAll helper won't throw if documents fail individually. You must subscribe to the observable and log failed documents separately. Silently ignoring bulk errors means losing data.

Key takeaways

• Typed requests and responses in Elastic.Clients catch bugs at compile time and make refactoring safe.
• Fluent query builders are more readable and maintainable than hand-crafted JSON DSL.
• Register the Elasticsearch client as a singleton in your DI container; creating new instances per request degrades performance and exhausts resources.
• Use the BulkAll helper for large-scale indexing; it handles batching, parallelism, and backoff automatically.
• Always check response validity and handle partial failures, especially in bulk operations.
• Define index mappings before indexing to ensure field types match your domain model.