Elasticsearch with Node.js: patterns, pitfalls and fixes

Elasticsearch with Node.js: patterns, pitfalls and fixes

When you move from prototyping Elasticsearch queries in Kibana to shipping them in production Node.js applications, the gap between theory and practice widens fast. The official @elastic/elasticsearch client handles connection pooling, retries, and typed responses, but only if you wire it up correctly. We've learned these patterns the hard way—through production incidents, memory leaks, and timeouts at 2 AM—and we're sharing what actually works.

Why the official client matters

The JavaScript ecosystem has no shortage of Elasticsearch wrappers, but using the official client directly gives you predictable behaviour, first-class support for new Elasticsearch features, and honest error handling. You won't find hidden retry loops or magical query transformations; what you write is what hits the cluster.

The cost of getting this wrong is high. A misconfigured client can exhaust your Node.js process memory, leave connections hanging, or hammer your Elasticsearch cluster with retry storms. We'll walk through the patterns that prevent these issues.

Client setup—one client instance per app

The most common mistake we see is creating a new client for each request. This sounds obvious, but it happens constantly in frameworks where dependency injection isn't the default. Every client instance opens its own connection pool; create one per request and you've built a connection leak.

The fix is simple: instantiate the client once at application startup, store it in a singleton or inject it into your dependency container, and reuse it for the lifetime of your app.

import { Client } from '@elastic/elasticsearch'

let es = null

export function initElasticsearch() {
  es = new Client({
    node: process.env.ES_URL || 'http://localhost:9200',
    auth: {
      username: process.env.ES_USER,
      password: process.env.ES_PASSWORD
    },
    maxRetries: 3,
    requestTimeout: 30000
  })
  return es
}

export function getClient() {
  if (!es) throw new Error('Elasticsearch client not initialized')
  return es
}

Call initElasticsearch() once during app startup, then use getClient() everywhere else. In NestJS, register it as a provider. In Express, call it before starting the server. In serverless environments (AWS Lambda), initialize it at the handler level or use provisioned concurrency so the same client persists across invocations.

The connection pool is managed automatically—the client will queue requests if all connections are busy, retry on transient failures, and respect backpressure from Elasticsearch. Set maxRetries and requestTimeout explicitly so your timeouts match your SLA, not Elasticsearch's defaults.

Typed results—use generics for document types

JavaScript is dynamically typed, but that doesn't mean your search results should be. The official client accepts TypeScript generics; use them. When your code expects product.price to be a number but gets back a string, you want that bug at compile time or caught by your linter, not at runtime when a customer gets charged the wrong amount.

Define your document types explicitly:

interface Product {
  id: string
  name: string
  price: number
  category: string
  inStock: boolean
  tags?: string[]
}

interface SearchResponse {
  hits: Array<{
    _id: string
    _score: number
    _source: Product
  }>
  total: { value: number }
}

Now use these types in your search:

async function searchProducts(query: string, limit: number = 10) {
  const client = getClient()
  
  const result = await client.search({
    index: 'products',
    size: limit,
    query: {
      multi_match: {
        query,
        fields: ['name^2', 'description', 'tags']
      }
    }
  })

  return result.hits.hits.map(hit => ({
    id: hit._id,
    score: hit._score,
    ...hit._source
  }))
}

The generic <Product> tells TypeScript that hit._source is a Product, so accessing hit._source.price is type-safe. If your mapping changes and a field becomes optional or changes type, your IDE will flag it immediately.

For aggregations, create separate interfaces that mirror your aggregation structure:

interface CategoryFacet {
  key: string
  doc_count: number
}

interface PriceBuckets {
  buckets: Array<{ key_as_string: string; doc_count: number }>
}

const aggs = await client.search({
  index: 'products',
  size: 0,
  aggs: {
    categories: { terms: { field: 'category' } },
    priceRanges: {
      date_histogram: { field: 'createdAt', calendar_interval: 'month' }
    }
  }
})

// Type-safe access
const categories = aggs.aggregations.categories as {
  buckets: CategoryFacet[]
}

This pattern scales. As your search logic grows—adding filters, scripting, complex aggregations—typed results let you refactor with confidence.

Bulk helper—client.helpers.bulk for large imports

When you need to index thousands of documents, the bulk API is non-negotiable. The official client provides a helper that abstracts away the formatting and error handling:

async function importProducts(products: Product[]) {
  const client = getClient()

  const result = await client.helpers.bulk({
    datasource: products,
    onDocument(doc) {
      return [
        { index: { _index: 'products', _id: doc.id } },
        doc
      ]
    },
    onDrop(doc) {
      console.warn(`Failed to index product ${doc.id}`)
    },
    flushBytes: 5000000, // 5MB batches
    concurrency: 4 // 4 concurrent bulk requests
  })

  console.log(`Indexed ${result.successful} products, failed: ${result.failed}`)
}

The helper does several things for you: it batches documents, respects Elasticsearch's backpressure, retries failed documents, and lets you handle failures without blocking the entire import. Set flushBytes based on your document size—too small and you'll spam Elasticsearch, too large and you'll risk timeouts. concurrency controls how many bulk requests run in parallel; start conservative (2–4) and increase if Elasticsearch is still not saturated.

Compare this to manual bulk formatting, which is error-prone and leaves error handling to you:

// Don't do this—error handling is fragile
const body = []
products.forEach(doc => {
  body.push({ index: { _index: 'products' } })
  body.push(doc)
})

const result = await client.bulk({ body })
// If one document fails, you don't know which one. Good luck.

The helper gives you per-document error callbacks and detailed success/failure counts. For large imports, pair it with a queue—index documents in batches scheduled during off-peak hours so you don't starve real-time queries.

Error handling—deal with 429 and backpressure

Elasticsearch returns 429 Too Many Requests when it's overloaded. The official client retries automatically, but only up to maxRetries. If your app is doing sustained bulk indexing or running expensive aggregations during peak traffic, you'll hit the limit.

Implement exponential backoff at the application level for critical operations:

async function searchWithRetry(
  index: string,
  query: any,
  maxAttempts: number = 3
) {
  const client = getClient()
  
  for (let attempt = 1; attempt <= maxAttempts; attempt++) {
    try {
      return await client.search({ index, query })
    } catch (err) {
      if (err.statusCode === 429 && attempt < maxAttempts) {
        const delay = Math.min(1000 * Math.pow(2, attempt - 1), 30000)
        console.warn(
          `Search throttled, retrying in ${delay}ms (attempt ${attempt}/${maxAttempts})`
        )
        await new Promise(resolve => setTimeout(resolve, delay))
      } else {
        throw err
      }
    }
  }
}

For bulk operations, the helper already handles retries, but you can make it smarter by scaling concurrency dynamically:

async function adaptiveBulkImport(products: Product[]) {
  const client = getClient()
  let concurrency = 4

  const result = await client.helpers.bulk({
    datasource: products,
    onDocument(doc) {
      return [{ index: { _index: 'products', _id: doc.id } }, doc]
    },
    concurrency,
    flushInterval: 5000 // Flush every 5 seconds if < flushBytes
  })

  if (result.failed > products.length * 0.01) {
    // More than 1% failed—cluster was struggling
    console.warn('High failure rate; reduce concurrency next time')
  }

  return result
}

Monitor response times and error rates. If 429 errors spike, reduce indexing concurrency. If search latency creeps up during bulk imports, you're competing for resources—schedule bulk work during off-peak hours or dedicate separate Elasticsearch nodes to heavy lifting.

Common mistakes

• Creating a new client per request: Drains connection pools and memory. Use a singleton.
• Ignoring document types: Leads to runtime type errors. Use TypeScript generics.
• Parsing bulk responses without checking failures: Missing documents silently. Always inspect result.failed.
• Not setting timeouts: Requests hang indefinitely. Set requestTimeout explicitly.
• Bulk importing during peak hours: Starves real-time queries. Import off-peak or use separate shards.
• Catching all errors the same way: Network errors are different from validation errors. Handle 429, 503, and connection errors separately.
• Not testing failover: Your app works fine until a node goes down. Test with a cluster that has failures.

Key takeaways

• Initialize the Elasticsearch client once at app startup and reuse it everywhere; connection pooling relies on a single instance.
• Use TypeScript generics to type your documents and responses; it catches data shape mismatches at compile time.
• Leverage client.helpers.bulk() for large imports; it handles batching, concurrency, and per-document error tracking.
• Handle 429 (throttling) and 503 (unavailable) separately from other errors; implement exponential backoff for critical paths.
• Monitor bulk import failure rates and adjust concurrency dynamically; a high failure rate signals cluster overload.
• Set explicit requestTimeout and maxRetries`; defaults don't match production SLAs.
• Test failover scenarios—single-node Elasticsearch clusters hide real-world resilience issues.