XML sitemaps at scale: a practical guide for developers

Most teams discover their sitemap problem too late: your crawl budget vanishes on redirects and soft 404s, Search Console shows half your URLs as "Discovered but not indexed," and you're manually rebuilding XML files every deploy. When you scale to thousands of products, blog posts, or user-generated content, XML sitemaps stop being a nice-to-have and become infrastructure. XML sitemaps at scale require a shift from static files to automated, database-driven generation—and strict discipline about what belongs inside them.

Only canonical, indexable URLs

Every URL in your sitemap must satisfy three conditions: it returns HTTP 200, has no noindex meta tag, and isn't a redirect. The moment you add a 301, a soft 404, or a noindex page, you're wasting Google's crawl budget and confusing search engines about your site's actual structure.

Let's say you run an e-commerce platform with product variations. A URL like https://example.com/products/blue-shirt-m might redirect to https://example.com/products/blue-shirt?variant=m if you refactored your URL structure. Never include the old URL in your sitemap. Instead, update your data source—your database or CMS—to point only to the canonical destination.

For a marketplace or content platform, treat soft 404s with zero tolerance. A soft 404 is a page that returns 200 but signals "this resource doesn't exist"—often through empty state messaging or a 404 in the page title. Search Console flags these, but the real damage is harder to measure: Google wastes crawl budget on pages that add no value. Before you generate a URL for the sitemap, verify it has substantive content.

Here's a practical filter for generation:

-- Pseudo-query: only live, published content with no noindex flag
SELECT url, lastmod, priority 
FROM pages 
WHERE status = 'published' 
  AND published_at IS NOT NULL 
  AND no_index_flag = false 
  AND deleted_at IS NULL 
ORDER BY lastmod DESC;

This query ensures every URL in your sitemap is worth crawling. If a product is marked as deleted, archived, or draft, it stays out. If it's marked noindex (perhaps for internal use or low-quality variations), it stays out.

Split large sitemaps

XML sitemaps have a hard limit: 50,000 URLs per file, 50 MB uncompressed. If you run an e-commerce store with 200,000 SKUs or a content platform with millions of posts, a single monolithic sitemap will choke your server during generation and confuse crawlers about freshness.

The solution is a sitemap index file—a wrapper that points to multiple sitemaps, each under the limit. Google treats the index as a manifest; it crawls each sitemap file independently and prioritizes them separately.

A typical structure might look like:

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://expona.dev/sitemap-posts-1.xml</loc>
    <lastmod>2024-01-15T10:00:00Z</lastmod>
  </sitemap>
  <sitemap>
    <loc>https://expona.dev/sitemap-posts-2.xml</loc>
    <lastmod>2024-01-15T10:00:00Z</lastmod>
  </sitemap>
  <sitemap>
    <loc>https://expona.dev/sitemap-products.xml</loc>
    <lastmod>2024-01-14T08:30:00Z</lastmod>
  </sitemap>
</sitemapindex>

Notice each sitemap entry includes <lastmod>. Google uses this to decide which sitemaps to crawl first; if a product sitemap hasn't changed since yesterday, Google may skip it this round. This is crucial for load management.

The grouping strategy depends on your content model. For blogs, split by date range or category. For e-commerce, split by product category, then by pagination within each category. For user-generated content (forum posts, reviews, etc.), split by publish date or by user ID ranges. The goal is stability: each sitemap should contain URLs that share similar freshness patterns.

Here's a pattern for generating multiple product sitemaps in NestJS:

async generateSitemaps() {
  const pageSize = 50000;
  const totalProducts = await this.db.count('products', { status: 'published' });
  const numSitemaps = Math.ceil(totalProducts / pageSize);

  for (let page = 0; page < numSitemaps; page++) {
    const offset = page * pageSize;
    const products = await this.db.query(
      `SELECT id, slug, updated_at FROM products 
       WHERE status = 'published' ORDER BY id 
       LIMIT ? OFFSET ?`,
      [pageSize, offset]
    );
    const xml = this.buildSitemapXml(products);
    await this.storage.write(
      `sitemap-products-${page + 1}.xml`,
      xml
    );
  }
  
  // Generate index file
  const index = this.buildSitemapIndex(numSitemaps);
  await this.storage.write('sitemap-index.xml', index);
}

This approach scales linearly. Even with a million URLs, you're generating 20 sitemap files—fast, memory-efficient, and easy to debug.

Generate automatically

Static sitemaps are a trap. You commit a sitemap to git, deploy it, and within hours it's stale. New products appear, old ones are archived, URLs change. Teams then either fall back to manual regeneration (error-prone, infrequent) or deploy broken sitemaps (hurts SEO and wastes crawl budget).

The right approach: generate sitemaps from your source of truth—your database—on every deploy or on a scheduled trigger. This ensures the sitemap always reflects what's actually live.

Two strategies work well:

1. On-deploy generation: When you ship code, your deploy pipeline queries the database and generates fresh sitemaps. This is simple for stable content (blogs, documentation) but can be slow for massive catalogs.
2. Scheduled generation: A cron job (every 6–24 hours) or a hook triggered by database changes regenerates sitemaps. This is ideal for high-volume platforms where content changes frequently.

For a SaaS platform with moderate content volume, on-deploy generation works:

// deploy.js - runs after code deployment
const db = require('./db');
const storage = require('./storage');

async function regenerateSitemaps() {
  const posts = await db.query(
    `SELECT id, slug, updated_at FROM posts 
     WHERE published = true AND deleted_at IS NULL 
     ORDER BY updated_at DESC`
  );

  const sitemapXml = `<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
${posts.map(post => `  <url>
    <loc>https://example.com/posts/${post.slug}</loc>
    <lastmod>${post.updated_at.toISOString()}</lastmod>
    <priority>0.8</priority>
  </url>`).join('\n')}
</urlset>`;

  await storage.write('sitemap.xml', sitemapXml);
  console.log(`✓ Generated sitemap with ${posts.length} URLs`);
}

regenerateSitemaps().catch(err => {
  console.error('Sitemap generation failed:', err);
  process.exit(1);
});

For high-volume platforms (marketplace with 50k+ SKUs, UGC platform with millions of pieces of content), scheduled generation is safer:

// sitemapWorker.js - runs via cron or message queue
const schedule = require('node-schedule');

schedule.scheduleJob('0 2 * * *', async () => {
  console.log('Starting sitemap generation...');
  await generateSitemaps();
  console.log('✓ Sitemaps ready. Uploading to CDN...');
  await uploadToS3('sitemaps/');
  console.log('✓ Deployment complete');
});

In both cases, you avoid human error and keep your sitemaps in sync with reality. Search engines see a reliable, fresh source of truth.

Submit and monitor

Once your sitemaps are live, submit them to Google Search Console and Bing Webmaster Tools. More importantly, monitor them. Search Console shows you for each sitemap:

• Discovered URLs: How many URLs Google found in your sitemap.
• Indexed URLs: How many Google actually indexed.
• Excluded URLs: How many Google found but didn't index, and why (typically due to noindex, redirect, or crawl errors).

A healthy sitemap shows most discovered URLs in the indexed column. A gap—large numbers of excluded URLs—signals a problem: pages returning 404, missing title/meta tags, or noindex flags you didn't intend.

A practical monitoring workflow:

1. Generate sitemaps automatically (deploy or cron).
2. Submit the index URL to Search Console: https://example.com/sitemap-index.xml.
3. Every Monday morning, review Search Console's sitemap report. Look for sudden drops in indexed count or spikes in excluded URLs.
4. If you see anomalies, check your database for accidental noindex flags, published_at changes, or deleted content that's still in the database.
5. Fix the data, redeploy, and Search Console will pick up the change within hours (crawl budget permitting).

Use an alert or dashboard to flag when indexed / discovered ratio drops below 80%. That's your canary: something broke.

Common mistakes

• Including redirects or soft 404s: Teams assume "if Google sees the URL, it will follow the redirect." Wrong. Redirects waste crawl budget and confuse canonicalization. Only include final destination URLs.
• Putting all URLs in one file: Even if you stay under 50k, generating a 40 MB sitemap file blocks your deploy pipeline and stalls crawlers. Split early.
• Static sitemaps committing to version control: A sitemap in git becomes a lie the moment you ship code. Generate from the database every time.
• Ignoring the lastmod field: If you always set lastmod to "now," Google treats every URL as recently updated and crawls aggressively, wasting budget. Only update lastmod when content genuinely changed.
• Never checking Search Console: You generate a perfect sitemap, submit it, and assume it works. Then half your URLs sit in "discovered but not indexed" for weeks, and you never notice. Check at least monthly.
• Over-prioritizing with priority tags: Marking everything priority 0.9 means nothing. Use it only for true high-value pages (homepage, core products, top blog posts).

Key takeaways

• Every URL in your sitemap must return 200, have no noindex tag, and not be a redirect. Filter your data source rigorously before generation.
• Split sitemaps at 50,000 URLs per file. Use a sitemap index to group related content (posts, products, etc.) for independent prioritization and freshness signals.
• Generate sitemaps automatically from your database on deploy or via cron. Static files become stale and break SEO. Automation is non-negotiable at scale.
• Submit your sitemap index to Search Console and monitor the discovered vs. indexed ratio weekly. A ratio below 80% is a warning sign to audit your content model.
• Use lastmod honestly. It tells Google which URLs changed; false signals waste crawl budget.
• Start building sitemap generation into your deploy pipeline now, before you scale. It's easier to automate early than retrofit it later.