Crawl budget at scale: a practical guide for developers

Crawl budget at scale

When you're running a site with hundreds of thousands of URLs, Google doesn't crawl everything every day. The search engine allocates a crawl budget—a finite amount of resources it's willing to spend on your domain—and if you're not strategic, most of your URLs stay invisible. Crawl budget at scale is the difference between a thriving discovery engine and a graveyard of orphaned pages that never rank.

I learned this the hard way building the product catalog for a regional e-commerce platform. We had 400,000 SKUs, aggressive filtering options, and dynamic URLs for nearly every user combination. Our crawl budget was hemorrhaging on duplicate content and trap pages while our core products sat stale. Fixing this wasn't about doing SEO—it was about engineering smarter systems upstream.

Who needs to care

Not every site faces crawl budget constraints. If you're running a content blog with a few hundred pages, you're fine. But if you're building any of these, this is critical:

• Large e-commerce: Tens or hundreds of thousands of product pages, category combinations, and variants.
• Marketplaces: Listings, user profiles, transaction records—volume scales fast.
• Programmatic sites: Job boards, real estate portals, travel aggregators where content is generated dynamically from user input.
• SaaS with public portfolios: If you expose customer data or user-generated content at unique URLs, crawl budget matters.

The common thread: you generate URLs faster than you can manually manage them. At that scale, Google's crawl budget becomes a real bottleneck. It's not theoretical—it's the constraint between being indexed and being invisible.

Remove crawl traps

A crawl trap is any URL or URL pattern that generates infinite or near-infinite variations without adding real value. Google will keep crawling these endlessly, burning your crawl budget on garbage.

Calendars and date pickers are the classic trap. If your site has an event listing page and users can pick date ranges, you could theoretically generate millions of unique URLs:

example.com/events?date=2024-01-01
example.com/events?date=2024-01-02
example.com/events?date=2024-01-03
...infinitely forward and backward

Google's crawler will follow these links and keep going. The solution: handle date filtering server-side without changing the URL, or use URL parameters that you explicitly canonicalize to a single version. For a travel site I worked on, we used Next.js dynamic routes with canonical tags:

<link rel="canonical" href="https://example.com/events" />

All date-filtered views pointed to the single canonical URL. Crawlers saw the parameter, understood it wasn't a new page, and moved on.

Infinite filters are another trap. E-commerce sites often let users combine filters: size, color, price range, brand, stock status. Each combination is technically a different URL:

example.com/products?size=M&color=red&price=50-100&brand=Nike
example.com/products?size=M&color=red&price=50-100&brand=Adidas
example.com/products?size=L&color=blue&price=100-200&brand=Nike

Multiply these across your catalog and you have millions of permutations. The fix: limit which filter combinations you allow. Decide which filters drive real user intent (size and brand matter for apparel; maybe price matters; obscure combinations don't). Generate a finite set of combinations and exclude the rest via robots.txt or meta robots tags. On one marketplace rebuild, we reduced filterable URLs from 12 million to 180,000 just by disallowing low-conversion filter chains.

Session parameters are sneaky. Some older systems append session IDs to URLs:

example.com/products?sessionid=abc123xyz
example.com/products?sessionid=def456uvw

Every visitor session creates a new URL. Google treats it as a fresh page. The fix: move session data to cookies or server-side storage. If you must use URL parameters for tracking, use fixed values (not user-specific IDs) and declare them as URL parameters in Google Search Console so Google ignores them.

The rule: if a parameter doesn't genuinely change the content a user sees, it shouldn't create a new URL.

Speed up responses

Google allocates crawl budget based partly on server health. If your server is slow—takes 3 seconds to respond to a crawl request—Google assumes you're resource-constrained and reduces crawl rate. Faster servers get crawled more aggressively.

This is concrete: I profiled crawl patterns for a trading platform with 2M+ market data pages. Response times averaged 1.2 seconds. We optimized the database queries (added indexes, reduced N+1 problems), moved static data to Redis, and cut response time to 180ms. Three weeks later, Search Console showed crawl rate doubled. Same crawl budget, but faster responses meant Google could crawl more pages in the same time window.

Practical optimizations:

• Cache aggressively: Use Redis or Memcached for product data, categories, and taxonomies. On a ride-hailing search, we cached location hierarchies (cities → neighborhoods → zones) to reduce database round trips by 70%.
• Optimize database queries: Crawlers often hit your most expensive endpoints (listings pages, search results). Slow queries here mean Google crawls fewer pages. Use explain plans, add covering indexes, and consider materialized views for complex aggregations.
• Use CDNs for static assets: Crawlers need to parse your HTML. If CSS, JS, and images are slow to fetch, the whole page load suffers. Serve everything via CloudFront or similar.
• Monitor real crawl performance: Set up alerts for response times to Googlebot. Search Console shows crawl stats; track it alongside your server metrics.

Speed isn't a luxury here—it's math. If you halve response time, you double the pages Google can crawl in your budget window.

Signal priority

Even with unlimited budget, Google prioritizes what to crawl. You can influence that with smart signals.

Fresh sitemaps: A stale sitemap tells Google "I'm not actively managing this site." Update your sitemap regularly—weekly if you add or remove URLs frequently. For programmatic sites, generate it dynamically:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/products/123</loc>
    <lastmod>2024-01-15T10:30:00Z</lastmod>
    <priority>0.8</priority>
  </url>
</urlset>

Use the lastmod tag accurately. Google uses it to decide how often to recrawl. If a product was updated yesterday, mark it as fresh. If it hasn't changed in six months, Google knows it can crawl it less frequently and allocate budget to newer content.

Strong internal links: Pages with more internal links get crawled more often. This seems obvious, but most teams don't leverage it. On a marketplace, we added contextual links from category pages to high-value listings. We updated our navigation to highlight seasonal or trending products. Result: those pages got crawled 40% more frequently. If you have URLs you want indexed urgently, build more links to them—from navigation, category pages, or related content.

Consolidated duplicates: Duplicates waste crawl budget. Every duplicate URL is a wasted crawl. If you have multiple URLs pointing to the same content (different parameters, mobile vs desktop versions, HTTP vs HTTPS), consolidate them.

Use canonical tags:

<link rel="canonical" href="https://example.com/products/shoe-model" />

Or 301 redirects for older patterns. On a travel site rebuild, we found 60,000 URLs indexed from old filter combinations that no longer worked. We set up 301s to canonical versions. Crawl budget freed up immediately, and the site crawled more fresh content within weeks.

Common mistakes

Ignoring parameter handling: Teams often ignore URL parameters, thinking they don't matter. Google treats ?color=red&size=M and ?size=M&color=red as different URLs. Standardize parameter order, or declare them as insignificant in Search Console.

Over-prioritizing everything: If you set every URL to priority 1.0 in your sitemap, you signal nothing. Prioritization only works if it's selective. High priority should mean "crawl this urgently"—maybe 10-20% of your URLs.

Neglecting response time under load: You optimize response times in dev and staging. Then production traffic spikes, servers slow down, and crawl budget drops. Test under realistic load and set up alerts for response time degradation.

Letting duplicate content accumulate: Teams often fix one duplicate, then let others build up. Establish a process. On every product launch, ask: is this URL unique, or does it duplicate existing content? If it's not unique, should it exist at all?

Key takeaways

• Crawl budget constraints are real at scale—hundreds of thousands of URLs demand strategic engineering, not just SEO tactics.
• Remove crawl traps: calendars, infinite filters, and session parameters waste budget on non-content. Consolidate them or exclude them.
• Faster responses directly increase crawl efficiency. Optimize queries, cache aggressively, and monitor response times to Googlebot specifically.
• Signal priority through fresh sitemaps with accurate lastmod dates, strategic internal links to important content, and canonical tags to eliminate duplicates.
• Crawl budget is finite—treat it like you'd treat memory or API rate limits in code. Measure it, optimize for it, and revisit regularly as your site grows.