Building an SEO audit crawler: best practices explained

Building an SEO audit crawler: best practices explained

When you're tasked with analyzing thousands of pages across a live production site, a poorly designed crawler can become your worst enemy. We've learned this the hard way while building SEOlenz, our internal SEO audit tool. Building an SEO audit crawler: best practices explained means understanding that crawling isn't just about fetching URLs—it's about doing so responsibly, efficiently, and in a way that extracts the insights you actually need without breaking the site or your infrastructure.

A production-grade audit crawler sits at the intersection of respect, efficiency, and technical rigor. Over the past 72 days of this technical SEO series, we've covered on-page factors, structured data, and rendering issues. Now we're pulling it all together into a system that can systematically assess an entire domain. Let's walk through the four pillars that separate a tool you're proud of from one that gets blacklisted.

Respect the site — Obey robots.txt, throttle requests and identify your crawler

The first rule of crawling is simple: don't be the crawler that takes down a site. This isn't just etiquette—it's practical engineering. A misconfigured crawler sending 100 requests per second to a small WordPress site can exhaust database connections and trigger DDoS protections.

Start by parsing and respecting robots.txt. Most sites allow audit crawlers explicitly in this file, but you need to check. If a path is disallowed and you're not running a search engine, skip it.

User-agent: *
Disallow: /admin/
Disallow: /temp/
Allow: /public/

User-agent: SEOlenz
Allow: /

Parse this correctly. A crawler should check whether its user agent (the one you set in your HTTP headers) matches any specific user-agent rules before falling back to the wildcard rules. Many sites explicitly permit audit crawlers by name, so honor that.

Next, implement throttling. We typically run at 1–2 requests per second per domain, distributed across multiple threads or workers. If you're crawling a large site, space requests out so you don't hammer a single origin. Use request headers to identify yourself clearly:

User-Agent: SEOlenz/1.0 (+https://seo-audit-example.com/bot)
Accept-Language: en-US,en;q=0.9
Accept-Encoding: gzip, deflate

This transparency helps site owners understand what's hitting their logs. When we built SEOlenz, we included a link to our bot documentation so administrators could whitelist us if needed. We also added a configurable delay parameter so users could tell our crawler to go slower on sensitive infrastructure.

Finally, implement a backoff strategy. If you receive 429 (Too Many Requests) or 503 (Service Unavailable) responses, back off exponentially. Log these events. They're often your first signal that you're being too aggressive.

Queue everything — A URL frontier with deduplication and depth limits

A naive crawler fetches URLs in the order it discovers them. A production crawler manages a URL frontier: a queue that respects deduplication, enforces depth limits, and prioritizes strategically.

Start with deduplication. URLs like /page, /page?, /page?utm_source=social, and /page?utm_source=email are often the same page. If you don't deduplicate, you'll crawl the same content repeatedly and never reach pages deeper in your site structure. Normalize URLs before queuing:

function normalizeUrl(url) {
  const parsed = new URL(url);
  
  // Remove query params that don't affect content
  const ignoredParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_content', 'utm_term', 'fbclid', 'gclid'];
  const params = new URLSearchParams(parsed.search);
  
  ignoredParams.forEach(param => params.delete(param));
  
  // Reconstruct and remove trailing slash for consistency
  parsed.search = params.toString();
  let normalized = parsed.toString();
  
  if (normalized.endsWith('/') && parsed.pathname !== '/') {
    normalized = normalized.slice(0, -1);
  }
  
  return normalized;
}

Use a Set or database to track visited URLs. In smaller crawls (under 100k pages), an in-memory Set works fine. For larger sites, consider a database like Redis or PostgreSQL with a unique index. When you encounter a URL, check if you've seen it before. If yes, skip it. If no, add it to both your frontier queue and your visited set.

Implement depth limits. You don't want to follow every link infinitely. Typically, we set a maximum crawl depth of 3–4 levels from the root. This keeps crawls bounded and prevents exploring irrelevant subsections or user-generated content that links outward.

const MAX_DEPTH = 3;

async function crawl(startUrl, maxDepth = MAX_DEPTH) {
  const queue = [{ url: startUrl, depth: 0 }];
  const visited = new Set();
  
  while (queue.length > 0) {
    const { url, depth } = queue.shift();
    
    if (visited.has(url)) continue;
    visited.add(url);
    
    // Fetch and process...
    const html = await fetchPage(url);
    const links = extractLinks(html, url);
    
    if (depth < maxDepth) {
      links.forEach(link => {
        const normalized = normalizeUrl(link);
        if (!visited.has(normalized)) {
          queue.push({ url: normalized, depth: depth + 1 });
        }
      });
    }
  }
}

This approach ensures you explore systematically without runaway crawls. We've seen crawlers stuck in infinite loops because they followed pagination links without depth limits, crawling the same products in different sort orders indefinitely.

Render selectively — Headless Chrome is expensive. Render only when raw HTML is not enough.

Headless browser rendering is powerful but costly. Starting a headless Chrome instance takes 500ms to 2 seconds. Rendering a page adds another 2–5 seconds. If you render every page, a 10,000-page site crawl becomes a multi-hour operation that consumes enormous memory and CPU.

Be selective. Fetch raw HTML first via a simple HTTP GET. Analyze it for SEO signals: meta tags, headers, structured data, internal links. This is fast and covers 95% of auditing needs.

Render only when you detect signals that the page is client-rendered:

• You find a <div id="app"></div> or similar placeholder with minimal content
• Meta tags are missing or generic ("React App")
• The page declares a next.js, nuxt, or other framework in the HTML
• You detect JavaScript bundles but no actual content

Build a heuristic to decide:

function shouldRender(html, url) {
  // Check for common SPA patterns
  if (html.includes('<div id="__next"></div>') || 
      html.includes('<div id="app"></div>') ||
      html.includes('<div id="root"></div>')) {
    return true;
  }
  
  // Check if page has meaningful content in raw HTML
  const contentLength = html.replace(/<[^>]*>/g, '').length;
  if (contentLength < 500) {
    return true;
  }
  
  // Check for meta tags
  const hasMetaDescription = html.includes('meta name="description"');
  const hasTitle = html.includes('<title>') && html.includes('</title>');
  if (!hasMetaDescription || !hasTitle) {
    return true;
  }
  
  return false;
}

When you do render, use a pool of reusable browser instances rather than spawning a new one per page. Libraries like puppeteer with connection pooling let you efficiently parallelize rendering across a limited set of instances.

Cache rendering results. If you re-audit a site later, store the rendered HTML so you don't re-render unchanged pages. We store the rendering timestamp and check for page modification via ETags or Last-Modified headers before deciding to re-render.

Store raw responses — Status, headers and HTML snapshots make re-analysis cheap

One of the biggest mistakes we made early was discarding the raw HTTP response after extraction. We'd pull out the title, meta description, and links, then throw away the HTML. Weeks later, someone would ask us to check structured data or Core Web Vitals metadata, and we'd have to re-crawl the entire site.

Instead, store everything. For each crawled URL, save:

• HTTP status code and response headers
• The raw HTML body (gzipped for efficiency)
• Crawl timestamp
• Whether the page was rendered via headless browser

A simple schema might look like:

This approach transforms your crawler into a content archive. Later analysis tools can extract whatever signals they need without re-crawling. We've used this to retroactively check for certain markup patterns, validate XML-RPC changes, and audit structured data evolution over time—all without touching the live site again.

Common mistakes

Ignoring robots.txt for your own domain: We once had a new team member point out that our crawler was hitting /admin/ pages because we assumed "it's our domain, robots.txt doesn't apply." Wrong. Respect it anyway. It keeps crawls focused and predictable.

Not handling redirects properly: A 301 redirect from /old-page to /new-page should follow the redirect and audit the final destination, not the redirect source. But don't follow redirect chains infinitely. Set a maximum of 5 redirects per URL.

Assuming all 200 responses are valid content: A 200 status code with a 404 message in the body exists more often than you'd think. Check both HTTP status and page title/content to detect soft 404s.

Rendering everything because it's "safer": Rendering every page feels safer but wastes resources. The cost rarely justifies the marginal gain. Selective rendering based on content detection is the right balance.

Forgetting to handle timeouts and errors gracefully: Network issues, timeouts, and crashes happen. Log them, retry with backoff, and continue the crawl. A single bad URL shouldn't halt the entire audit.

Key takeaways

• Respect robots.txt, throttle requests (1–2 per second), and identify your crawler clearly in the User-Agent header. This keeps you off blocklists and maintains the trust of site owners.
• Implement a URL frontier with deduplication and depth limits. Normalize URLs, track visited pages, and cap crawl depth at 3–4 levels to ensure systematic, bounded exploration.
• Fetch raw HTML first; render selectively only when heuristics detect client-side rendering. This keeps crawls fast and cost-effective while still handling modern JavaScript frameworks.
• Store the full HTTP response—status, headers, and gzipped HTML—so you can re-analyze the same crawl without hitting the live site again. This transforms your crawler into a reusable content archive.
• Plan for errors. Implement backoff strategies, handle redirects correctly (with limits), and gracefully continue past failed URLs so individual issues don't break entire crawls.