Crawlers must respect robots.txt: lessons from production

Crawlers must respect robots.txt—it's not just good etiquette, it's essential

If you're building a web crawler or scraper, ignoring robots.txt is like walking into someone's house without knocking. Over the past five years, I've seen production systems tank because they hammered domains without respecting rate limits or bot directives. This article walks through the practical reality of being a responsible crawler on the internet.

Why robots.txt Matters

The robots.txt file is a plain text file served at the root of a domain (e.g., example.com/robots.txt) that tells crawlers which areas they can and cannot access. It's part of the Robots Exclusion Protocol, a standard that's been around since 1994.

Most websites publish a robots.txt file to manage crawl traffic. Search engines respect it. Legitimate scrapers respect it. When you build a crawler, respecting it signals that your bot is trustworthy and not out to cause harm.

Here's what a basic robots.txt looks like:

User-agent: *
Disallow: /admin/
Disallow: /private/
Disallow: /api/
Crawl-delay: 2
Allow: /api/public/

This says: all bots should wait 2 seconds between requests and never crawl /admin/, /private/, or /api/ (except /api/public/).

Parse and Obey robots.txt Rules

Parsing robots.txt sounds simple, but the rules are surprisingly nuanced. The file uses pattern matching (not regex), and directives apply to specific user agents. You need to:

• Fetch the file from https://domain.com/robots.txt
• Match your bot's user agent to the appropriate rules
• Handle wildcard user agents (the * user agent matches all bots)
• Respect Disallow and Allow directives
• Parse the Sitemap directive if present

Let me show you a practical parser in Node.js using a library like robots-parser:

const fetch = require('node-fetch');
const { parse } = require('robots-parser');

async function checkRobotsRules(domain, path, userAgent) {
  try {
    const robotsTxtUrl = `https://${domain}/robots.txt`;
    const response = await fetch(robotsTxtUrl);
    const content = await response.text();
    
    const robots = parse(robotsTxtUrl, content);
    
    // Check if the path is allowed for this user agent
    const isAllowed = robots.isAllowed(path, userAgent);
    
    console.log(`${path} allowed: ${isAllowed}`);
    return isAllowed;
  } catch (error) {
    // If robots.txt doesn't exist, assume everything is allowed
    console.log('robots.txt not found, assuming allowed');
    return true;
  }
}

// Example usage
checkRobotsRules('example.com', '/products/shoes', 'MyBot/1.0');

A few critical details:

• If robots.txt doesn't exist (404), treat all paths as allowed
• If the fetch fails due to network error, back off and retry later
• Cache the robots.txt file for up to 24 hours to avoid constant re-fetching
• Match your user agent string exactly or fall back to the wildcard rule

I once deployed a crawler that didn't cache robots.txt, and it fetched the file on every single request. The site eventually rate-limited us, and we looked irresponsible. Now I store it in Redis with a TTL of 86,400 seconds.

Rate Limit Per Host

Respect the crawl-delay and request-rate directives. These tell you how fast you can crawl:

• Crawl-delay: 2 means wait 2 seconds between requests
• Request-rate: 10/1m means 10 requests per minute

Beyond robots.txt, common sense dictates rate limiting. A responsible crawler waits between requests and doesn't hammer a single host. Here's a practical rate limiter using a queue:

const pQueue = require('p-queue');

class HostRateLimiter {
  constructor(delayMs = 1000) {
    this.queues = new Map(); // one queue per host
    this.delayMs = delayMs;
  }

  async fetchUrl(url, userAgent) {
    const { hostname } = new URL(url);
    
    // Create queue for this host if it doesn't exist
    if (!this.queues.has(hostname)) {
      this.queues.set(hostname, new pQueue({
        concurrency: 1,
        interval: this.delayMs,
        intervalCap: 1
      }));
    }
    
    const queue = this.queues.get(hostname);
    
    return queue.add(async () => {
      console.log(`Fetching ${url} with delay ${this.delayMs}ms`);
      const response = await fetch(url, {
        headers: { 'User-Agent': userAgent }
      });
      return response.text();
    });
  }
}

// Usage
const limiter = new HostRateLimiter(2000); // 2 second delay per host
await limiter.fetchUrl('https://example.com/page1', 'MyBot/1.0');
await limiter.fetchUrl('https://example.com/page2', 'MyBot/1.0');
// The second fetch will wait 2 seconds after the first completes

The key insight: maintain separate rate limiters per host. You can crawl multiple domains concurrently, but each domain gets one request at a time (or whatever the rate allows).

Identify Your Bot With a Clear User Agent

Your user agent string tells the server who you are. Make it descriptive and honest. A vague or misleading user agent (like pretending to be Chrome) is disrespectful and can get you blocked.

Good user agent examples:

• MyDataCollector/1.0 (+http://mydomain.com/bot)
• PriceMonitor/2.1 (compatible; +https://monitoring-service.com/crawler)
• NewsAggregator/1.5 (+contact@newsagg.com)

Bad user agent examples:

• Mozilla/5.0 (Windows NT 10.0; Win64; x64) – looks like a browser, misleading
• Bot – too vague, no contact info
• MyBot – still no way to contact you or find documentation

When you set your user agent, include:

1. A descriptive name and version
2. A website or email where someone can learn about your bot
3. Make it easy for site owners to contact you if there's an issue

In Node.js, set the user agent on every request:

const userAgent = 'MyBot/1.0 (+https://mybot.example.com/crawler)';

const response = await fetch(url, {
  headers: {
    'User-Agent': userAgent,
    'Accept': 'text/html,application/xhtml+xml'
  }
});

// Or with axios
axios.get(url, {
  headers: { 'User-Agent': userAgent }
});

Back Off on 429 and 5xx Responses

HTTP response codes tell you when you're being too aggressive:

• 429 Too Many Requests – You're crawling too fast. Back off immediately.
• 503 Service Unavailable – The server is overloaded or down. Stop crawling temporarily.
• 500, 502, 504 – Server errors. Retry with exponential backoff.

A responsible crawler implements exponential backoff and respects rate-limit headers. Here's a practical example:

async function fetchWithBackoff(url, userAgent, maxRetries = 5) {
  let retries = 0;
  
  while (retries < maxRetries) {
    try {
      const response = await fetch(url, {
        headers: { 'User-Agent': userAgent }
      });
      
      // Handle rate limiting
      if (response.status === 429) {
        const retryAfter = response.headers.get('Retry-After') || 60;
        const delaySeconds = parseInt(retryAfter, 10);
        console.log(`Rate limited. Backing off for ${delaySeconds} seconds`);
        await sleep(delaySeconds * 1000);
        retries++;
        continue;
      }
      
      // Handle server errors
      if (response.status >= 500) {
        const backoffSeconds = Math.pow(2, retries); // exponential backoff
        console.log(`Server error (${response.status}). Retrying in ${backoffSeconds}s`);
        await sleep(backoffSeconds * 1000);
        retries++;
        continue;
      }
      
      // Success
      if (response.ok) {
        return response.text();
      }
      
      // Other errors (4xx) shouldn't retry
      console.log(`Client error (${response.status}). Not retrying`);
      return null;
      
    } catch (error) {
      console.error(`Fetch error: ${error.message}`);
      const backoffSeconds = Math.pow(2, retries);
      await sleep(backoffSeconds * 1000);
      retries++;
    }
  }
  
  throw new Error(`Failed after ${maxRetries} retries`);
}

function sleep(ms) {
  return new Promise(resolve => setTimeout(resolve, ms));
}

Key behaviors:

• Read the Retry-After header if present and respect it
• Use exponential backoff (1s, 2s, 4s, 8s, etc.) for retries
• Don't retry on 4xx errors (they indicate a permanent problem)
• Stop crawling a domain if you get repeated 5xx or 429 responses

Common Mistakes

Ignoring robots.txt entirely. Some developers assume robots.txt is just for search engines. It's not. Respecting it is the social contract of web crawling. Ignore it and expect to be blocked.

Not caching robots.txt. Fetching it on every request tanks performance and annoys site owners. Cache it for at least 24 hours.

Using a browser user agent. Pretending to be Chrome or Firefox might work short-term, but it's deceptive. Site owners can detect and block crawler traffic easily. Be honest about who you are.

Crawling everything at once. Launching 100 concurrent requests without rate limiting is a denial-of-service attack. Respect per-host concurrency limits (usually 1 or 2 concurrent requests per domain).

Not handling 429 and 5xx responses. If a server tells you to back off, listen. Retrying immediately is aggressive and ineffective.

Parsing robots.txt manually. The rules are more complex than they seem (e.g., pattern matching, user agent specificity). Use a library like robots-parser instead of rolling your own.

Key Takeaways

• Fetch and cache robots.txt for every domain you crawl. Respect the Disallow, Allow, and Crawl-delay rules.
• Implement per-host rate limiting. Use a queue to ensure only one request per host at a time (or according to robots.txt rules).
• Set a descriptive, honest user agent string with a link to your bot's documentation or contact info.
• Back off aggressively on 429 (Too Many Requests) and 5xx (server errors) responses. Use exponential backoff and respect Retry-After headers.
• Being a good citizen on the web is good business. Respectful crawlers don't get blocked, and they build trust with site owners.