Verify real Googlebot

Anyone can fake the Googlebot user agent by spoofing the User-Agent header in an HTTP request, which means your application can't simply trust what the request claims to be. If you're building a platform that needs to serve different content, apply rate limits, or trigger special handling for search engine crawlers, you need a reliable way to confirm that traffic is actually coming from Google. Over my 5+ years building production platforms, I've seen teams get burned by this—either letting fake crawlers waste resources or accidentally blocking real Google traffic through overly aggressive verification.

The good news is that Google publishes guidance on how to verify crawler authenticity, and it's straightforward once you understand the DNS lookup process. Let me walk you through the exact steps I use in production systems.

Reverse DNS lookup on the IP

The foundation of Googlebot verification is reverse DNS lookup. When a request arrives at your server, you capture the client IP address and perform a reverse lookup to see what hostname resolves to that IP. This is the first filter—if the reverse lookup fails or returns a generic hostname with no Google affiliation, you can safely assume it's not Googlebot.

Here's what a real reverse lookup looks like from the command line:

html

host 66.249.66.1
# Output: 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com

In that example, the IP 66.249.66.1 reverse-resolves to crawl-66-249-66-1.googlebot.com. The hostname itself tells you this is a Google crawler. But here's the critical part: an attacker can fake reverse DNS if they control the DNS server for their domain. That's why we need the next step.

In Node.js or Python, you'd implement this like:

html

const dns = require('dns').promises;

async function reverseResolve(ipAddress) {
  try {
    const hostname = await dns.reverse(ipAddress);
    return hostname[0];
  } catch (error) {
    return null;
  }
}

Domain must end in googlebot.com or google.com

After getting the reverse DNS hostname, check that it ends with .googlebot.com or .google.com. This is a quick gate—if the domain isn't from Google's own namespace, it's not Googlebot, full stop.

I typically see hostnames like:

crawl-66-249-66-1.googlebot.com
msnbot-207-46-13-140.bing.com (not Google)
baiduspider-180-76-5-144.crawl.baidu.com (not Google)

The reason we restrict to .googlebot.com or .google.com is simple: these are Google's official reverse DNS zones. A bad actor can't register a domain under .googlebot.com because they don't own that TLD. Here's a practical check:

html

function isGoogleDomain(hostname) {
  if (!hostname) return false;
  return hostname.endsWith('.googlebot.com') || 
         hostname.endsWith('.google.com');
}

Combine this with your reverse lookup to filter out non-Google traffic immediately. Many platforms stop here and call it done, but there's one more crucial validation step.

Forward lookup must match the IP

This is where I catch most fake crawlers. After confirming the reverse DNS hostname looks legitimate (ends in .googlebot.com), perform a forward DNS lookup on that hostname and verify the result matches the original client IP. If an attacker spoofed their reverse DNS, the forward lookup will fail or point to a different IP address.

Let's test with our example IP again:

html

host crawl-66-249-66-1.googlebot.com
# Output: crawl-66-249-66-1.googlebot.com has address 66.249.66.1

The hostname forward-resolves back to the same IP. That's the match we want. If forward lookup returned a different IP or failed, we'd reject the request.

Here's how to implement bidirectional verification in Node.js:

html

const dns = require('dns').promises;

async function verifyGooglebot(clientIp) {
  // Step 1: Reverse DNS
  let hostname;
  try {
    const hostnames = await dns.reverse(clientIp);
    hostname = hostnames[0];
  } catch (error) {
    console.log('Reverse DNS failed for', clientIp);
    return false;
  }

  // Step 2: Check domain
  if (!hostname.endsWith('.googlebot.com') && 
      !hostname.endsWith('.google.com')) {
    console.log('Hostname not from Google:', hostname);
    return false;
  }

  // Step 3: Forward DNS verification
  try {
    const addresses = await dns.resolve4(hostname);
    if (!addresses.includes(clientIp)) {
      console.log('Forward DNS mismatch:', addresses, clientIp);
      return false;
    }
  } catch (error) {
    console.log('Forward DNS failed for', hostname);
    return false;
  }

  return true;
}

// Usage in middleware
app.use(async (req, res, next) => {
  const clientIp = req.ip;
  const isRealGooglebot = await verifyGooglebot(clientIp);
  req.isGooglebot = isRealGooglebot;
  next();
});

This three-step process is the standard that Google itself recommends. By doing both reverse and forward lookups, you confirm that the IP actually corresponds to a Google-owned hostname, and that the hostname correctly resolves back to that IP.

Google publishes crawler IP ranges too

As a supplementary layer, Google publishes official IP ranges for its crawlers. You can cross-reference the client IP against these ranges to add an extra signal of legitimacy. This is optional but useful if you want to be extra cautious or if you're implementing rate-limiting logic.

Google maintains this list in their public documentation and you can also check via their Search Console API or published CIDR blocks. Some platforms subscribe to updated IP lists from Google Cloud or use Google's published ranges.

In practice, I'd combine DNS verification with an optional IP range check:

html

const GOOGLE_CRAWLER_RANGES = [
  '66.249.64.0/19',
  '203.208.39.0/24',
  '204.11.215.0/24',
  // Check Google's official list for complete ranges
];

function isInGoogleRange(ip) {
  // Use a CIDR library to check if IP falls within ranges
  const ipaddr = require('ipaddr.js');
  const addr = ipaddr.process(ip);
  
  return GOOGLE_CRAWLER_RANGES.some(range => {
    try {
      return addr.match(ipaddr.process(range));
    } catch {
      return false;
    }
  });
}

async function verifyGooglebotFull(clientIp) {
  // DNS verification (required)
  const isDnsVerified = await verifyGooglebot(clientIp);
  
  // IP range check (optional but useful)
  const isInRange = isInGoogleRange(clientIp);
  
  // Pass if DNS check passes; IP range is supplementary
  return isDnsVerified && isInRange;
}

The DNS verification is the primary check; IP ranges are a nice-to-have signal. Don't rely on IP ranges alone because they can change and aren't always kept perfectly current. Always lead with DNS.

Common mistakes

I've seen teams make several errors when implementing Googlebot verification:

Only checking the User-Agent header. This is the most common mistake. User-Agent strings are trivial to spoof. You need DNS verification for real confidence.
Skipping the forward lookup. Stopping after reverse DNS and domain check leaves you vulnerable to spoofed reverse DNS. Always verify both directions.
Doing synchronous DNS lookups on every request. DNS calls have latency. Cache the results or do verification asynchronously to avoid blocking your request handler.
Trusting old IP ranges. Google's crawler IP blocks change occasionally. If you're using a hardcoded list, update it periodically or just rely on DNS verification, which is always current.
Not handling DNS timeout gracefully. If a reverse lookup times out, don't assume the request is legitimate. Treat lookup failures as failed verification and reject the request unless you have a good reason to allow it.

Key takeaways

Never trust the User-Agent header alone; fake crawlers spoof it trivially.
Perform reverse DNS lookup on the client IP to get the hostname.
Verify the hostname ends with .googlebot.com or .google.com.
Do a forward DNS lookup on that hostname and confirm it resolves back to the same IP.
Use IP range checks as a supplementary signal, not a primary verification method.
Cache DNS results and handle timeouts gracefully to avoid latency and reliability issues.
Consider logging failed verification attempts to monitor for malicious crawler spoofing.

Verify real Googlebot: a practical guide for developers