X-Robots-Tag for non-HTML files: lessons from production

Search engines crawl far more than HTML pages—they index PDFs, images, XML feeds, and exported documents. But here's the problem: you can't embed meta tags in a PDF or tell an image file to block indexing from within itself. That's where X-Robots-Tag for non-HTML files saves you. Using HTTP headers, you control indexing behavior for any asset your server delivers, giving you the same fine-grained control over robots as you'd have with HTML meta tags.

Why X-Robots-Tag matters for non-HTML content

In my years building platforms—e-commerce sites with thousands of product PDFs, SaaS dashboards with exported reports, travel apps with generated itineraries—I've seen the same mistake repeatedly: teams assume search engines ignore non-HTML files. They don't. Google crawls and indexes PDFs natively. Bing indexes images. And if your private documentation, draft designs, or sensitive spreadsheets are publicly accessible, they can all end up in search results.

The X-Robots-Tag HTTP response header is the answer. It works exactly like a robots meta tag, but it lives in the HTTP headers instead of the HTML document. This means you can use it on PDFs, images, JSON feeds, CSV exports, or any file type your server delivers over HTTP.

From a technical SEO perspective, this is crucial. You're not just preventing indexation where it matters—you're also signaling intent to search engine crawlers. Properly configured X-Robots-Tag headers reduce crawl waste, improve your crawl budget allocation, and ensure sensitive assets stay private.

Set X-Robots-Tag: noindex on private PDFs

Let's start with the most common use case: you have PDFs that shouldn't appear in search results. These might be draft proposals, internal process documents, or temporary reports. Without X-Robots-Tag, Google will happily index them.

Here's a concrete scenario from a CRM platform I built: clients uploaded confidential sales forecasts as PDFs to a shared folder. The folder was technically public (needed for authorized users to access via direct link), but we didn't want these showing up when someone searched for the client's company name. Solution: add X-Robots-Tag headers to all PDFs in that directory.

The header syntax is simple:

X-Robots-Tag: noindex

This tells all robots to not index the asset. You can also be more specific:

X-Robots-Tag: noindex, nofollow

Or target specific search engines:

X-Robots-Tag: googlebot: noindex
X-Robots-Tag: bingbot: nofollow

The key advantage: this works for any file type. The browser downloads the file and the user can access it normally—but search crawlers see the header and respect it.

Configure in Nginx or your CDN

Where you implement X-Robots-Tag depends on your infrastructure. I'll walk through the most common approaches I've used in production.

Nginx configuration

In Nginx, you add headers in your server or location block. Here's how to add X-Robots-Tag to all PDFs in a specific directory:

location /private-documents/ {
    add_header X-Robots-Tag "noindex, nofollow";
    proxy_pass http://backend;
}

location ~ \.pdf$ {
    add_header X-Robots-Tag "noindex";
    proxy_pass http://backend;
}

I often use a hybrid approach. Public PDFs get normal indexing, but files in admin or draft directories get noindex automatically:

location /public/documents/ {
    # No X-Robots-Tag header—allow indexing
    proxy_pass http://backend;
}

location /drafts/ {
    add_header X-Robots-Tag "noindex, nofollow";
    proxy_pass http://backend;
}

location /exports/ {
    add_header X-Robots-Tag "noindex";
    proxy_pass http://backend;
}

This approach is clean because it's declarative—your infrastructure code reflects your indexing policy.

Application-level configuration

If you're using Node.js (NestJS, Express) or .NET, you can also set headers in your controller or middleware. Here's a NestJS example from a SaaS platform I worked on:

import { Controller, Get, Header, Param } from '@nestjs/common';

@Controller('exports')
export class ExportController {
  @Get(':id')
  @Header('X-Robots-Tag', 'noindex, nofollow')
  async downloadExport(@Param('id') exportId: string) {
    // Return PDF or CSV
    return this.exportService.getExport(exportId);
  }
}

This gives you per-endpoint control. You could make the header conditional too:

@Get(':id')
async downloadExport(@Param('id') exportId: string, @Res() res) {
  const file = await this.exportService.getExport(exportId);
  
  if (file.isPrivate) {
    res.header('X-Robots-Tag', 'noindex, nofollow');
  }
  
  return res.download(file.path);
}

CDN configuration

If you serve assets through a CDN like CloudFront or Cloudflare, you can configure headers there too. On Cloudflare, I use Page Rules or Transform Rules to add headers based on path patterns:

If URI path contains "/private"
  Add response header: X-Robots-Tag: noindex, nofollow

On AWS CloudFront, you'd add headers at the distribution level or use Lambda@Edge for dynamic logic. This is especially powerful if you have thousands of files—one rule handles them all.

Works for images, feeds, and exports

X-Robots-Tag isn't just for PDFs. It applies to any content type your server delivers. Let me give you practical examples from actual projects.

Images

On an e-commerce platform, we had placeholder images and draft product shots in the same S3 bucket. We configured Cloudflare to add noindex headers to URLs matching /drafts/*. This kept draft images out of Google Images while public product photos remained discoverable.

You can also use this for user-generated content you want to serve but not index—profile photos in private groups, for instance.

XML feeds and sitemaps

Sometimes you have internal feeds (analytics exports, admin dashboards in feed format) that shouldn't be indexed. A simple noindex header prevents them from becoming search results clutter:

location /admin/feeds/ {
    add_header X-Robots-Tag "noindex, nofollow";
    proxy_pass http://backend;
}

Your actual sitemaps should never have this header—but private feeds should.

Exports and generated files

On a trading platform I built, users could export transaction history as CSV. These files were temporary and user-specific. We cached them briefly but marked them noindex:

location /user-exports/ {
    add_header X-Robots-Tag "noindex";
    add_header Cache-Control "private, max-age=3600";
    proxy_pass http://backend;
}

This prevented Google from indexing thousands of identical export variations while still allowing users to download their own data.

Check with curl -I

You've configured your headers—now verify they're actually being sent. This is where I see developers guess or assume. Don't. Test it.

The simplest approach is curl with the -I flag, which fetches headers only:

curl -I https://yoursite.com/private-documents/report.pdf

Look for your header in the response:

HTTP/2 200
content-type: application/pdf
x-robots-tag: noindex, nofollow
content-length: 524288

If X-Robots-Tag isn't there, your configuration didn't work. Common issues I've debugged:

• Nginx location blocks aren't being matched (test with nginx -T to validate syntax and trace which block matches)
• Middleware is executing in the wrong order (in NestJS, ensure your header middleware runs before the response is sent)
• Your proxy is stripping headers (check if your reverse proxy or CDN is removing them)
• The file is being served from a different origin or subdomain (headers only apply to the response you're checking)

To debug, also check that your file is actually being served with a 200 status, not 304 (cached) or 403 (forbidden). A 304 response won't include headers, and curl won't show them.

For a more detailed test, use curl with verbose output:

curl -Iv https://yoursite.com/private-documents/report.pdf 2>&1 | grep -i x-robots

Or use online header checkers, but I trust curl for production verification. It's reliable and you control the request completely.

Common mistakes

Mistake 1: Only adding noindex, forgetting nofollow. If a PDF contains links, Google will still crawl them even if the PDF itself isn't indexed. Use noindex, nofollow together for private files.

Mistake 2: Setting X-Robots-Tag on HTML responses incorrectly. You can absolutely use X-Robots-Tag on HTML (instead of or alongside meta tags), but forgetting the syntax is common. Make sure it's an HTTP header, not an HTML attribute. The header is sent by the server; meta tags live in <head>.

Mistake 3: Inconsistency across CDN and origin. If your CDN strips or modifies headers, your origin server's X-Robots-Tag might never reach crawlers. Always test through your full stack—through your CDN or reverse proxy, not just your origin.

Mistake 4: Not testing with robots that actually matter. Google and Bing both respect X-Robots-Tag, but lesser-known crawlers might not. If you're concerned about other bots, add explicit headers for them: X-Robots-Tag: googlebot: noindex, bingbot: noindex.

Mistake 5: Using X-Robots-Tag as your only access control. This header prevents indexing, but it doesn't prevent access. If a file is sensitive, use authentication (401, 403) or robots.txt. X-Robots-Tag is SEO tooling, not security tooling.

Key takeaways

• X-Robots-Tag is an HTTP header that controls indexing for any file type—PDFs, images, exports—without requiring embedded meta tags.
• Configure at the infrastructure level (Nginx, CDN) for broad patterns or at the application level for fine-grained control per endpoint.
• Use noindex for files you want served but not indexed, and combine it with nofollow to prevent crawling links within private files.
• Always verify headers with curl -I before assuming they're working. Testing is non-negotiable.
• X-Robots-Tag reduces crawl waste and ensures your crawl budget focuses on content that deserves indexing, not drafts or temporary exports.
• This is complementary to robots.txt—use robots.txt for broad directives, X-Robots-Tag for surgical, per-asset control.