Crawl the old site before migrating: a step-by-step guide

Crawl the old site before migrating: Your first line of defense

I've seen too many site migrations go sideways because teams skipped the baseline. You launch the new site, traffic drops, and suddenly you're trying to figure out which 200 pages you forgot to redirect. Crawl the old site before migrating, document everything, then compare post-launch. It's boring work, but it saves weeks of firefighting.

This is day 19 of the 100-day Technical SEO for Developers series, and we're covering something non-negotiable: creating a complete inventory of your current domain before you touch anything. Think of it as an audit checkpoint. You can't protect what you haven't inventoried.

Full crawl with titles, canonicals and status codes

Start with a dedicated crawl tool. I use Screaming Frog for smaller sites (under 50k URLs) and custom Python scripts for larger ones. The goal is simple: capture every URL your site serves, along with its HTTP status code, title tag, canonical tag, and metadata.

Here's a basic Python crawler using requests and BeautifulSoup that gets you started:

import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin, urlparse
import csv
from datetime import datetime

class SiteCrawler:
    def __init__(self, base_url, output_file):
        self.base_url = base_url
        self.visited = set()
        self.results = []
        self.output_file = output_file
        self.domain = urlparse(base_url).netloc
    
    def crawl(self, url, depth=0, max_depth=10):
        if depth > max_depth or url in self.visited:
            return
        
        if urlparse(url).netloc != self.domain:
            return
        
        self.visited.add(url)
        
        try:
            response = requests.get(url, timeout=10, allow_redirects=True)
            soup = BeautifulSoup(response.content, 'html.parser')
            
            title = soup.find('title')
            title_text = title.string if title else ''
            
            canonical = soup.find('link', {'rel': 'canonical'})
            canonical_href = canonical.get('href', '') if canonical else ''
            
            meta_desc = soup.find('meta', {'name': 'description'})
            meta_desc_text = meta_desc.get('content', '') if meta_desc else ''
            
            self.results.append({
                'url': response.url,
                'status_code': response.status_code,
                'title': title_text,
                'canonical': canonical_href,
                'meta_description': meta_desc_text,
                'content_length': len(response.content),
                'crawl_time': datetime.now().isoformat()
            })
            
            for link in soup.find_all('a', href=True):
                next_url = urljoin(url, link['href'])
                self.crawl(next_url, depth + 1, max_depth)
        
        except Exception as e:
            self.results.append({
                'url': url,
                'status_code': 'ERROR',
                'title': str(e),
                'canonical': '',
                'meta_description': '',
                'content_length': 0,
                'crawl_time': datetime.now().isoformat()
            })
    
    def export_csv(self):
        with open(self.output_file, 'w', newline='', encoding='utf-8') as f:
            writer = csv.DictWriter(f, fieldnames=['url', 'status_code', 'title', 'canonical', 'meta_description', 'content_length', 'crawl_time'])
            writer.writeheader()
            writer.writerows(self.results)
        print(f"Crawl complete. {len(self.results)} URLs exported to {self.output_file}")

crawler = SiteCrawler('https://yoursite.com', 'crawl_report.csv')
crawler.crawl('https://yoursite.com')
crawler.export_csv()

What you're capturing here matters. The status code tells you which pages actually exist (200), which redirect (301/302), which are broken (404), and which are forbidden (403). The title and canonical are your SEO anchors—if a page has a self-referencing canonical pointing to itself, that's good. If it points somewhere else, you need to understand why before migration.

For a real-world example: I migrated an e-commerce platform from /products/item-123 to /shop/products/item-123. The old crawl showed 15 product pages had incorrect self-referencing canonicals pointing to the category page instead. We fixed those before launch. Post-launch, zero duplicate content issues. If we'd skipped the crawl, we'd have wasted a week debugging.

Export top pages by traffic and backlinks

Raw crawl data is a foundation, but you need context. Pull your top pages by organic traffic from Google Analytics and top pages by backlink count from a tool like Ahrefs or SEMrush. These are your VIPs during migration.

Create a merged spreadsheet with three columns: URL, monthly organic traffic, backlink count. Sort by traffic. The pages at the top—especially anything getting 100+ monthly visits—need special attention. They need proper 301 redirects, updated internal links, and verification post-launch.

Here's how I structure this in a Google Sheet or local CSV:

This matrix becomes your migration checklist. You know exactly which URLs matter. When your new site launches, you verify each one manually or with an automated checker.

Save structured data and internal links

Structured data—JSON-LD, microdata, RDFa—lives in your HTML. If your old site has schema markup for products, articles, FAQs, or breadcrumbs, extract it before migration. You want to replicate this on the new site or improve it.

Write a quick script to pull all JSON-LD blocks:

import json
import requests
from bs4 import BeautifulSoup

def extract_json_ld(url):
    response = requests.get(url)
    soup = BeautifulSoup(response.content, 'html.parser')
    
    json_ld_blocks = []
    for script in soup.find_all('script', {'type': 'application/ld+json'}):
        try:
            data = json.loads(script.string)
            json_ld_blocks.append({
                'url': url,
                'type': data.get('@type', 'unknown'),
                'schema': json.dumps(data, indent=2)
            })
        except json.JSONDecodeError:
            pass
    
    return json_ld_blocks

# Test on a single page
blocks = extract_json_ld('https://yoursite.com/product/item-123')
for block in blocks:
    print(f"Found {block['type']} schema")
    print(block['schema'])

For internal links, build a map of which pages link to which. This helps you understand navigation flow and identify critical link paths. If page A links to pages B, C, and D, make sure those relationships survive the migration.

In a real scenario: I migrated a SaaS knowledge base and discovered that 40% of internal traffic came through a single hub page. The new site's structure buried that hub deeper in navigation. We restructured the new site's internal links to preserve that flow, and traffic recovery was much faster.

Compare after launch

This is the final step and where many teams falter. After you launch the new site, run the same crawl again and compare. Did you get the same number of 200 responses? Are all your redirects working? Are there new 404s?

A basic comparison script:

import pandas as pd

old_crawl = pd.read_csv('old_site_crawl.csv')
new_crawl = pd.read_csv('new_site_crawl.csv')

# Merge on URL
comparison = pd.merge(old_crawl, new_crawl, on='url', how='outer', suffixes=('_old', '_new'))

# Find status code changes
status_changes = comparison[comparison['status_code_old'] != comparison['status_code_new']]
print(f"Status code changes: {len(status_changes)}")
print(status_changes[['url', 'status_code_old', 'status_code_new']].head(20))

# Find URLs missing in new crawl
missing = comparison[comparison['status_code_new'].isna()]
print(f"\nURLs not found in new crawl: {len(missing)}")
print(missing[['url', 'status_code_old']].head(20))

# Find new 404s
new_404s = comparison[(comparison['status_code_old'] == 200) & (comparison['status_code_new'] == 404)]
print(f"\nPages that are now 404: {len(new_404s)}")
print(new_404s[['url']].head(20))

Run this 48 hours after launch. Any new 404s, missing redirects, or title tag drops need immediate attention. I once caught 127 broken redirects this way—they would've silently tanked search traffic within a week.

Common mistakes

• Crawling only the homepage and sitemap URLs: Use a proper crawler that follows links. Plenty of pages hide behind navigation menus or are only accessible through internal links. Screaming Frog's spider mode catches these; a simple sitemap parser won't.
• Ignoring query parameters: If you have /products?sort=price and /products?sort=date, they might be the same page with different sorting. Mark them as duplicates in your crawl or configure the crawler to ignore parameters correctly.
• Not testing redirects under load: A 301 redirect works fine in a curl test, but if your old site's server collapses under redirect traffic before new site caching kicks in, you have a problem. Load test your redirect endpoints.
• Forgetting PDFs, images, and other assets: If your site serves downloadable PDFs or has image sitemap URLs with SEO value, include those in your crawl. They matter for completeness.
• Comparing crawls taken days apart: If old crawl is from Monday and new crawl is from Thursday, content changes, caches clear, and your comparison is unreliable. Ideally, both crawls happen within hours of launch.

Key takeaways

• Before migrating, run a full site crawl capturing status codes, title tags, canonicals, and meta descriptions. This is your baseline.
• Export your top pages by organic traffic and backlinks. These are your VIPs and require manual verification post-launch.
• Document all structured data and internal link relationships. You need to replicate or improve these on the new site.
• Compare the old crawl to a new crawl 48 hours after launch. Look for unexpected status code changes, new 404s, and missing redirects.
• Automate this where possible. A few hours of scripting now saves weeks of debugging later.