Detecting SEO issues automatically: a step-by-step guide

// Pseudo-code: indexability check
async function checkIndexability(url) {
  const response = await fetch(url);
  const status = response.status;
  const html = await response.text();
  
  const issues = [];
  
  if (status >= 400) {
    issues.push(`HTTP ${status} - page not indexable`);
  }
  
  const metaRobots = html.match(/

// Pseudo-code: on-page check
function checkOnPage(html, url) {
  const issues = [];
  
  const titleMatch = html.match(/([^<]+)<\/title>/i);
  const title = titleMatch ? titleMatch[1].trim() : null;
  
  if (!title) {
    issues.push('missing title tag');
  } else if (title.length < 30) {
    issues.push(`title too short: ${title.length} chars`);
  } else if (title.length > 70) {
    issues.push(`title too long: ${title.length} chars`);
  }
  
  const descMatch = html.match(/<meta\s+name=["']description["']\s+content=["']([^"']*)/i);
  const desc = descMatch ? descMatch[1].trim() : null;
  
  if (!desc) {
    issues.push('missing meta description');
  } else if (desc.length < 120) {
    issues.push(`description too short: ${desc.length} chars`);
  }
  
  const h1s = html.match(/<h1[^>]*>([^<]+)<\/h1>/gi) || [];
  if (h1s.length === 0) {
    issues.push('missing H1 tag');
  } else if (h1s.length > 1) {
    issues.push(`multiple H1 tags (${h1s.length})`);
  }
  
  return { url, title, description: desc, h1Count: h1s.length, issues };
}
</code></pre>

<h2>Links — Broken internal links, redirect chains, orphan pages</h2>

<p>Link health is critical. A broken internal link wastes crawl budget and hurts user experience. A redirect chain (A → B → C → final destination) slows down crawling and bleeds authority. An orphan page has no internal links pointing to it, so users and crawlers can't find it.</p>

<p>Your audit should: crawl all internal links on every page and check their status codes, detect redirect chains longer than two hops, identify pages with no inbound internal links, and log pages that only link to themselves via footer navigation.</p>

<p>When crawling internal links, extract them from href attributes and follow them. For each destination, record the final status code. A 404 is broken. A 301 or 302 that redirects to another URL that redirects again is a chain. An inbound link count of zero (excluding the homepage, which is always reachable) signals an orphan.</p>

<pre><code class="language-html">// Pseudo-code: link check
function checkLinks(html, url, crawlMap) {
  const linkRegex = /<a\s+[^>]*href=["']([^"']+)["']/gi;
  const links = [];
  let match;
  
  while ((match = linkRegex.exec(html)) !== null) {
    const href = match[1];
    // normalize relative URLs, skip external links, skip anchors
    if (href.startsWith('/') || href.startsWith('http')) {
      links.push(href);
    }
  }
  
  const issues = [];
  links.forEach(link => {
    const target = crawlMap[link];
    if (!target) {
      issues.push(`broken link: ${link}`);
    } else if (target.redirectChainLength > 2) {
      issues.push(`redirect chain too long: ${link} (${target.redirectChainLength} hops)`);
    }
  });
  
  return { url, outboundLinks: links.length, issues };
}

// Identify orphan pages: run this after crawling all URLs
function findOrphans(crawlMap) {
  const inboundCount = {};
  
  for (const [url, data] of Object.entries(crawlMap)) {
    data.outboundLinks.forEach(link => {
      inboundCount[link] = (inboundCount[link] || 0) + 1;
    });
  }
  
  const orphans = Object.keys(crawlMap).filter(
    url => inboundCount[url] === 0 && url !== '/'
  );
  
  return orphans;
}
</code></pre>

<h2>Performance — Page weight, render-blocking resources, Core Web Vitals data</h2>

<p>Search engines increasingly weight performance in rankings. Your audit engine should measure three categories: page weight (total bytes downloaded), render-blocking resources (CSS, JavaScript loaded in the head), and Core Web Vitals metrics (Largest Contentful Paint, First Input Delay, Cumulative Layout Shift).</p>

<p>Page weight over 5 MB is excessive. Render-blocking scripts in the head delay first paint—move them to the end of the body or mark them as async/defer. For Core Web Vitals, you need real user data or field data from Google's PageSpeed Insights API or CrUX API, not just lab testing.</p>

<p>In a travel platform I built, the homepage was 8 MB because someone embedded an unoptimized video. The audit flagged it immediately. Without that signal, we'd have had poor Core Web Vitals for weeks before noticing organic traffic drop.</p>

<pre><code class="language-html">// Pseudo-code: performance check (lab data via Lighthouse API)
async function checkPerformance(url) {
  const response = await fetch(`https://www.googleapis.com/pagespeedonline/v5/runPagespeed?url=${url}&key=YOUR_KEY`);
  const data = await response.json();
  
  const issues = [];
  const lighthouseResult = data.lighthouseResult;
  
  const totalBytes = lighthouseResult.audits['total-byte-weight'].numericValue;
  if (totalBytes > 5000000) {
    issues.push(`page weight too high: ${(totalBytes / 1024 / 1024).toFixed(1)} MB`);
  }
  
  const renderBlockingScore = lighthouseResult.audits['render-blocking-resources'].score;
  if (renderBlockingScore < 0.9) {
    issues.push('render-blocking resources detected');
  }
  
  const cwvMetrics = {
    LCP: lighthouseResult.audits['largest-contentful-paint']?.numericValue,
    FID: lighthouseResult.audits['max-potential-fid']?.numericValue,
    CLS: lighthouseResult.audits['cumulative-layout-shift']?.numericValue
  };
  
  return { url, totalBytes, cwvMetrics, issues };
}
</code></pre>

<h2>Structured data — Invalid JSON-LD and missing required properties</h2>

<p>Structured data (JSON-LD) helps search engines understand your content and qualify you for rich snippets. Your audit should validate that any JSON-LD on the page is syntactically correct and contains required properties for its schema type.</p>

<p>For an e-commerce product page, the Product schema requires name, image, and price. Missing any of these and Google won't render a rich result. For an article, Article requires headline, datePublished, and author. For local business, LocalBusiness requires name, address, and telephone.</p>

<p>Parse each JSON-LD block, identify its @type, then cross-reference it against the Google Schema.org requirements. If a required property is missing or null, flag it. Also validate that @type, @context, and the structure itself is valid JSON.</p>

<pre><code class="language-html">// Pseudo-code: structured data check
function checkStructuredData(html, url) {
  const jsonLdBlocks = html.match(/<script\s+type=["']application\/ld\+json["']>([^<]+)<\/script>/gi) || [];
  const issues = [];
  
  jsonLdBlocks.forEach((block, idx) => {
    try {
      const jsonStr = block.match(/>([^<]+)</)[1];
      const json = JSON.parse(jsonStr);
      
      const schemaType = json['@type'];
      const required = {
        'Product': ['name', 'image', 'offers'],
        'Article': ['headline', 'datePublished', 'author'],
        'LocalBusiness': ['name', 'address', 'telephone']
      };
      
      if (required[schemaType]) {
        required[schemaType].forEach(prop => {
          if (!json[prop]) {
            issues.push(`${schemaType} missing required property: ${prop}`);
          }
        });
      }
    } catch (e) {
      issues.push(`JSON-LD block ${idx} has invalid syntax: ${e.message}`);
    }
  });
  
  return { url, jsonLdCount: jsonLdBlocks.length, issues };
}
</code></pre>

<h2>Common mistakes</h2>

<ul>
<li><strong>Not following redirects in the crawler.</strong> If you only check the initial response and don't follow a 301, you'll miss issues on the final destination. Always crawl the final URL.</li>
<li><strong>Ignoring case sensitivity in URLs.</strong> Most servers treat /product and /Product the same, but your audit might see them as duplicates. Normalize URLs before comparing.</li>
<li><strong>Only checking Core Web Vitals from lab data.</strong> Lab data (Lighthouse) is useful but doesn't reflect real user experience. Integrate with the CrUX API to get field data for actual user sessions.</li>
<li><strong>Flagging duplicate H1s without context.</strong> Some pages legitimately have multiple H1s if they're structured as a grid or list. A better check: H1 count should match the primary content section, not the entire page.</li>
<li><strong>Not handling dynamically injected content.</strong> If your page renders content via JavaScript, a simple HTML parser will miss it. Use a headless browser (Puppeteer, Playwright) to render before analyzing.</li>
</ul>

<h2>Key takeaways</h2>

<ul>
<li>Detect indexability issues first: status codes, noindex, canonicals, robots blocks. These are hard blockers on visibility.</li>
<li>Check on-page signals systematically: title length, description presence, H1 count. Aim for consistency across the site.</li>
<li>Crawl internal links and measure redirect chain depth. Orphan pages and broken links waste crawl budget and harm user experience.</li>
<li>Integrate performance metrics into your audit: page weight, render-blocking resources, Core Web Vitals from both lab and field data.</li>
<li>Validate structured data schema types and required properties. Missing JSON-LD properties means lost rich snippet opportunities.</li>
<li>Automate these checks and run them on every deployment or daily crawl. Manual audits miss the long tail of issues that accumulate as your site grows.</li>
</ul>