Building an AI SEO assistant: a practical AI & ML guide
SEO tools are everywhere, but most teams still struggle to scale content optimization across dozens or hundreds of pages. Building an AI SEO assistant: a practical AI & ML guide walks you through a real-world product that crawls your site, scores content against known SEO rules, uses an LLM to suggest improvements, and validates every suggestion before surfacing it. By the end, you'll have a working system that catches missing meta tags, flags short titles, and rewrites descriptions—all automatically and safely.
Why build an AI SEO assistant
Manual SEO reviews are slow. Your team can't check every page for missing og:image tags or title lengths that exceed 60 characters. LLMs are powerful, but raw LLM output often violates your constraints (a rewritten title might be 120 characters instead of 55). The solution: a layered pipeline that crawls, scores, generates suggestions, and validates them all—with humans reviewing the final output.
This approach has shipped in production across SaaS and e-commerce teams. It's how you turn SEO into a product feature rather than a spreadsheet.
Crawl and extract—Collect titles, headings and content from pages
Start by building a lightweight crawler that fetches your site's HTML and pulls out the key SEO signals. You don't need Puppeteer or Selenium for most cases; a simple HTTP client + BeautifulSoup or jsdom will do.
import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin, urlparse
def crawl_page(url, base_url):
"""Fetch a single page and extract SEO metadata."""
try:
response = requests.get(url, timeout=10)
response.raise_for_status()
except requests.RequestException as e:
return None
soup = BeautifulSoup(response.content, 'html.parser')
# Extract core SEO elements
title = soup.find('title')
meta_desc = soup.find('meta', attrs={'name': 'description'})
og_image = soup.find('meta', attrs={'property': 'og:image'})
og_title = soup.find('meta', attrs={'property': 'og:title'})
og_desc = soup.find('meta', attrs={'property': 'og:description'})
# Extract headings
h1_tags = [h.get_text(strip=True) for h in soup.find_all('h1')]
h2_tags = [h.get_text(strip=True) for h in soup.find_all('h2')]
# Extract body text for context
body_text = soup.get_text(separator=' ', strip=True)[:2000]
return {
'url': url,
'title': title.get_text() if title else None,
'meta_description': meta_desc.get('content') if meta_desc else None,
'og_image': og_image.get('content') if og_image else None,
'og_title': og_title.get('content') if og_title else None,
'og_description': og_desc.get('content') if og_desc else None,
'h1_tags': h1_tags,
'h2_tags': h2_tags,
'body_preview': body_text,
}
def crawl_sitemap(sitemap_url):
"""Parse a sitemap.xml and crawl all listed URLs."""
response = requests.get(sitemap_url)
soup = BeautifulSoup(response.content, 'xml')
urls = [loc.get_text() for loc in soup.find_all('loc')]
base_url = urlparse(sitemap_url).netloc
results = []
for url in urls:
page_data = crawl_page(url, base_url)
if page_data:
results.append(page_data)
return resultsStore the extracted data in a database (PostgreSQL with a JSONB column works well). You now have a snapshot of your site's current state. In a production system, you'd run this on a schedule—nightly or weekly—and track changes over time.
Score with rules—Length checks and missing tags in code
Before you involve an LLM, establish a deterministic scoring system. This catches obvious issues and gives you a baseline to compare suggestions against.
class SEOScorer:
def __init__(self):
self.rules = {
'title_min': 30,
'title_max': 60,
'meta_desc_min': 120,
'meta_desc_max': 160,
'og_image_required': True,
'h1_count_max': 1,
}
def score_page(self, page_data):
"""Return a score (0–100) and a list of issues."""
issues = []
score = 100
# Title checks
title = page_data.get('title') or ''
if not title:
issues.append('Missing page title')
score -= 20
elif len(title) < self.rules['title_min']:
issues.append(f'Title too short: {len(title)} chars (min {self.rules["title_min"]})')
score -= 10
elif len(title) > self.rules['title_max']:
issues.append(f'Title too long: {len(title)} chars (max {self.rules["title_max"]})')
score -= 10
# Meta description checks
meta_desc = page_data.get('meta_description') or ''
if not meta_desc:
issues.append('Missing meta description')
score -= 15
elif len(meta_desc) < self.rules['meta_desc_min']:
issues.append(f'Meta description too short: {len(meta_desc)} chars')
score -= 10
elif len(meta_desc) > self.rules['meta_desc_max']:
issues.append(f'Meta description too long: {len(meta_desc)} chars')
score -= 10
# OG tags
if not page_data.get('og_image'):
issues.append('Missing og:image tag')
score -= 10
if not page_data.get('og_title'):
issues.append('Missing og:title tag')
score -= 5
# H1 checks
h1_count = len(page_data.get('h1_tags', []))
if h1_count != 1:
issues.append(f'Expected 1 H1 tag, found {h1_count}')
score -= 15
return max(0, score), issues
# Usage
scorer = SEOScorer()
for page in pages:
score, issues = scorer.score_page(page)
page['baseline_score'] = score
page['issues'] = issuesThis deterministic layer is crucial. It's fast, auditable, and doesn't require an API call. Use it to prioritize which pages to improve: pages with a score below 70 are good candidates for LLM suggestions.
Suggest with an LLM—Rewrite titles and descriptions within limits
Now bring in the LLM. The key is to prompt it with context and hard constraints on output length. Use a structured prompt that includes the current content, the scoring issues, and explicit character limits.
import anthropic
def generate_suggestions(page_data, issues, max_retries=2):
"""Use Claude to suggest SEO improvements."""
client = anthropic.Anthropic()
prompt = f"""You are an SEO expert. Improve this page's SEO.
Current state:
- URL: {page_data['url']}
- Title: "{page_data.get('title', '')}"
- Meta description: "{page_data.get('meta_description', '')}"
- H1 tags: {page_data.get('h1_tags', [])}
- Body preview: {page_data['body_preview'][:500]}
Issues to fix:
{chr(10).join(f"- {issue}" for issue in issues)}
Requirements:
1. New title: 30–60 characters exactly. No brand name at the end.
2. New meta description: 120–160 characters exactly.
3. If missing, suggest an og:title (same as page title) and og:description (same as meta description).
4. Suggest a single, compelling H1 tag if needed.
Output as JSON:
{{
"suggested_title": "...",
"suggested_meta_description": "...",
"suggested_og_title": "...",
"suggested_og_description": "...",
"suggested_h1": "...",
"reasoning": "..."
}}
"""
for attempt in range(max_retries):
try:
message = client.messages.create(
model="claude-3-5-sonnet-20241022",
max_tokens=500,
messages=[
{"role": "user", "content": prompt}
]
)
import json
response_text = message.content[0].text
# Extract JSON from response (Claude may include extra text)
start = response_text.find('{')
end = response_text.rfind('}') + 1
suggestions = json.loads(response_text[start:end])
return suggestions
except (json.JSONDecodeError, IndexError) as e:
if attempt == max_retries - 1:
raise
continue
return NoneThe prompt is structured to be unambiguous. By stating "30–60 characters exactly," you guide the model toward compliance. In practice, Claude respects these constraints about 95% of the time; the remaining cases are caught by the validation layer.
Validate—Check every suggestion against the same rules before showing it
Never show an LLM suggestion without validating it first. Use the same scorer you built earlier.
def validate_suggestions(suggestions, page_data, scorer):
"""Validate LLM suggestions against SEO rules."""
validated = {}
errors = []
# Validate title
title = suggestions.get('suggested_title', '')
if not title:
errors.append('LLM did not suggest a title')
elif len(title) < 30 or len(title) > 60:
errors.append(f'Suggested title is {len(title)} chars; must be 30–60')
else:
validated['title'] = title
# Validate meta description
meta_desc = suggestions.get('suggested_meta_description', '')
if not meta_desc:
errors.append('LLM did not suggest a meta description')
elif len(meta_desc) < 120 or len(meta_desc) > 160:
errors.append(f'Suggested meta description is {len(meta_desc)} chars; must be 120–160')
else:
validated['meta_description'] = meta_desc
# Validate OG tags
og_title = suggestions.get('suggested_og_title', '')
og_desc = suggestions.get('suggested_og_description', '')
if og_title and len(og_title) <= 60:
validated['og_title'] = og_title
if og_desc and len(og_desc) <= 160:
validated['og_description'] = og_desc
# Validate H1
h1 = suggestions.get('suggested_h1', '')
if h1:
validated['h1'] = h1
return validated, errors
# Usage
suggestions = generate_suggestions(page_data, issues)
validated, errors = validate_suggestions(suggestions, page_data, scorer)
if errors:
# Log but don't surface; ask LLM to retry
print(f"Validation failed for {page_data['url']}: {errors}")
else:
# Ready to show to user
page_data['suggestions'] = validatedThis two-stage approach—generate, then validate—is simpler and more reliable than trying to force the LLM to be perfect on the first attempt. If validation fails, you can either retry the LLM with a stricter prompt or mark the page as needing manual review.
Putting it together: an end-to-end example
Here's how the pieces fit into a production workflow:
- Crawl your sitemap nightly and store page data in PostgreSQL.
- Run the scorer on every page; flag pages with a score below 70.
- For flagged pages, call the LLM and validate suggestions.
- Store validated suggestions in a separate table.
- Surface suggestions in a dashboard where your team can review and approve changes.
- On approval, generate the updated HTML and push it via your CMS API.
The entire pipeline for 500 pages takes about 5 minutes. Most of that time is the crawl and LLM calls (which you can parallelize with asyncio or batch requests).
Common mistakes
- Skipping validation. Trust the LLM's intent, not its output. Validate every suggestion, every time.
- Vague constraints in the prompt. "Make the title better" fails. "30–60 characters, front-load the keyword, no brand at the end" works.
- Crawling without respecting robots.txt. Check it before you crawl. Add a User-Agent header. Use exponential backoff if you get 429s.
- Storing suggestions as-is. If you plan to auto-apply changes, log every diff and make rollback easy. SEO changes can tank traffic if they're wrong.
- Ignoring context. Feed the LLM the page's body text or a summary of its main topic. A rewritten title that doesn't match the content hurts more than the original.
Key takeaways
- Build a deterministic scorer first. It's your source of truth and a safety net for LLM output.
- Crawl and store page data on a schedule. You'll want historical trends and the ability to audit changes.
- Use a strict, structured prompt for the LLM. Explicit character limits and constraints reduce hallucinations and improve compliance.
- Validate every LLM suggestion before surfacing it. This is cheaper and more reliable than retraining or fine-tuning.
- Start with review-and-approve workflows, then graduate to auto-apply for low-risk changes (e.g., missing meta tags) once you've proven quality.
- Log everything: crawl results, scores, suggestions, validations, approvals, and applied changes. You'll need this for debugging and improving the system over time.
Comments
Loading comments…