Python for web scraping, responsibly: a practical guide
Web scraping with Python can feel like a superpower—gather market data, monitor competitor pricing, or collect research samples at scale. But that power comes with responsibility: scrape badly and you'll get IP-banned, face legal trouble, or take down someone's server. This guide walks through the practices and tools we use on real projects to collect data ethically and sustainably.
Understanding robots.txt and terms of service
Before you write a single line of scraping code, open a browser and check two places. First, visit https://example.com/robots.txt to see what the site explicitly permits. Second, read the terms of service. These aren't just legal cover—they tell you whether the site owner wants scrapers at all.
A robots.txt file might look like this:
User-agent: *
Disallow: /admin/
Disallow: /api/private/
Allow: /products/
Crawl-delay: 2That Crawl-delay: 2 is a signal. It means "wait 2 seconds between requests." Respect it. The Disallow rules tell crawlers which paths are off-limits. If you scrape /admin/, you're ignoring explicit instruction.
In our experience, the sites that get scrapped hardest are the ones nobody asked permission from. Reach out to the site owner or API team first. Many will grant access directly—and you'll get cleaner, faster data through an official feed than by scraping HTML.
For Python, parse robots.txt with the built-in urllib.robotparser:
from urllib.robotparser import RobotFileParser
rp = RobotFileParser()
rp.set_url("https://example.com/robots.txt")
rp.read()
if rp.can_fetch("*", "https://example.com/products/"):
print("✓ Allowed to scrape /products/")
else:
print("✗ Disallowed by robots.txt")
# Check the crawl delay
delay = rp.request_rate("*")
if delay:
print(f"Recommended delay: {delay.requests} requests per {delay.seconds} seconds")Choosing the right library: httpx with selectolax or BeautifulSoup
For static HTML pages, you don't need a heavyweight browser automation tool. Use httpx to fetch and selectolax or BeautifulSoup to parse. We reach for selectolax when speed matters; BeautifulSoup when we need stability and clarity.
httpx is a modern HTTP client that's async-friendly and handles cookies naturally:
import httpx
from selectolax.parser import HTMLParser
async def fetch_products(url: str) -> list[dict]:
async with httpx.AsyncClient() as client:
response = await client.get(url, timeout=10.0)
response.raise_for_status()
parser = HTMLParser(response.text)
products = []
for item in parser.css("div.product-card"):
name = item.css_first("h2")
price = item.css_first(".price")
if name and price:
products.append({
"name": name.text(),
"price": price.text(),
})
return productsIf you prefer BeautifulSoup's syntax or need its error-handling, swap the parser:
from bs4 import BeautifulSoup
import httpx
def fetch_products(url: str) -> list[dict]:
response = httpx.get(url, timeout=10.0)
soup = BeautifulSoup(response.text, "html.parser")
products = []
for item in soup.select("div.product-card"):
name_tag = item.select_one("h2")
price_tag = item.select_one(".price")
if name_tag and price_tag:
products.append({
"name": name_tag.get_text(strip=True),
"price": price_tag.get_text(strip=True),
})
return productsThe main trade-off: selectolax is faster (it uses the native Lexbor C parser), but BeautifulSoup is more forgiving with broken HTML and has deeper community documentation. For most projects, either works well. Choose based on your team's comfort and your speed requirements.
Handling JavaScript-heavy pages with Playwright
Some sites load content dynamically with JavaScript. A static HTTP request won't see it. That's when you need Playwright, a browser automation library. It's heavier than httpx but it renders the page like a real browser would.
Install it with:
pip install playwright
playwright install chromiumHere's a practical example scraping a page that loads products via JavaScript:
from playwright.async_api import async_playwright
from selectolax.parser import HTMLParser
async def scrape_js_products(url: str) -> list[dict]:
async with async_playwright() as p:
browser = await p.chromium.launch(headless=True)
page = await browser.new_page()
await page.goto(url, wait_until="networkidle")
await page.wait_for_selector("div.product-card", timeout=5000)
# Wait a bit longer if the page has animations
await page.wait_for_timeout(2000)
html = await page.content()
await browser.close()
parser = HTMLParser(html)
products = []
for item in parser.css("div.product-card"):
name = item.css_first("h2")
price = item.css_first(".price")
if name and price:
products.append({
"name": name.text(),
"price": price.text(),
})
return productsKey points on Playwright:
wait_until="networkidle"ensures the page is fully loaded before we extract content.wait_for_selectorblocks until the target element is in the DOM.- Always
await browser.close()to avoid resource leaks. - Set reasonable timeouts; sites that take more than 10 seconds probably shouldn't be scraped.
Use Playwright only when necessary. It's resource-intensive and slow compared to static parsing. If the site has an API, use that. If the HTML is plain HTTP, use httpx. Reserve Playwright for truly dynamic content.
Rate-limiting and request caching
Hammering a server with rapid requests is how you get banned—and rightfully so. Implement rate-limiting and caching to scrape sustainably and reduce server load.
Here's a practical rate-limiter using asyncio and a simple cache with SQLite:
import asyncio
import hashlib
import sqlite3
from datetime import datetime, timedelta
import httpx
class ResponsibleScraper:
def __init__(self, delay_seconds: float = 2.0, cache_ttl_hours: int = 24):
self.delay_seconds = delay_seconds
self.cache_ttl = timedelta(hours=cache_ttl_hours)
self.last_request_time = 0
self._init_cache()
def _init_cache(self):
self.conn = sqlite3.connect(":memory:")
self.conn.execute("""
CREATE TABLE cache (
url_hash TEXT PRIMARY KEY,
url TEXT,
content TEXT,
cached_at TIMESTAMP
)
""")
self.conn.commit()
def _get_cache_key(self, url: str) -> str:
return hashlib.md5(url.encode()).hexdigest()
def _get_cached(self, url: str) -> str | None:
url_hash = self._get_cache_key(url)
cursor = self.conn.execute(
"SELECT content, cached_at FROM cache WHERE url_hash = ?",
(url_hash,)
)
row = cursor.fetchone()
if not row:
return None
content, cached_at = row
cached_dt = datetime.fromisoformat(cached_at)
if datetime.now() - cached_dt > self.cache_ttl:
self.conn.execute("DELETE FROM cache WHERE url_hash = ?", (url_hash,))
self.conn.commit()
return None
return content
def _set_cached(self, url: str, content: str):
url_hash = self._get_cache_key(url)
self.conn.execute(
"""
INSERT OR REPLACE INTO cache (url_hash, url, content, cached_at)
VALUES (?, ?, ?, ?)
""",
(url_hash, url, content, datetime.now().isoformat())
)
self.conn.commit()
async def fetch(self, url: str) -> str:
# Check cache first
cached = self._get_cached(url)
if cached:
print(f"Cache hit: {url}")
return cached
# Rate-limit: ensure minimum delay between requests
elapsed = asyncio.get_event_loop().time() - self.last_request_time
if elapsed < self.delay_seconds:
await asyncio.sleep(self.delay_seconds - elapsed)
print(f"Fetching: {url}")
async with httpx.AsyncClient() as client:
response = await client.get(url, timeout=10.0)
response.raise_for_status()
self.last_request_time = asyncio.get_event_loop().time()
# Store in cache
self._set_cached(url, response.text)
return response.text
# Usage
async def main():
scraper = ResponsibleScraper(delay_seconds=2.0, cache_ttl_hours=24)
urls = [
"https://example.com/page1",
"https://example.com/page2",
"https://example.com/page1", # Will be cached
]
for url in urls:
html = await scraper.fetch(url)
print(f"Fetched {len(html)} bytes from {url}")
# asyncio.run(main())This scraper enforces a 2-second delay between requests and caches responses for 24 hours. The second request to page1 returns instantly from cache.
For distributed scraping across multiple machines, consider using a dedicated library like redis or RabbitMQ to coordinate rate-limits. A shared Redis instance can track request counts per domain and ensure your entire team respects the same delays.
Common mistakes
- Ignoring robots.txt. It's not legally binding in most jurisdictions, but violating it is a signal you're acting in bad faith. Respect it.
- Scraping too fast. Even if a site doesn't ban you, rapid requests consume their resources. Rate-limit generously—2–5 seconds is reasonable.
- Not setting timeouts. If a request hangs, it'll block your scraper indefinitely. Always set
timeouton HTTP calls. - Using headless browsers for static content. Playwright is slow. Use it only when JavaScript rendering is essential.
- Parsing without error handling. Websites change. Your selectors will break. Use try-catch and log failures.
- Not respecting the Crawl-delay directive. If robots.txt says wait 5 seconds, wait 5 seconds.
- Scraping behind a login without permission. Even if you have credentials, scraping authenticated content often violates the TOS. Ask first.
Key takeaways
- Check
robots.txtand the site's terms of service before you start. Scraping responsibly means respecting the site owner's wishes. - Use
httpxwithselectolaxorBeautifulSoupfor static HTML; reserve Playwright for JavaScript-heavy pages. - Always implement rate-limiting and caching. Respect the Crawl-delay directive and cache responses to reduce server load.
- Set reasonable timeouts on all network requests and handle parse failures gracefully.
- When possible, ask for API access or a data partnership. Official feeds beat scraped HTML every time.
Comments
Loading comments…