Most web applications render content with JavaScript, making traditional HTTP scraping tools insufficient for real-world data extraction needs. Playwright bridges this gap by automating a real browser, letting you interact with dynamic pages exactly as users do. Web scraping with Playwright, responsibly for production, means respecting legal boundaries, optimizing performance, and building systems that won't overwhelm target servers or damage your reputation.
Check robots.txt and terms of service
Before writing a single line of scraping code, verify you have permission. Start by examining the target site's robots.txt file and terms of service. Many sites explicitly prohibit automated access, and ignoring that is both legally risky and ethically wrong. Some sites allow scraping under specific conditions—perhaps with rate limits or for research purposes only.
Reading robots.txt is straightforward. If you're scraping example.com, fetch https://example.com/robots.txt and look for directives targeting your user agent. Here's a practical approach:
import https from 'https';
async function checkRobotsTxt(domain: string): Promise {
return new Promise((resolve, reject) => {
https.get(`https://${domain}/robots.txt`, (res) => {
let data = '';
res.on('data', (chunk) => (data += chunk));
res.on('end', () => resolve(data));
}).on('error', reject);
});
}
const robotsContent = await checkRobotsTxt('example.com');
console.log(robotsContent);
// Check for "Disallow: /" or "Disallow: /api" rules
Beyond robots.txt, review the terms of service. Some sites allow scraping for personal use but not commercial purposes. Others permit scraping of public data but prohibit storing or reselling it. Document your findings—if you ever face questions about your scraping practices, this evidence protects you.
Use locators to extract data reliably
Playwright's locator system is built for reliable element selection in dynamic environments. Unlike CSS selectors alone, locators can wait for elements to appear, handle stale references, and retry automatically. For production scraping, this resilience is critical because pages change, network delays happen, and elements render asynchronously.
Let's say you're scraping product listings from an e-commerce site. Rather than querying the DOM once and hoping the structure remains stable, use locators with explicit waits:
import { chromium } from 'playwright';
async function scrapeProducts(url: string) {
const browser = await chromium.launch();
const page = await browser.newPage();
await page.goto(url, { waitUntil: 'networkidle' });
// Locator waits for element and retries automatically
const productLocator = page.locator('[data-testid="product-card"]');
// Wait for at least one product to load
await productLocator.first().waitFor({ state: 'visible' });
const products = [];
const count = await productLocator.count();
for (let i = 0; i < count; i++) {
const card = productLocator.nth(i);
const title = await card.locator('[data-testid="product-title"]').textContent();
const price = await card.locator('[data-testid="product-price"]').textContent();
const url = await card.locator('a').getAttribute('href');
products.push({
title: title?.trim(),
price: price?.trim(),
url
});
}
await browser.close();
return products;
}
This approach uses data-testid attributes, which are more stable than CSS classes that change frequently. If data-testid isn't available, build selectors from stable semantic HTML. Avoid overly specific selectors that break when page layouts shift slightly.
For complex pages, consider using page.evaluate() to run JavaScript in the page context. This is faster than multiple round-trips to Playwright's API:
const products = await page.evaluate(() => {
return Array.from(document.querySelectorAll('[data-testid="product-card"]'))
.map((card) => ({
title: card.querySelector('[data-testid="product-title"]')?.textContent?.trim(),
price: card.querySelector('[data-testid="product-price"]')?.textContent?.trim(),
url: card.querySelector('a')?.getAttribute('href')
}));
});
Throttle requests to be respectful
Sending rapid requests to a server is disruptive and can trigger IP bans or legal action. Production scraping requires deliberate rate limiting. The key is matching your request rate to what the server can handle while remaining invisible as just another user.
Implement throttling at multiple levels: delays between page loads, backoff for errors, and concurrent connection limits.
import { chromium } from 'playwright';
class ResponsibleScraper {
private requestsPerSecond = 2;
private lastRequestTime = 0;
private browser: any;
async initialize() {
this.browser = await chromium.launch();
}
private async throttleRequest() {
const now = Date.now();
const timeSinceLastRequest = now - this.lastRequestTime;
const minInterval = 1000 / this.requestsPerSecond;
if (timeSinceLastRequest < minInterval) {
await new Promise((resolve) =>
setTimeout(resolve, minInterval - timeSinceLastRequest)
);
}
this.lastRequestTime = Date.now();
}
async scrapePage(url: string) {
await this.throttleRequest();
const page = await this.browser.newPage();
try {
await page.goto(url, { waitUntil: 'networkidle' });
// Extract data here
return await page.locator('[data-testid="content"]').textContent();
} catch (error) {
console.error(`Failed to scrape ${url}:`, error);
// Implement exponential backoff on failure
await new Promise((resolve) => setTimeout(resolve, 5000));
return null;
} finally {
await page.close();
}
}
async scrapeMultiple(urls: string[], concurrency = 3) {
const results = [];
for (let i = 0; i < urls.length; i += concurrency) {
const batch = urls.slice(i, i + concurrency);
const batchResults = await Promise.all(
batch.map((url) => this.scrapePage(url))
);
results.push(...batchResults);
}
return results;
}
async close() {
await this.browser.close();
}
}
const scraper = new ResponsibleScraper();
await scraper.initialize();
const data = await scraper.scrapeMultiple(['url1', 'url2', 'url3']);
await scraper.close();
Also, set a realistic user agent and add random delays between requests to mimic human browsing patterns:
const page = await browser.newPage({
userAgent: 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'
});
// Random delay between 2-5 seconds
const delay = Math.random() * 3000 + 2000;
await new Promise((resolve) => setTimeout(resolve, delay));
Cache results to minimize repeated requests
The most respectful scraping doesn't scrape at all—it reuses cached results. Caching reduces server load, speeds up your application, and makes development cycles faster. Design your scraping pipeline around data freshness requirements.
Use a simple file-based cache for smaller datasets or Redis for distributed systems:
import { createHash } from 'crypto';
import fs from 'fs/promises';
import path from 'path';
class CachedScraper {
private cacheDir = './scrape_cache';
private cacheTTL = 3600000; // 1 hour in milliseconds
async initializeCache() {
await fs.mkdir(this.cacheDir, { recursive: true });
}
private getCacheKey(url: string): string {
return createHash('md5').update(url).digest('hex');
}
async getFromCache(url: string): Promise {
const cacheKey = this.getCacheKey(url);
const cachePath = path.join(this.cacheDir, `${cacheKey}.json`);
try {
const stats = await fs.stat(cachePath);
const age = Date.now() - stats.mtimeMs;
if (age < this.cacheTTL) {
const data = await fs.readFile(cachePath, 'utf-8');
console.log(`Cache hit for ${url}`);
return JSON.parse(data);
}
// Cache expired
await fs.unlink(cachePath);
} catch (error) {
// File doesn't exist or other error
}
return null;
}
async saveToCache(url: string, data: any): Promise {
const cacheKey = this.getCacheKey(url);
const cachePath = path.join(this.cacheDir, `${cacheKey}.json`);
await fs.writeFile(cachePath, JSON.stringify(data, null, 2));
}
async scrapeWithCache(url: string, scrapeFn: () => Promise) {
const cached = await this.getFromCache(url);
if (cached) return cached;
const fresh = await scrapeFn();
await this.saveToCache(url, fresh);
return fresh;
}
}
// Usage
const scraper = new CachedScraper();
await scraper.initializeCache();
const data = await scraper.scrapeWithCache(
'https://example.com/products',
async () => {
// Your scraping logic
return { products: [] };
}
);
For production systems handling thousands of URLs, consider tiered caching: in-memory for hot data, disk for medium-term storage, and a refresh strategy that fetches only outdated records.
Common mistakes
- Ignoring robots.txt and terms: "The site didn't block me" isn't legal permission. Document your research and get explicit approval if uncertain.
- Scraping without caching: Hitting the same page repeatedly wastes bandwidth and increases ban risk. Implement caching from day one.
- Using overly specific selectors: Selectors tied to CSS classes or IDs break when design changes. Prefer data attributes or semantic HTML.
- No error handling: Network timeouts, page rendering failures, and server errors are inevitable. Use try-catch blocks, retries, and logging.
- Running infinite scraping loops: Always set batch limits and exit conditions. Uncontrolled scraping can drain resources or trigger DDoS protections.
- Storing sensitive data carelessly: If scraping includes personally identifiable information, encrypt it and follow data protection regulations like GDPR.
Key takeaways
- Always verify permissions via robots.txt and terms of service before scraping any site.
- Use Playwright's locators with explicit waits for reliable data extraction on dynamic pages.
- Implement request throttling, backoff on errors, and realistic user agents to avoid overwhelming servers and getting blocked.
- Cache results aggressively—caching reduces load, speeds development, and demonstrates respect for the target server.
- Treat production scraping as you would any critical data pipeline: add logging, error handling, and monitoring from the start.
Comments
Loading comments…