What is web scraping?

What is web scraping?

Web scraping is the automated extraction of data from websites. A scraper sends requests to a web server, receives the HTML response, and parses out the specific data it needs: product prices, article text, inventory counts.

How web scraping works

At its simplest, a scraper is a program that downloads web pages and extracts data from the HTML. Modern scrapers have become far more sophisticated. They render JavaScript, solve CAPTCHAs, rotate through proxy networks to avoid detection, and mimic real browser behavior down to mouse movements and scroll patterns.

Why companies scrape websites

Web scraping serves many legitimate and illegitimate purposes. Price comparison sites scrape retailer pricing. Research firms scrape public data for analysis. Recruiters scrape LinkedIn profiles. Competitors scrape each other for market intelligence. And increasingly, AI companies scrape the entire web to train their models.

The AI scraping difference

Traditional scraping typically targets specific data points from specific sites. AI scraping is different in scale and purpose. AI companies need massive volumes of diverse text data to train large language models. They scrape broadly, deeply, and continuously — often returning to the same sites multiple times as they update their training data.

Tollbit reports that across 550 billion website visits they analyzed, 9 billion were AI bot scrapes, and 2.9 billion of those bypassed robots.txt instructions.

Legal landscape

The legality of web scraping varies by jurisdiction and context. In the US, the Computer Fraud and Abuse Act and copyright law provide some protections, but enforcement is inconsistent. The EU's Database Directive offers stronger protections for structured data. Several high-profile lawsuits — including actions against OpenAI and Anthropic — are testing whether AI training constitutes fair use.

Commercial scraping services

A growing ecosystem of scraping-as-a-service providers (BrightData, Oxylabs, ScraperAPI, and others) makes it trivial to scrape at scale. These services provide rotating residential proxies, browser automation, and CAPTCHA solving — making detection significantly harder.

Protecting against unwanted scraping

Effective anti-scraping requires multiple layers: rate limiting, IP reputation analysis, TLS fingerprinting, behavioral analysis, and crawler identification. No single technique is sufficient because scrapers constantly adapt their methods. The goal is to make scraping expensive enough that attackers move to easier targets.