Nearly 1 in 5 top websites block AI search crawlers
On 19 November 2026 we read the robots.txt file of the 1,000 most-visited domains on the web and checked, for each AI crawler, whether the site lets it read the home page. 475 of the domains had a robots.txt we could read. Of those:
- 18.3% (87 sites) block at least one AI search crawler: OpenAI’s OAI-SearchBot, Anthropic’s Claude-SearchBot or PerplexityBot. According to the companies’ own documentation, blocking them keeps a site out of, or less visible in, those assistants’ search answers.
- 11.8% (56 sites) block all three.
- The top 100 block twice as often: 33.3% of the top-100 domains with a readable robots.txt block an AI search crawler, against 15.4% of those ranked 101 to 500 and 17.4% of those ranked 501 to 1,000.
- Training is blocked more than search. 21.5% block GPTBot, which collects training data for OpenAI’s models, but only 12.0% block OAI-SearchBot. Of the 102 sites that block GPTBot, 45 still let OAI-SearchBot in.
You can download the full data (one row per domain) and check any site yourself with our free AI crawler check.
Every crawler, side by side
Share of the 475 sites whose robots.txt keeps each crawler away from the home page, and share that name the crawler at all.
| Crawler | Run by | Used for | Blocked | Named in robots.txt |
|---|---|---|---|---|
| GPTBot | OpenAI | Training | 21.5% | 26.5% |
| OAI-SearchBot | OpenAI | ChatGPT search | 12.0% | 18.1% |
| ChatGPT-User | OpenAI | Pages a ChatGPT user asks about | 13.5% | 19.6% |
| ClaudeBot | Anthropic | Training | 22.3% | 26.1% |
| Claude-SearchBot | Anthropic | Claude search | 13.9% | 15.4% |
| Claude-User | Anthropic | Pages a Claude user asks about | 13.9% | 14.7% |
| PerplexityBot | Perplexity | Perplexity search | 17.9% | 24.4% |
| Perplexity-User | Perplexity | Pages a Perplexity user asks about | 15.6% | 13.5% |
| Google-Extended | Gemini training and grounding | 18.1% | 22.7% | |
| Applebot-Extended | Apple | Apple’s AI models | 17.7% | 18.7% |
| CCBot | Common Crawl | Open web archive used to train many models | 24.6% | 23.8% |
| Bytespider | ByteDance | ByteDance’s crawler, widely reported as used for training | 26.1% | 22.5% |
| Meta-ExternalAgent | Meta | Training Meta’s AI models | 20.4% | 18.7% |
| Amazonbot | Amazon | Amazon services, including AI | 17.5% | 15.4% |
| Googlebot | Google Search | 3.2% | 16.8% | |
| Bingbot | Microsoft | Bing, Copilot | 3.4% | 12.8% |
A crawler can be blocked without being named: a rule for all crawlers (User-agent: *) applies to every crawler the file doesn’t name separately. That’s why some “blocked” figures are higher than the “named” ones.
What stands out
Big sites draw a line between training and search
OpenAI and Anthropic each run separate crawlers for training and for search, and say each robots.txt setting works independently. Perplexity says its crawlers aren’t used for training at all. Many large sites use that split: 45 of the 102 sites that block GPTBot still allow OAI-SearchBot, and 40 of the 106 that block ClaudeBot still allow Claude-SearchBot. They keep their content out of model training but stay eligible to be cited in AI search answers.
Perplexity is blocked more often than ChatGPT search
PerplexityBot is blocked by 17.9% of sites, against 12.0% for OAI-SearchBot and 13.9% for Claude-SearchBot, although all three do the same job for their assistant: finding pages to quote in answers.
Many rules lag behind the crawlers
124 sites name ClaudeBot, but only 73 name Claude-SearchBot, which Anthropic lists separately for search. 68 still name anthropic-ai, which isn’t one of the three crawlers on Anthropic’s crawler page today. A rule written for one crawler doesn’t cover its newer siblings, so a site that meant to block only training can end up allowing search, or the other way round.
Some sites shut AI search out without naming it
20 sites block all three AI search crawlers without naming any of them, through a rule for all crawlers. 8 of those still let Googlebot in by name. Sites that let in only the search engines they list keep out every new AI assistant by default, whether or not that was the intention.
The most visited sites block the most
17 of the 51 top-100 domains with a readable robots.txt block at least one AI search crawler, including facebook.com, instagram.com, linkedin.com, amazon.com, x.com, pinterest.com and tiktok.com. Large platforms often license their content or keep it for their own products, so blocking is a business decision for them.
What this means for a small business
The sites that block AI crawlers on purpose are mostly large platforms with something to protect. For a small business, the trade usually runs the other way: AI assistants answer questions like “who fixes boilers near me” or “best accountant for freelancers”, and a site they can’t read can’t be named in the answer.
Three things are worth checking on your own site:
- Your robots.txt doesn’t block OAI-SearchBot, Claude-SearchBot or PerplexityBot, either by name or through a rule for all crawlers.
- Your host, CDN or security plugin doesn’t block AI crawlers. A robots.txt that allows everything doesn’t help if a firewall refuses the request; some CDNs offer a “block AI bots” switch.
- Your pages don’t say noindex, which keeps them out of search and AI answers alike.
Our free AI crawler check does all three for a page in under a minute.
How we did it
- Sites: the top 1,000 domains of the Tranco list (list L5KW4), a research ranking that combines several popularity sources.
- Request: one request per domain for
https://<domain>/robots.txt, following redirects, with a user agent that names OOO Works and links to this survey. At most 16 sites were read at once. - Reading: a file counted as readable if it came back with status 200 and wasn’t an HTML page. Each crawler was tested against the home page path
/with the open-source robotstxt parser for Go (temoto/robotstxt). “Named” means the crawler appears in aUser-agentline. - Who dropped out: 525 domains had no readable robots.txt. 222 had no website at that name (many top domains are infrastructure, such as content-delivery or API domains), 115 failed for other connection reasons, 69 refused us (403), 59 returned a web page instead of a robots.txt, 47 had none (404), and 13 returned other errors.
Limits
- robots.txt is a request, not a lock. Crawlers that follow it stay out; it doesn’t show what a firewall does. OpenAI and Perplexity also say their user-triggered fetchers (ChatGPT-User, Perplexity-User) may not follow robots.txt.
- Some sites send different robots.txt files to different crawlers, or check real crawlers by IP address. We read the copy served to a named research crawler.
- We tested the home page only. Many sites allow the home page and block other sections, or the reverse.
- The list measures popularity of domains, not of businesses; it is a snapshot of one day.
Sources for what each crawler does: OpenAI’s crawler documentation, Anthropic’s crawler page, Perplexity’s crawler documentation and Google’s list of common crawlers.