Blog

Nearly 1 in 5 top websites block AI search crawlers

OOO Works ·

On 19 November 2026 we read the robots.txt file of the 1,000 most-visited domains on the web and checked, for each AI crawler, whether the site lets it read the home page. 475 of the domains had a robots.txt we could read. Of those:

You can download the full data (one row per domain) and check any site yourself with our free AI crawler check.

Every crawler, side by side

Share of the 475 sites whose robots.txt keeps each crawler away from the home page, and share that name the crawler at all.

Crawler Run by Used for Blocked Named in robots.txt
GPTBot OpenAI Training 21.5% 26.5%
OAI-SearchBot OpenAI ChatGPT search 12.0% 18.1%
ChatGPT-User OpenAI Pages a ChatGPT user asks about 13.5% 19.6%
ClaudeBot Anthropic Training 22.3% 26.1%
Claude-SearchBot Anthropic Claude search 13.9% 15.4%
Claude-User Anthropic Pages a Claude user asks about 13.9% 14.7%
PerplexityBot Perplexity Perplexity search 17.9% 24.4%
Perplexity-User Perplexity Pages a Perplexity user asks about 15.6% 13.5%
Google-Extended Google Gemini training and grounding 18.1% 22.7%
Applebot-Extended Apple Apple’s AI models 17.7% 18.7%
CCBot Common Crawl Open web archive used to train many models 24.6% 23.8%
Bytespider ByteDance ByteDance’s crawler, widely reported as used for training 26.1% 22.5%
Meta-ExternalAgent Meta Training Meta’s AI models 20.4% 18.7%
Amazonbot Amazon Amazon services, including AI 17.5% 15.4%
Googlebot Google Google Search 3.2% 16.8%
Bingbot Microsoft Bing, Copilot 3.4% 12.8%

A crawler can be blocked without being named: a rule for all crawlers (User-agent: *) applies to every crawler the file doesn’t name separately. That’s why some “blocked” figures are higher than the “named” ones.

What stands out

OpenAI and Anthropic each run separate crawlers for training and for search, and say each robots.txt setting works independently. Perplexity says its crawlers aren’t used for training at all. Many large sites use that split: 45 of the 102 sites that block GPTBot still allow OAI-SearchBot, and 40 of the 106 that block ClaudeBot still allow Claude-SearchBot. They keep their content out of model training but stay eligible to be cited in AI search answers.

PerplexityBot is blocked by 17.9% of sites, against 12.0% for OAI-SearchBot and 13.9% for Claude-SearchBot, although all three do the same job for their assistant: finding pages to quote in answers.

Many rules lag behind the crawlers

124 sites name ClaudeBot, but only 73 name Claude-SearchBot, which Anthropic lists separately for search. 68 still name anthropic-ai, which isn’t one of the three crawlers on Anthropic’s crawler page today. A rule written for one crawler doesn’t cover its newer siblings, so a site that meant to block only training can end up allowing search, or the other way round.

Some sites shut AI search out without naming it

20 sites block all three AI search crawlers without naming any of them, through a rule for all crawlers. 8 of those still let Googlebot in by name. Sites that let in only the search engines they list keep out every new AI assistant by default, whether or not that was the intention.

The most visited sites block the most

17 of the 51 top-100 domains with a readable robots.txt block at least one AI search crawler, including facebook.com, instagram.com, linkedin.com, amazon.com, x.com, pinterest.com and tiktok.com. Large platforms often license their content or keep it for their own products, so blocking is a business decision for them.

What this means for a small business

The sites that block AI crawlers on purpose are mostly large platforms with something to protect. For a small business, the trade usually runs the other way: AI assistants answer questions like “who fixes boilers near me” or “best accountant for freelancers”, and a site they can’t read can’t be named in the answer.

Three things are worth checking on your own site:

  1. Your robots.txt doesn’t block OAI-SearchBot, Claude-SearchBot or PerplexityBot, either by name or through a rule for all crawlers.
  2. Your host, CDN or security plugin doesn’t block AI crawlers. A robots.txt that allows everything doesn’t help if a firewall refuses the request; some CDNs offer a “block AI bots” switch.
  3. Your pages don’t say noindex, which keeps them out of search and AI answers alike.

Our free AI crawler check does all three for a page in under a minute.

How we did it

Limits

Sources for what each crawler does: OpenAI’s crawler documentation, Anthropic’s crawler page, Perplexity’s crawler documentation and Google’s list of common crawlers.

Check your own site. Our free AI crawler check shows, for ChatGPT, Claude, Perplexity, Google's AI answers and Copilot, whether they can read your pages, and what to change if they can't.