Blog

Free AI crawler checkers compared (November 2026)

OOO Works ·

If you want to know whether ChatGPT, Claude, Perplexity or Google’s AI answers can read your website, there are now a dozen free tools that will tell you. They don’t all test the same thing. Some only read your robots.txt file; others actually request your page the way each AI crawler would, which also catches blocks from your host, CDN or firewall.

We make one of these tools, the OOO Works AI crawler check, so read this with that in mind. To keep it fair, everything below about another tool comes from that tool’s own page on 19 November 2026, or from a check we ran on it that day. Where a page doesn’t say whether it does something, the table says not stated, which doesn’t mean it doesn’t. Bot counts are each tool’s own claim. Pages like these change often; if you spot something out of date, tell us through the feedback box on our tool.

What a full check covers

Four things decide whether an AI assistant can read a page:

  1. robots.txt rules for each crawler. The file can allow one crawler and block another.
  2. What the server actually returns to each crawler. A host, CDN or firewall can refuse a crawler whatever robots.txt says. The only way to see this from outside is to request the page under each crawler’s name and compare with a normal browser.
  3. noindex. A page marked noindex is kept out of search results and AI answers that rely on them.
  4. Which kind of crawler is blocked. Each AI company runs separate crawlers for training, for search and for fetching a page when a user asks. Blocking training doesn’t stop you being found in AI search; blocking the search crawler does. A useful report keeps them apart.

Our guide How to check if ChatGPT can see your website explains each of these and how to fix them.

The comparison

Key: robots = reads robots.txt per crawler. Live fetch = requests the page under each crawler’s name. Split = reports training, search and user crawlers separately. New crawlers = Claude-SearchBot, Claude-User, OAI-SearchBot and Perplexity-User, the search and user crawlers added most recently.

Tool Free, no sign-up Robots Live fetch noindex Split Crawlers New crawlers Fix advice
OOO Works Yes (10 checks an hour) Yes Yes, against a browser and a neutral control request Yes Yes 20 All four Yes, with robots.txt lines to copy
TechnicalSEO.com AI Bot Access Tester (Merkle) Yes Yes Yes not stated No 35 All four not stated
Knowatoa AI Search Console Yes not stated Yes not stated No 25 OAI-SearchBot, Perplexity-User not stated
Known Agents robots.txt checker (formerly Dark Visitors) Yes Yes No sign of it not stated Yes, by agent type 2,000+ not verified Yes, ranked
365i AI Crawler Checker Yes Yes Yes, with a neutral control request Yes Yes 14 All four Yes
Layer3Labs AI Crawler Checker Yes Yes Yes, against a browser not stated Yes 11 All four Partly (robots.txt lines to copy)
Trakkr AI Robots.txt Checker Yes Yes No (robots.txt only) not stated Yes 94 All four Yes (writes the lines)
GEO Toolbox AI Crawler Checker Yes Yes No (robots.txt only) not stated Yes 34 not verified Shows the blocking line
Exploding Topics robots.txt checker (Semrush) Free; sign-up not stated Yes not clear from the page not stated Yes 17 All four Limited
Backlinko robots.txt checker (Semrush) Yes (3 checks a day) Yes not clear from the page not stated Yes 19 All four Yes, ranked
Rank Prompt robots.txt checker Yes Yes No sign of it not stated not verified 19 not verified Partly
CrawlCheck Yes (paid report extra) Yes Yes, 8 crawlers Paid report Yes 60+ (8 live) Three of four Yes

What stands out

Which one to use

Where ours falls short

Update, 19 November 2026: since this post first went up we added seven crawlers (Applebot, Amazon’s three, DuckAssistBot and Mistral’s two), a neutral control request like 365i’s, an llms.txt check, and robots.txt lines to copy. Those were the gaps this comparison found in our own tool.

What we do differently: we read robots.txt as each crawler receives it (some servers send different files to different crawlers), compare each crawler’s result with a normal browser, and say “can’t test from outside” when a site checks real crawlers by their IP address, instead of guessing.

Run the free check on your site →

Check your own site. Our free AI crawler check shows, for ChatGPT, Claude, Perplexity, Google's AI answers and Copilot, whether they can read your pages, and what to change if they can't.