Free AI crawler checkers compared (November 2026)
If you want to know whether ChatGPT, Claude, Perplexity or Google’s AI answers can read your website, there are now a dozen free tools that will tell you. They don’t all test the same thing. Some only read your robots.txt file; others actually request your page the way each AI crawler would, which also catches blocks from your host, CDN or firewall.
We make one of these tools, the OOO Works AI crawler check, so read this with that in mind. To keep it fair, everything below about another tool comes from that tool’s own page on 19 November 2026, or from a check we ran on it that day. Where a page doesn’t say whether it does something, the table says not stated, which doesn’t mean it doesn’t. Bot counts are each tool’s own claim. Pages like these change often; if you spot something out of date, tell us through the feedback box on our tool.
What a full check covers
Four things decide whether an AI assistant can read a page:
- robots.txt rules for each crawler. The file can allow one crawler and block another.
- What the server actually returns to each crawler. A host, CDN or firewall can refuse a crawler whatever robots.txt says. The only way to see this from outside is to request the page under each crawler’s name and compare with a normal browser.
- noindex. A page marked noindex is kept out of search results and AI answers that rely on them.
- Which kind of crawler is blocked. Each AI company runs separate crawlers for training, for search and for fetching a page when a user asks. Blocking training doesn’t stop you being found in AI search; blocking the search crawler does. A useful report keeps them apart.
Our guide How to check if ChatGPT can see your website explains each of these and how to fix them.
The comparison
Key: robots = reads robots.txt per crawler. Live fetch = requests the page under each crawler’s name. Split = reports training, search and user crawlers separately. New crawlers = Claude-SearchBot, Claude-User, OAI-SearchBot and Perplexity-User, the search and user crawlers added most recently.
| Tool | Free, no sign-up | Robots | Live fetch | noindex | Split | Crawlers | New crawlers | Fix advice |
|---|---|---|---|---|---|---|---|---|
| OOO Works | Yes (10 checks an hour) | Yes | Yes, against a browser and a neutral control request | Yes | Yes | 20 | All four | Yes, with robots.txt lines to copy |
| TechnicalSEO.com AI Bot Access Tester (Merkle) | Yes | Yes | Yes | not stated | No | 35 | All four | not stated |
| Knowatoa AI Search Console | Yes | not stated | Yes | not stated | No | 25 | OAI-SearchBot, Perplexity-User | not stated |
| Known Agents robots.txt checker (formerly Dark Visitors) | Yes | Yes | No sign of it | not stated | Yes, by agent type | 2,000+ | not verified | Yes, ranked |
| 365i AI Crawler Checker | Yes | Yes | Yes, with a neutral control request | Yes | Yes | 14 | All four | Yes |
| Layer3Labs AI Crawler Checker | Yes | Yes | Yes, against a browser | not stated | Yes | 11 | All four | Partly (robots.txt lines to copy) |
| Trakkr AI Robots.txt Checker | Yes | Yes | No (robots.txt only) | not stated | Yes | 94 | All four | Yes (writes the lines) |
| GEO Toolbox AI Crawler Checker | Yes | Yes | No (robots.txt only) | not stated | Yes | 34 | not verified | Shows the blocking line |
| Exploding Topics robots.txt checker (Semrush) | Free; sign-up not stated | Yes | not clear from the page | not stated | Yes | 17 | All four | Limited |
| Backlinko robots.txt checker (Semrush) | Yes (3 checks a day) | Yes | not clear from the page | not stated | Yes | 19 | All four | Yes, ranked |
| Rank Prompt robots.txt checker | Yes | Yes | No sign of it | not stated | not verified | 19 | not verified | Partly |
| CrawlCheck | Yes (paid report extra) | Yes | Yes, 8 crawlers | Paid report | Yes | 60+ (8 live) | Three of four | Yes |
What stands out
- Robots.txt-only tools can’t see firewall blocks. Trakkr says so plainly: “Robots.txt asks; it doesn’t prove who visited.” If your host or CDN blocks AI crawlers, a robots.txt-only check will tell you everything is fine. TechnicalSEO.com, Knowatoa, 365i, Layer3Labs, CrawlCheck and ours request the page for real.
- Coverage of the newer crawlers varies. Several tools now list Claude-SearchBot and Claude-User, the crawlers behind Claude’s search and its fetches for users. Knowatoa’s list didn’t include them on the day we looked.
- Big lists aren’t the same as better answers. Known Agents tracks over 2,000 bots and Trakkr 94, which is useful if you want to manage scrapers. For the question “can AI assistants find and quote my page”, a dozen crawlers from the main assistants matter most.
- llms.txt is rarely checked. CrawlCheck’s free scan checks it, and ours now does too. It’s optional: robots.txt and your server decide whether assistants can read you, llms.txt doesn’t.
Which one to use
- One page, and you want to know if anything at all is blocking AI assistants: a tool that does a live fetch and a noindex check: 365i or ours.
- Many pages at once: TechnicalSEO.com takes up to 100 addresses per run and exports the results.
- Writing robots.txt rules: Trakkr, Layer3Labs and ours give you lines to paste.
- Managing a long tail of scrapers and bots: Known Agents.
- From an AI agent or a script: ours has a free JSON API and an MCP server, described in its OpenAPI file. We didn’t see an API for the check itself on the other tools’ pages.
Where ours falls short
- Fewer crawlers: 20, against 34 to 94 on several others. We cover the search, user and training crawlers of ChatGPT, Claude, Perplexity, Google, Bing, Apple, Amazon (Alexa), DuckDuckGo, Mistral and Common Crawl, not every bot.
- One page per check, 10 checks an hour. Fine for a small business; not for auditing a large site.
Update, 19 November 2026: since this post first went up we added seven crawlers (Applebot, Amazon’s three, DuckAssistBot and Mistral’s two), a neutral control request like 365i’s, an llms.txt check, and robots.txt lines to copy. Those were the gaps this comparison found in our own tool.
What we do differently: we read robots.txt as each crawler receives it (some servers send different files to different crawlers), compare each crawler’s result with a normal browser, and say “can’t test from outside” when a site checks real crawlers by their IP address, instead of guessing.