How to check if ChatGPT can see your website (and fix it)
When someone asks ChatGPT, Claude, Perplexity or Google’s AI answers for a business like yours, the assistant can only name and link sites its crawlers are allowed to read. Three things decide that, and each takes a few minutes to check:
- robots.txt, the file that tells crawlers what they may read.
- Your host, CDN or firewall, which can refuse crawlers whatever robots.txt says.
- noindex, a tag that keeps a page out of search and AI answers.
The quickest way is our free AI crawler check: enter your address and it runs all three for every major assistant. Below is how to check and fix each one yourself.
Which crawlers matter
Each assistant finds pages with a search crawler. That’s the one to allow if you want to be named and linked:
| Assistant | Search crawler to allow |
|---|---|
| ChatGPT | OAI-SearchBot |
| Claude | Claude-SearchBot |
| Perplexity | PerplexityBot |
| Google AI Overviews and AI Mode | Googlebot (the same crawler as Google Search) |
| Microsoft Copilot | Bingbot |
OpenAI says sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. Google says its AI features use pages that are indexed in Google Search and eligible to show a snippet; there’s no separate AI crawler to allow.
The companies also run training crawlers (OpenAI’s GPTBot, Anthropic’s ClaudeBot, Google-Extended for Gemini). Those decide whether your pages may be used to train AI models. Blocking them is your choice and doesn’t keep you out of AI search answers: OpenAI, for example, says each of its robots.txt settings works independently.
1. Check your robots.txt
Open yourdomain.com/robots.txt in a browser. Look for:
- A block for all crawlers,
User-agent: *followed byDisallow: /. That shuts out every crawler the file doesn’t name separately, including AI search crawlers, even if Googlebot is let in by name. - A block naming an AI search crawler, such as
User-agent: OAI-SearchBotfollowed byDisallow: /.
If you find either and want to be found by AI assistants, add this block to the file (it can sit next to any rules for training crawlers):
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
On WordPress, robots.txt is usually managed by an SEO plugin (Yoast, Rank Math and others have a robots.txt editor). On Shopify, Wix and Squarespace it’s generated for you; Shopify lets you edit it through a robots.txt.liquid theme template.
A missing robots.txt isn’t a problem: crawlers treat it as “everything allowed”.
2. Check your host, CDN and firewall
robots.txt only asks. A firewall decides. If your site sits behind a CDN or a security service, it may refuse AI crawlers even when robots.txt allows them. Common places to look:
- Cloudflare: in the bot settings (Security → Settings → Bot traffic at the time of writing), check whether “Block AI bots” is on. If Cloudflare manages your robots.txt, its AI setting changes the file too.
- Security plugins and web application firewalls: look for rules that block “bad bots” or unfamiliar user agents.
- Your host’s control panel: some hosts offer AI-bot blocking as a one-click option.
This is the hardest part to test yourself, because real crawlers come from their own IP addresses and firewalls can tell them apart from an imitation. Our check compares what your site serves a normal browser with what it serves each crawler’s name, and says plainly when a site treats imitations differently and can’t be tested from outside.
3. Check for noindex
A page with a noindex instruction asks search engines to leave it out. Google’s AI features only use pages that can be indexed, so a noindex page won’t appear in AI Overviews or AI Mode either. In your browser, view the page source and search for noindex. It can appear in a tag like <meta name="robots" content="noindex"> or in an X-Robots-Tag header sent by the server.
Website builders often add noindex to draft pages, password-protected pages, or the whole site while it’s “under construction”. If a page should be found, remove it.
After you’ve fixed it
Crawlers don’t come back instantly. Perplexity says changes to its robots.txt settings can take up to 24 hours to take effect, and search indexes take longer to fill in new pages. Two things speed it up:
- Submit your sitemap in Google Search Console and Bing Webmaster Tools. Copilot is built on Bing, and studies find that the sources ChatGPT search cites overlap heavily with Bing’s top results.
- Run the AI crawler check again to confirm every assistant can read the page.
Being readable is the first step, not the last. Assistants still choose which businesses to name. That depends on whether your pages answer the questions your customers ask, and on what other sites say about you. Our study of which top sites block AI crawlers shows how differently large sites handle this.
Sources: OpenAI’s crawler documentation, Anthropic’s crawler page, Perplexity’s crawler documentation, Google: AI features and your website.