Blog

How to check if ChatGPT can see your website (and fix it)

OOO Works ·

When someone asks ChatGPT, Claude, Perplexity or Google’s AI answers for a business like yours, the assistant can only name and link sites its crawlers are allowed to read. Three things decide that, and each takes a few minutes to check:

  1. robots.txt, the file that tells crawlers what they may read.
  2. Your host, CDN or firewall, which can refuse crawlers whatever robots.txt says.
  3. noindex, a tag that keeps a page out of search and AI answers.

The quickest way is our free AI crawler check: enter your address and it runs all three for every major assistant. Below is how to check and fix each one yourself.

Which crawlers matter

Each assistant finds pages with a search crawler. That’s the one to allow if you want to be named and linked:

Assistant Search crawler to allow
ChatGPT OAI-SearchBot
Claude Claude-SearchBot
Perplexity PerplexityBot
Google AI Overviews and AI Mode Googlebot (the same crawler as Google Search)
Microsoft Copilot Bingbot

OpenAI says sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers, though they can still appear as navigational links. Google says its AI features use pages that are indexed in Google Search and eligible to show a snippet; there’s no separate AI crawler to allow.

The companies also run training crawlers (OpenAI’s GPTBot, Anthropic’s ClaudeBot, Google-Extended for Gemini). Those decide whether your pages may be used to train AI models. Blocking them is your choice and doesn’t keep you out of AI search answers: OpenAI, for example, says each of its robots.txt settings works independently.

1. Check your robots.txt

Open yourdomain.com/robots.txt in a browser. Look for:

If you find either and want to be found by AI assistants, add this block to the file (it can sit next to any rules for training crawlers):

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /

On WordPress, robots.txt is usually managed by an SEO plugin (Yoast, Rank Math and others have a robots.txt editor). On Shopify, Wix and Squarespace it’s generated for you; Shopify lets you edit it through a robots.txt.liquid theme template.

A missing robots.txt isn’t a problem: crawlers treat it as “everything allowed”.

2. Check your host, CDN and firewall

robots.txt only asks. A firewall decides. If your site sits behind a CDN or a security service, it may refuse AI crawlers even when robots.txt allows them. Common places to look:

This is the hardest part to test yourself, because real crawlers come from their own IP addresses and firewalls can tell them apart from an imitation. Our check compares what your site serves a normal browser with what it serves each crawler’s name, and says plainly when a site treats imitations differently and can’t be tested from outside.

3. Check for noindex

A page with a noindex instruction asks search engines to leave it out. Google’s AI features only use pages that can be indexed, so a noindex page won’t appear in AI Overviews or AI Mode either. In your browser, view the page source and search for noindex. It can appear in a tag like <meta name="robots" content="noindex"> or in an X-Robots-Tag header sent by the server.

Website builders often add noindex to draft pages, password-protected pages, or the whole site while it’s “under construction”. If a page should be found, remove it.

After you’ve fixed it

Crawlers don’t come back instantly. Perplexity says changes to its robots.txt settings can take up to 24 hours to take effect, and search indexes take longer to fill in new pages. Two things speed it up:

Being readable is the first step, not the last. Assistants still choose which businesses to name. That depends on whether your pages answer the questions your customers ask, and on what other sites say about you. Our study of which top sites block AI crawlers shows how differently large sites handle this.

Sources: OpenAI’s crawler documentation, Anthropic’s crawler page, Perplexity’s crawler documentation, Google: AI features and your website.

Check your own site. Our free AI crawler check shows, for ChatGPT, Claude, Perplexity, Google's AI answers and Copilot, whether they can read your pages, and what to change if they can't.