Free tools › AI robots.txt generator

Write robots.txt rules for AI crawlers

Choose which AI crawlers may read your site: the search crawlers that let assistants find and quote you, the fetchers that open a page when someone asks, and the crawlers that collect pages for training. Paste your current robots.txt and we add the new rules without deleting any of yours. Everything runs in your browser.

1. Choose

"Be found, no training" lets AI search and user crawlers in and blocks training crawlers.

Loading the crawler list…

4. Your new robots.txt

    

Search, user and training crawlers

AI companies send different crawlers for different jobs, and each one can be allowed or blocked on its own:

Googlebot, Bingbot and Applebot are ordinary search crawlers. Blocking them takes you out of Google Search, Bing or Apple's search features, not just their AI answers. Google says a page must be indexed and eligible to be shown in Google Search with a snippet to appear as a link in AI Overviews or AI Mode. The generator warns you if you block one of them.

How your rules are kept

After you upload it

Save the file as robots.txt at the root of your site, then run the AI crawler check to confirm each assistant sees what you chose. Crawlers read robots.txt again from time to time; OpenAI says its systems can take about 24 hours to adjust. If your site is behind Cloudflare, its "Block AI bots" setting can block crawlers whatever robots.txt says.

Questions

Will blocking GPTBot remove my site from ChatGPT?

No. OpenAI uses OAI-SearchBot to show websites in ChatGPT's search answers. GPTBot only signals whether your pages may be used to train its models.

Does blocking Google-Extended hurt my Google ranking?

No. Google says Google-Extended doesn't affect a site's inclusion in Google Search and isn't used as a ranking signal. It only controls whether Google may use your pages for Gemini's training and grounding.

Can I block ChatGPT-User and Perplexity-User?

You can add the rule, but it may not be followed. OpenAI says robots.txt rules may not apply to ChatGPT-User because a person asked for the page, Perplexity says Perplexity-User generally ignores robots.txt, and Amazon says Amzn-User may not follow all robots.txt directives.

Why does the generator copy my User-agent: * rules?

A crawler follows only the group that names it most specifically. Google's rules say groups for a named crawler and the * group aren't combined. So when we add a group for a crawler, we copy your * rules into it, so the pages you keep private stay private.

Does this tool send my robots.txt anywhere?

No. Your rules are put together in your browser. Only the list of crawlers is loaded from our server.

Where does the file go?

At the root of your site, for example yourbusiness.com/robots.txt. Many website builders and SEO plugins have a robots.txt setting; some hosted builders don't let you change it.

Related free tools

Why this matters: in our study of AI recommendations for local businesses, ChatGPT, Claude and Gemini rarely agreed on which businesses to name.

Who made this

OOO Works, which is building done-for-you services for small businesses worldwide, including OOO Visibility: it tracks whether AI assistants name a business for the questions its customers ask, and fixes what keeps it out. OOO Works is opening soon; nothing is on sale yet. About OOO Works.