robots.txt and AI crawlers: which bots to allow, and which to block
· 4 min read · MAALI team
Your robots.txt file used to be about one thing: telling search engines which parts of your site not to crawl. Now a dozen AI companies send their own crawlers, and many website owners are asking: should I block them?
The honest answer is "it depends which ones". Blocking every AI bot can quietly stop ChatGPT, Claude or Perplexity from recommending you. Allowing every one means your content can be used to train models. Here's how to decide.
Training bots vs search bots
AI crawlers mostly fall into three groups:
- Training crawlers collect pages to train future AI models. Blocking them keeps your content out of training data. It doesn't change whether today's assistants can find you.
- Search crawlers build the index an assistant searches when someone asks a question. Blocking them can stop that assistant from citing or recommending your site.
- User-initiated fetchers load a page because a person asked the assistant to look at it, like a browser acting for that person.
For most small businesses that want customers, the sensible default is: allow search crawlers, and decide for yourself about training crawlers.
The main AI crawlers
| User agent | Company | What it's for |
|---|---|---|
GPTBot |
OpenAI | Training data for OpenAI models |
OAI-SearchBot |
OpenAI | Search results inside ChatGPT |
ChatGPT-User |
OpenAI | Pages a ChatGPT user asks it to visit |
ClaudeBot |
Anthropic | Training data for Claude models |
Claude-SearchBot |
Anthropic | Search results inside Claude |
Claude-User |
Anthropic | Pages a Claude user asks it to visit |
PerplexityBot |
Perplexity | Perplexity's search index |
Google-Extended |
Whether your content is used for Gemini models (not a separate crawler) | |
Applebot-Extended |
Apple | Whether your content is used for Apple's AI models (not a separate crawler) |
CCBot |
Common Crawl | A public web archive that many AI models are trained on |
Bytespider |
ByteDance | ByteDance's crawler, including AI uses |
meta-externalagent |
Meta | Training and AI features at Meta |
Two details people often get wrong:
- Google-Extended doesn't affect Google Search. Blocking it doesn't remove you from search results or from Google's AI Overviews, which use the normal Googlebot index. It only controls use of your content for Gemini models.
- Blocking a crawler isn't retroactive. Content already collected isn't removed by changing robots.txt today.
Companies rename and add crawlers from time to time, so check each company's own crawler documentation before you rely on a list (including this one).
A robots.txt you can start from
This version allows normal search and AI search, and opts out of AI training:
User-agent: *
Disallow:
# Opt out of AI training
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: meta-externalagent
Disallow: /
Sitemap: https://yourbusiness.com/sitemap.xml
If you'd rather be included in training too (some businesses want AI models to "know" them), simply leave those blocks out. If you want to block everything AI-related, also add OAI-SearchBot, Claude-SearchBot and PerplexityBot, but understand the trade-off: assistants that rely on those crawlers will have less to say about you.
Our free robots.txt generator lets you tick the crawlers you want to block and builds the file for you.
How robots.txt works (the 60-second version)
- It lives at the root:
https://yourbusiness.com/robots.txt. - Each
User-agentline starts a group; theDisallowlines under it apply to that bot.Disallow: /blocks the whole site; an emptyDisallow:allows everything. - A bot follows the most specific group that names it, and falls back to
User-agent: *otherwise. - It's a request, not a lock. Reputable companies follow it; it doesn't stop someone who ignores it. Anything truly private needs a login.
- Always include a
Sitemap:line so search engines find all your pages.
Common mistakes
- Accidentally blocking everything. A leftover
Disallow: /underUser-agent: *from a staging site hides your whole website from search engines. If your traffic suddenly drops, check this first. - Blocking CSS and JavaScript folders. Search engines need them to see your pages as visitors do.
- Using robots.txt to hide a page from Google. A blocked page can still appear in results if other sites link to it. Use a
noindextag instead, and don't block that page in robots.txt, or Google can't see the tag. - Blocking AI search by accident while trying to block training. Read the table above carefully.
Where to edit it
- WordPress: most SEO plugins (Yoast, Rank Math and others) have a robots.txt editor; otherwise upload the file to the site root.
- Shopify: edit the
robots.txt.liquidtheme template (Online Store → Themes → Edit code → Add a new template → robots). - Webflow: Site settings → SEO → robots.txt.
- Squarespace and Wix: both manage robots.txt for you, with limited options. Check their current help pages for AI crawler settings.
Check what bots can see
After any change, open yourbusiness.com/robots.txt to confirm it's live, and use Google Search Console's robots.txt report to check Google can read it. MAALI's own crawler, MaaliBot, follows robots.txt too: here's what it reads and how to opt out.
MAALI's AI Visibility check also reads your robots.txt each time it runs and tells you, for free, if it blocks the crawlers ChatGPT, Perplexity, Claude, Gemini or Google use to answer questions, alongside whether those assistants actually recommend you.
Keep reading
A step-by-step way to see what AI assistants say when your customers ask for what you sell, which competitors they name instead, and how to track it over time.
AI assistants recommend businesses they find on directories, review sites and roundups. How to find the ones that matter, get listed honestly and stay accurate.
The seven most common reasons ChatGPT, Perplexity, Gemini and Claude name a competitor instead of you, how to spot each one, and the fix.
What generative engine optimization (GEO) means for a small business, how AI assistants decide what to recommend, and the practical steps that help.