[ Training ]
Collects pages to train models
Blocking these keeps your content out of future training sets. It doesn't remove you from AI search.
[ Search ]
Builds the index answers cite
Blocking these removes your pages from that engine's answers, the way blocking Googlebot removes you from Google.
OAI-SearchBot · Claude-SearchBot · PerplexityBot
[ On demand ]
Fetches a page a person asked for
These act for one person in real time. OpenAI and Perplexity say robots.txt may not apply to them.
ChatGPT-User · Claude-User · Perplexity-User
Three common files
Each example is a complete file. Pick the one that matches what you want each kind of machine to do.
Allow every machine
User-agent: *Allow: /
Search yes, training no
User-agent: GPTBotUser-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /
User-agent: *
Allow: /
Block AI, keep search engines
User-agent: GPTBotUser-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: CCBot
User-agent: Google-Extended
Disallow: /
User-agent: *
Allow: /
Before you block
robots.txt is a request, not a lock. The operators listed here say they honor it, and you can check that against your own logs with the verification guide. A machine that ignores robots.txt usually also skips the checks that would identify it.