prerender[ OBSERVATORY ]
[ Guide ] [ robots.txt ]

robots.txt for AI answer engines

AI companies send different machines for training, for search and for fetching a page someone asked about. Each one has its own robots.txt token, so you can keep your pages out of training and still be cited in AI answers.

[ Training ]

Collects pages to train models

Blocking these keeps your content out of future training sets. It doesn't remove you from AI search.

GPTBot · ClaudeBot · CCBot
Google-Extended (token) · Applebot-Extended (token)
[ Search ]

Builds the index answers cite

Blocking these removes your pages from that engine's answers, the way blocking Googlebot removes you from Google.

OAI-SearchBot · Claude-SearchBot · PerplexityBot
[ On demand ]

Fetches a page a person asked for

These act for one person in real time. OpenAI and Perplexity say robots.txt may not apply to them.

ChatGPT-User · Claude-User · Perplexity-User

Three common files

Each example is a complete file. Pick the one that matches what you want each kind of machine to do.

Allow every machine
User-agent: *
Allow: /
Search yes, training no
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

User-agent: *
Allow: /
Block AI, keep search engines
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: CCBot
User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /

Before you block

robots.txt is a request, not a lock. The operators listed here say they honor it, and you can check that against your own logs with the verification guide. A machine that ignores robots.txt usually also skips the checks that would identify it.