robots.txt for AI Bots: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Others
Contents
The robots.txt file serves as the main tool for managing how automated crawlers access website pages. With the advent of generative systems, artificial intelligence developers divided their bots into two categories: real-time search indexing bots and data collection bots for pre-training neural networks. Proper configuration of this file allows a company to gain visitors from conversational systems while retaining control over the use of proprietary materials.
In this practical guide, we will analyze the current list of bots from leading tech companies in detail, review three ready-made file configuration options, and point out typical mistakes that block business visibility.
Why AI bots need access and what is the difference between training and search
To make informed decisions about server configuration, a business needs to understand the functional separation between the tasks of different bots. Crawlers that visit your website on behalf of artificial intelligence labs pursue two fundamentally different goals.
The first goal is fast, real-time information retrieval. Bots like OAI-SearchBot or PerplexityBot crawl web pages similarly to traditional search engines. They download text, analyze current offers, and generate answers for users with active, clickable citations pointing to the original sources. Allowing access for search bots directly affects whether your company receives organic traffic and direct visits from chats.
The second goal is collecting massive training datasets. Bots like GPTBot, ClaudeBot, or CCBot download content to train next-generation foundation language models. This data dissolves into the numerical weights of the neural network, is not accompanied by a direct link during generation, and produces no immediate visits to your website.
The decision to open or block content for training crawlers is a strategic business decision for every website owner. There is no pressure here: if you consider your original articles, documentation, or analytics to be proprietary intellectual property, you have every right to restrict access for training scrapers, giving the green light only to search agents.
Full table of AI bots and search crawlers
The table below brings together confirmed crawlers, their developers, their purpose, and the consequences of blocking access via the Disallow directive.
| Crawler (User-agent) | Developer | Bot purpose | What happens when blocked |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Indexing for ChatGPT search | Your website will not appear in ChatGPT answers with links |
| GPTBot | OpenAI | Data collection for training GPT models | Content will not enter OpenAI training datasets |
| ChatGPT-User | OpenAI | Following a specific link from a user | The model will not be able to open the page upon a user's request |
| Claude-SearchBot | Anthropic | Search crawl for Anthropic services | Your website will disappear from Claude search results |
| ClaudeBot | Anthropic | Collecting training data for Claude models | Texts will not be included in Anthropic training datasets |
| Claude-User | Anthropic | Loading a URL upon a direct request in chat | The model will show an access error when a link is sent |
| PerplexityBot | Perplexity | Indexing for the Perplexity search engine | Your website will stop being cited in Perplexity answers |
| Perplexity-User | Perplexity | Checking a page upon a user request | The service will not be able to analyze the specified URL |
| Googlebot | Main crawler for search and AI Overviews | Your website will drop out of Google Search and AI Overviews blocks | |
| Google-Extended | Data collection for training Gemini models | Data is not collected for Gemini, and Google Search is not affected | |
| Bingbot | Microsoft | Search indexing for Bing and Copilot | Your website will lose positions in Bing and Copilot results |
| Applebot-Extended | Apple | Content collection for training Apple AI | Data is excluded from Apple training algorithms |
| CCBot | Common Crawl | Building an open web archive | Texts will not end up in the publicly available training archive |
| YandexBot | Yandex | Indexing for Yandex search | Pages will drop out of the Yandex search engine |
Three ready-made robots.txt configuration scenarios
Depending on your company's position on generative content, you can choose one of three standard configurations.
Scenario 1: Keep your website completely open to all search engines and neural networks
This option suits companies interested in the widest possible distribution of information about their products, services, and brand across all digital environments without restrictions.
User-agent: *
Allow: /
Sitemap: https://yourdomain.com/sitemap.xml
In this configuration, any legitimate bot (including Googlebot, YandexBot, OAI-SearchBot, and training crawlers) gets unrestricted access to public sections of the website.
Scenario 2: Allow search, but block data collection for training
An optimal solution for service companies, content media, and expert blogs. You maintain a full presence across all search engines and receive visits with clickable links from ChatGPT search and Perplexity, without handing over your original content to training archives.
User-agent: *
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
Sitemap: https://yourdomain.com/sitemap.xml
With this setup, training bots are blocked, while crawlers that generate real-time links continue scanning the website.
Scenario 3: Completely block your website from specialized AI systems
If your business has decided to completely exclude content from the ecosystems of independent artificial intelligence labs, retaining presence solely in classic search engines Google and Yandex:
User-agent: *
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: Claude-User
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Perplexity-User
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
Sitemap: https://yourdomain.com/sitemap.xml
Warning: with this configuration, ChatGPT and Perplexity users will not see links to your resource, even if search is active in the conversation.
Limitations and technical specifics of robots.txt
When administering the robots.txt file, remember its fundamental characteristics:
- The robots.txt file represents a public agreement and a request, not a software wall. Legitimate bots from large corporations strictly adhere to these rules. However, dubious scripts, spam scrapers, and illicit database harvesters may completely ignore the file. If a page must be physically protected from prying eyes, use mandatory password authorization, access tokens, or IP address restrictions.
- Applying changes takes time. Search crawlers cache a copy of the robots.txt file to avoid requesting it before every single URL. According to official OpenAI data, after uploading a new version of the file to the server, the company's crawlers take about 24 hours to update their local cache and apply the updated directives.
Common mistakes when configuring the file
In practice, website indexing issues with AI bots most often stem from minor technical oversights:
- Accidental complete blocking. The Disallow: / directive inside the generic User-agent: * section completely blocks the entire resource from all search engines in the world. This error often happens when copying templates.
- Blocking layout asset files. If the file blocks paths to folders containing CSS stylesheets and JavaScript scripts, modern search engine crawlers cannot correctly render the page and understand its actual content.
- Forgotten test server (staging) settings. When moving a website from a staging domain to production, webmasters often forget to update the robots.txt file, leaving the live website closed to crawling for weeks and months.
- Crawler blocking at the WAF or Cloudflare firewall level. Often, the robots.txt file is configured perfectly, but a DDoS protection system or the hosting web firewall treats frequent requests from OAI-SearchBot or PerplexityBot as malicious activity and drops the network connection with a 403 code. You need to check server logs and add trusted bots to security whitelists.
Frequently asked questions
If you block GPTBot, will your website stop appearing in ChatGPT search?
No, blocking GPTBot only affects whether your materials are used to train future OpenAI models. Searching and generating links in ChatGPT conversations is handled by a separate bot, OAI-SearchBot. If OAI-SearchBot is allowed, your website still has a chance to appear in responses with citations.
How quickly do bots respond to opening a previously blocked section?
The timing depends on how frequently the crawler visits your website. After removing the restriction in the robots.txt file, the OpenAI crawler needs about 24 hours to update its instructions, after which the bot resumes crawling pages according to the general search indexing queue.
Is it mandatory to specify Allow directives if everything is open by default?
In the Robots Exclusion Standard, all paths that are not explicitly forbidden by a Disallow directive are considered allowed. However, explicitly specifying the Allow: / directive for specific search agents helps avoid ambiguity when there are complex restrictions for individual subdirectories.
Why is the CCBot agent included in the disallow list?
The CCBot crawler belongs to the non-profit organization Common Crawl, which regularly collects giant archives of the open web. These archives are public, and many research teams and companies use them to train their systems. Blocking CCBot prevents your pages from being swept into these public datasets on a massive scale.
Does robots.txt help hide your website from unauthorized text copying?
No, the file is not designed to protect against bad-faith scraping. Any third-party developer can configure their own script without adhering to the robots.txt protocol. For reliable protection of trade secrets and sensitive information, strictly closed perimeters with mandatory user authentication are used.
