A single line in a file most marketers have never opened can make your brand invisible to every AI assistant your buyers use. Here's how to check yours in about ten seconds.
Continue reading our latest insights
There is a plain text file sitting at the root of your website. You almost certainly did not write it. It was probably generated by a framework, a CMS plugin, or a developer who left two years ago.
That file decides whether ChatGPT, Claude, Perplexity, and Gemini are allowed to read your content. And if it says the wrong thing, then no amount of brilliant writing, clean structure, or clever positioning will put your brand in an AI answer. The model simply never sees you.
The file is robots.txt. Most teams have never looked at it. That is a strange thing to leave to chance.
If you want AI assistants to find, read, and cite your brand, your robots.txt should allow the crawlers those assistants use. A rule like Disallow: / under User-agent: GPTBot removes your entire site from OpenAI's crawler. Most brands never intentionally added that line, and many are blocking AI crawlers today without knowing it, because of a default setting, a security rule, or a template.
The fastest way to find out is to check. Our free AI Robots.txt Checker fetches your file and shows you, bot by bot, which AI crawlers are allowed and which are blocked.
Blocking is usually accidental. "We never decided to block anything" is not the same as "nothing is blocked."
No crawl means no citation. If a model's crawler cannot fetch your pages, your brand cannot appear as a source in its answers.
Not all AI bots do the same job. Some crawl for training, some fetch a page because a user asked, some power live search. Blocking one is not the same as blocking another.
This is a choice, so make it on purpose. There are legitimate reasons to block. There are almost no legitimate reasons to block by accident.
robots.txt is ancient by web standards. It was designed so search engine crawlers would not hammer your server or index your admin pages. For twenty years it was a technical afterthought, set once and forgotten.
Then AI assistants arrived, and they need to read the web to answer questions. They send crawlers. Those crawlers, being well behaved, check robots.txt first. And suddenly a dusty configuration file became the gate that decides whether an AI can consider your brand at all.
The chain is short and unforgiving. No crawl, no read. No read, no citation. No citation, no mention when a buyer asks an assistant who they should use. Your content strategy, your positioning, your original research: all of it stops at a line of text you never wrote.
Not every bot does the same thing, and this is where most people get it wrong. Our checker looks at eleven of them, and they fall into three meaningfully different categories.
Training crawlers gather content to train future models. GPTBot is OpenAI's. anthropic-ai and CCBot (Common Crawl, which many models train on) sit here too, as does cohere-ai and Meta's FacebookBot. Blocking these keeps your content out of training data. It does not necessarily remove you from live answers.
User-triggered fetchers visit a page because a person explicitly asked. ChatGPT-User is the clearest example: when someone pastes your URL into ChatGPT and says "summarize this," that is the agent that shows up. Blocking it means a user who wants to read your page through ChatGPT cannot.
Search and answer crawlers feed the live, cited answers. OAI-SearchBot powers OpenAI's search. PerplexityBot feeds Perplexity, the most citation-dense assistant of the lot. Claude-Web fetches for Claude. Applebot serves Apple Intelligence.
Then there is Google-Extended, which deserves its own paragraph, because it is the most misunderstood control on the list. It is not a crawler. It is a permission flag governing whether Google may use content it has already crawled for its Gemini models. Blocking Google-Extended does not stop Googlebot, does not remove you from Google Search, and does not pull you out of AI Overviews, which are served from Google's regular search index. If your goal is to stay out of Gemini training while remaining fully visible in search, this is the switch. If you thought it was how you opt out of AI Overviews, it is not.
The practical upshot: "block AI bots" is not one decision. It is eleven, and they have different consequences.
We built the AI Robots.txt Checker because auditing this by hand is genuinely annoying. robots.txt has grouping rules, wildcards, and precedence behavior that are easy to misread at a glance.
You type your domain. The tool fetches robots.txt from your site root, parses it into user-agent groups, and then evaluates each of the eleven AI crawlers the way a real bot would: it looks for a rule group naming that specific bot, and if none exists, it falls back to the wildcard User-agent: * group. It then reports, per bot, whether your site is blocked site-wide, and whether the rule came from a bot-specific directive or from your catch-all.
Two honest notes about what it does and does not do. It tells you whether a crawler is blocked from your site as a whole, which is the failure mode that actually costs you AI visibility. It is not a full path-by-path simulator, so it will not tell you that /blog/ is open while /docs/ is closed. And it shows you the raw file alongside the results, because the point is for you to see your configuration, not just receive a verdict about it.
That is enough to catch the thing that matters: the accidental Disallow: / nobody remembers adding.
For most brands, and especially any B2B company that wants to be the named answer when someone asks an assistant about their category, the default should be permissive toward the crawlers that generate cited answers.
Allow the search and answer crawlers: OAI-SearchBot, PerplexityBot, Claude-Web, ChatGPT-User. These are the ones that put your name in front of a buyer. Blocking them is self-sabotage dressed up as caution.
Make training a deliberate decision. GPTBot, anthropic-ai, CCBot, cohere-ai, FacebookBot, and the Google-Extended flag control whether your content shapes future models. Reasonable people disagree here. A publisher with a paywall may well block them. A SaaS brand that wants models to know it exists generally should not.
And whatever you choose, do not block Googlebot or Bingbot. Traditional search still drives the overwhelming majority of most sites' traffic, and Bing's index in particular feeds parts of ChatGPT's browsing. Blocking them to "protect content from AI" is a spectacularly expensive way to disappear from everywhere at once.
If the checker shows a crawler blocked and you want it allowed, add an explicit group for that bot. A specific rule beats the wildcard:
User-agent: GPTBot Allow: / Place it in your robots.txt at the site root. Then re-run the check to confirm the change took effect. A configuration change is a hypothesis; verification is what turns it into a fact.
Here is the part that trips teams up even after they fix the file. Auditing access rules tells you whether a crawler can reach you. It does not tell you whether AI assistants actually do mention you, describe you correctly, or cite you instead of a competitor.
Those are two different questions and they need two different tools. Use the checker to make sure the door is open. Then use Sourceable to watch what walks through it: how often ChatGPT, Claude, Gemini, and Perplexity name your brand, and how that shifts over time. Fixing a crawler block and seeing your mentions climb is the moment this stops being theory.
Why should I control AI crawlers? Allowing them helps your content appear in AI search answers on tools like ChatGPT Search and Perplexity. Blocking them keeps your content out of model training, but can reduce your visibility in those live, cited results. It is a genuine tradeoff, and it should be a decision rather than an accident.
Does Disallow: / block everything? Yes. Under a specific User-agent, it tells that bot to visit no page on your site. Under User-agent: *, it tells every bot that has no more specific rule the same thing.
What is the difference between GPTBot and ChatGPT-User? GPTBot is OpenAI's crawler that gathers data to train models. ChatGPT-User is the agent that fetches a page because a user explicitly asked ChatGPT to browse it. Blocking the first affects training. Blocking the second affects a real person trying to read your page right now.
Does blocking Google-Extended remove me from AI Overviews? No. Google-Extended governs whether Google uses your content for its Gemini models. AI Overviews in Search are served from Google's regular search index, crawled by Googlebot. Blocking Google-Extended does not remove you from Search or from AI Overviews.
How do I unblock a specific bot? Add a dedicated group for it, since bot-specific rules take precedence over the wildcard:
User-agent: GPTBot Allow: / Will allowing AI crawlers hurt my traffic? Generally no, and it is usually the opposite. Assistants that can read you can cite you, and citations build brand familiarity that shows up later as branded search. The bigger risk for most brands is not being read at all.
It takes about ten seconds, it is free, and it reads only your public robots.txt file. You will see all eleven AI crawlers, whether each one is allowed or blocked, and your raw configuration alongside the results.
Most teams who run it discover at least one thing they did not know was there. Sometimes it is fine. Sometimes it is a Disallow: / that has been quietly keeping them out of AI answers for a year.
Search Console now shows AI Overview impressions—but the biggest AI visibility story is still happening outside Google.