You can find out in about five minutes whether your website is blocking ChatGPT. Open your robots.txt, check the six AI crawler user-agents (GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Google-Extended), and confirm none are disallowed. If one is, the AI answer engines your customers use are being told to skip you.
This is one of the quietest failures we see. A contractor can rank fine in Google, have a fast site, collect reviews, and still be missing from every AI-generated answer because a single line in a config file is turning the crawlers away. It is not a design problem or a content problem, it is an access problem, and access problems never show up in the traffic report. We audit this on every client site before launch, and we find blocked crawlers more often than you would think, usually switched on by a plugin or a security setting nobody remembers touching.
Here is how to find out where you stand, what each crawler does, and how to fix it if you are locked out.
Why would my website block ChatGPT without me knowing?
Because the block is almost never something you did on purpose. It is a default setting, a plugin option, or a security toggle that shipped switched on, and none of them announce themselves. There are three places it hides, and a contractor who has never opened a config file can be blocked by all three at once.
The first is robots.txt, a plain text file at the root of your domain that tells crawlers what they may and may not read. A single Disallow: / under an AI bot's name, or a blanket rule that catches everything, removes you from that engine. The second is Cloudflare. If your site sits behind Cloudflare (a huge share of small-business sites do), its bot-fight-mode and its newer one-click "block AI bots" control can challenge or refuse the exact crawlers that feed ChatGPT and Perplexity. The third is your CMS. Several WordPress SEO and security plugins now ship an AI-blocking option, and on some it defaults to on.
The result is the same in every case: your site loads perfectly for humans and is closed to the machines that decide whether you show up when a customer asks an AI for a recommendation.
What are the AI crawler user-agents I need to allow?
There are six that matter, and each one controls a different door into AI search. Allow all six and you are readable everywhere that counts. Block any one and you disappear from that engine's answers.
| User-agent | Who runs it | What it does | Block it and you lose | |---|---|---|---| | GPTBot | OpenAI | Crawls pages for training and to inform ChatGPT's answers | Your presence in ChatGPT's knowledge of your business | | OAI-SearchBot | OpenAI | Powers ChatGPT Search results and citations | Citations in ChatGPT Search | | ChatGPT-User | OpenAI | Fetches your page live when a user asks ChatGPT about it | The ability for ChatGPT to read your site on demand | | PerplexityBot | Perplexity | Indexes pages for Perplexity's answer engine | Visibility and citations in Perplexity | | ClaudeBot | Anthropic | Crawls pages for Anthropic's Claude | Presence in Claude's answers | | Google-Extended | Google | Controls whether Google uses your content for Gemini and AI Overviews | Inclusion in AI Overviews and Gemini |
A note that trips people up: Google-Extended is not the same as Googlebot. Blocking Google-Extended does not hurt your normal Google search ranking, and allowing it does not change your ranking either. It only governs whether Google may use your pages to generate AI answers. Many "privacy-first" plugin presets block Google-Extended by default, which quietly pulls you out of AI Overviews while leaving classic search untouched, so the owner sees no traffic drop and never suspects it.
How do I check my robots.txt for AI blocking in 5 minutes?
Open your site's robots.txt in a browser and read it for any rule that applies to the six crawlers above. That is the whole check, and here is the exact sequence.
- Open the file. Type your domain followed by
/robots.txtinto any browser, for examplehttps://yoursite.com/robots.txt. Every site serves this file publicly, and you do not need a login to see it. - Scan for the crawler names. Look for any
User-agent:line that namesGPTBot,OAI-SearchBot,ChatGPT-User,PerplexityBot,ClaudeBot, orGoogle-Extended, and check whatDisallowrule sits under it.Disallow: /under any of those names means that crawler is fully blocked. - Check the wildcard rule. Find the
User-agent: *block, which applies to every crawler that is not named specifically. If it containsDisallow: /, you are blocking essentially everything, AI crawlers included. - See it the way the bot sees it. On a Mac or Linux terminal, run
curl -A "GPTBot" https://yoursite.com/robots.txtto fetch the file while identifying as GPTBot. On Windows, PowerShell'scurl.exe -A "GPTBot" https://yoursite.com/robots.txtdoes the same. Compare what comes back to what your browser showed. Some setups serve different rules to bots than to humans, and this is how you catch that. - Test a real page load. Run
curl -A "GPTBot" -I https://yoursite.com/and look at the status code. A200means the crawler can reach your homepage. A403means something (usually Cloudflare or a security plugin) is refusing the bot even if robots.txt looks clean.
If steps 2 through 5 all come back clean, you are open for business in AI search. If any of them turns up a block, keep reading.
What does a correct robots.txt look like?
A correct file explicitly welcomes the AI crawlers instead of staying silent or blocking them. Silence usually works, since most crawlers treat "not mentioned" as "allowed," but explicit Allow lines remove all doubt and make your intent obvious to anyone auditing the file later. Here is a clean, paste-ready block that opens the door to all six:
# AI search crawlers, explicitly allowed
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Google-Extended
Allow: /
# Everyone else
User-agent: *
Allow: /
Sitemap: https://yoursite.com/sitemap.xml
Swap yoursite.com for your real domain and point the sitemap line at your actual sitemap. If your existing robots.txt already has rules you need (say, disallowing an admin path), keep those and just make sure none of them catch the six crawlers above. The order matters less than the absence of a Disallow: / that applies to a bot you want reading your pages.
Cloudflare and WordPress plugins block these by default, so where do I look?
The two places most likely to be blocking you silently are Cloudflare's bot controls and a WordPress plugin's AI setting, and neither one touches your robots.txt, so a clean robots.txt does not mean you are in the clear.
In Cloudflare, go to your domain, then Security, then Bots. Check two things: bot-fight-mode (which challenges automated traffic broadly and can catch AI crawlers), and the newer "Block AI bots" or AI-scraper control, which some plans nudge you to enable. If your goal is to be found in AI search, both of those should be off for the crawlers you want. Cloudflare also lets you write firewall rules that block by user-agent, so if someone set one up to stop scrapers, it may be catching GPTBot along with the bad actors.
In WordPress, the usual suspects are the SEO and security plugins. Check these:
- Yoast SEO, Rank Math, and All in One SEO have added AI-crawler controls, and some presets block Google-Extended or the OpenAI bots by default in the name of protecting your content.
- Wordfence and other security plugins can rate-limit or block crawlers by user-agent, and their default aggressive settings sometimes catch AI bots as "suspicious."
- Cache and firewall plugins occasionally ship their own bot rules that overlap with the above.
Open each one, find its crawler or bot settings, and confirm nothing is blocking the six user-agents. This is the step that catches the blocks a robots.txt review misses, and it is the one most site owners never think to check.
Should I actually want AI crawlers reading my site?
For a local service business, yes, almost without exception. The instinct to block AI crawlers usually comes from a "protect my content" argument, and for a contractor, cleaner, tiler, landscaper, or HVAC company, that argument is backwards. You are not sitting on proprietary research that a model will steal. You are a business that needs to be recommended, and blocking the crawlers is the digital equivalent of taking your name out of the phone book to stop competitors from reading it.
Here is the plain math. When a homeowner asks ChatGPT or Perplexity for the best roofer in their town, the engine builds its answer from the sites it is allowed to read. If GPTBot cannot reach you, you are not a candidate, full stop. Blocking the crawler does not protect anything worth protecting, and it does not remove you from models that already trained on the open web. All it reliably does is delete you from the live answer your next customer is reading. Being cited in that answer is the whole game now, and we broke down how that citation actually gets earned in how to get found in ChatGPT and AI search and in how contractors get cited in Google's AI Overviews. Step one of all of it is simply letting the crawlers in.
What do I do after I confirm the crawlers can read my site?
Getting unblocked is the floor, not the ceiling. Access lets the AI read you; it does not make you the business it names. Once the crawlers can reach your pages, the work shifts to giving them something worth citing: clean LocalBusiness and FAQ schema, answer-first content that responds to the questions people actually ask an AI, consistent name, address, and phone across your listings, and a claim on Bing Places, since ChatGPT routes its web search through Bing. That is the difference between being readable and being recommended, and it is the core of AEO and GEO for small businesses.
If you want the short version of the whole stack, it is this: let the crawlers in, structure your pages so an AI can quote them cleanly, and make sure your business says the exact same thing everywhere it appears online. Do those three and you go from invisible to citable. Once you are unblocked, the honest test of whether it worked is to ask the AI directly: here is how to check if ChatGPT can actually see your business in about ten minutes.
Updated July 2026 with the current AI crawler user-agent list and the Cloudflare and WordPress default-blocking checks.
FAQ
Is my website blocking AI crawlers?
It might be, and you would never know from looking at the site. The three common culprits are a Disallow line in robots.txt that names GPTBot or uses a wildcard, Cloudflare's bot-fight-mode or AI-crawler toggle challenging the bots, and a WordPress SEO or security plugin (Yoast, Rank Math, All in One SEO, Wordfence) that ships an AI-blocking option turned on by default. Pull up yoursite.com/robots.txt in a browser and look for any Disallow line that applies to GPTBot, OAI-SearchBot, ClaudeBot, or PerplexityBot. If you see one, ChatGPT and the others are being told to stay out.
What are the AI crawler user-agents I should allow?
The six that matter most in 2026 are GPTBot (OpenAI's training and answer crawler), OAI-SearchBot (ChatGPT Search), ChatGPT-User (fetches a page when a user asks ChatGPT about it live), PerplexityBot (Perplexity), ClaudeBot (Anthropic's Claude), and Google-Extended (governs whether Google uses your content for Gemini and AI Overviews). Allowing all six in robots.txt is the baseline for being visible in AI answers. Blocking any of them removes you from that engine's results.
How do I check my robots.txt for AI blocking in 5 minutes?
Type yoursite.com/robots.txt into a browser and read it. Look for User-agent lines naming GPTBot, ClaudeBot, PerplexityBot, or a User-agent star followed by Disallow slash, which blocks everything. On a Mac or Linux terminal you can run curl -A GPTBot https://yoursite.com/robots.txt to see exactly what the crawler sees. If you find blocking rules, remove them or replace them with explicit Allow lines, then re-check. The whole audit takes about five minutes.
Does Cloudflare block AI crawlers by default?
It can. Cloudflare added a one-click "Block AI bots" control and its bot-fight-mode challenges automated traffic, and on some plans the AI-crawler block is easy to switch on without realizing it applies to the bots that feed ChatGPT and Perplexity. If your site is behind Cloudflare, check Security, then Bots, and confirm the AI-scraper blocking and bot-fight-mode settings are not shutting out the crawlers you want to reach customers through.
Will blocking AI crawlers protect my content?
It mostly just makes you invisible. Blocking GPTBot or ClaudeBot does not remove content the models were already trained on, and it does not stop determined scrapers, but it does reliably remove you from the live AI-search answers your customers are reading right now. For a local service business trying to get cited when someone asks ChatGPT for the best roofer or cleaner in town, blocking the crawlers is self-sabotage, not protection.
How is blocking AI crawlers different from a Google noindex?
A Google noindex tag removes a page from Google's regular search index. Blocking an AI crawler in robots.txt tells a specific AI bot not to read your pages at all, which is a separate control that most SEO tools do not check. A site can rank fine in Google and still be completely invisible in ChatGPT because a robots.txt line is blocking GPTBot. The two systems do not talk to each other, which is exactly why this slips past so many owners.
The bottom line
Being blocked from AI search is a five-minute problem to diagnose and usually a five-minute problem to fix, but only if you go looking, because nothing on your site tells you it is happening. Open your robots.txt, check the six crawler user-agents, look behind Cloudflare, and audit your WordPress plugins. If any of them is turning the AI bots away, you are handing every ChatGPT and Perplexity recommendation to the competitor down the road who left the door open.
If you would rather have someone confirm it for you, run your site through the free grader at currentdigital.co/grade and it will flag crawler-access and AI-readiness issues along with the rest of your visibility gaps. If you already know you are locked out and want it fixed and your whole AI-search layer built properly, book a call and we will handle the audit, the robots.txt, the schema, and the listings in one pass.