Free tool · AI Crawler Checker

AI Crawler Checker: can AI bots read your website?

Check if ChatGPT, Claude, Perplexity and Google’s AI bots are allowed to read your site, and see the exact rule that blocks them.

example.com/robots.txtFix needed

GPTBotAllowed

OAI-SearchBotAllowed

ClaudeBotAllowed

PerplexityBotBlocked

Google-ExtendedAllowed

CCBotAllowed

robots.txtUser-agent: PerplexityBot
Disallow: /

What your result means

Each bot gets a clear answer

Allowed

This bot can read the page under your robots.txt rules.

Blocked

A robots.txt rule stops this bot. You see the rule that does it.

Check the firewall

robots.txt says yes, but a firewall or CDN setting may still stop the bot.

No rule

No rule names this bot, so it follows your general rules for all bots.

How to use

Step by step

1
Type your website
Your home page is a good place to start.
2
The tool reads your robots.txt
Rules are checked for each major AI bot.
3
See the answer per bot
Allowed or blocked, with the rule behind it.
4
Decide what you want
Choose which AI uses you are happy with.
5
Fix and check again
Change robots.txt, then run the check once more.

Guide

Know the bots before you block them

Search bots

OAI-SearchBot, Claude-SearchBot and PerplexityBot help your site show up and get linked in AI search answers.

Training bots

GPTBot and ClaudeBot collect content that may be used to train AI models. Blocking them is a choice about training, not about search.

User-triggered bots

ChatGPT-User, Claude-User and Perplexity-User visit when a person asks the AI to open a page. OpenAI and Perplexity say robots.txt may not apply to these visits.

Google-Extended

A robots.txt token, not a separate bot. It controls use for Gemini, and Google says it does not affect Google Search or rankings.

Bot	Company	Main job
OAI-SearchBot	OpenAI	Search in ChatGPT
GPTBot	OpenAI	Training
ChatGPT-User	OpenAI	Visits a user asks for
Claude-SearchBot	Anthropic	Search in Claude
ClaudeBot	Anthropic	Training
PerplexityBot	Perplexity	Search in Perplexity
Google-Extended	Google	Gemini use, not Search
CCBot	Common Crawl	Open web dataset

Why it matters

Why it matters for a Thai business

Hotels and tours

Travellers ask AI where to stay and what to do. If search bots are blocked, your pages cannot be read for those answers.

Hosting defaults

Some hosting, security plugins and CDN settings block bots without telling you. A check shows it in plain words.

Thai and English pages

Rules apply per site and subdomain, so your /th/ pages and any other subdomain need checking too.

When to use

When to use it

Before you start any AEO work

After you change hosting, CDN or a security plugin

When AI tools never mention your business

After a developer edits robots.txt

Who it’s for

Who it’s for, and who it isn’t

Good for

Business owners who want AI to find them
Marketers starting AEO
Developers checking a site launch

Not for

Checking if AI actually mentions you (use the AI Visibility Checker)
Removing content that AI has already collected

Common mistakes

Mistakes to avoid

Blocking every AI bot at once

Fix: Block training bots if you like, but keep search bots open if you want to show up in AI answers.

Thinking Google-Extended affects Google Search

Fix: Google says it does not. It only controls Gemini use.

Forgetting subdomains

Fix: Each subdomain has its own robots.txt. Check every one you use.

Trusting robots.txt alone

Fix: A firewall or bot protection setting can still block bots. Check both.

Blocking by IP address

Fix: Anthropic notes IP blocking is not reliable. Use robots.txt rules.

Worked example

Example: a Phuket hotel

Before

A security plugin added “Disallow: /” for every unknown bot. AI search bots could not read the rooms page.

→

After

The owner allowed OAI-SearchBot, Claude-SearchBot and PerplexityBot, kept GPTBot blocked, and checked again.

Illustrative example.

Honest limits

What this tool can’t tell you

It reads rules. It cannot promise that any AI will mention or link to you.
It cannot see every firewall setting from outside.
Bots that act for a user may not follow robots.txt.
Rules change. Check again after any change to your site.

FAQ

Questions, answered simply

What is an AI crawler?

An AI crawler is a bot that reads web pages for an AI company, for search answers, for model training, or when a user asks the AI to open a page.

Should I block GPTBot?

That is your choice. OpenAI says GPTBot is used for training, and blocking it does not stop OAI-SearchBot, which is used for search in ChatGPT.

Does blocking AI bots hurt my Google ranking?

Blocking Google-Extended does not. Google says it does not affect inclusion in Google Search and is not a ranking signal. Do not block Googlebot.

What is the difference between OAI-SearchBot and GPTBot?

OAI-SearchBot helps sites show up in ChatGPT search answers. GPTBot collects content that may be used for training. OpenAI treats each setting separately.

Do AI bots respect robots.txt?

Search and training bots from OpenAI, Anthropic and Perplexity say they follow robots.txt. Visits a user asks for, such as ChatGPT-User or Perplexity-User, may not.

My robots.txt allows bots. Why am I still blocked?

A firewall, CDN or bot protection setting can stop bots before robots.txt is read. Check those settings with your host.

Is robots.txt the same as llms.txt?

No. robots.txt tells bots what they may read. llms.txt is an optional file that describes your site. Google says llms.txt is not needed for Google Search.

Does this work for Thai websites?

Yes. Rules work the same for Thai and English pages. Check each subdomain you use.

Is it free?

Yes. The AI Crawler Checker is free and you do not need a card.

Related tools