AI BOT CHECKER

Testing Tool

Check whether known AI bots can crawl, read and use your web pages. The tool checks the robots.txt file and the HTTP status code for up to 10 URLs.

COMMON QUESTIONS

What exactly does this tool check?
The tool checks two things: first, whether the URL is blocked in your robots.txt for the specific bot (Disallow). Second, whether the server returns a successful status code (200 OK) to the bot or denies access (e.g. 403 Forbidden).
Does the tool also check JavaScript content?
No. This tool only checks the access permission at the HTTP level. It performs no rendering. That means: we don't see whether the bot can actually "see" or render your JavaScript-generated content (e.g. texts in a single page application). That would require a full render check.
Why am I shown 'BLOCKED'?
That can be due to your robots.txt (Disallow rule) or a firewall (WAF) that blocks known bot IPs or user agents. Many hosting providers or security plugins block AI bots by default.
Which robots rules (user agents) apply to what?

To control specific bots, use these user agents in your robots.txt:

  • GPTBot: OpenAI crawler (training)
  • ChatGPT-User: live browsing of ChatGPT
  • OAI-SearchBot: SearchGPT indexing
  • PerplexityBot: Perplexity AI
  • ClaudeBot: Anthropic / Claude
  • Google-Extended: Google Gemini (training)
  • CCBot: Common Crawl (basis for many LLMs)

Example:

User-agent: GPTBot
Disallow: /