Verified crawl accessibility across ChatGPT Search, Perplexity, Google AI Overviews & Claude.
Directives governing LLM training crawlers (GPTBot, ClaudeBot, Meta Llama) from robots.txt.
| User Agent | robots.txt | Status code | Word count / Redirect URL |
|---|
Make sure your content is cited in ChatGPT, Perplexity and Google AI Overviews. Get a Generative Engine Optimization (GEO) audit.
Enter any live URL to scan how search engines, AI crawlers, and LLM bots see your content.
Select whether you want to allow AI search engines, block training scrapers, or maximize total search visibility.
View live crawler accessibility scores, inspect individual bot responses, and copy a tailored robots.txt configuration with instant one-click fixes.
As AI search engines like ChatGPT, Perplexity, and Google AI Overviews replace traditional search, being accessible to AI bots is essential to being cited as a source.
Ensure bots from OpenAI, Perplexity, and Anthropic can access your core content to cite your brand in AI search results.
Keep model training scrapers from indexing private content while keeping live search bots allowed.
Detect if Cloudflare, AWS WAF, or robots.txt are unintentionally blocking high-value AI search crawlers.
AI search bots (like ChatGPT-User and PerplexityBot) fetch real-time content to cite in user queries. Model training scrapers (like GPTBot and ClaudeBot) crawl websites to train future AI models.
Yes. You can use robots.txt to allow ChatGPT-User and PerplexityBot while disallowing GPTBot and CCBot. Our tool generates the exact rules for you.
Many Web Application Firewalls (Cloudflare, AWS WAF) or security plugins block automated crawlers by default or require JavaScript challenges that AI bots cannot solve.
To allow an AI search engine, add its specific user-agent (such as User-agent: OAI-SearchBot) followed by Allow: / in your robots.txt file. Our tool provides the exact copy-paste configuration under the Proposed robots.txt card.
No. AI training bots (like GPTBot, ClaudeBot, and CCBot) only harvest content to train foundation models. Blocking them does not impact your standard rankings on Google Search or Bing.
AEO and GEO focus on optimizing your website structure and crawler permissions so that AI search engines (like ChatGPT Search, Perplexity, and Google AI Overviews) accurately cite and recommend your brand in generated answers.