AI Readiness Scanner — Check If ChatGPT, Claude and Perplexity Can Read Your Site
Free & No Signup Required
Check whether GPTBot, ClaudeBot, PerplexityBot, Googlebot and 11 other AI agents can actually reach your page. Tests robots.txt rules and live server responses, then shows exactly what to fix.
Initializing Module
Optimizing assets and preparing high-fidelity workspace...
What is AI Readiness Scanner — Check If ChatGPT, Claude and Perplexity Can Read Your Site?
Most site owners have no idea which AI systems can read their pages. The information is spread across a robots.txt file few people read carefully, a CDN dashboard with a toggle nobody remembers flipping, and a rendering choice made by whoever built the front end.
Those three places disagree more often than not. A robots.txt can welcome every AI crawler while Cloudflare's "Block AI Scrapers" rule quietly returns 403 to all of them. A site can pass every technical check and still ship an empty div to anything that does not run JavaScript.
AI Readiness Scanner checks all three in one pass.
First, robots.txt. The scanner implements the RFC 9309 matching rules properly: the longest matching user-agent token wins, the longest matching path pattern wins within it, and Allow beats Disallow on a tie. That matters, because a naive check reports `User-agent: Google-Extended / Disallow: /` as blocking Googlebot. It does not — those are different product tokens with completely different consequences.
Second, the live response. For every agent the scanner issues a real request carrying that agent's User-Agent and compares the result against a baseline browser request. A 403, a 429, a Cloudflare interstitial or an HTTP 402 pay-per-crawl response all show up here, even when robots.txt says the agent is welcome.
Third, rendering. The scanner reads the HTML your server actually sends, before any script runs, and looks for the signature of an unhydrated single-page app: a framework mount point that arrived empty. If your content only appears after hydration, every AI crawler except Googlebot sees a blank page.
The report separates crawlers that build an index from agents that fetch a page live when a user asks, because blocking one is a very different decision from blocking the other. Each finding comes with the exact rule that caused it and the specific change that fixes it.
Key Features
- 15 AI agents tested, grouped by what a block actually costs you: search indexing, live fetching, or model training
- RFC 9309 robots.txt matching — longest user-agent token, longest path pattern, Allow wins ties
- Live User-Agent probes that catch CDN and WAF blocks your robots.txt knows nothing about
- Detects Cloudflare pay-per-crawl (HTTP 402) and bot-challenge interstitials
- Client-side rendering detection: finds content that exists in the browser but not in the HTML crawlers receive
- Checks structured data, meta robots, X-Robots-Tag, noai signals, sitemap, llms.txt and TTFB
- Every issue includes the exact offending rule and a concrete fix
- States plainly what the method cannot see, instead of implying certainty it does not have
Common Use Cases
- Marketers checking why a well-ranked page never appears in ChatGPT or Perplexity answers
- Developers verifying that a Next.js or React site server-renders content for AI crawlers
- Site owners auditing a robots.txt they inherited and never fully read
- Agencies producing a fast, evidence-backed AI visibility check for a prospect
- Anyone who enabled a CDN bot-protection toggle and wants to know what it actually blocked
This tool runs on our servers
AI Readiness Scanner tells you whether AI assistants can actually read your website. Enter a URL and it checks 15 AI agents — GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Googlebot, bingbot, Applebot-Extended, meta-externalagent, Amazonbot and CCBot — against your robots.txt rules, then sends a live request carrying each agent's User-Agent to see whether your server or CDN refuses it anyway. It also checks whether your content exists in the raw HTML, because no AI crawler except Googlebot executes JavaScript.
How to use it
Enter your URL
Paste any public page address. The scheme is optional — example.com works as well as https://example.com/page.
Run the scan
The scanner fetches your robots.txt, sends a live probe for each of the 15 AI agents, and reads the raw HTML your server returns.
Read the access matrix
Agents are grouped by purpose. Red means blocked — either by a robots.txt rule or by your server refusing the request outright.
Work through the issues
Findings are sorted by severity. Open any one to see the exact rule that caused it and the change that fixes it.
Re-scan after fixing
Results are cached for 24 hours per URL, so allow for that when verifying a change.
FAQ
?Does blocking GPTBot remove me from ChatGPT?
No, and this is the single most common misunderstanding. GPTBot collects data for model training. ChatGPT's search results come from OAI-SearchBot, and ChatGPT-User is what opens a page when someone asks about a specific link. Blocking GPTBot while allowing the other two is a perfectly coherent choice: your content stays out of training but still appears in answers.
?Does Google-Extended affect AI Overviews?
No. Google-Extended controls whether your content is used for Gemini training and grounding. AI Overviews are generated from Google's regular search index, which is built by Googlebot. If you want out of AI Overviews, Google-Extended is not the control that does it.
?Why does the scanner say a bot is blocked when my robots.txt allows it?
Because robots.txt is not the only thing that decides. That result means your server or CDN returned a refusal — a 403, a rate limit, or a bot challenge — to a request carrying that agent's User-Agent. Cloudflare's "Block AI Scrapers and Crawlers" setting is the most common cause, and it overrides whatever your robots.txt says.
?Are the probes sent from the real crawler IP addresses?
No. They come from our servers with the crawler's User-Agent set. That catches every block based on User-Agent, which is the overwhelming majority. It cannot catch blocks based on IP range or ASN — for those you need to look at your own server logs, which is the only place the ground truth lives.
?Do I need an llms.txt file?
Not really, and we would rather say so than sell you a checkbox. No major AI crawler consumes llms.txt today. The scanner reports whether you have one because it costs nothing to check, but it carries a small weight in the score and should not be a priority over server-rendering your content.
?Does being accessible mean I will get cited?
No. Access is necessary, not sufficient. An open, fast, well-marked-up page can still go unmentioned by every assistant, because citation depends on whether your content actually answers the questions people ask. This scanner measures whether the door is open, not whether anyone walks through it.
?Is the scan free?
Yes. It runs on plain HTTP checks with no AI model calls involved, so it costs us almost nothing to run. There is an hourly limit per visitor to keep the tool from being used to hammer other people's sites.
Need a custom tool?
We're constantly adding new tools. Suggest one or report a bug.
Related Tools You Might Need
Robots.txt Generator with AI Shield
Free online tool to create robots.txt files and instantly block AI bots from scraping your content.
Free AGENTS.md / CLAUDE.md Generator
Generate a ready-to-use AGENTS.md or CLAUDE.md project-instructions file for AI coding and writing agents. Pick your project type, answer a short wizard, done.
Free Online Color Blindness Simulator
Simulate various forms of color blindness (Deuteranopia, Protanopia, Tritanopia) on your images. Compare side-by-side or in a grid and download results. 100% local.
Free Online Markdown Table Generator & Editor
Create, edit, and style Markdown tables visually with our interactive editor. Import CSV or paste Markdown, align columns, and copy formatted code.
Free Online Token Counter — GPT-5.4, Claude 4.6, Gemini 3.5
Count tokens for GPT-5.4, Claude 4.6, Gemini 3.5, and Llama 4 instantly. See context window usage, estimated cost, and trim text to model limits — 100% in your browser.
Free Online Word to Markdown Converter — DOCX to MD
Convert Word (.docx) documents to clean Markdown instantly in your browser. Preserves headings, bold, italics, lists, tables, and links. No upload, 100% private.