Free tools / AI Crawler Checker
Can AI crawlers read your site?
Check your robots.txt for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended, whether your content shows without JavaScript, and whether a firewall blocks AI bots.
By Raphaël Aubry, founder of Howseen AI · Updated
Free, no signup. Your result shows right here.
How it works
The checker reads your robots.txt the way each AI crawler does, fetches your page as raw HTML and with a GPTBot user agent, and flags any rule, firewall or noindex that blocks AI.
“A robots.txt file tells search engine crawlers which URLs the crawler can access on your site.”
Google Search Central, Introduction to robots.txt
Enter your site
Any public URL, no signup.
We read it like an AI crawler
Your robots.txt rules for 10 AI and search bots, your raw HTML, and a request with a GPTBot user agent.
Fix what blocks AI
Allowed or blocked, bot by bot, with the exact rule and how to fix it.
FAQ
Short answers to the questions people ask most before they run the ai crawler checker, based on what the tool actually checks.
Should I block GPTBot in robots.txt?+
Usually no. GPTBot is OpenAI's training crawler, while OAI-SearchBot is the one behind ChatGPT search, so blocking GPTBot alone does not remove you from ChatGPT answers. A blanket rule that blocks every bot is the real risk.
Does Google-Extended control AI Overviews?+
No. Google-Extended only controls use of your content for Gemini training. AI Overviews and AI Mode use standard Googlebot access.
Why does content without JavaScript matter?+
Most AI crawlers fetch the raw HTML and do not run JavaScript. If your page builds its text in the browser, they may see an empty page.
What is llms.txt?+
A short markdown file at your site root that lists your key pages for AI agents. It is optional and not a ranking factor, but it helps agents find the right pages. You can generate one with our LLMs.txt generator.
Go further: GPTBot, ClaudeBot, PerplexityBot: the robots.txt guide
More free tools
Each free tool answers one question about how AI assistants see your brand. Together they cover visibility, crawling, llms.txt, searches and schema.
Sources
The official documentation this tool follows, from the companies that run the AI crawlers and the standards it checks against.
