Skip to content
Centinel AnalyticaCentinel Analytica
Free tool

Is your robots.txt blocking AI crawlers?

Paste your robots.txt below. We'll check it against 15 known AI crawlers and show you which ones can still access your content.

How it works

This tool parses your robots.txt file and checks each AI crawler user agent against your rules. It shows you exactly which crawlers are blocked and which are still allowed to scrape.

Paste your robots.txt

Copy the contents of your robots.txt file (usually at yoursite.com/robots.txt) and paste it into the text box.

We check 15 AI crawlers

We parse your directives against GPTBot, ClaudeBot, Bytespider, and 12 other known AI crawler user agents.

See your exposure

Get a clear report showing which crawlers are blocked and which can still access your site — plus what to do about it.

robots.txt AI Crawler Analyzer | Centinel | Centinel Analytica