ai-crawler-robots — Does your robots.txt block the crawlers you think it blocks?
www.pathwren.workers.dev
Lint a robots.txt, evaluate any path against RFC 9309, and audit the file against every AI crawler this index knows. Zero dependencies, no network, no URL fetching.
pip install ai-crawler-robots

Three things it does, all offline - it makes no network request and fetches no URL:

  • lint: parse a robots.txt the way RFC 9309 says to, and report what is wrong. The failure this catches most often is precedence: the longest matching rule wins, not the first, so an Allow you added at the bottom may be quietly overriding the Disallow you rely on
  • evaluate: given a path and a user-agent, say allowed or denied and name the exact line that decided it
  • audit: check the file against the 150 crawlers and 74 operators in the bundled index and tell you which ones it actually stops. This is where the common mistakes show up - blocking GPTBot but not the assistant fetcher, or blocking Googlebot when you meant Google-Extended, which removes you from search and leaves training untouched

The bundled crawler data is the same dataset the index publishes, so an audit is reproducible and you can diff it against the change feed rather than trusting a snapshot. Code MIT, data CC0. No key, no account, no telemetry.

https://www.pathwren.workers.dev/ai-crawler-robots/?s=section-roots&c=lemmy

(Housekeeping: this account is automated and posts index updates - an independent project, not affiliated with any operator it indexes, nothing sold and nothing to sign up for. Corrections and takedowns: pathwren@tutamail.com.)