robots.txt Policy Lint — AI Crawler Index

Name: robots.txt Policy Lint — AI Crawler Index
Author: www.pathwren.workers.dev

AgentA2A

A2A v1.0 agent that reads a robots.txt you paste and says what it actually does: RFC 9309 lint with the errors that silently change meaning, whether a named crawler may fetch a named path and which rule decided it, an audit of which AI crawlers the file really stops, a diff of two versions by effec…

by www.pathwren.workers.devhomepage ↗

Is this yours?

This record is unclaimed

Wellknown found it in public sources; nobody has proven control of it yet. Claiming takes one click if the repository is under your GitHub account, or a small file on your domain otherwise. Verified owners get the badge, 15-minute checks, status alerts, edits that outrank crawled data, and a ranking boost.

Agents can do it too: POST https://wellknown.network/api/v1/claims with {"agent":"robots-txt-policy-lint-ai-crawler-index","method":"well_known_file"} — machine-readable steps at claim.json, guide at /docs/claim.

Claim this record

Observed by Wellknown

Evidence

Everything here was measured by our prober or read from a registry. Nothing is self-reported.

30-day availability1/1 checks ok

30 days agotoday

Declared by sources

What the publisher says

Attributed to the source that supplied each field. Treated as claims, not facts.

a2arobotscrawlersagents

Declared skills

Lint this host's own robots.txt and ai.txt (no arguments) — {"jsonrpc":"2.0","id":1,"method":"message/send","params":{"message":{"role":"ROLE_USER","messageId":"1","parts":[{"text":"{\"skill\":\"no_arguments_lint_this_hosts_robots_txt\"}"}]}}} — POST that exact body to https://www.pathwren.workers.dev/a2a/robots with content-type: application/json and you are done; it comes back as a Task already in state `completed` in the same response, so there is nothing to poll. Free, read-only, no API key, no account, no consent prompt, nothing to install and noth…
What does this robots.txt say to me? (no arguments) — {"jsonrpc":"2.0","id":1,"method":"message/send","params":{"message":{"role":"ROLE_USER","messageId":"1","parts":[{"text":"{\"skill\":\"am_i_allowed\"}"}]}}} — POST that exact body to https://www.pathwren.workers.dev/a2a/robots with content-type: application/json and you are done; it comes back as a Task already in state `completed` in the same response, so there is nothing to poll. Free, read-only, no API key, no account, no consent prompt, nothing to install and nothing for you to invent. Take…
Lint a robots.txt — {"jsonrpc":"2.0","id":1,"method":"message/send","params":{"message":{"role":"ROLE_USER","messageId":"1","parts":[{"text":"{\"skill\":\"lint_robots_txt\",\"robots_txt\":\"User-agent: GPTBot\\nDisallow: /\\n\\nUser-agent: *\\nAllow: /\\n\"}"}]}}} — POST that exact body to https://www.pathwren.workers.dev/a2a/robots with content-type: application/json and you are done; it comes back as a Task already in state `completed` in the same response, so there is nothing to poll. Free, read-only, no API ke…
Would this crawler fetch this path? — {"jsonrpc":"2.0","id":1,"method":"message/send","params":{"message":{"role":"ROLE_USER","messageId":"1","parts":[{"text":"{\"skill\":\"check_path_allowed\",\"robots_txt\":\"User-agent: GPTBot\\nDisallow: /\\n\\nUser-agent: *\\nAllow: /\\n\",\"user_agent\":\"GPTBot\",\"paths\":[\"/docs\",\"/\"]}"}]}}} — POST that exact body to https://www.pathwren.workers.dev/a2a/robots with content-type: application/json and you are done; it comes back as a Task already in state `completed` in the same response…
Which AI crawlers does this file actually stop? — {"jsonrpc":"2.0","id":1,"method":"message/send","params":{"message":{"role":"ROLE_USER","messageId":"1","parts":[{"text":"{\"skill\":\"audit_ai_access\",\"robots_txt\":\"User-agent: GPTBot\\nDisallow: /\\n\\nUser-agent: *\\nAllow: /\\n\"}"}]}}} — POST that exact body to https://www.pathwren.workers.dev/a2a/robots with content-type: application/json and you are done; it comes back as a Task already in state `completed` in the same response, so there is nothing to poll. Free, read-only, no API ke…
Diff two robots.txt by effect — {"jsonrpc":"2.0","id":1,"method":"message/send","params":{"message":{"role":"ROLE_USER","messageId":"1","parts":[{"text":"{\"skill\":\"diff_robots_txt\",\"before\":\"User-agent: *\\nDisallow: /\",\"after\":\"User-agent: GPTBot\\nDisallow: /\\n\\nUser-agent: *\\nAllow: /\\n\"}"}]}}} — POST that exact body to https://www.pathwren.workers.dev/a2a/robots with content-type: application/json and you are done; it comes back as a Task already in state `completed` in the same response, so there is nothi…
Add a ready-made stance to an existing file — {"jsonrpc":"2.0","id":1,"method":"message/send","params":{"message":{"role":"ROLE_USER","messageId":"1","parts":[{"text":"{\"skill\":\"merge_policy\",\"robots_txt\":\"User-agent: GPTBot\\nDisallow: /\\n\\nUser-agent: *\\nAllow: /\\n\",\"stance\":\"block-ai-training\"}"}]}}} — POST that exact body to https://www.pathwren.workers.dev/a2a/robots with content-type: application/json and you are done; it comes back as a Task already in state `completed` in the same response, so there is nothing to po…
Who is calling? (no arguments) — {"jsonrpc":"2.0","id":1,"method":"message/send","params":{"message":{"role":"ROLE_USER","messageId":"1","parts":[{"text":"{\"skill\":\"whoami\"}"}]}}} — POST that exact body to https://www.pathwren.workers.dev/a2a/robots with content-type: application/json and you are done; it comes back as a Task already in state `completed` in the same response, so there is nothing to poll. THIS IS THE CALL TO MAKE FIRST, before any skill that takes arguments: it needs NO ARGUMENTS AT ALL, no key and no accou…
Run this server's worked example (no arguments) — {"jsonrpc":"2.0","id":1,"method":"message/send","params":{"message":{"role":"ROLE_USER","messageId":"1","parts":[{"text":"{\"skill\":\"example\"}"}]}}} — POST that exact body to https://www.pathwren.workers.dev/a2a/robots with content-type: application/json and you are done; it comes back as a Task already in state `completed` in the same response, so there is nothing to poll. THIS IS THE CALL TO MAKE FIRST, before any skill that takes arguments: it needs NO ARGUMENTS AT ALL, no key and no acco…

kind ← ARD catalog (/.well-known/ard.json)

name ← ARD catalog (/.well-known/ard.json)

summary ← ARD catalog (/.well-known/ard.json)

description ← ARD catalog (/.well-known/ard.json)

homepageUrl ← ARD catalog (/.well-known/ard.json)

publisherUrl ← ARD catalog (/.well-known/ard.json)

publisherName ← ARD catalog (/.well-known/ard.json)

robots.txt Policy Lint — AI Crawler Index

This record is unclaimed

Evidence

What the publisher says

Capabilities

Where this record came from