AI Crawler Index — MCP servers

Name: AI Crawler Index — MCP servers
Author: www.pathwren.workers.dev

MCP serverMCP

Is this yours?

This record is unclaimed

Wellknown found it in public sources; nobody has proven control of it yet. Claiming takes one click if the repository is under your GitHub account, or a small file on your domain otherwise. Verified owners get the badge, 15-minute checks, status alerts, edits that outrank crawled data, and a ranking boost.

Agents can do it too: POST https://wellknown.network/api/v1/claims with {"agent":"ai-crawler-index-mcp-servers","method":"well_known_file"} — machine-readable steps at claim.json, guide at /docs/claim.

Claim this record

Observed by Wellknown

Evidence

Everything here was measured by our prober or read from a registry. Nothing is self-reported.

30-day availability1/1 checks ok

30 days agotoday

Tools observed via MCP handshake

whoami — {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — POST that exact body to https://www.pathwren.workers.dev/mcp and you
example — {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"example","arguments":{}}} — POST that exact body to https://www.pathwren.workers.dev/mcp and you
classify_user_agent — FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — Identify the crawler behind a raw User-Age
lookup_crawler — FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — The full index record for one crawler: ope
list_crawlers — FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — The index, filtered or whole: each row say
generate_robots_txt — FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — A ready-to-paste robots.txt block for one
is_verified_crawler_ip — FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — Test an IPv4 or IPv6 address against every
whats_changed — FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — Freshness and change state: when the index
changes_since — FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — Only what changed since your last call: pr

Declared by sources

What the publisher says

Attributed to the source that supplied each field. Treated as claims, not facts.

Five remote MCP servers (Streamable HTTP, JSON-RPC 2.0, no auth) on one host. The endpoint to connect to is POST https://www.pathwren.workers.dev/c/ard/mcp — this URL is the server card that describes it, not the endpoint. /mcp identifies one crawler at a time, /mcp/triage takes a pasted access log and returns who crawled you, /mcp/doctor grades which agent-discovery documents a host actually serves, /mcp/robots reads a robots.txt and says which AI crawlers it really stops, and /mcp/netcheck answers whether an address belongs to a declared crawler range. The card names every endpoint and mirrors the record the official MCP registry holds.

mcpcrawlerslog-analysisdiscovery

kind ← ARD catalog (/.well-known/ard.json)

name ← ARD catalog (/.well-known/ard.json)

summary ← ARD catalog (/.well-known/ard.json)

description ← ARD catalog (/.well-known/ard.json)

homepageUrl ← ARD catalog (/.well-known/ard.json)

publisherUrl ← ARD catalog (/.well-known/ard.json)

publisherName ← ARD catalog (/.well-known/ard.json)

Provenance

Where this record came from

Every source is kept verbatim. Field changes are logged as events.

ARD catalog (/.well-known/ard.json)urn:air:www.pathwren.workers.dev:mcp:ai-crawler-index

first seen 7 h ago · fetched 7 h agoopen ↗

Field history (6 recent events)

7 h ago · declared · summary → "Five remote MCP servers (Streamable HTTP, JSON-RPC 2.0, no auth) on one host. The endpoint to connect to is POST https:
7 h ago · declared · description → "Five remote MCP servers (Streamable HTTP, JSON-RPC 2.0, no auth) on one host. The endpoint to connect to is POST https:
7 h ago · declared · publisherName → "www.pathwren.workers.dev"
7 h ago · declared · publisherUrl → "https://www.pathwren.workers.dev"
7 h ago · declared · homepageUrl → "https://www.pathwren.workers.dev/c/ard/mcp.html"
7 h ago · declared · protocols → ["mcp"]

This record is unclaimed

Evidence

What the publisher says

Capabilities

Where this record came from