WellknownBot

Our crawler and prober identify themselves with the user agent WellknownBot/0.1 (+https://wellknown.network/bot). They exist to keep the index accurate: to read public registries and well-known documents, and to check whether declared endpoints respond.

What it fetches

• Public registry APIs (npm, the official MCP registry) and public list files on GitHub.
• /.well-known/agent-card.json and /.well-known/agent.json on hosts that indexed records point at.
• Endpoint liveness: a HEAD or GET of a declared URL, or the protocol's own handshake (an MCP initialize plus the read-only tools/list). It never invokes a tool, never sends untrusted input, never authenticates, and never follows more than three redirects.
• /.well-known/wellknown-verify.txt and DNS TXT records only when a publisher starts an ownership claim.

Limits

Responses are capped at 128 KB, requests time out after 8 seconds, one request is in flight per host at a time, and every endpoint is checked at most every 15 minutes (claimed records) or every 6–24 hours (everything else), with exponential backoff up to 7 days when a host fails.

Opting out

Return 403 or 410 to our user agent, or disallow it in robots.txt; we honour both on the next cycle and mark the record accordingly. To remove a record entirely, claim it and use the dashboard, or write to hello@wellknown.network.