Crawl Census

Name: Crawl Census
Author: taylorsmithgg

MCP serverMCP

Ask before you fetch: will this domain serve your crawler, refuse it, or charge it?

by taylorsmithggrepository ↗homepage ↗v1.0.0

Is this yours?

This record is unclaimed

Wellknown found it in public sources; nobody has proven control of it yet. Claiming takes one click if the repository is under your GitHub account, or a small file on your domain otherwise. Verified owners get the badge, 15-minute checks, status alerts, edits that outrank crawled data, and a ranking boost.

Agents can do it too: POST https://wellknown.network/api/v1/claims with {"agent":"crawl-census","method":"well_known_file"} — machine-readable steps at claim.json, guide at /docs/claim.

Claim this record

Observed by Wellknown

Evidence

Everything here was measured by our prober or read from a registry. Nothing is self-reported.

30-day availability109/109 checks ok

when	check	result	http	latency	detail
3 h ago	mcp initialize	ok	200	21 ms	server crawl-census 1.0.0 · protocol 2025-06-18 · 7 tools
9 h ago	mcp initialize	ok	200	56 ms	server crawl-census 1.0.0 · protocol 2025-06-18 · 7 tools
16 h ago	mcp initialize	ok	200	19 ms	server crawl-census 1.0.0 · protocol 2025-06-18 · 7 tools
23 h ago	mcp initialize	ok	200	35 ms	server crawl-census 1.0.0 · protocol 2025-06-18 · 7 tools
30 h ago	mcp initialize	ok	200	30 ms	server crawl-census 1.0.0 · protocol 2025-06-18 · 7 tools
2 d ago	mcp initialize	ok	200	30 ms	server crawl-census 1.0.0 · protocol 2025-06-18 · 7 tools
2 d ago	mcp initialize	ok	200	12 ms	server crawl-census 1.0.0 · protocol 2025-06-18 · 7 tools
2 d ago	mcp initialize	ok	200	21 ms	server crawl-census 1.0.0 · protocol 2025-06-18 · 7 tools
2 d ago	mcp initialize	ok	200	44 ms	server crawl-census 1.0.0 · protocol 2025-06-18 · 7 tools
3 d ago	mcp initialize	ok	200	45 ms	server crawl-census 1.0.0 · protocol 2025-06-18 · 7 tools

Tools observed via MCP handshake

scan_site — Run a live AI-accessibility audit of a domain: robots.txt policy for every tracked AI crawler, live user-agent probes, JavaScript-free readability, structured d
site_report — Return the most recent stored audit for a domain without triggering a new scan. Faster and free of load on the target site.
census_stats — Corpus-level statistics: how many measured domains block each AI crawler, mean access score, llms.txt adoption.
crawl_preflight — Decide whether a crawler may fetch a list of domains before spending requests on them. Works for any crawler token, not only the ones this census tracks: an unr
agent_profile — What this census measures and publishes about one AI crawler: how often it is disallowed in robots.txt, how often live requests carrying its user agent are refu
census_facts — Every headline finding from the census as discrete, dated records rather than prose. Each carries its value, unit, denominator, measurement date, the page it co
submit_domains — Queue domains the census has not measured yet so a later crawl_preflight can answer them. This closes the loop crawl_preflight starts: anything it returns as un

Declared by sources

What the publisher says

Attributed to the source that supplied each field. Treated as claims, not facts.

No long description beyond the summary.

kind Official MCP Registry

name Official MCP Registry

repoUrl Official MCP Registry

summary Official MCP Registry

version Official MCP Registry

description Official MCP Registry

homepageUrl Official MCP Registry

publisherName Official MCP Registry

Provenance

Where this record came from

Every source is kept verbatim. Field changes are logged as events.

Official MCP Registryio.github.taylorsmithgg/crawl-census

first seen 1 mo ago · fetched 2 d agoopen ↗

Field history (7 recent events)

1 mo ago · declared · summary → "Ask before you fetch: will this domain serve your crawler, refuse it, or charge it?"
1 mo ago · declared · description → "Ask before you fetch: will this domain serve your crawler, refuse it, or charge it?"
1 mo ago · declared · publisherName → "taylorsmithgg"
1 mo ago · declared · homepageUrl → "https://crawlcensus.com/for-crawlers"
1 mo ago · declared · repoUrl → "https://github.com/taylorsmithgg/crawl-census-client"
1 mo ago · declared · version → "1.0.0"
1 mo ago · declared · protocols → ["mcp"]

This record is unclaimed

Evidence

What the publisher says

Capabilities

Where this record came from