CrawlCheck

Name: CrawlCheck
Author: emmanuelorta

MCP serverMCP

Verification layer for the agentic web: a signed answer about any site before an agent acts.

by emmanuelortarepository ↗homepage ↗v1.1.0

Is this yours?

This record is unclaimed

Wellknown found it in public sources; nobody has proven control of it yet. Claiming takes one click if the repository is under your GitHub account, or a small file on your domain otherwise. Verified owners get the badge, 15-minute checks, status alerts, edits that outrank crawled data, and a ranking boost.

Agents can do it too: POST https://wellknown.network/api/v1/claims with {"agent":"crawlcheck","method":"well_known_file"} — machine-readable steps at claim.json, guide at /docs/claim.

Claim this record

Observed by Wellknown

Evidence

Everything here was measured by our prober or read from a registry. Nothing is self-reported.

30-day availability41/41 checks ok

when	check	result	http	latency	detail
2 h ago	mcp initialize	ok	200	746 ms	server crawlcheck 1.1.0 · protocol 2025-06-18 · 25 tools
9 h ago	mcp initialize	ok	200	42 ms	server crawlcheck 1.1.0 · protocol 2025-06-18 · 25 tools
16 h ago	mcp initialize	ok	200	23 ms	server crawlcheck 1.1.0 · protocol 2025-06-18 · 25 tools
23 h ago	mcp initialize	ok	200	357 ms	server crawlcheck 1.1.0 · protocol 2025-06-18 · 25 tools
29 h ago	mcp initialize	ok	200	616 ms	server crawlcheck 1.1.0 · protocol 2025-06-18 · 25 tools
2 d ago	mcp initialize	ok	200	661 ms	server crawlcheck 1.1.0 · protocol 2025-06-18 · 25 tools
2 d ago	mcp initialize	ok	200	593 ms	server crawlcheck 1.1.0 · protocol 2025-06-18 · 25 tools
2 d ago	mcp initialize	ok	200	811 ms	server crawlcheck 1.1.0 · protocol 2025-06-18 · 25 tools
2 d ago	mcp initialize	ok	200	779 ms	server crawlcheck 1.1.0 · protocol 2025-06-18 · 25 tools
3 d ago	mcp initialize	ok	200	40 ms	server crawlcheck 1.1.0 · protocol 2025-06-18 · 25 tools

Tools observed via MCP handshake

preflight — Call BEFORE reading, citing, connecting to or transacting with a domain. Returns a signed decision: allow, warn, require_confirmation, block or unsupported, wit
best_source — Call when choosing WHICH business or site to cite, recommend or call. Give domains (2-10 candidate sites you found) or trade and city for a measured market. Ret
resolve_domain — Call before fetching from, citing, connecting to or transacting with a domain. Returns one signed answer: what the domain claims, what it actually serves each c
field_check — Call before using a phone number, address, opening hours or service area you hold for a business. One signed verdict per business field. field=phone|address|hou
scan_domain — Scan a public domain as several crawler identities and return the graded record: grade, AI-visibility score, per-section scores, findings with evidence, and whe
list_verified_capabilities — What a domain declares agents can do (OpenAPI operations, MCP tools, A2A skills, API catalog entries), each classed by safety (read_only_public up to financial)
mcp_servers — Which remote MCP servers the official MCP Registry lists on a domain, and what each endpoint actually answered when CrawlCheck sent the MCP handshake (initializ
verify_crawler_log — Given raw web-server access-log lines, decide for each line that names a crawler (GPTBot, ClaudeBot, Googlebot, PerplexityBot...) whether the source IP falls in
corpus_state — Coverage of CrawlCheck's public dataset: how many distinct domains have been measured and how they split by platform, rendering, size, language and kind, with t
resolve_robots — Resolve every named answer engine, search index and training crawler against a domain's robots.txt the way a crawler does: most-specific group only, longest mat
draft_llms — Draft an llms.txt from the domain's own homepage and up to 25 declared pages, using their titles and descriptions. A draft for the owner to cut down, not a publ
fix_robots — The domain's served robots.txt, corrected: * group rules copied into named groups that lacked them (shadowing), a Sitemap line added when missing, a minimal rep
fix_entitymap — A starter entitymap.json built from the domain's latest scan record: name, phone, coordinates and declared service areas as entities with SERVES relations. Fiel
video_check — One YouTube video against the video anchor model: metadata, chapters, captions and storyboard where observed, twelve checks each naming what it reads.
video_channel — Every upload on a YouTube channel, read 50 at a time, sorted by views. The upload list is free; the per-check tally needs a Watch licence and is not offered her
video_site — The site half of video: homepage plus up to 60 sitemap pages read for YouTube embeds, facades and page-builder widgets, VideoObject nodes and their required pro
registry_lookup — Whether a domain is in CrawlCheck's registry and what it serves: llms.txt, agents.md, a media kit at /.well-known/media-kit.json, a reciprocity-tested entity gr
crawler_path — Where each crawler's path into a domain dies, as an observed data flow: edge decision, robots.txt as a file, robots.txt rules, then the page, JSON-LD, sitemap,
public_counts — The figures CrawlCheck quotes about itself, read from the source: sites_measured, domains_in_corpus, crawler_visits (a rolling window), sections, sections_score
telemetry — Verified-crawler traffic observed at crawlcheck.io itself: which named agents arrived, how many claims were confirmed against operator ranges, how many were for
fix_mostly_code — A fix plan for PAGE_IS_MOSTLY_CODE: the homepage's measured bytes by category, the largest blocks with what to do with each, ordered steps with the text ratio a
fix_stale_cache — A fix plan for STALE_CACHE_SERVED: the homepage fetched now, its Age, the cache layers that name themselves in the headers in purge order (innermost first), the
fix_machine_chain — A fix plan for MACHINE_CHAIN_HEAVY: the files a crawler reads before the page and the 404s absent agent files return, then measured reductions (llms.txt to an i
explain_finding — Why CrawlCheck decided a finding: the rule and its revision, the exact fetches it compared (identity, status, bytes, sha256), when the condition held on the sit
machine_record — The unified machine record for one observation: subject, grade, findings (the top one in full without a licence), capabilities with verification, and the eviden

CrawlCheck

This record is unclaimed

Evidence

What the publisher says

Capabilities

Where this record came from