RobotsGate

Name: RobotsGate
Author: dev.workers.mike-tusa.robotsgate

MCP serverMCP

Generate, validate and check robots.txt rules for AI crawlers. Free; optional Pro key.

by dev.workers.mike-tusa.robotsgatehomepage ↗v0.3.2

Is this yours?

This record is unclaimed

Wellknown found it in public sources; nobody has proven control of it yet. Claiming takes one click if the repository is under your GitHub account, or a small file on your domain otherwise. Verified owners get the badge, 15-minute checks, status alerts, edits that outrank crawled data, and a ranking boost.

Agents can do it too: POST https://wellknown.network/api/v1/claims with {"agent":"robotsgate","method":"well_known_file"} — machine-readable steps at claim.json, guide at /docs/claim.

Claim this record

Observed by Wellknown

Evidence

Everything here was measured by our prober or read from a registry. Nothing is self-reported.

30-day availability44/44 checks ok

when	check	result	http	latency	detail
2 h ago	mcp initialize	ok	200	83 ms	server robotsgate 0.3.3 · protocol 2025-06-18 · 4 tools
9 h ago	mcp initialize	ok	200	76 ms	server robotsgate 0.3.3 · protocol 2025-06-18 · 4 tools
16 h ago	mcp initialize	ok	200	78 ms	server robotsgate 0.3.3 · protocol 2025-06-18 · 4 tools
23 h ago	mcp initialize	ok	200	175 ms	server robotsgate 0.3.3 · protocol 2025-06-18 · 4 tools
31 h ago	mcp initialize	ok	200	48 ms	server robotsgate 0.3.3 · protocol 2025-06-18 · 4 tools
2 d ago	mcp initialize	ok	200	103 ms	server robotsgate 0.3.3 · protocol 2025-06-18 · 4 tools
2 d ago	mcp initialize	ok	200	100 ms	server robotsgate 0.3.3 · protocol 2025-06-18 · 4 tools
2 d ago	mcp initialize	ok	200	47 ms	server robotsgate 0.3.3 · protocol 2025-06-18 · 4 tools
3 d ago	mcp initialize	ok	200	161 ms	server robotsgate 0.3.3 · protocol 2025-06-18 · 4 tools
3 d ago	mcp initialize	ok	200	122 ms	server robotsgate 0.3.3 · protocol 2025-06-18 · 4 tools

Tools observed via MCP handshake

list_crawlers — Return RobotsGate's AI crawler registry: each crawler's robots.txt token, operator, category (training, dataset, search, user_fetch or other), whether the opera
generate_robots — Build a robots.txt from per-category choices (allow, block or omit for training, dataset, search, user_fetch or other), optional per-token overrides, a default
validate_robots — Validate robots.txt text against RFC 9309 and report errors, warnings and, for one URL path, which registry AI crawlers are allowed or blocked and by which rule
check_site — Fetch a site's /robots.txt (http/https on the default port only; private, loopback and internal addresses are refused; up to 512,000 bytes; results cached for 5

Declared by sources

What the publisher says

Attributed to the source that supplied each field. Treated as claims, not facts.

No long description beyond the summary.

kind Official MCP Registry

name Official MCP Registry

summary Official MCP Registry

version Official MCP Registry

description Official MCP Registry

homepageUrl Official MCP Registry

publisherName Official MCP Registry

Provenance

Where this record came from

Every source is kept verbatim. Field changes are logged as events.

Official MCP Registrydev.workers.mike-tusa.robotsgate/robotsgate

first seen 13 d ago · fetched 2 d agoopen ↗

Field history (6 recent events)

13 d ago · declared · summary → "Generate, validate and check robots.txt rules for AI crawlers. Free; optional Pro key."
13 d ago · declared · description → "Generate, validate and check robots.txt rules for AI crawlers. Free; optional Pro key."
13 d ago · declared · publisherName → "dev.workers.mike-tusa.robotsgate"
13 d ago · declared · homepageUrl → "https://robotsgate.mike-tusa.workers.dev/"
13 d ago · declared · version → "0.3.1"
13 d ago · declared · protocols → ["mcp"]

This record is unclaimed

Evidence

What the publisher says

Capabilities

Where this record came from