HTTP web extraction, browser header rotation, proxy pool management, and clean Markdown MCP server for AI agents
Wellknown found it in public sources; nobody has proven control of it yet. Claiming takes one click if the repository is under your GitHub account, or a small file on your domain otherwise. Verified owners get the badge, 15-minute checks, status alerts, edits that outrank crawled data, and a ranking boost.
Agents can do it too: POST https://wellknown.network/api/v1/claims with {"agent":"crawlengine-mcp","method":"well_known_file"} — machine-readable steps at claim.json, guide at /docs/claim.
Everything here was measured by our prober or read from a registry. Nothing is self-reported.
Attributed to the source that supplied each field. Treated as claims, not facts.
# crawlengine-mcp [](https://pypi.org/project/crawlengine-mcp/) [](https://github.com/johannschiffer/crawlengine-mcp/actions/workflows/ci.yml) [](LICENSE) [](https://python.org) [](https://modelcontextprotocol.io) [](https://github.com/TensorBlock/awesome-mcp-servers) <a href="https://glama.ai/mcp/servers/@johannschiffer/crawlengine-mcp"> <img width="380" height="200" src="https://glama.ai/mcp/servers/@johannschiffer/crawlengine-mcp/badge" alt="crawlengine-mcp MCP server" /> </a> **CrawlEngine MCP** is a **Model Context Protocol (2024-11-05)** web extraction server built on `httpx` and `beautifulsoup4` that converts HTML pages into clean Markdown, resolves relative links, extracts JSON-LD and HTML tables, and rotates browser headers and HTTP/SOCKS5 proxies. Listed in [TensorBlock/awesome-mcp-servers](https://github.com/TensorBlock/awesome-mcp-servers). ## Why CrawlEngine MCP? 1. **Context Window Efficiency**: Strips `<script>`, `<style>`, `<nav>`, `<footer>`, cookie banners, and ads before returning Markdown to the model. 2. **Resolved Canonical Links**: Converts relative `<a href="/docs/start">` paths into full URLs using the page's `base_url`. 3. **Outbound SSRF Protection**: Blocks loopback, RFC1918 private networks, and cloud metadata (`169.254.169.254`) on both initial requests and HTTP redirects. ## Features - **Semantic Markdown Extraction**: Preserves headings, lists, code blocks, blockquotes, and resolved inline links without duplicating nested list paragraphs. - **Browser He…
Mapped onto the structured taxonomy from declared text and observed tool names. Confidence shown for derived entries.
Every source is kept verbatim. Field changes are logged as events.