{"$schema":"https://wellknown.network/schemas/agent-record-v1.json","schemaVersion":"1","id":"ag_9thsnnn8j5jq","handle":"crawlengine-mcp","url":"https://wellknown.network/agents/crawlengine-mcp","links":{"self":"https://wellknown.network/agents/crawlengine-mcp/record.json","html":"https://wellknown.network/agents/crawlengine-mcp","markdown":"https://wellknown.network/agents/crawlengine-mcp/record.md","api":"https://wellknown.network/api/v1/agents/crawlengine-mcp","status":"https://wellknown.network/api/v1/agents/crawlengine-mcp/status","claim":"https://wellknown.network/agents/crawlengine-mcp/claim","claimApi":"https://wellknown.network/api/v1/claims","claimDescriptor":"https://wellknown.network/agents/crawlengine-mcp/claim.json","badge":"https://wellknown.network/agents/crawlengine-mcp/badge.svg","openapi":"https://wellknown.network/openapi.json","history":"https://wellknown.network/api/v1/agents/crawlengine-mcp/history","tools":"https://wellknown.network/api/v1/agents/crawlengine-mcp/tools"},"ard":{"identifier":"urn:air::server:crawlengine-mcp","type":"application/mcp-server-card+json"},"kind":"mcp_server","declared":{"name":"crawlengine-mcp","summary":"HTTP web extraction, browser header rotation, proxy pool management, and clean Markdown MCP server for AI agents","description":"# crawlengine-mcp\n\n[![PyPI version](https://img.shields.io/pypi/v/crawlengine-mcp.svg)](https://pypi.org/project/crawlengine-mcp/)\n[![CI](https://github.com/johannschiffer/crawlengine-mcp/actions/workflows/ci.yml/badge.svg)](https://github.com/johannschiffer/crawlengine-mcp/actions/workflows/ci.yml)\n[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)\n[![Python 3.10+](https://img.shields.io/badge/Python-3.10+-brightgreen.svg)](https://python.org)\n[![MCP Compliant](https://img.shields.io/badge/MCP-2024--11--05-purple.svg)](https://modelcontextprotocol.io)\n[![Awesome MCP Servers](https://img.shields.io/badge/Awesome--MCP--Servers-TensorBlock-blue?style=flat-square)](https://github.com/TensorBlock/awesome-mcp-servers)\n\n<a href=\"https://glama.ai/mcp/servers/@johannschiffer/crawlengine-mcp\">\n  <img width=\"380\" height=\"200\" src=\"https://glama.ai/mcp/servers/@johannschiffer/crawlengine-mcp/badge\" alt=\"crawlengine-mcp MCP server\" />\n</a>\n\n**CrawlEngine MCP** is a **Model Context Protocol (2024-11-05)** web extraction server built on `httpx` and `beautifulsoup4` that converts HTML pages into clean Markdown, resolves relative links, extracts JSON-LD and HTML tables, and rotates browser headers and HTTP/SOCKS5 proxies.\n\nListed in [TensorBlock/awesome-mcp-servers](https://github.com/TensorBlock/awesome-mcp-servers).\n\n## Why CrawlEngine MCP?\n\n1. **Context Window Efficiency**: Strips `<script>`, `<style>`, `<nav>`, `<footer>`, cookie banners, and ads before returning Markdown to the model.\n2. **Resolved Canonical Links**: Converts relative `<a href=\"/docs/start\">` paths into full URLs using the page's `base_url`.\n3. **Outbound SSRF Protection**: Blocks loopback, RFC1918 private networks, and cloud metadata (`169.254.169.254`) on both initial requests and HTTP redirects.\n\n## Features\n\n- **Semantic Markdown Extraction**: Preserves headings, lists, code blocks, blockquotes, and resolved inline links without duplicating nested list paragraphs.\n- **Browser He…","publisher":{"name":"Johann Schiffer","url":null},"homepage":"https://github.com/johannschiffer/crawlengine-mcp#readme","repository":"https://github.com/johannschiffer/crawlengine-mcp#readme","version":"2.0.1","license":null,"protocols":["mcp"],"tags":["mcp","model-context-protocol","web-scraping","crawler","markdown","claude","cursor","ai-agent"],"pricing":null,"endpoints":[{"url":"pypi:crawlengine-mcp","type":"package_pypi","auth":null,"probeable":false}],"skills":null,"tools":null,"extra":null,"attribution":{"kind":"pypi","name":"pypi","repoUrl":"pypi","summary":"pypi","version":"pypi","description":"pypi","homepageUrl":"pypi","publisherName":"pypi"}},"derived":{"capabilities":[{"slug":"data.web-scraping","name":"Web Scraping","confidence":1,"provenance":"declared"}],"categories":["data"],"language":"en"},"observed":{"status":"unknown","statusReason":"Distributed as a package to run locally; no network endpoint to check.","lastOkAt":null,"lastProbedAt":null,"statusComputedAt":null,"reliability30d":null,"latestObservations":[],"tools":null,"package":{"name":"crawlengine-mcp","registry":"pypi","observedAt":"2026-11-05T21:26:34.966Z","publishedAt":"2026-10-28T15:50:45.312335Z","latestVersion":"2.0.1"},"toolSurface":null,"endpointFacts":[]},"verification":{"claimed":false,"claimedAt":null,"proofs":[]},"provenance":{"sources":[{"source":"pypi","key":"crawlengine-mcp","url":"https://pypi.org/project/crawlengine-mcp/","firstSeenAt":"2026-10-24T17:27:05.982Z","fetchedAt":"2026-11-05T21:24:59.089Z","normalizedAt":"2026-11-05T21:24:59.089Z"}]},"firstSeenAt":"2026-10-24T17:27:05.982Z","updatedAt":"2026-11-05T21:26:34.966Z"}