Headless web-scraping MCP server built on Scrapy: fetch, extract (CSS/XPath), links, tables, sitemaps, robots, and async crawls.
Wellknown found it in public sources; nobody has proven control of it yet. Claiming takes one click if the repository is under your GitHub account, or a small file on your domain otherwise. Verified owners get the badge, 15-minute checks, status alerts, edits that outrank crawled data, and a ranking boost.
Agents can do it too: POST https://wellknown.network/api/v1/claims with {"agent":"scrapy-mcp","method":"well_known_file"} — machine-readable steps at claim.json, guide at /docs/claim.
Everything here was measured by our prober or read from a registry. Nothing is self-reported.
Attributed to the source that supplied each field. Treated as claims, not facts.
# scrapy-mcp A headless web-scraping [MCP](https://modelcontextprotocol.io) server built on [Scrapy](https://scrapy.org). It exposes Scrapy's scraping primitives — polite fetching, CSS/XPath extraction, link and table extraction, sitemap and robots.txt reading, and bounded asynchronous crawls — as MCP tools an agent can call over stdio. - **Headless, no rendering.** Pages are fetched and parsed as HTML; no browser, no JavaScript execution. This keeps the footprint tiny — it runs comfortably on weak machines. - **Reactor-safe.** Every operation runs in a short-lived Scrapy subprocess, so Twisted's reactor never lives inside the asyncio MCP server (no `ReactorNotRestartable`), and memory is reclaimed after each call. - **Polite by default.** Obeys `robots.txt`, throttles with AutoThrottle, and enforces hard page/depth caps so a crawl can't run away. ## Install / run Run straight from PyPI with [uv](https://docs.astral.sh/uv/) — no install step: ```bash uvx scrapy-mcp ``` Or install it: ```bash uv pip install scrapy-mcp scrapy-mcp ``` The server speaks MCP over **stdio**. Point any MCP client at it. For Claude Desktop, add to `claude_desktop_config.json`: ```json { "mcpServers": { "scrapy": { "command": "uvx", "args": ["scrapy-mcp"] } } } ``` ## Tools | Tool | What it does | |------|--------------| | `fetch_page(url, format, max_bytes, obey_robots)` | Fetch one page as `markdown` (default), `text`, or `html`. | | `extract(url, selectors, obey_robots)` | Pull structured fields with CSS/XPath selectors. | | `extract_tables(url, max_tables, obey_robots)` | Extract every HTML `<table>` as `{headers, rows}`. | | `extract_links(url, same_domain, pattern, limit, obey_robots)` | List de-duplicated links on a page. | | `get_sitemap(url, limit, obey_robots)` | Read a sitemap (gzip + sitemap-index aware). | | `check_robots(url, user_agent)` | Is a URL crawlable? Returns the crawl-delay and sitemaps. | | `start_crawl(start_url, allow_pat…
Mapped onto the structured taxonomy from declared text and observed tool names. Confidence shown for derived entries.
Every source is kept verbatim. Field changes are logged as events.