{"$schema":"https://wellknown.network/schemas/agent-record-v1.json","schemaVersion":"1","id":"ag_55rktj2d4e3h","handle":"scrapy-mcp","url":"https://wellknown.network/agents/scrapy-mcp","links":{"self":"https://wellknown.network/agents/scrapy-mcp/record.json","html":"https://wellknown.network/agents/scrapy-mcp","markdown":"https://wellknown.network/agents/scrapy-mcp/record.md","api":"https://wellknown.network/api/v1/agents/scrapy-mcp","status":"https://wellknown.network/api/v1/agents/scrapy-mcp/status","claim":"https://wellknown.network/agents/scrapy-mcp/claim","claimApi":"https://wellknown.network/api/v1/claims","claimDescriptor":"https://wellknown.network/agents/scrapy-mcp/claim.json","badge":"https://wellknown.network/agents/scrapy-mcp/badge.svg","openapi":"https://wellknown.network/openapi.json"},"ard":{"identifier":"urn:air::server:scrapy-mcp","type":"application/mcp-server-card+json"},"kind":"mcp_server","declared":{"name":"scrapy-mcp","summary":"Headless web-scraping MCP server built on Scrapy: fetch, extract (CSS/XPath), links, tables, sitemaps, robots, and async crawls.","description":"# scrapy-mcp\n\nA headless web-scraping [MCP](https://modelcontextprotocol.io) server built on\n[Scrapy](https://scrapy.org). It exposes Scrapy's scraping primitives — polite fetching,\nCSS/XPath extraction, link and table extraction, sitemap and robots.txt reading, and bounded\nasynchronous crawls — as MCP tools an agent can call over stdio.\n\n- **Headless, no rendering.** Pages are fetched and parsed as HTML; no browser, no JavaScript\n  execution. This keeps the footprint tiny — it runs comfortably on weak machines.\n- **Reactor-safe.** Every operation runs in a short-lived Scrapy subprocess, so Twisted's\n  reactor never lives inside the asyncio MCP server (no `ReactorNotRestartable`), and memory\n  is reclaimed after each call.\n- **Polite by default.** Obeys `robots.txt`, throttles with AutoThrottle, and enforces hard\n  page/depth caps so a crawl can't run away.\n\n## Install / run\n\nRun straight from PyPI with [uv](https://docs.astral.sh/uv/) — no install step:\n\n```bash\nuvx scrapy-mcp\n```\n\nOr install it:\n\n```bash\nuv pip install scrapy-mcp\nscrapy-mcp\n```\n\nThe server speaks MCP over **stdio**. Point any MCP client at it. For Claude Desktop, add to\n`claude_desktop_config.json`:\n\n```json\n{\n  \"mcpServers\": {\n    \"scrapy\": {\n      \"command\": \"uvx\",\n      \"args\": [\"scrapy-mcp\"]\n    }\n  }\n}\n```\n\n## Tools\n\n| Tool | What it does |\n|------|--------------|\n| `fetch_page(url, format, max_bytes, obey_robots)` | Fetch one page as `markdown` (default), `text`, or `html`. |\n| `extract(url, selectors, obey_robots)` | Pull structured fields with CSS/XPath selectors. |\n| `extract_tables(url, max_tables, obey_robots)` | Extract every HTML `<table>` as `{headers, rows}`. |\n| `extract_links(url, same_domain, pattern, limit, obey_robots)` | List de-duplicated links on a page. |\n| `get_sitemap(url, limit, obey_robots)` | Read a sitemap (gzip + sitemap-index aware). |\n| `check_robots(url, user_agent)` | Is a URL crawlable? Returns the crawl-delay and sitemaps. |\n| `start_crawl(start_url, allow_pat…","publisher":{"name":"Eitan Hadar","url":null},"homepage":"https://github.com/eitan3/Scrapy_MCP_Scraper","repository":"https://github.com/eitan3/Scrapy_MCP_Scraper","version":"0.1.0","license":"MIT","protocols":["mcp"],"tags":["crawler","mcp","model-context-protocol","scraper","scrapy","web-scraping"],"pricing":null,"endpoints":[{"url":"pypi:scrapy-mcp","type":"package_pypi","auth":null,"probeable":false}],"skills":null,"tools":null,"extra":null,"attribution":{"kind":"pypi","name":"pypi","license":"pypi","repoUrl":"pypi","summary":"pypi","version":"pypi","description":"pypi","homepageUrl":"pypi","publisherName":"pypi"}},"derived":{"capabilities":[{"slug":"data.web-scraping","name":"Web Scraping","confidence":1,"provenance":"declared"},{"slug":"infra.browser-automation","name":"Browser Automation","confidence":0.814,"provenance":"derived"}],"categories":["data","infra"],"language":"en"},"observed":{"status":"unknown","statusReason":"Distributed as a package to run locally; no network endpoint to check.","lastOkAt":null,"lastProbedAt":null,"statusComputedAt":null,"reliability30d":null,"latestObservations":[],"tools":null,"package":{"name":"scrapy-mcp","registry":"pypi","observedAt":"2026-09-10T12:23:12.442Z","publishedAt":"2026-06-10T05:05:54.346568Z","latestVersion":"0.1.0"}},"verification":{"claimed":false,"claimedAt":null,"proofs":[]},"provenance":{"sources":[{"source":"pypi","key":"scrapy-mcp","url":"https://pypi.org/project/scrapy-mcp/","firstSeenAt":"2026-09-10T12:21:01.985Z","fetchedAt":"2026-09-10T12:21:01.985Z","normalizedAt":"2026-09-10T12:21:01.985Z"}]},"firstSeenAt":"2026-09-10T12:21:01.985Z","updatedAt":"2026-09-10T12:23:12.442Z"}