{"$schema":"https://wellknown.network/schemas/agent-record-v1.json","schemaVersion":"1","id":"ag_jp72u6gpq2e3","handle":"webcrawl-mcp","url":"https://wellknown.network/agents/webcrawl-mcp","links":{"self":"https://wellknown.network/agents/webcrawl-mcp/record.json","html":"https://wellknown.network/agents/webcrawl-mcp","markdown":"https://wellknown.network/agents/webcrawl-mcp/record.md","api":"https://wellknown.network/api/v1/agents/webcrawl-mcp","status":"https://wellknown.network/api/v1/agents/webcrawl-mcp/status","claim":"https://wellknown.network/agents/webcrawl-mcp/claim","claimApi":"https://wellknown.network/api/v1/claims","claimDescriptor":"https://wellknown.network/agents/webcrawl-mcp/claim.json","badge":"https://wellknown.network/agents/webcrawl-mcp/badge.svg","openapi":"https://wellknown.network/openapi.json"},"ard":{"identifier":"urn:air::server:webcrawl-mcp","type":"application/mcp-server-card+json"},"kind":"mcp_server","declared":{"name":"webcrawl-mcp","summary":"Lightweight MCP server for web scraping, search, and crawling — local-first with optional Firecrawl fallback","description":"# webcrawl-mcp\n\n[![PyPI version](https://img.shields.io/pypi/v/webcrawl-mcp.svg)](https://pypi.org/project/webcrawl-mcp/)\n[![Python versions](https://img.shields.io/pypi/pyversions/webcrawl-mcp.svg)](https://pypi.org/project/webcrawl-mcp/)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)\n\nA lightweight MCP server that gives Claude Code (or any MCP client) the ability to scrape, search, map, and crawl the web — using free, open-source libraries. Firecrawl is supported as an **optional** fallback for JS-heavy sites when you have a key.\n\n## Why\n\nMost scraping doesn't actually need a headless browser. `trafilatura` handles the ~80% case (articles, docs, blogs) locally, which is faster and keeps external API usage to a minimum. This server routes the easy stuff through local extraction and only falls back to Firecrawl when content quality is genuinely poor.\n\n## Tools\n\n| Tool | Purpose |\n|------|---------|\n| `webcrawl_scrape` | Fetch a single URL → markdown |\n| `webcrawl_search` | DuckDuckGo search (optionally scrape results) |\n| `webcrawl_map` | Discover same-domain URLs from a starting page |\n| `webcrawl_crawl` | BFS crawl multiple pages |\n\n## Install\n\n```bash\npip install webcrawl-mcp\n```\n\nRequires Python 3.12+.\n\nQuick smoke test (should print `Webcrawl MCP server running` then exit cleanly with Ctrl-C):\n\n```bash\nwebcrawl-mcp\n```\n\n### Install from source (for development)\n\n```bash\ngit clone https://github.com/andyliszewski/webcrawl.git\ncd webcrawl\npython -m venv venv\nsource venv/bin/activate      # Windows: venv\\Scripts\\activate\npip install -e .\n```\n\n## Configure your MCP client\n\n### Claude Code\n\nCreate `.mcp.json` in your project root (or merge into `~/.claude/settings.json`):\n\n```json\n{\n  \"mcpServers\": {\n    \"webcrawl\": {\n      \"command\": \"uvx\",\n      \"args\": [\"webcrawl-mcp\"]\n    }\n  }\n}\n```\n\nThis uses [`uvx`](https://docs.astral.sh/uv/guides/tools/) to run the package in a temporary environment — no manual install needed. If `uvx` is u…","publisher":{"name":"Andy Liszewski","url":null},"homepage":"https://github.com/andyliszewski/webcrawl","repository":"https://github.com/andyliszewski/webcrawl","version":"0.1.0","license":"MIT","protocols":["mcp"],"tags":["claude","crawling","mcp","model-context-protocol","search","trafilatura","web-scraping"],"pricing":null,"endpoints":[{"url":"pypi:webcrawl-mcp","type":"package_pypi","auth":null,"probeable":false}],"skills":null,"tools":null,"extra":null,"attribution":{"kind":"pypi","name":"pypi","license":"pypi","repoUrl":"pypi","summary":"pypi","version":"pypi","description":"pypi","homepageUrl":"pypi","publisherName":"pypi"}},"derived":{"capabilities":[{"slug":"data.web-scraping","name":"Web Scraping","confidence":1,"provenance":"declared"},{"slug":"dev.package-management","name":"Packages & Dependencies","confidence":1,"provenance":"derived"},{"slug":"infra.browser-automation","name":"Browser Automation","confidence":0.814,"provenance":"derived"},{"slug":"dev.version-control","name":"Version Control","confidence":0.745,"provenance":"derived"}],"categories":["data","dev","infra"],"language":"en"},"observed":{"status":"unknown","statusReason":"Distributed as a package to run locally; no network endpoint to check.","lastOkAt":null,"lastProbedAt":null,"statusComputedAt":null,"reliability30d":null,"latestObservations":[],"tools":null,"package":{"name":"webcrawl-mcp","registry":"pypi","observedAt":"2026-09-10T15:25:25.052Z","publishedAt":"2026-04-17T18:57:46.140539Z","latestVersion":"0.1.0"}},"verification":{"claimed":false,"claimedAt":null,"proofs":[]},"provenance":{"sources":[{"source":"pypi","key":"webcrawl-mcp","url":"https://pypi.org/project/webcrawl-mcp/","firstSeenAt":"2026-09-10T15:24:21.764Z","fetchedAt":"2026-09-10T15:24:21.764Z","normalizedAt":"2026-09-10T15:24:21.764Z"}]},"firstSeenAt":"2026-09-10T15:24:21.764Z","updatedAt":"2026-09-10T15:25:25.052Z"}