Lightweight MCP server for web scraping, search, and crawling — local-first with optional Firecrawl fallback
Wellknown found it in public sources; nobody has proven control of it yet. Claiming takes one click if the repository is under your GitHub account, or a small file on your domain otherwise. Verified owners get the badge, 15-minute checks, status alerts, edits that outrank crawled data, and a ranking boost.
Agents can do it too: POST https://wellknown.network/api/v1/claims with {"agent":"webcrawl-mcp","method":"well_known_file"} — machine-readable steps at claim.json, guide at /docs/claim.
Everything here was measured by our prober or read from a registry. Nothing is self-reported.
Attributed to the source that supplied each field. Treated as claims, not facts.
# webcrawl-mcp [](https://pypi.org/project/webcrawl-mcp/) [](https://pypi.org/project/webcrawl-mcp/) [](LICENSE) A lightweight MCP server that gives Claude Code (or any MCP client) the ability to scrape, search, map, and crawl the web — using free, open-source libraries. Firecrawl is supported as an **optional** fallback for JS-heavy sites when you have a key. ## Why Most scraping doesn't actually need a headless browser. `trafilatura` handles the ~80% case (articles, docs, blogs) locally, which is faster and keeps external API usage to a minimum. This server routes the easy stuff through local extraction and only falls back to Firecrawl when content quality is genuinely poor. ## Tools | Tool | Purpose | |------|---------| | `webcrawl_scrape` | Fetch a single URL → markdown | | `webcrawl_search` | DuckDuckGo search (optionally scrape results) | | `webcrawl_map` | Discover same-domain URLs from a starting page | | `webcrawl_crawl` | BFS crawl multiple pages | ## Install ```bash pip install webcrawl-mcp ``` Requires Python 3.12+. Quick smoke test (should print `Webcrawl MCP server running` then exit cleanly with Ctrl-C): ```bash webcrawl-mcp ``` ### Install from source (for development) ```bash git clone https://github.com/andyliszewski/webcrawl.git cd webcrawl python -m venv venv source venv/bin/activate # Windows: venv\Scripts\activate pip install -e . ``` ## Configure your MCP client ### Claude Code Create `.mcp.json` in your project root (or merge into `~/.claude/settings.json`): ```json { "mcpServers": { "webcrawl": { "command": "uvx", "args": ["webcrawl-mcp"] } } } ``` This uses [`uvx`](https://docs.astral.sh/uv/guides/tools/) to run the package in a temporary environment — no manual install needed. If `uvx` is u…
Mapped onto the structured taxonomy from declared text and observed tool names. Confidence shown for derived entries.
Every source is kept verbatim. Field changes are logged as events.