A powerful web crawling tool that integrates with AI assistants via the MCP (Machine Conversation Protocol)
Wellknown found it in public sources; nobody has proven control of it yet. Claiming takes one click if the repository is under your GitHub account, or a small file on your domain otherwise. Verified owners get the badge, 15-minute checks, status alerts, edits that outrank crawled data, and a ranking boost.
Agents can do it too: POST https://wellknown.network/api/v1/claims with {"agent":"iflow-mcp-laurentvv-crawl4ai-mcp","method":"well_known_file"} — machine-readable steps at claim.json, guide at /docs/claim.
Everything here was measured by our prober or read from a registry. Nothing is self-reported.
Attributed to the source that supplied each field. Treated as claims, not facts.
# Web Crawler MCP [](README.md) [](lang/README.zh.md) [](lang/README.hi.md) [](lang/README.es.md) [](lang/README.fr.md) [](lang/README.ar.md) [](lang/README.bn.md) [](lang/README.ru.md) [](lang/README.pt.md) [](lang/README.id.md)   A powerful web crawling tool that integrates with AI assistants via the MCP (Machine Conversation Protocol). This project allows you to crawl websites and save their content [...] ## 📋 Features - Website crawling with configurable depth - Support for internal and external links - Generation of structured Markdown files - Native integration with AI assistants via MCP - Detailed crawl result statistics - Error and not found page handling ## 🚀 Installation ### Prerequisites - Python 3.9 or higher ### Installation Steps 1. Clone this repository: ```bash git clone laurentvv/crawl4ai-mcp cd crawl4ai-mcp ``` 2. Create and activate a virtual environment: ```bash # Windows python -m venv .venv .venv\Scripts\activate # Linux/MacOS python -m venv .venv source .venv/bin/activate ``` 3. Install the required dependencies: ```bash pip install -r requirements.txt ``` ## 🔧 Configuration ### MCP Configuration for AI Assistants To use this crawler with AI assistants like VScode Cline, configure your `cline_mcp_settings.json` file: ```json { "mcpServers": {…
Mapped onto the structured taxonomy from declared text and observed tool names. Confidence shown for derived entries.
Every source is kept verbatim. Field changes are logged as events.