A Model Context Protocol server for web crawling using Crawl4ai
Wellknown found it in public sources; nobody has proven control of it yet. Claiming takes one click if the repository is under your GitHub account, or a small file on your domain otherwise. Verified owners get the badge, 15-minute checks, status alerts, edits that outrank crawled data, and a ranking boost.
Agents can do it too: POST https://wellknown.network/api/v1/claims with {"agent":"crawl4ai-mcp-server","method":"well_known_file"} — machine-readable steps at claim.json, guide at /docs/claim.
Everything here was measured by our prober or read from a registry. Nothing is self-reported.
Attributed to the source that supplied each field. Treated as claims, not facts.
# Crawl4AI MCP Server A Model Context Protocol server for web crawling using the Crawl4ai library. ## 📋 Overview Crawl4AI MCP Server provides a set of tools and prompts for web crawling through the Model Context Protocol (MCP). It allows AI assistants to autonomously crawl websites, extract content, and save information as Markdown files. ## ✨ Features - 🕸️ **Single Page Crawling**: Extract content from a single webpage in Markdown format - 🌐 **Deep Website Crawling**: Crawl multiple pages of a website with configurable depth and limits - 🔍 **Structured Data Extraction**: Use CSS selectors to extract specific structured data from webpages - 💾 **Markdown Export**: Save crawled content directly as Markdown files ## 🚀 Installation ```bash pip install crawl4ai-mcp-server ``` ## 🛠️ Usage ### Command Line Run the server directly from the command line: ```bash crawl4ai-mcp ``` ### Python API ```python import asyncio from crawl4ai_mcp import serve # Run the server asyncio.run(serve()) ``` ## 📝 Available Tools ### crawl_webpage Crawls a single webpage and returns its content as markdown. **Parameters**: - `url` (string, required): URL to crawl - `include_images` (boolean, optional): Whether to include images in the result (default: true) - `bypass_cache` (boolean, optional): Whether to bypass cache (default: false) ### crawl_website Crawls a website starting from the given URL, with specified depth and page limit. **Parameters**: - `url` (string, required): Starting URL - `max_depth` (integer, optional): Maximum crawl depth (default: 1) - `max_pages` (integer, optional): Maximum number of pages to crawl (default: 5) - `include_images` (boolean, optional): Whether to include images (default: true) ### extract_structured_data Extracts structured data from a webpage using CSS selectors. **Parameters**: - `url` (string, required): URL to extract data from - `schema` (object, optional): Schema defining what to extract - `css_selector` (string, …
Mapped onto the structured taxonomy from declared text and observed tool names. Confidence shown for derived entries.
Every source is kept verbatim. Field changes are logged as events.