FreeCrawl MCP Server - Self-hosted web scraping and document processing as a Firecrawl replacement
Wellknown found it in public sources; nobody has proven control of it yet. Claiming takes one click if the repository is under your GitHub account, or a small file on your domain otherwise. Verified owners get the badge, 15-minute checks, status alerts, edits that outrank crawled data, and a ranking boost.
Agents can do it too: POST https://wellknown.network/api/v1/claims with {"agent":"freecrawl-mcp","method":"well_known_file"} — machine-readable steps at claim.json, guide at /docs/claim.
Everything here was measured by our prober or read from a registry. Nothing is self-reported.
Attributed to the source that supplied each field. Treated as claims, not facts.
# FreeCrawl MCP Server A production-ready Model Context Protocol (MCP) server for web scraping and document processing, designed as a self-hosted replacement for Firecrawl. ## 🚀 Features - **JavaScript-enabled web scraping** with Playwright and anti-detection measures - **Document processing** with fallback support for various formats - **Concurrent batch processing** with configurable limits - **Intelligent caching** with SQLite backend - **Rate limiting** per domain - **Comprehensive error handling** with retry logic - **Easy installation** via `uvx` or local development setup - **Health monitoring** and metrics collection ## MCP Config (using `uvx`) ```json { "mcpServers": { "freecrawl": { "command": "uvx", "args": ["freecrawl-mcp"], } } } ``` ## 📦 Installation & Usage ### Quick Start with uvx (Recommended) The easiest way to use FreeCrawl is with `uvx`, which automatically manages dependencies: ```bash # Install browsers on first run uvx freecrawl-mcp --install-browsers # Test functionality uvx freecrawl-mcp --test ``` ### Local Development Setup For local development or customization: 1. **Clone from GitHub:** ```bash git clone https://github.com/dylan-gluck/freecrawl-mcp.git cd freecrawl-mcp ``` 2. **Set up environment:** ```bash # Sync dependencies uv sync # Install browser dependencies uv run freecrawl-mcp --install-browsers # Run tests uv run freecrawl-mcp --test ``` 3. **Run the server:** ```bash uv run freecrawl-mcp ``` ## 🛠 Configuration Configure FreeCrawl using environment variables: ### Basic Configuration ```bash # Transport (stdio for MCP, http for REST API) export FREECRAWL_TRANSPORT=stdio # Browser pool settings export FREECRAWL_MAX_BROWSERS=3 export FREECRAWL_HEADLESS=true # Concurrency limits export FREECRAWL_MAX_CONCURRENT=10 export FREECRAWL_MAX_PER_DOMAIN=3 # Cache settings export FREECRAWL_CACHE=true export FREECRAWL_CACHE_DIR=/tmp/freecrawl_cache…
Mapped onto the structured taxonomy from declared text and observed tool names. Confidence shown for derived entries.
Every source is kept verbatim. Field changes are logged as events.