CrawlNest MCP server for web scraping, Twitter, Reddit, GitHub, Hacker News, YC directory, Product Hunt, Google Play Store, Apple App Store, Google Maps, IndieHackers, Quora, SERP web search, LinkedIn, AppSumo, Trustpilot, YouTube, video download, image download, and transcription. Use with Claude …
Wellknown found it in public sources; nobody has proven control of it yet. Claiming takes one click if the repository is under your GitHub account, or a small file on your domain otherwise. Verified owners get the badge, 15-minute checks, status alerts, edits that outrank crawled data, and a ranking boost.
Agents can do it too: POST https://wellknown.network/api/v1/claims with {"agent":"crawlnest-mcp","method":"well_known_file"} — machine-readable steps at claim.json, guide at /docs/claim.
Everything here was measured by our prober or read from a registry. Nothing is self-reported.
Attributed to the source that supplied each field. Treated as claims, not facts.
# crawlnest-mcp MCP server for web scraping, Twitter, and Reddit — 17 tools for any MCP-compatible LLM client. Works with Claude Code, Claude Desktop, Cursor, Windsurf, and any other MCP client. ## Quick Start ```bash pip install crawlnest-mcp ``` Then configure your client with two env vars: | Variable | Description | |----------|-------------| | `CRAWLNEST_API_URL` | Your CrawlNest API endpoint (e.g., `https://api.crawlnest.com` or `http://100.x.x.x:8000` for Tailscale) | | `CRAWLNEST_API_KEY` | Your API key (starts with `cn_...`) | That's it. No SSH, no scripts, no cloning repos. ## Client Configuration ### Claude Code ```bash claude mcp add crawlnest \ -e CRAWLNEST_API_URL=https://api.crawlnest.com \ -e CRAWLNEST_API_KEY=cn_your_key \ -- crawlnest-mcp ``` Or add to `~/.claude.json` manually: ```json { "mcpServers": { "crawlnest": { "command": "crawlnest-mcp", "env": { "CRAWLNEST_API_URL": "https://api.crawlnest.com", "CRAWLNEST_API_KEY": "cn_your_key" } } } } ``` ### Claude Desktop Add to your config file: | Platform | Config Path | |----------|------------| | macOS | `~/Library/Application Support/Claude/claude_desktop_config.json` | | Windows | `%APPDATA%\Claude\claude_desktop_config.json` | | Linux | `~/.config/Claude/claude_desktop_config.json` | ```json { "mcpServers": { "crawlnest": { "command": "crawlnest-mcp", "env": { "CRAWLNEST_API_URL": "https://api.crawlnest.com", "CRAWLNEST_API_KEY": "cn_your_key" } } } } ``` Restart Claude Desktop after saving. ### Cursor Add to `~/.cursor/mcp.json`: ```json { "mcpServers": { "crawlnest": { "command": "crawlnest-mcp", "env": { "CRAWLNEST_API_URL": "https://api.crawlnest.com", "CRAWLNEST_API_KEY": "cn_your_key" } } } } ``` ### Windsurf Add to `~/.codeium/windsurf/mcp_config.json`: ```json { "mcpServers": { "crawlnest": { "command": "crawln…
Mapped onto the structured taxonomy from declared text and observed tool names. Confidence shown for derived entries.
Every source is kept verbatim. Field changes are logged as events.