a web cralwer for claude mcp
Wellknown found it in public sources; nobody has proven control of it yet. Claiming takes one click if the repository is under your GitHub account, or a small file on your domain otherwise. Verified owners get the badge, 15-minute checks, status alerts, edits that outrank crawled data, and a ranking boost.
Agents can do it too: POST https://wellknown.network/api/v1/claims with {"agent":"mcp-server-spidy","method":"well_known_file"} — machine-readable steps at claim.json, guide at /docs/claim.
Everything here was measured by our prober or read from a registry. Nothing is self-reported.
Attributed to the source that supplied each field. Treated as claims, not facts.
# mcp-server-spidy MCP server a web crawler for claude mcp ## Components ### Resources The server implements a simple note storage system with: - Custom note:// URI scheme for accessing individual notes - Each note resource has a name, description, and text/plain mimetype ### Prompts The server provides a single prompt: - summarize-notes: Creates summaries of all stored notes - Optional "style" argument to control detail level (brief/detailed) - Generates prompt combining all current notes with style preference ### Tools The server implements two tools: - web-crawl: Crawls a website and saves the results to a text file - Takes "url" and "output_file" as required string arguments - Fetches the content from the specified URL and writes it to the specified text file ## Configuration [TODO: Add configuration details specific to your implementation] ## Quickstart ### Install #### Claude Desktop On MacOS: `~/Library/Application\ Support/Claude/claude_desktop_config.json` On Windows: `%APPDATA%/Claude/claude_desktop_config.json` ### Usage To use the web crawling tool, you can call it with the following parameters: - `url`: The URL of the website you want to crawl. - `output_file`: The name of the file where the crawled content will be saved. <details> <summary>Published Servers Configuration</summary> ``` "mcpServers": { "mcp-server-spidy": { "command": "uvx", "args": [ "mcp-server-spidy" ] } } ``` </details> ## Development ### Building and Publishing To prepare the package for distribution: 1. Sync dependencies and update lockfile: ```bash uv sync ``` 2. Build package distributions: ```bash uv build ``` This will create source and wheel distributions in the `dist/` directory. 3. Publish to PyPI: ```bash uv publish ``` Note: You'll need to set PyPI credentials via environment variables or command flags: - Token: `--token` or `UV_PUBLISH_TOKEN` - Or username/password: `--username`/`UV_PUBLISH_USER…
Mapped onto the structured taxonomy from declared text and observed tool names. Confidence shown for derived entries.
Every source is kept verbatim. Field changes are logged as events.