Scrapy middleware for MCP request tracking
Wellknown found it in public sources; nobody has proven control of it yet. Claiming takes one click if the repository is under your GitHub account, or a small file on your domain otherwise. Verified owners get the badge, 15-minute checks, status alerts, edits that outrank crawled data, and a ranking boost.
Agents can do it too: POST https://wellknown.network/api/v1/claims with {"agent":"scrapy-mcp-middleware","method":"well_known_file"} — machine-readable steps at claim.json, guide at /docs/claim.
Everything here was measured by our prober or read from a registry. Nothing is self-reported.
Attributed to the source that supplied each field. Treated as claims, not facts.
# Scrapy MCP Middleware Scrapy middleware for tracking requests and responses with MCP (Model Context Protocol) integration. ## Installation ```bash pip install scrapy-mcp-middleware ``` ## Usage Add the middleware to your Scrapy settings: ```python SPIDER_MIDDLEWARES = { 'scrapy_mcp_middleware.SessionTrackerMiddleware': 100, } # Optional: Configure the session store URL (defaults to http://127.0.0.1:8765) SESSION_STORE_URL = 'http://127.0.0.1:8765' ``` ## Features - Tracks all requests and responses - Hierarchical request tree tracking - Base64 encoding for request/response bodies - Configurable session store URL - Automatic storage cleanup on spider start ## Requirements - Python >= 3.8 - Scrapy >= 2.0.0 - requests >= 2.25.0
Mapped onto the structured taxonomy from declared text and observed tool names. Confidence shown for derived entries.
Every source is kept verbatim. Field changes are logged as events.