Behavioral evaluation and description linting for MCP servers
Wellknown found it in public sources; nobody has proven control of it yet. Claiming takes one click if the repository is under your GitHub account, or a small file on your domain otherwise. Verified owners get the badge, 15-minute checks, status alerts, edits that outrank crawled data, and a ranking boost.
Agents can do it too: POST https://wellknown.network/api/v1/claims with {"agent":"mcp-tool-evals","method":"well_known_file"} — machine-readable steps at claim.json, guide at /docs/claim.
Everything here was measured by our prober or read from a registry. Nothing is self-reported.
Attributed to the source that supplied each field. Treated as claims, not facts.
# mcp-evals Behavioral evaluation and description linting for [Model Context Protocol](https://modelcontextprotocol.io) (MCP) servers. [](https://pypi.org/project/mcp-tool-evals/) [](LICENSE) [](https://www.python.org/downloads/) [](https://github.com/yashshah9/mcp-evals/actions/workflows/ci.yml) > **Status:** v0.8 — stdio/HTTP/SSE discovery, description lint, mock/OpenAI evals with **argument scoring**, GitHub Action PR comments. ## 60-second try ```bash pip install mcp-tool-evals mcp-evals health mcp-evals run examples/eval-suite.yaml --catalog examples/tools.yaml --model mock # or with Docker (no local Python): docker compose run --rm run-example ``` ## Why this vs alternatives | Approach | Strength | Gap | |----------|----------|-----| | **mcp-evals** | Lint + YAML evals + mock CI runner | Not a full agent harness | | MCP Inspector | Interactive debugging | No CI lint/eval suite | | Protocol conformance tests | JSON-RPC correctness | Do not test tool selection | | Hand-written LLM mocks | Full control | Drift from real descriptions | ## Problem Protocol conformance tests verify JSON-RPC correctness. They do **not** verify whether an LLM agent selects the right tool, passes sensible arguments, or completes the task. Tool descriptions are now load-bearing API design, and there is no standard way to test them. **mcp-evals** fills the behavioral layer: lint tool descriptions, define eval cases in YAML, and (next) measure tool-selection accuracy in CI. ## Key features (v0.8) - **Description linter** — missing descriptions, undocumented required params, overlapping tools, ambiguous verbs, `name-description-mismatch`, `missing-examples` - **Live discover** — handshake a stdio, HTTP JSON-RP…
Mapped onto the structured taxonomy from declared text and observed tool names. Confidence shown for derived entries.
Every source is kept verbatim. Field changes are logged as events.