Pytest-style framework for evaluating Model Context Protocol (MCP) servers.
Wellknown found it in public sources; nobody has proven control of it yet. Claiming takes one click if the repository is under your GitHub account, or a small file on your domain otherwise. Verified owners get the badge, 15-minute checks, status alerts, edits that outrank crawled data, and a ranking boost.
Agents can do it too: POST https://wellknown.network/api/v1/claims with {"agent":"eval-mcp","method":"well_known_file"} — machine-readable steps at claim.json, guide at /docs/claim.
Everything here was measured by our prober or read from a registry. Nothing is self-reported.
Attributed to the source that supplied each field. Treated as claims, not facts.
# MCP-Eval: An Evaluation Framework for MCP Servers MCP-Eval is a developer-first testing framework for Model Context Protocol (MCP) servers, built on the `mcp-agent` library. It enables you to write clear, concise, and powerful tests to evaluate the performance, reliability, and correctness of your AI agents and the MCP servers they connect to. ## Core Features - **Task-Based Testing**: Define tests as async functions where an agent performs a task. - **Automatic Metrics**: Automatically collect detailed metrics on latency, token usage, cost, and tool calls for every test run. - **Rich Assertions**: A powerful set of assertions designed for AI testing, including: - `contains()`: Checks for substrings in responses. - `tool_was_called()`: Verifies that a specific tool was used. - `tool_arguments_match()`: Checks if a tool was called with the correct arguments. - `cost_under()`: Asserts that a test run stays within a defined cost budget. - `number_of_steps_under()`: Ensures an agent completes a task efficiently. - `objective_succeeded()`: Uses an LLM to verify if the agent's response achieved the overall goal. - `plan_is_efficient()`: Uses an LLM to check for redundant or inefficient steps in the agent's execution path. - **Tool Coverage Reporting**: Automatically calculates the percentage of a server's tools that are exercised by your test suite. - **Automated Test Generation**: A CLI tool to automatically generate a baseline test suite for any MCP server. - **Detailed Reports**: Get immediate feedback from rich console reports and generate detailed JSON reports for CI/CD or further analysis. ## Getting Started ### 1. Installation Install `mcp_eval` and its dependencies. Make sure `mcp-agent` is also installed in your environment. ```bash pip install "typer[all]" rich pydantic jinja2
Mapped onto the structured taxonomy from declared text and observed tool names. Confidence shown for derived entries.
Every source is kept verbatim. Field changes are logged as events.