A code-first evaluation framework for testing LLM agents' ability to use MCP tools
Wellknown found it in public sources; nobody has proven control of it yet. Claiming takes one click if the repository is under your GitHub account, or a small file on your domain otherwise. Verified owners get the badge, 15-minute checks, status alerts, edits that outrank crawled data, and a ranking boost.
Agents can do it too: POST https://wellknown.network/api/v1/claims with {"agent":"pydantic-ai-mcp-evals","method":"well_known_file"} — machine-readable steps at claim.json, guide at /docs/claim.
Everything here was measured by our prober or read from a registry. Nothing is self-reported.
Attributed to the source that supplied each field. Treated as claims, not facts.
# MCP Evals A code-first evaluation framework for testing LLM agents' ability to use MCP (Model Context Protocol) tools and accomplish tasks. Related projects: [MCPMark](https://github.com/eval-sys/mcpmark), [MCPUniverse](https://github.com/SalesforceAIResearch/MCP-Universe). ## Installation For a basic setup use uv/pip/pdm/etc: ```bash pip install pydantic-ai-mcp-evals ``` To run `contrib` tasks there are `domain-filesystem` and `domain-postgres` extras provided (see [example script](./scripts/run_domain_tasks.py)). ## Overview This library provides infrastructure for running structured evaluations of LLM agents against MCP-enabled environments. Each task is defined by: - **Domain**: A collection of MCP servers providing tools - **Goal**: A text-described objective for the agent - **Evaluators**: Functions that verify task completion and compute metrics ## Key Principles | Principle | Implementation | |-----------|----------------| | **Code-first** | All tasks defined in Python, no YAML/JSON configs | | **Simple user API** | Users define domains and tasks; library handles orchestration | | **No wheel reinvention** | pydantic-ai for LLM + MCP, pydantic_evals for evaluation, loguru for logging, logfire for observability | | **Resource lifecycle** | Async context managers with safe cleanup | | **Maintainability** | pytest, mypy, ruff | ## Quick Start ```python from mcp_evals import DomainRunner, PlainGrouper from mcp_evals.contrib.filesystem import FilesystemDomain from pydantic_ai import Agent async def main(): agent = Agent("openai:gpt-4o") runner = DomainRunner(agent=agent, grouper=PlainGrouper()) report = await runner.run(FilesystemDomain(), experiment_name="quickstart") report.print() ``` Requires `uv sync --extra domain-filesystem` and Docker. To run the full filesystem benchmark (defaults to `fs`; set `OPENAI_MODEL` or use `--model` to override): ```bash uv run python scripts/run_domain_tasks.py ``` Use `uv run python scripts/r…
Mapped onto the structured taxonomy from declared text and observed tool names. Confidence shown for derived entries.
Every source is kept verbatim. Field changes are logged as events.