{"$schema":"https://wellknown.network/schemas/agent-record-v1.json","schemaVersion":"1","id":"ag_uqsauc5hmykj","handle":"pydantic-ai-mcp-evals","url":"https://wellknown.network/agents/pydantic-ai-mcp-evals","links":{"self":"https://wellknown.network/agents/pydantic-ai-mcp-evals/record.json","html":"https://wellknown.network/agents/pydantic-ai-mcp-evals","markdown":"https://wellknown.network/agents/pydantic-ai-mcp-evals/record.md","api":"https://wellknown.network/api/v1/agents/pydantic-ai-mcp-evals","status":"https://wellknown.network/api/v1/agents/pydantic-ai-mcp-evals/status","claim":"https://wellknown.network/agents/pydantic-ai-mcp-evals/claim","claimApi":"https://wellknown.network/api/v1/claims","claimDescriptor":"https://wellknown.network/agents/pydantic-ai-mcp-evals/claim.json","badge":"https://wellknown.network/agents/pydantic-ai-mcp-evals/badge.svg","openapi":"https://wellknown.network/openapi.json","history":"https://wellknown.network/api/v1/agents/pydantic-ai-mcp-evals/history","tools":"https://wellknown.network/api/v1/agents/pydantic-ai-mcp-evals/tools"},"ard":{"identifier":"urn:air::server:pydantic-ai-mcp-evals","type":"application/mcp-server-card+json"},"kind":"mcp_server","declared":{"name":"pydantic-ai-mcp-evals","summary":"A code-first evaluation framework for testing LLM agents' ability to use MCP tools","description":"# MCP Evals\n\nA code-first evaluation framework for testing LLM agents' ability to use MCP (Model Context Protocol) tools and accomplish tasks.\n\nRelated projects: [MCPMark](https://github.com/eval-sys/mcpmark), [MCPUniverse](https://github.com/SalesforceAIResearch/MCP-Universe).\n\n## Installation\n\nFor a basic setup use uv/pip/pdm/etc:\n```bash\npip install pydantic-ai-mcp-evals\n```\n\nTo run `contrib` tasks there are `domain-filesystem` and `domain-postgres` extras provided (see [example script](./scripts/run_domain_tasks.py)).\n\n## Overview\n\nThis library provides infrastructure for running structured evaluations of LLM agents against MCP-enabled environments. Each task is defined by:\n\n- **Domain**: A collection of MCP servers providing tools\n- **Goal**: A text-described objective for the agent\n- **Evaluators**: Functions that verify task completion and compute metrics\n\n## Key Principles\n\n| Principle | Implementation |\n|-----------|----------------|\n| **Code-first** | All tasks defined in Python, no YAML/JSON configs |\n| **Simple user API** | Users define domains and tasks; library handles orchestration |\n| **No wheel reinvention** | pydantic-ai for LLM + MCP, pydantic_evals for evaluation, loguru for logging, logfire for observability |\n| **Resource lifecycle** | Async context managers with safe cleanup |\n| **Maintainability** | pytest, mypy, ruff | \n\n## Quick Start\n\n```python\nfrom mcp_evals import DomainRunner, PlainGrouper\nfrom mcp_evals.contrib.filesystem import FilesystemDomain\nfrom pydantic_ai import Agent\n\nasync def main():\n    agent = Agent(\"openai:gpt-4o\")\n    runner = DomainRunner(agent=agent, grouper=PlainGrouper())\n    report = await runner.run(FilesystemDomain(), experiment_name=\"quickstart\")\n    report.print()\n```\n\nRequires `uv sync --extra domain-filesystem` and Docker. To run the full filesystem benchmark (defaults to `fs`; set `OPENAI_MODEL` or use `--model` to override):\n\n```bash\nuv run python scripts/run_domain_tasks.py\n```\n\nUse `uv run python scripts/r…","publisher":{"name":"voorhs","url":null},"homepage":null,"repository":null,"version":"0.1.1","license":null,"protocols":["mcp"],"tags":["mcp"],"pricing":null,"endpoints":[{"url":"pypi:pydantic-ai-mcp-evals","type":"package_pypi","auth":null,"probeable":false}],"skills":null,"tools":null,"extra":null,"attribution":{"kind":"pypi","name":"pypi","summary":"pypi","version":"pypi","description":"pypi","publisherName":"pypi"}},"derived":{"capabilities":[{"slug":"ai.evaluation","name":"Evaluation & Benchmarks","confidence":1,"provenance":"derived"},{"slug":"dev.filesystem","name":"Filesystem","confidence":0.882,"provenance":"derived"}],"categories":["ai","dev"],"language":"en"},"observed":{"status":"unknown","statusReason":"Distributed as a package to run locally; no network endpoint to check.","lastOkAt":null,"lastProbedAt":null,"statusComputedAt":null,"reliability30d":null,"latestObservations":[],"tools":null,"package":{"name":"pydantic-ai-mcp-evals","registry":"pypi","observedAt":"2026-10-28T20:23:10.484Z","publishedAt":"2026-05-12T17:33:55.537452Z","latestVersion":"0.1.1"},"toolSurface":null,"endpointFacts":[]},"verification":{"claimed":false,"claimedAt":null,"proofs":[]},"provenance":{"sources":[{"source":"pypi","key":"pydantic-ai-mcp-evals","url":"https://pypi.org/project/pydantic-ai-mcp-evals/","firstSeenAt":"2026-09-10T10:24:31.247Z","fetchedAt":"2026-10-28T20:21:52.544Z","normalizedAt":"2026-10-28T20:21:52.544Z"}]},"firstSeenAt":"2026-09-10T10:24:31.247Z","updatedAt":"2026-10-28T20:23:10.484Z"}