{"$schema":"https://wellknown.network/schemas/agent-record-v1.json","schemaVersion":"1","id":"ag_ashsjawns49v","handle":"eval-mcp","url":"https://wellknown.network/agents/eval-mcp","links":{"self":"https://wellknown.network/agents/eval-mcp/record.json","html":"https://wellknown.network/agents/eval-mcp","markdown":"https://wellknown.network/agents/eval-mcp/record.md","api":"https://wellknown.network/api/v1/agents/eval-mcp","status":"https://wellknown.network/api/v1/agents/eval-mcp/status","claim":"https://wellknown.network/agents/eval-mcp/claim","claimApi":"https://wellknown.network/api/v1/claims","claimDescriptor":"https://wellknown.network/agents/eval-mcp/claim.json","badge":"https://wellknown.network/agents/eval-mcp/badge.svg","openapi":"https://wellknown.network/openapi.json","history":"https://wellknown.network/api/v1/agents/eval-mcp/history","tools":"https://wellknown.network/api/v1/agents/eval-mcp/tools"},"ard":{"identifier":"urn:air::server:eval-mcp","type":"application/mcp-server-card+json"},"kind":"mcp_server","declared":{"name":"eval-mcp","summary":"Pytest-style framework for evaluating Model Context Protocol (MCP) servers.","description":"# MCP-Eval: An Evaluation Framework for MCP Servers\n\nMCP-Eval is a developer-first testing framework for Model Context Protocol (MCP) servers, built on the `mcp-agent` library. It enables you to write clear, concise, and powerful tests to evaluate the performance, reliability, and correctness of your AI agents and the MCP servers they connect to.\n\n## Core Features\n\n- **Task-Based Testing**: Define tests as async functions where an agent performs a task.\n- **Automatic Metrics**: Automatically collect detailed metrics on latency, token usage, cost, and tool calls for every test run.\n- **Rich Assertions**: A powerful set of assertions designed for AI testing, including:\n    - `contains()`: Checks for substrings in responses.\n    - `tool_was_called()`: Verifies that a specific tool was used.\n    - `tool_arguments_match()`: Checks if a tool was called with the correct arguments.\n    - `cost_under()`: Asserts that a test run stays within a defined cost budget.\n    - `number_of_steps_under()`: Ensures an agent completes a task efficiently.\n    - `objective_succeeded()`: Uses an LLM to verify if the agent's response achieved the overall goal.\n    - `plan_is_efficient()`: Uses an LLM to check for redundant or inefficient steps in the agent's execution path.\n- **Tool Coverage Reporting**: Automatically calculates the percentage of a server's tools that are exercised by your test suite.\n- **Automated Test Generation**: A CLI tool to automatically generate a baseline test suite for any MCP server.\n- **Detailed Reports**: Get immediate feedback from rich console reports and generate detailed JSON reports for CI/CD or further analysis.\n\n## Getting Started\n\n### 1. Installation\n\nInstall `mcp_eval` and its dependencies. Make sure `mcp-agent` is also installed in your environment.\n\n```bash\npip install \"typer[all]\" rich pydantic jinja2","publisher":null,"homepage":null,"repository":null,"version":"0.1.1","license":"Apache License Version 2.0, January 2004 http://www.apache.org/…","protocols":["mcp"],"tags":["mcp"],"pricing":null,"endpoints":[{"url":"pypi:eval-mcp","type":"package_pypi","auth":null,"probeable":false}],"skills":null,"tools":null,"extra":null,"attribution":{"kind":"pypi","name":"pypi","license":"pypi","summary":"pypi","version":"pypi","description":"pypi"}},"derived":{"capabilities":[{"slug":"dev.ci-cd","name":"CI/CD & Deploy","confidence":1,"provenance":"derived"},{"slug":"code.testing","name":"Testing","confidence":0.859,"provenance":"derived"}],"categories":["code","dev"],"language":"en"},"observed":{"status":"unknown","statusReason":"Distributed as a package to run locally; no network endpoint to check.","lastOkAt":null,"lastProbedAt":null,"statusComputedAt":null,"reliability30d":null,"latestObservations":[],"tools":null,"package":{"name":"eval-mcp","registry":"pypi","observedAt":"2026-09-15T14:26:58.100Z","publishedAt":"2025-08-20T03:23:37.923623Z","latestVersion":"0.1.1"},"toolSurface":null,"endpointFacts":[]},"verification":{"claimed":false,"claimedAt":null,"proofs":[]},"provenance":{"sources":[{"source":"pypi","key":"eval-mcp","url":"https://pypi.org/project/eval-mcp/","firstSeenAt":"2026-09-09T15:21:54.753Z","fetchedAt":"2026-09-15T14:25:01.960Z","normalizedAt":"2026-09-15T14:25:01.960Z"}]},"firstSeenAt":"2026-09-09T15:21:54.753Z","updatedAt":"2026-09-15T14:26:58.100Z"}