{"$schema":"https://wellknown.network/schemas/agent-record-v1.json","schemaVersion":"1","id":"ag_j6dpz6g7pbj3","handle":"mcp-tool-evals","url":"https://wellknown.network/agents/mcp-tool-evals","links":{"self":"https://wellknown.network/agents/mcp-tool-evals/record.json","html":"https://wellknown.network/agents/mcp-tool-evals","markdown":"https://wellknown.network/agents/mcp-tool-evals/record.md","api":"https://wellknown.network/api/v1/agents/mcp-tool-evals","status":"https://wellknown.network/api/v1/agents/mcp-tool-evals/status","claim":"https://wellknown.network/agents/mcp-tool-evals/claim","claimApi":"https://wellknown.network/api/v1/claims","claimDescriptor":"https://wellknown.network/agents/mcp-tool-evals/claim.json","badge":"https://wellknown.network/agents/mcp-tool-evals/badge.svg","openapi":"https://wellknown.network/openapi.json","history":"https://wellknown.network/api/v1/agents/mcp-tool-evals/history","tools":"https://wellknown.network/api/v1/agents/mcp-tool-evals/tools"},"ard":{"identifier":"urn:air::server:mcp-tool-evals","type":"application/mcp-server-card+json"},"kind":"mcp_server","declared":{"name":"mcp-tool-evals","summary":"Behavioral evaluation and description linting for MCP servers","description":"# mcp-evals\n\nBehavioral evaluation and description linting for [Model Context Protocol](https://modelcontextprotocol.io) (MCP) servers.\n\n[![PyPI](https://img.shields.io/pypi/v/mcp-tool-evals.svg)](https://pypi.org/project/mcp-tool-evals/)\n[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)\n[![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/)\n[![CI](https://github.com/yashshah9/mcp-evals/actions/workflows/ci.yml/badge.svg)](https://github.com/yashshah9/mcp-evals/actions/workflows/ci.yml)\n\n> **Status:** v0.8 — stdio/HTTP/SSE discovery, description lint, mock/OpenAI evals with **argument scoring**, GitHub Action PR comments.\n\n## 60-second try\n\n```bash\npip install mcp-tool-evals\nmcp-evals health\nmcp-evals run examples/eval-suite.yaml --catalog examples/tools.yaml --model mock\n# or with Docker (no local Python):\ndocker compose run --rm run-example\n```\n\n## Why this vs alternatives\n\n| Approach | Strength | Gap |\n|----------|----------|-----|\n| **mcp-evals** | Lint + YAML evals + mock CI runner | Not a full agent harness |\n| MCP Inspector | Interactive debugging | No CI lint/eval suite |\n| Protocol conformance tests | JSON-RPC correctness | Do not test tool selection |\n| Hand-written LLM mocks | Full control | Drift from real descriptions |\n\n## Problem\n\nProtocol conformance tests verify JSON-RPC correctness. They do **not** verify whether an LLM agent selects the right tool, passes sensible arguments, or completes the task. Tool descriptions are now load-bearing API design, and there is no standard way to test them.\n\n**mcp-evals** fills the behavioral layer: lint tool descriptions, define eval cases in YAML, and (next) measure tool-selection accuracy in CI.\n\n## Key features (v0.8)\n\n- **Description linter** — missing descriptions, undocumented required params, overlapping tools, ambiguous verbs, `name-description-mismatch`, `missing-examples`\n- **Live discover** — handshake a stdio, HTTP JSON-RP…","publisher":null,"homepage":"https://github.com/yashshah9/mcp-evals#readme","repository":"https://github.com/yashshah9/mcp-evals#readme","version":"0.8.0","license":null,"protocols":["mcp"],"tags":["agents","llm","mcp","model-context-protocol","testing"],"pricing":null,"endpoints":[{"url":"pypi:mcp-tool-evals","type":"package_pypi","auth":null,"probeable":false}],"skills":null,"tools":null,"extra":null,"attribution":{"kind":"pypi","name":"pypi","repoUrl":"pypi","summary":"pypi","version":"pypi","description":"pypi","homepageUrl":"pypi"}},"derived":{"capabilities":[{"slug":"ai.evaluation","name":"Evaluation & Benchmarks","confidence":1,"provenance":"derived"},{"slug":"code.testing","name":"Testing","confidence":1,"provenance":"declared"}],"categories":["ai","code"],"language":"en"},"observed":{"status":"unknown","statusReason":"Distributed as a package to run locally; no network endpoint to check.","lastOkAt":null,"lastProbedAt":null,"statusComputedAt":null,"reliability30d":null,"latestObservations":[],"tools":null,"package":{"name":"mcp-tool-evals","registry":"pypi","observedAt":"2026-10-28T14:24:48.103Z","publishedAt":"2026-09-18T18:13:13.197005Z","latestVersion":"0.8.0"},"toolSurface":null,"endpointFacts":[]},"verification":{"claimed":false,"claimedAt":null,"proofs":[]},"provenance":{"sources":[{"source":"pypi","key":"mcp-tool-evals","url":"https://pypi.org/project/mcp-tool-evals/","firstSeenAt":"2026-09-16T04:19:19.292Z","fetchedAt":"2026-10-28T14:24:01.659Z","normalizedAt":"2026-10-28T14:24:01.659Z"}]},"firstSeenAt":"2026-09-16T04:19:19.292Z","updatedAt":"2026-10-28T14:24:48.103Z"}