{"$schema":"https://wellknown.network/schemas/agent-record-v1.json","schemaVersion":"1","id":"ag_ttyhbw5gdcrf","handle":"mcp-llm-eval","url":"https://wellknown.network/agents/mcp-llm-eval","links":{"self":"https://wellknown.network/agents/mcp-llm-eval/record.json","html":"https://wellknown.network/agents/mcp-llm-eval","markdown":"https://wellknown.network/agents/mcp-llm-eval/record.md","api":"https://wellknown.network/api/v1/agents/mcp-llm-eval","status":"https://wellknown.network/api/v1/agents/mcp-llm-eval/status","claim":"https://wellknown.network/agents/mcp-llm-eval/claim","claimApi":"https://wellknown.network/api/v1/claims","claimDescriptor":"https://wellknown.network/agents/mcp-llm-eval/claim.json","badge":"https://wellknown.network/agents/mcp-llm-eval/badge.svg","openapi":"https://wellknown.network/openapi.json","history":"https://wellknown.network/api/v1/agents/mcp-llm-eval/history","tools":"https://wellknown.network/api/v1/agents/mcp-llm-eval/tools"},"ard":{"identifier":"urn:air::server:mcp-llm-eval","type":"application/mcp-server-card+json"},"kind":"mcp_server","declared":{"name":"mcp-llm-eval","summary":"MCP server that packages LLM evaluation gates as reusable CI/CD primitives","description":"# mcp-llm-eval\n\n[](https://pypi.org/project/mcp-llm-eval/)\n[](https://www.python.org/downloads/)\n[](LICENSE)\n\n> **Status:** stable. Used in CI gates by [`mcp-content-pipeline`](https://github.com/berkayildi/mcp-content-pipeline). Live benchmarks at [llmshot.vercel.app](https://llmshot.vercel.app).\n\nA local **Model Context Protocol (MCP) server** that packages LLM evaluation gates as reusable CI/CD primitives. Run datasets against multiple models, score responses with an LLM-as-judge, and enforce quality thresholds — all through MCP tools that AI agents can call.\n\n```mermaid\nflowchart LR\n A[PR opened] --> B[Run dataset
through models]\n B --> C[Judge scores
faithfulness + relevance]\n C --> D{Thresholds met?}\n D -->|Yes| E[PR passes]\n D -->|No| F[PR blocked
with diff comment]\n```\n\n---\n\n## Why?\n\nThere's no unit test for LLM quality. Teams ship prompt changes, swap models, or update system prompts with no automated way to verify that output quality didn't regress. Manual spot-checking doesn't scale, and existing eval frameworks are heavy, opinionated, and hard to wire into CI/CD.\n\n**mcp-llm-eval** gives AI agents structured access to a lightweight eval pipeline. Instead of building custom scripts for every project, you define a dataset, point the agent at it, and get scored results with pass/fail gates — the same workflow whether you're testing locally or gating a deployment.\n\n---\n\n## Role in ecosystem\n\nmcp-llm-eval is the evaluation engine for a small ecosystem of repos:\n\n- **Producers** that consume this library: [mcp-content-pipeline](https://github.com/berkayildi/mcp-content-pipeline) and this repo itself (dogfoods its own engine on a self-defined dataset)\n- **Data layer**: each produc…","publisher":{"name":"Berkay Yildirim","url":null},"homepage":"https://github.com/berkayildi/mcp-llm-eval","repository":"https://github.com/berkayildi/mcp-llm-eval/blob/main/CHANGELOG.md","version":"0.9.2","license":"MIT","protocols":["mcp"],"tags":["benchmark","ci-cd","evaluation","llm","mcp"],"pricing":null,"endpoints":[{"url":"pypi:mcp-llm-eval","type":"package_pypi","auth":null,"probeable":false}],"skills":null,"tools":null,"extra":null,"attribution":{"kind":"pypi","name":"pypi","license":"pypi","repoUrl":"pypi","summary":"pypi","version":"pypi","description":"pypi","homepageUrl":"pypi","publisherName":"pypi"}},"derived":{"capabilities":[{"slug":"dev.ci-cd","name":"CI/CD & Deploy","confidence":1,"provenance":"declared"},{"slug":"ai.evaluation","name":"Evaluation & Benchmarks","confidence":1,"provenance":"declared"},{"slug":"code.testing","name":"Testing","confidence":0.825,"provenance":"derived"},{"slug":"dev.package-management","name":"Packages & Dependencies","confidence":0.768,"provenance":"derived"}],"categories":["ai","code","dev"],"language":"en"},"observed":{"status":"unknown","statusReason":"Distributed as a package to run locally; no network endpoint to check.","lastOkAt":null,"lastProbedAt":null,"statusComputedAt":null,"reliability30d":null,"latestObservations":[],"tools":null,"package":{"name":"mcp-llm-eval","registry":"pypi","observedAt":"2026-10-28T12:22:40.316Z","publishedAt":"2026-05-03T21:54:03.000145Z","latestVersion":"0.9.2"},"toolSurface":null,"endpointFacts":[]},"verification":{"claimed":false,"claimedAt":null,"proofs":[]},"provenance":{"sources":[{"source":"pypi","key":"mcp-llm-eval","url":"https://pypi.org/project/mcp-llm-eval/","firstSeenAt":"2026-09-10T02:23:23.226Z","fetchedAt":"2026-10-28T12:20:34.021Z","normalizedAt":"2026-10-28T12:20:34.021Z"}]},"firstSeenAt":"2026-09-10T02:23:23.226Z","updatedAt":"2026-10-28T12:22:40.316Z"}