mcp-llm-eval

# mcp-llm-eval [](https://pypi.org/project/mcp-llm-eval/) [](https://www.python.org/downloads/) [](LICENSE) > **Status:** stable. Used in CI gates by [`mcp-content-pipeline`](https://github.com/berkayildi/mcp-content-pipeline). Live benchmarks at [llmshot.vercel.app](https://llmshot.vercel.app). A local **Model Context Protocol (MCP) server** that packages LLM evaluation gates as reusable CI/CD primitives. Run datasets against multiple models, score responses with an LLM-as-judge, and enforce quality thresholds — all through MCP tools that AI agents can call. ```mermaid flowchart LR A[PR opened] --> B[Run dataset<br/>through models] B --> C[Judge scores<br/>faithfulness + relevance] C --> D{Thresholds met?} D -->|Yes| E[PR passes] D -->|No| F[PR blocked<br/>with diff comment] ``` --- ## Why? There's no unit test for LLM quality. Teams ship prompt changes, swap models, or update system prompts with no automated way to verify that output quality didn't regress. Manual spot-checking doesn't scale, and existing eval frameworks are heavy, opinionated, and hard to wire into CI/CD. **mcp-llm-eval** gives AI agents structured access to a lightweight eval pipeline. Instead of building custom scripts for every project, you define a dataset, point the agent at it, and get scored results with pass/fail gates — the same workflow whether you're testing locally or gating a deployment. --- ## Role in ecosystem mcp-llm-eval is the evaluation engine for a small ecosystem of repos: - **Producers** that consume this library: [mcp-content-pipeline](https://github.com/berkayildi/mcp-content-pipeline) and this repo itself (dogfoods its own engine on a self-defined dataset) - **Data layer**: each produc…