Evaluation & Benchmarks

An MCP server for BigQuery exploration with cost guardrails and a built-in NL→SQL eval harness.

Detect pytest config-leakage that corrupts agent-benchmark grading. Deterministic, no-LLM.

A benchmark for MCP.

Agentic security testing for MCP servers

MCP as a Judge: a behavioral MCP that strengthens AI coding assistants via explicit LLM evaluations

Quality verification for AI agents and MCP servers. 6-axis scoring, adversarial probes.

Fail-closed quality gate and hash-chained receipt ledger for AI agent workflows.

MCP Server for LLM comparison, benchmarks, and pricing — find the best model for any task

Runs a throughput benchmark for LLM APIs, measuring generation throughput, prompt throughput, and Time To First Token (TTFT) under various concurrency levels.

Pre-curated canonical memory + prose/code provenance for any AI coding client. MCP-native.

Model Context Protocol server for Yahoo Fantasy Sports API with advanced historical analysis and manager profiling

MCP server for kensa, the open source agent evals harness

Judge QA for agents — measures whether your LLM judge agrees with humans, drifted between runs, or is biased, before you ship the number

ToolGym: An Open-world Tool-using Environment for LLM Agent Evaluation

A comprehensive Model Context Protocol server for YouTube Data API access with 14 functions including transcript extraction and intelligent content evaluation

OSWorld-MCP: A comprehensive MCP server for computer-use agents with 158 validated tools

MCP server exposing VMAF scoring, model listing, and benchmark runs via JSON-RPC.

LiveMCPBench is a benchmark for evaluating the ability of agents to navigate and utilize a large-scale MCP toolset. It provides a comprehensive set of tasks that challenge agents to effectively use various tools in daily scenarios.

MCP as a Judge: a behavioral MCP that strengthens AI coding assistants via explicit LLM evaluations

Qualify dataset columns and process them against a target with AutoCarver, fully on your machine.