Evaluation & Benchmarks

OSWorld-MCP: A comprehensive MCP server for computer-use agents with 158 validated tools

MCP server exposing VMAF scoring, model listing, and benchmark runs via JSON-RPC.

LiveMCPBench is a benchmark for evaluating the ability of agents to navigate and utilize a large-scale MCP toolset. It provides a comprehensive set of tasks that challenge agents to effectively use various tools in daily scenarios.

MCP as a Judge: a behavioral MCP that strengthens AI coding assistants via explicit LLM evaluations

Qualify dataset columns and process them against a target with AutoCarver, fully on your machine.

A custom MCP server using FastAPI, AWS, Firebase, and LangChain for real-time audio analysis.

House of Wisdom MCP — multi-model consultation council. Diverse AI model families investigate independently and return their own perspectives; the orchestrator weighs them. Inspired by the medieval Bayt al-Hikma.

Version-controlled golden datasets and RAG evaluation, no API key needed.

Find the cheapest AI model that passes your tests: audit repo models, build evals, routing, sizing

Read-only GitHub repos, issues, PRs and commits, with tool descriptions tuned on a benchmark.

MCP server for fraud prevention — IP reputation, email breach detection, URL safety, phone validation and risk scoring via free APIs (IPQualityScore, HaveIBeenPwned).

The trustworthy FHIR bridge for AI agents — MCP server with reproducible clinical evals, reversible de-identification, and CDS Hooks.

Modal-backed MCP server for parallel evaluation of agent-generated code variants

MCP server binary for eval-hub

MCP server exposing EssenceTeacher courses, assignments, grading and student support as agent tools

MCP server for running an EssenceStudy course from the terminal: Draft with AI, assignments, documents, real data, rubrics, enrolment, grading and usage

Hardening and trajectory evals for MCP servers that let agents act on booking systems: scoped, audience-bound tokens, OAuth JWTs, an encrypted audit log, encryption at rest, rate limits and a CI eval gate.

Score Python MCP defences on the mcp-defense-bench corpus, with the same metrics as the JS leaderboard.

Guardrailed video editing for AI agents: FFmpeg, captions, effects, Hyperframes, and receipts.

Behavior-compatible Python MCP server exposing TypeSafe's Jev judgment tools.