{"$schema":"https://wellknown.network/schemas/agent-record-v1.json","schemaVersion":"1","id":"ag_ztcr4mnwp78a","handle":"judge-audit-mcp","url":"https://wellknown.network/agents/judge-audit-mcp","links":{"self":"https://wellknown.network/agents/judge-audit-mcp/record.json","html":"https://wellknown.network/agents/judge-audit-mcp","markdown":"https://wellknown.network/agents/judge-audit-mcp/record.md","api":"https://wellknown.network/api/v1/agents/judge-audit-mcp","status":"https://wellknown.network/api/v1/agents/judge-audit-mcp/status","claim":"https://wellknown.network/agents/judge-audit-mcp/claim","claimApi":"https://wellknown.network/api/v1/claims","claimDescriptor":"https://wellknown.network/agents/judge-audit-mcp/claim.json","badge":"https://wellknown.network/agents/judge-audit-mcp/badge.svg","openapi":"https://wellknown.network/openapi.json","history":"https://wellknown.network/api/v1/agents/judge-audit-mcp/history","tools":"https://wellknown.network/api/v1/agents/judge-audit-mcp/tools"},"ard":{"identifier":"urn:air::server:judge-audit-mcp","type":"application/mcp-server-card+json"},"kind":"mcp_server","declared":{"name":"judge-audit-mcp","summary":"Judge QA for agents — measures whether your LLM judge agrees with humans, drifted between runs, or is biased, before you ship the number","description":"# judge-audit-mcp\n\n[![PyPI](https://img.shields.io/pypi/v/judge-audit-mcp)](https://pypi.org/project/judge-audit-mcp/)\n[![Python](https://img.shields.io/pypi/pyversions/judge-audit-mcp)](https://pypi.org/project/judge-audit-mcp/)\n[![License: MIT](https://img.shields.io/badge/license-MIT-green)](https://github.com/asif786ka/judge-audit-mcp/blob/main/LICENSE)\n[![Tests](https://img.shields.io/badge/tests-171%20passing-brightgreen)](https://github.com/asif786ka/judge-audit-mcp/blob/main/smoke_test.py)\n\n**Your eval scores went up 6%. Nobody checked whether the judge changed.**\n\nAn MCP server that audits the LLM judge grading your evals: whether it agrees\nwith humans, whether it drifted between runs, and whether it's biased. Everyone\nruns LLM-as-judge. Platforms ship judges. **None of them ship judge QA.**\n\n```\n$ judge-audit drift run_v1.jsonl run_v2.jsonl --fail-on serious\n\n🛑 Judge changed between runs — 1 field(s) differ\n    These two runs were scored by different instruments. Their means are not on\n    the same scale, so the difference between them is not an improvement or a\n    regression — it is not a quantity at all until the judge's own shift is\n    measured and removed.\n\n🛑 85% of the +6.0% score change is the judge, not the system\n    Headline: 6.873 → 7.287 (+6.0%) over 150 common item(s).\n    On 40 anchor item(s) — items whose output text is byte-identical in both\n    runs, so the system provably did not change — the judge scored +0.350\n    (+5.1%) higher in run B.\n    That movement is the judge's, by construction. Removing it leaves +0.063\n    for your system.\n    The judge accounts for 85% of the headline.\n    The improvement is the judge.\n    delta (judge shift on anchors): +0.350  95% CI [+0.200, +0.500]  n=40\n\n✗ failing: 2 finding(s) at or above serious\n```\n\nThat's the whole product. Someone bumped the judge model between two eval runs.\nNobody wrote it down, because **the judge isn't the thing under test** — its\nversion doesn't belong in the changelog of…","publisher":{"name":"Asif","url":null},"homepage":"https://github.com/asif786ka/judge-audit-mcp","repository":"https://github.com/asif786ka/judge-audit-mcp","version":"0.1.1","license":"MIT","protocols":["mcp"],"tags":["bias","calibration","cohens-kappa","evals","evaluation","judge-drift","llm-as-judge","llm-evaluation","mcp","position-bias","self-preference"],"pricing":null,"endpoints":[{"url":"pypi:judge-audit-mcp","type":"package_pypi","auth":null,"probeable":false}],"skills":null,"tools":null,"extra":null,"attribution":{"kind":"pypi","name":"pypi","license":"pypi","repoUrl":"pypi","summary":"pypi","version":"pypi","description":"pypi","homepageUrl":"pypi","publisherName":"pypi"}},"derived":{"capabilities":[{"slug":"ai.evaluation","name":"Evaluation & Benchmarks","confidence":1,"provenance":"declared"}],"categories":["ai"],"language":"en"},"observed":{"status":"unknown","statusReason":"Distributed as a package to run locally; no network endpoint to check.","lastOkAt":null,"lastProbedAt":null,"statusComputedAt":null,"reliability30d":null,"latestObservations":[],"tools":null,"package":{"name":"judge-audit-mcp","registry":"pypi","observedAt":"2026-10-28T08:27:57.003Z","publishedAt":"2026-07-15T16:42:40.149723Z","latestVersion":"0.1.1"},"toolSurface":null,"endpointFacts":[]},"verification":{"claimed":false,"claimedAt":null,"proofs":[]},"provenance":{"sources":[{"source":"pypi","key":"judge-audit-mcp","url":"https://pypi.org/project/judge-audit-mcp/","firstSeenAt":"2026-09-09T22:22:45.515Z","fetchedAt":"2026-10-28T08:25:55.408Z","normalizedAt":"2026-10-28T08:25:55.408Z"}]},"firstSeenAt":"2026-09-09T22:22:45.515Z","updatedAt":"2026-10-28T08:27:57.003Z"}