Judge QA for agents — measures whether your LLM judge agrees with humans, drifted between runs, or is biased, before you ship the number
Wellknown found it in public sources; nobody has proven control of it yet. Claiming takes one click if the repository is under your GitHub account, or a small file on your domain otherwise. Verified owners get the badge, 15-minute checks, status alerts, edits that outrank crawled data, and a ranking boost.
Agents can do it too: POST https://wellknown.network/api/v1/claims with {"agent":"judge-audit-mcp","method":"well_known_file"} — machine-readable steps at claim.json, guide at /docs/claim.
Everything here was measured by our prober or read from a registry. Nothing is self-reported.
Attributed to the source that supplied each field. Treated as claims, not facts.
# judge-audit-mcp [](https://pypi.org/project/judge-audit-mcp/) [](https://pypi.org/project/judge-audit-mcp/) [](https://github.com/asif786ka/judge-audit-mcp/blob/main/LICENSE) [](https://github.com/asif786ka/judge-audit-mcp/blob/main/smoke_test.py) **Your eval scores went up 6%. Nobody checked whether the judge changed.** An MCP server that audits the LLM judge grading your evals: whether it agrees with humans, whether it drifted between runs, and whether it's biased. Everyone runs LLM-as-judge. Platforms ship judges. **None of them ship judge QA.** ``` $ judge-audit drift run_v1.jsonl run_v2.jsonl --fail-on serious 🛑 Judge changed between runs — 1 field(s) differ These two runs were scored by different instruments. Their means are not on the same scale, so the difference between them is not an improvement or a regression — it is not a quantity at all until the judge's own shift is measured and removed. 🛑 85% of the +6.0% score change is the judge, not the system Headline: 6.873 → 7.287 (+6.0%) over 150 common item(s). On 40 anchor item(s) — items whose output text is byte-identical in both runs, so the system provably did not change — the judge scored +0.350 (+5.1%) higher in run B. That movement is the judge's, by construction. Removing it leaves +0.063 for your system. The judge accounts for 85% of the headline. The improvement is the judge. delta (judge shift on anchors): +0.350 95% CI [+0.200, +0.500] n=40 ✗ failing: 2 finding(s) at or above serious ``` That's the whole product. Someone bumped the judge model between two eval runs. Nobody wrote it down, because **the judge isn't the thing under test** — its version doesn't belong in the changelog of…
Mapped onto the structured taxonomy from declared text and observed tool names. Confidence shown for derived entries.
Every source is kept verbatim. Field changes are logged as events.