A drop-in confidence gate for LLM agents: turn token logprobs into an accept / verify / ask-a-human routing decision, with honest per-model calibration.
Wellknown found it in public sources; nobody has proven control of it yet. Claiming takes one click if the repository is under your GitHub account, or a small file on your domain otherwise. Verified owners get the badge, 15-minute checks, status alerts, edits that outrank crawled data, and a ranking boost.
Agents can do it too: POST https://wellknown.network/api/v1/claims with {"agent":"mcp-confidence","method":"well_known_file"} — machine-readable steps at claim.json, guide at /docs/claim.
Everything here was measured by our prober or read from a registry. Nothing is self-reported.
Attributed to the source that supplied each field. Treated as claims, not facts.
<h1 align="center">mcp-confidence</h1> <p align="center"> <strong>A drop-in confidence gate for LLM agents.</strong><br> Turn any model's token logprobs into an <em>accept / verify / ask-a-human</em> routing decision —<br> with honest, per-model calibration. Local & open-model first. </p> <p align="center"> <a href="https://pypi.org/project/mcp-confidence/"><img src="https://img.shields.io/pypi/v/mcp-confidence.svg" alt="PyPI version"></a> <a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-blue.svg" alt="MIT License"></a> <img src="https://img.shields.io/badge/python-3.10%20%7C%203.11%20%7C%203.12%20%7C%203.13-3776AB?logo=python&logoColor=white" alt="Python 3.10+"> </p> --- A language model can tell you *how* surprised it was by every token it just produced — that's what `logprobs=True` returns. `mcp-confidence` reads those numbers and turns them into a routing decision your agent can act on: - **HIGH** → auto-accept the model's output. - **MID** → verify (hand off to a stronger model or a supervisor check). - **LOW** → ask a human. It's **local- and open-model-first** (vLLM, Ollama, llama.cpp, TGI, or any OpenAI-compatible endpoint), the core has **zero third-party runtime dependencies**, and it ships a one-command **MCP server** so a cloud "director" (e.g. Claude) can delegate generation to a local worker and get a confidence band back with every answer. ## Why The naive approach — "average the token probabilities, accept above 0.95" — is a weak and *uncalibrated* signal for open-domain free text, and quietly wrong on most models. High-frequency function words and MoE routing pull the per-token mean toward zero, so `exp(mean_logprob)` sits at 0.05–0.20 even for perfectly fluent output. A fixed probability threshold tuned on short structured outputs will reject good answers and accept bad ones once you point it at real prose. `mcp-confidence` takes four deliberate positions to make the signal usable: 1. **Band in mean-lo…
Mapped onto the structured taxonomy from declared text and observed tool names. Confidence shown for derived entries.
Every source is kept verbatim. Field changes are logged as events.