Evaluate MCP server accuracy against known questions and answers
Wellknown found it in public sources; nobody has proven control of it yet. Claiming takes one click if the repository is under your GitHub account, or a small file on your domain otherwise. Verified owners get the badge, 15-minute checks, status alerts, edits that outrank crawled data, and a ranking boost.
Agents can do it too: POST https://wellknown.network/api/v1/claims with {"agent":"mcp-data-check","method":"well_known_file"} — machine-readable steps at claim.json, guide at /docs/claim.
Everything here was measured by our prober or read from a registry. Nothing is self-reported.
Attributed to the source that supplied each field. Treated as claims, not facts.
# mcp-data-check Evaluate MCP server accuracy against known questions and answers. ## Installation ```bash pip install mcp-data-check ``` Or install from source: ```bash pip install -e . ``` ## Usage ### Python API **Anthropic (default)** ```python from mcp_data_check import run_evaluation results = run_evaluation( questions_filepath="questions.csv", api_key="sk-ant-...", server_url="https://mcp.example.com/sse" ) print(f"Pass rate: {results['summary']['pass_rate']:.1%}") print(f"Passed: {results['summary']['passed']}/{results['summary']['total']}") ``` **OpenAI** ```python from mcp_data_check import run_evaluation results = run_evaluation( questions_filepath="questions.csv", api_key="sk-...", server_url="https://mcp.example.com/sse", provider="openai", model="gpt-4o" ) ``` ### Command Line **Anthropic (default)** ```bash mcp-data-check https://mcp.example.com/sse -q questions.csv -k YOUR_API_KEY ``` **OpenAI** ```bash mcp-data-check https://mcp.example.com/sse -q questions.csv -p openai -m gpt-4o -k YOUR_API_KEY ``` **Baseline comparison (MCP vs no tools)** ```bash # Anthropic mcp-data-check https://mcp.example.com/sse -q questions.csv --compare # OpenAI mcp-data-check https://mcp.example.com/sse -q questions.csv -p openai -m gpt-4o --compare ``` Each question is run `--repeats` times per mode (default 5). Pass/fail is determined by majority vote. Results are saved to `./results/comparison_<timestamp>.json`. Options: - `-q, --questions`: Path to questions CSV file (required) - `-p, --provider`: LLM provider to use: `anthropic` (default) or `openai` - `-k, --api-key`: API key for the chosen provider (defaults to `ANTHROPIC_API_KEY` or `OPENAI_API_KEY` env var) - `-o, --output`: Output directory for results (default: `./results`) - `-m, --model`: Model to use for evaluation (default: `claude-sonnet-4-20250514`; use e.g. `gpt-4o` for OpenAI) - `-n, --server-name`: Name for the MCP server (default: `mcp-server`) …
Mapped onto the structured taxonomy from declared text and observed tool names. Confidence shown for derived entries.
Every source is kept verbatim. Field changes are logged as events.