IA-QA Eval & RAG — 38 tools

Name: IA-QA Eval & RAG — 38 tools
Author: JcJamet

MCP serverMCP

LLM & RAG evals: hallucination, eval contracts, semantic tests, tokens & model info. Free, no key.

by JcJametrepository ↗v1.0.1

Is this yours?

This record is unclaimed

Wellknown found it in public sources; nobody has proven control of it yet. Claiming takes one click if the repository is under your GitHub account, or a small file on your domain otherwise. Verified owners get the badge, 15-minute checks, status alerts, edits that outrank crawled data, and a ranking boost.

Agents can do it too: POST https://wellknown.network/api/v1/claims with {"agent":"ia-qa-eval-rag-38-tools","method":"well_known_file"} — machine-readable steps at claim.json, guide at /docs/claim.

Claim this record

Observed by Wellknown

Evidence

Everything here was measured by our prober or read from a registry. Nothing is self-reported.

30-day availability3/3 checks ok

30 days ago

Tools observed via MCP handshake

get_testing_guidelines — Query the IA-QA methodology knowledge base. Returns structured testing guidelines, assertion strategies, thresholds, best practices, and relevant MCP tools for
find_tool — Search available MCP tools by keyword or category before calling them. Returns matching tool names, descriptions, and optionally their inputSchemas. Call this w
hallucination_check — Lexical hallucination check: verifies an LLM answer's words, numbers and polarity against the provided source/context. Fast, deterministic, no API key needed. E
run_eval_contract — Parse a .ia-eval.yaml LLM test suite, call the specified LLM model for each scenario, run all configured scorers, and return a structured JSON report with per-s
run_semantic_tests — Semantic assertion primitive: compare actual vs expected text pairs using cosine similarity + ROUGE-L. Two modes: tfidf (default, free, no API key) or embedding
generate_ci_workflow — Generate a ready-to-commit GitHub Actions workflow that gates a build on IA-QA. Two gate types, combinable: "eval_contract" runs a .ia-eval.yaml through ia-qa-c
needle_haystack_generate — Generate a "needle in a haystack" test: embeds a target fact into a large block of filler text at a specified position. Use this to test LLM context window retr
generate_eval_yaml — Generate a complete .ia-eval.yaml evaluation contract from a plain-language description of what your LLM should do. Uses Groq openai/gpt-oss-20b (server-side, n
metamorphic_check — Reference-free stability primitive: instead of comparing an answer to a ground truth, it checks that an assistant's answer stays INVARIANT when the QUESTION is
test_skill — Test a SKILL.md in the open Agent Skills format (https://agentskills.io/specification — read by Claude Code, Codex, GitHub Copilot, Cursor, Gemini CLI, OpenCode
llm_output_validator — Validate an LLM response against QA criteria: format checks (JSON, code, markdown), content rules (must-include, must-not-include), length constraints, language
compare_responses — Compare two ALREADY-PRODUCED outputs (e.g. model A vs model B on the same task) side by side. Returns deterministic metrics (token cosine, ROUGE-L, Jaccard, len
analyze_responses — Semantically analyze N already-produced model outputs for the SAME task (the MCP counterpart to the LLM Sandbox). Without a reference: computes consensus — pair
prompt_test_suite — Define a test suite for a prompt: provide the system prompt, user prompt, and expected output criteria. Returns a test plan with scored rubric — use this as inp
llm_format_check — Validate that an LLM output matches an expected format: JSON, Markdown, code block, bullet list, numbered list, table, YAML, XML, or custom regex. Essential for
consistency_check — Compare multiple LLM responses to the same prompt and detect inconsistencies using Jaccard word-overlap similarity and fact drift (number comparison). Fast, det
llm_json_schema_check — Validate that an LLM JSON output matches a JSON Schema definition. Tests required fields, types, enums, nested objects, and arrays. Critical for function-callin
response_quality_score — Score an LLM response against the criteria you pass: coverage of expected_keywords and compliance with max_length. Returns a 0-100 score over the criteria actua
conversation_analyze — Analyze a multi-turn conversation for context retention, topic drift, instruction following, and repetition. Accepts messages array [{role, content}]. Essential
cot_analyzer — Analyze a Chain-of-Thought (CoT) or reasoning trace from an LLM. Detects step count, logical flow, conclusion presence, backtracking, and estimates reasoning de
ab_test_report — Generate an A/B test report comparing two prompts or model configurations. Accepts arrays of scores and returns mean, median, std deviation, improvement percent
similarity_score — Compute text similarity between reference and hypothesis using multiple metrics: Cosine (BoW, TF-IDF), Jaccard, ROUGE-1, ROUGE-2, ROUGE-L, and BLEU. No API key
run_vlm_test_suite — Run a test suite against a Vision-Language Model (VLM) — send an image (URL or base64) + N test cases (each with a question + assertion) to GPT-4o, Claude 3.5,
embedding_similarity — Compute text similarity using local algorithms (Bag of Words, TF-IDF, Character N-grams). No API key needed — runs entirely in-process. NOT real embeddings: for
rag_relevance_rank — Rank an array of text chunks by relevance to a query using TF-IDF scoring. Simulates retrieval ranking for RAG testing without needing embeddings or an API.
bm25_score — Compute BM25 relevance score between a query and one or more documents. BM25 is the industry-standard keyword-based ranking algorithm used in Elasticsearch, Ope
build_rag_prompt — Assemble a complete RAG (Retrieval-Augmented Generation) prompt from retrieved context chunks and a user query. Handles token budgeting, citation numbering, sys
rerank_evaluate — Evaluate RAG retrieval quality: rank passages against a query and compute Precision@k / Recall@k plus a PASS/FAIL CI verdict from ground-truth relevance labels.
count_tokens — Estimate the token count of a text string using the cl100k_base approximation (~4 chars/token). Call this BEFORE sending any text to an LLM API to check if it f
extract_json_from_text — Extract the first valid JSON object or array embedded in chaotic LLM output (surrounded by markdown fences, prose, or explanatory text). Handles ```json blocks
estimate_llm_cost — Estimate the API cost in USD for a given model and token counts. Supports all major 2024–2026 models: GPT-4o, GPT-4.1, o3, o4-mini, Claude Opus 4, Claude Sonnet
token_budget_calculator — Plan token allocation across system prompt, user input, context/RAG chunks, and expected output. Warns if budget exceeds model context window. Supports 25+ mode
context_window_check — Given an array of message objects [{role, content}], estimate total token usage and check if it fits in the target model's context window. Warns about truncatio
model_info — Get detailed specs for an AI model: context window, pricing per 1K tokens, knowledge cutoff, provider, multimodal support, reasoning capabilities, and feature l
compare_models — Compare 2-5 AI models side by side: context window, pricing, multimodal, reasoning capabilities, and provider. Returns a comparison table with a recommendation
list_llm_models — List all LLM models available on ia-qa.com with their provider, API endpoint, and capabilities. Filter by provider name (e.g. "Groq", "HuggingFace", "OpenAI") o
llm_generate — Generate text with an open-source model hosted on Groq, on the server's key — no API key needed. Models: openai/gpt-oss-20b, openai/gpt-oss-120b (default "opena
rate_tool — Give honest usage feedback on an IA-QA MCP tool. Provide a score (1-5) and a comment. Rate low (1-2) if the tool was wrong, irrelevant, or a poor fit; rate high

when	check	result	http	latency	detail
5 h ago	mcp initialize	ok	200	18 ms	server ia-qa-toolbox-eval 1.0.0 · protocol 2025-06-18 · 38 tools
12 h ago	mcp initialize	ok	200	18 ms	server ia-qa-toolbox-eval 1.0.0 · protocol 2025-06-18 · 38 tools
20 h ago	mcp initialize	ok	200	19 ms	server ia-qa-toolbox-eval 1.0.0 · protocol 2025-06-18 · 38 tools

IA-QA Eval & RAG — 38 tools

This record is unclaimed

Evidence

What the publisher says

Capabilities

Where this record came from