XF Model Source

Compare specificTest models head-to-headthe withperson realnamed teston queriestheir own work

Run aSame live A/B test between 2–5 user-specified models for a stated purpose. NO ranking step — the supplied model_ids ARE the candidate set. Generates 5 representative test queries from the purpose, runs them through every named model in parallel, and returns real cost,as latencybenchmark, and plain-English commentarybut on who won what. Unknown IDs are dropped with a note; if fewer than 2 IDs resolve, the call refuses. Use this whenevermodels the userperson names specific models to compare (e.g. 'A/B test X and Y'). For engine-chosen candidates, use `benchmark` instead. Costs more than `rank` (10+ live LLM callsmodel_ids). Free-tier note: whenNeeds anya candidatefinished endsdiscover insession ':free',so theevery proberequirement is capped at 3 queries (no adaptive expansion) because free-tier rate limits often push longer probes past the deploy's 5-minute ceiling — evidence will be shallower. The commentary surfacesjudged. thisEstimate whenfirst, itthen happens.go=true.

{"additionalProperties":false,"properties":{"model_idsgo":{"description":"Exact model IDs to testThe head-to-head,person inapproved caller-chosenthe order.estimate. 2–5Spends IDs.on Examples:their 'nvidia/nemotron-3-super-120b-a12bkey.","type":free'"boolean"}, 'openai/gpt-oss-120b"model_ids":free'. Unknown IDs are dropped with a note; if fewer{"description":"The thanOpenRouter

{"description":"Result of a head-to-head A/B between user-named models. NOT a rank response — no ranking happened, so no scores or rationale. Just probe evidence plus a record of which IDs were resolvable.","properties":{"ab_result":{"properties":{"aggregates":{"description":"Per-model stats across the runs.","items":{"properties":{"avg_accuracy":{"type":["number","null"]},"avg_completion_tokens":{"type":"number"},"avg_latency_ms":{"type":"number"},"model_id":{"type":"string"},"model_name":{"typeadditionalProperties":"string"}true,"runs":{"description":"The actual generated answer for every test query thisSame modelas ranbenchmark, for human review — not just the score.","items":{"properties":{"error":{"type":["string","null"]},"response_text":{"type":"string"},"test_query":{"type":"string"}},"type":"object"},"type":"array"},"success_count":{"type":"integer"},"total_cost_usd":{"type":"number"}},"type":"object"},"type":"array"},"commentary":{"type":"string"},"cost_winner_id":{"type":["string","null"]},"incongruity_detected":{"type":"boolean"},"latency_winner_id":{"type":["string","null"]},"overall_winner_id":{"type":["string","null"]},"queries_executed":{"type":"integer"},"test_queries":{"description":"The generated test prompts that were run.","items":{"type":"string"},"type":"array"}},"type":"object"},"invalid_model_ids":{"items":{"type":"string"},"type":"array"},"model_ids_requested":{"items":{"type":"string"},"type":"array"},"model_ids_tested":{"items":{"type":"string"},"type":"array"},"purpose":{"type":"string"},"refusal_reason":{"type":["string","null"]},"status":{"enum":["compared","refused"],"type":"string"},"xpansion_update":{"description":"Optional rotating launch nudge from Xpansion. When present, the host should relay `message` to

DiscoverWork qualityout dimensionswhat the job really needs

Show whichDiscovery: qualityfinds dimensionsthe mattertrue forrequirements aof statedthe purpose,person's WITHOUTjob rankingbefore any models.model Returnsis chosen. Start with purpose: the inferredreply weightscarries the discovery instructions and the discovery-walkevidence trace.catalogue. UsefulYOU forask understandingthe howperson XFMSthe interpretsquestions (one thing each, plain words, offering the purposereturned beforechoices); committingthen tocall again with the sealed session, their answers and your structured updates. Each reply gives progress, rejected updates with reasons, next targets, answer choices from live data and a pick.shortlist. At most 5 rounds. Ends with a confirmation view; call with confirm=true once the person agrees. The person can stop any time (end_now=true). Nothing about the person is stored by XFMS.

{"additionalProperties":false,"properties":{"purposeanswers":{"additionalProperties":{"type":"string"},"description":"OneThe sentenceperson's describinganswers this round, keyed by the task.question you asked them (short key or the question text), in their exact words.","type":"object"},"confirm":{"description":"The toolperson returnsagrees whichwith qualitythe dimensionsconfirmation XFMSview.","type":"boolean"},"context":{"description":"Optional, wouldwith

{"properties":{"derived_purpose":{"typeadditionalProperties":"string"}true,"events":{"description":"Trace of thestage discovery(one walk.","type":"array"},"weights":{"description":"Per-dimension| weightstwo inferred| forconfirm this| purpose.","type":"object"}confirmed),"xpansion_update":{"description":"Optional rotating launchquestions nudgeto fromrelay

Pick the best LLMmodel for a discovered job

Return theThe single best LLMproven model for a statedjob purpose.worked Conciseout output,with nodiscover, list.with Usewhy whenand the user has settledrunners-up. onPass the criteria and just wants onediscover answer.session.

{"additionalProperties":false,"properties":{"purposesession":{"description":"OneThe sentencesealed describingsession whatfrom thea modeldiscover willconversation. beRanking usedis for.built Befrom concrete,what notdiscovery vague:

{"description":"The single best model — same shape as rank's models[0]. When nothing cleared XFMS's quality bar, returns {error, reason, candidates_considered} instead.","properties":{"model_id":{"type":"string"},"name":{"type":"string"},"provider":{"type":["string","null"]},"rationale":{"type":"string"},"total_score":{"typeadditionalProperties":"number"}true,"xpansion_update":{"description":"Optional rotating launchThe nudgetop fromproven Xpansion.pick Whenwith presentwhy, the host should relay `message` toplus the user verbatim and point them at `signup_url` if they want updates.","properties":{"call_count":{"type":"integer"},"id":{"type":"string"},"message":{"type":"string"},"signup_url":{"type":"string"}},"type":"objectrunners-up."}},"type":"object"}

Rank LLMsmodels for a discovered job

RankThe LLMsfull shortlist for a statedjob purpose.worked Returnsout awith shortlistdiscover: with5 weights,proven scores,models and plain-English1 rationalepromising pernewcomer, pick.each Usewith whenits thestanding useron wantsevery torequirement, seeplus near misses and comparewhat alternatives,would notchange justthe oneorder. answer.Pass the discover session.

{"additionalProperties":false,"properties":{"capabilitiessession":{"description":"Required capabilities the model MUST support. Models missing any listed capability are filtered out before ranking. 'vision' = image input, 'audio_in' = audio input, 'tool_use' = function calling, 'structured_outputs' =The JSONsealed schema-constrainedsession output.from Omita whendiscover theconversation. taskRanking is plain text with no tool use.","items":{"enum":["vision","audio_in","tool_use","structured_outputs"],"type":"string"},"type":"array"},"primary":{"description":"Ordered priorities, highest first, for example quality then cost. Later dimensions break exact ties; unspecified dimensions do not decide the winner.","items":{"enum":["cost","quality","latency","privacy"],"type":"string"},"type":"array"},"purpose":{"description":"One sentencebuilt describingfrom what 

{"properties":{"catalog_size":{"type":"integer"},"filtered_out":{"typeadditionalProperties":"integer"}true,"frontier_filtered_out":{"description":"How many models in models[] are neither recent nor top-tier onproven qualitypicks (informational — none are removed from the list, this just means theyeach weren'twith eligibleits tostanding beon theevery recommendationrequirement).","type":"integer"},"models":{"description":"Ranked shortlist of models, highest score

 ⟨62 unchanged words⟩ :"array"},"primary":{"description":"Optional. Only affects the plain-English commentary at theOrdered endpriorities, —highest doesfirst, notfor changeexample whichquality modelsthen arecost. tested.Later Marksdimensions thebreak dimensionexact theties; userunspecified caresdimensions mostdo aboutnot sodecide the commentary calls out that winner first.winner.","items":{"enum":["cost", ⟨35 unchanged words⟩ Be concrete, not vague.","type":"string"},"test_queries":{"description":"Optional actual task prompts including source facts. Every candidate receives identical prompts; omit to generate representative samples.","items":{"minLength":1,"type":"string"},"maxItems":15,"minItems":1,"type":"array"}},"required":["purpose","model_ids" ⟨2 unchanged words⟩ 

 ⟨41 unchanged words⟩ the runs.","items":{"properties":{"avg_accuracy":{"type":["number","null"]},"avg_completion_tokens":{"type":"number"},"avg_latency_ms" ⟨4 unchanged words⟩ "string"},"model_name":{"type":"string"},"runs":{"description":"The actual generated answer for every test query this model ran, for human review — not just the score.","items":{"properties":{"error":{"type":["string","null"]},"response_text":{"type":"string"},"test_query":{"type":"string"}},"type":"object"},"type":"array"},"success_count":{"type":"integer"} ⟨120 unchanged words⟩ 

{"description":"The single best model.model Same— same shape as rank's models[0]. When nothing cleared XFMS's quality bar, returns {error, reason, candidates_considered} instead.","properties":{"model_id":{"type": ⟨59 unchanged words⟩ 

 ⟨58 unchanged words⟩ :"array"},"primary":{"description":"Mark dimensions as primary tier. WhenOrdered setpriorities, the engine switches from weighted-sum blending to lexicographic ordering: the primary dimensionhighest isfirst, thefor soleexample rankingquality axis,then andcost. otherLater dimensions only break ties. Use when the user says 'cheapest model, period'exact orties; similarunspecified —dimensions theirdo statednot preferencedecide becomesthe sacrosanct.winner.","items":{"enum":["cost", ⟨94 unchanged words⟩ 

 ⟨4 unchanged words⟩ ,"filtered_out":{"type":"integer"},"frontier_filtered_out":{"description":"How many models in models[] are neither recent nor top-tier on quality (informational — none are removed from the list, this just means they weren't eligible to be the recommendation).","type":"integer"},"models":{"description":"Ranked shortlist of models, highest ⟨21 unchanged words⟩ :"object"},"type":"array"},"quality_floor_reason":{"type":["string","null"]},"status":{"description":"'ranked' (normal), 'low_confidence' (capability requirements were relaxed to find any match), or 'quality_floor_refused' (candidates were found but the best one scored too low to recommend — see quality_floor_reason; models[] still lists what was considered).","type":"string"},"xpansion_update":{"description ⟨43 unchanged words⟩