Build and own a Tool Selection Specialist Model.
Which tool, API, or system (MCP server, CRM, billing, orders, internal API) should the agent call to handle this request?
Overview
Reduce failed or wrong tool calls that waste turns and erode user trust.
Who owns this decision
Head of AI/ML, VP Engineering
Why owned AI may fit
This is one of the areas where the case for a specialist model is genuinely weaker today: frontier models with well-documented tool schemas and good context (recent history, account state) are increasingly reliable at tool selection out of the box.
What this Specialist Model actually does
Decision: Which tool, API, or system (MCP server, CRM, billing, orders, internal API) should the agent call to handle this request?
This is a larger system with 2 specialist model candidates and 4 supporting or external components.
Who uses this Specialist Model
Accountable owners: Head of AI Platform or VP Engineering.
- Agent builder
- AI platform
- Workflow owner
- Support operations
- Tool owner
- Security reviewer
- Compliance reviewer
- Human agent
Where the specialist model sits
This is a larger system with 2 specialist model candidates and 4 supporting or external components.
Lightweight classifier or fine-tuned small model
Specialist Model
Lightweight classifier or fine-tuned small model for high-volume routing decisions
Fallback logic invoking a larger general-purpose model
Specialist Model
Fallback logic invoking a larger general-purpose model for ambiguous or novel cases
More Specialist Models in Software →
Tool/API schema registry (names, descriptions, input/output contracts)
Supporting component
How Oumi connects to your stack →
Execution monitor capturing success/failure per tool call
Supporting component
How Oumi connects to your stack →
Confidence threshold routing low-confidence decisions
Supporting component
Confidence threshold routing low-confidence decisions to human review or a safe default
Feedback capture from execution outcomes into retraining data
Supporting component
How Oumi connects to your stack →
Decision and authority boundary
Deterministic systems establish identity registry eligibility permission schema health policy and approval truth. Oumi proposes a tool selection. Tool Calling execution systems and people validate calls actions and outcomes.
Abstain or escalate when
- Tenant unresolved
- User unresolved
- Request unresolved
- Tool identity unresolved
- Registry version unresolved
- Schema unresolved
- Permission unresolved
- Credential scope unresolved
How to evaluate and improve it
Evaluation
- Selection versus binding execution and workflow separation
Feedback signals
- Accept
- Reject
- Correct
- Wrong tool
- Missing tool
- Ineligible tool
- Unavailable tool
Model candidates to evaluate
- Qwen3.5-2B/4B (Apache 2.0)
- gpt-oss-20b (Apache 2.0)
- ModernBERT (Apache 2.0)
Relevant benchmark starting points
- Tau2 Bench
These are candidates, not performance claims. Run them against your evaluation set.


