Benchmark Generation Engine reference profile.
What benchmark should exist to test this model or capability, and what should the test cases and expected answers be?
Overview
Get a working evaluation suite in place for a new capability without waiting on a slow manual test-writing process.
Who owns this decision
Head of AI/ML Platform, VP Data Science
What this Specialist Model actually does
Decision: What benchmark should exist to test this model or capability, and what should the test cases and expected answers be?
The final taxonomy records 6 supporting or external components, but it doesn't promote an Oumi-owned specialist model for this engine.
Who uses this Specialist Model
- Head of AI/ML Platform
- VP Data Science
- Head of Applied AI
- ML Eng Lead
Where the specialist model sits
The final taxonomy records 6 supporting or external components, but it doesn't promote an Oumi-owned specialist model for this engine.
Task-specification parser
Supporting component
Task-specification parser that turns a capability description into a test-case generation plan
Retrieval over documentation, policies, and prior human-graded examples
Supporting component
Retrieval over documentation, policies, and prior human-graded examples for grounding
Test-case generator producing inputs across a target difficulty distribution
Supporting component
Expected-answer generator
Supporting component
Expected-answer generator with citation back to grounding sources
Human review and correction interface
Supporting component
Human review and correction interface for generated test cases and answers
Deduplication and coverage-gap analysis against the existing evaluation suite
Supporting component
How Oumi connects to your stack →
How to evaluate and improve it
Evaluation
- Test-case validity rate (share of generated cases that are well-formed and non-trivial)
- Expected-answer accuracy against subject-matter-expert review
- Coverage of the target difficulty distribution and business taxonomy
Feedback signals
- Human reviewer accepts, edits, or rejects a generated test case or expected answer
- A generated benchmark does or does not catch a real regression on the next model version
- Subject-matter expert disagrees with a generated expected answer during review
- Benchmark pass rate is later found to be uncorrelated with real production quality (benchmark itself gets revised)
- New production failure reveals a gap the generated benchmark missed


