Case Study: Ada built a real-time guardrail for their agents with 50% fewer false positives
With Oumi's technology, Ada's custom guardrail model beat GPT-4.1 Mini by 4% in accuracy while slashing latency and false positives rate
By Stefan Webb
Problem
Ada builds AI-powered customer service agents deployed across industries including fintech, e-commerce, SaaS, and travel. For every deployment, each action the agent takes must strictly align with the customer’s operating policies—any deviation risks damaging the customer relationship.
To enforce this, Ada runs a real-time adherence check: for every proposed agent action, a guardrail model evaluates the action against the customer’s standard operating procedure (SOP) and classifies it as “adherent” or “non-adherent.” Because this check runs live during customer conversations, latency is critical.
Ada’s existing guardrail relied on GPT-4.1 Mini. While reasonably accurate at 87.5%, the model introduced unacceptable latency for real-time use. Ada also wanted greater ownership and control over the model powering this critical system.
Solution
Ada partnered with Oumi to develop a custom fine-tuned guardrail model optimized for policy adherence classification.
Since Ada had only around 20 labeled examples initially, the team built a fully synthetic data pipeline. Using Oumi’s platform, they generated 250 SOPs spanning Ada’s priority domains, varying attributes like complexity and tool count to ensure diversity. From these, they synthesized thousands of agent conversations with controlled variation across customer clarity, conversation length, and adherence outcomes.
This data was used to fine-tune Qwen3 4B, a compact language model well-suited to the latency requirements of Ada’s real-time pipeline.
Outcome
The fine-tuned 4B-parameter model outperformed GPT-4.1 Mini across all key metrics (see table above).
The bottom line: By moving from a general-purpose hosted model to a purpose-built fine-tuned LLM, Ada gained better accuracy, lower latency, and full ownership of a core component in their AI agent infrastructure.
What’s next
With Oumi’s technology, Ada were able to build a small custom AI model that solved pressing business needs. Theirs, however, is just one use case. Many other enterprises are discovering the benefits of custom AI models and the ease and economy of model development that the Oumi Platform makes possible.
Why not try it out today and see for yourself? You only need come with your task prompt and the Oumi Agent takes it from there!