oumi.judges.rules

oumi.judges.rules#

Rules module for deterministic judge evaluations.

class oumi.judges.rules.BaseRule[source]#

Bases: ABC

Base class for rules used for deterministic evals.

abstractmethod apply(input_data: dict[str, str], rule_config: dict) → tuple[bool, float][source]#

Apply the rule to input data.

Returns:

(judgement: bool, score: float)

Return type:

tuple

class oumi.judges.rules.RegexRule[source]#

Bases: BaseRule

Rule that checks if input text matches a regex pattern.

Config Parameters:

pattern (str): The regex pattern to match against input_field (str): The field name to extract text from input_data match_mode (str): How to match - “search”, “match”, “fullmatch” inverse (bool): If True, pass when pattern does NOT match (default: False) flags (int): Optional regex flags (e.g., re.IGNORECASE) (default: 0)

Examples

Match a phone number pattern: >>> rule_config = { … “pattern”: r”\d{3}-\d{4}”, … “input_field”: “text”, … “match_mode”: “search” … } >>> rule = RegexRule() >>> result, score = rule.apply({“text”: “Call 555-1234”}, rule_config) >>> print(result, score) True 1.0

Inverse matching (expect NOT to match): >>> rule_config = { … “pattern”: r”error|fail”, … “input_field”: “output”, … “inverse”: True … } >>> result, score = rule.apply({“output”: “Success!”}, rule_config) >>> print(result, score) True 1.0

apply(input_data: dict[str, str], rule_config: dict) → tuple[bool, float][source]#

Apply regex pattern matching to input data.

Parameters:

  • input_data – Dictionary containing input fields

  • (e.g. – “…”, “expected”: “…”})

  • {"text" – “…”, “expected”: “…”})

  • rule_config – Configuration with ‘pattern’, ‘input_field’, ‘inverse’, etc.

Returns:

bool, score: float) - judgment: True if test passes (matches, or doesn’t match if inverse=True) - score: 1.0 if judgment is True, 0.0 otherwise

Return type:

Tuple of (judgment

Raises:

ValueError – If required config parameters are missing or invalid

Contents