oumi.datasets.evaluation

Evaluation datasets module.

class oumi.datasets.evaluation.AlpacaEvalDataset(*, include_system_prompt: bool = False, unused_entries_to_metadata: bool = False, trust_remote_code: bool = True, **kwargs)

Bases: BaseSftDataset

dataset_name: str

default_dataset: str | None = 'tatsu-lab/alpaca_eval'

system_prompt_with_context = 'Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.'

system_prompt_without_context = 'Below is an instruction that describes a task. Write a response that appropriately completes the request.'

transform_conversation(example: dict | Series) → Conversation

Preprocesses the inputs of the example and returns a dictionary.

Parameters:

example (dict or Pandas Series) – An example containing input (optional), instruction entries.

Returns:

The input example converted to Alpaca dictionary format.

Return type:

dict

Note

If unused_entries_to_metadata is set: all example’s entries, other than the expected ones (i.e., input and instruction), are saved as metadata.

trust_remote_code: bool