oumi#

Oumi (Open Universal Machine Intelligence) library.

This library provides tools and utilities for training, evaluating, and inferring with machine learning models, particularly focused on language tasks.

Modules:

• models: Contains model architectures and related utilities.

• evaluate: Functions for evaluating models.

• evaluate_async: Asynchronous evaluation functionality.

• infer: Functions for model inference, including interactive mode.

• train: Training utilities for machine learning models.

• utils: Utility functions, including logging configuration.

• judges: Functions for judging datasets and conversations.

Functions:

• train(): Train a machine learning model.

• evaluate_async(): Asynchronously evaluate a model.

• evaluate(): Evaluate a model using LM Harness.

• infer(): Perform inference with a trained model.

• infer_interactive(): Run interactive inference with a model.

• quantize(): Quantize a model to reduce size and memory usage.

• judge_dataset(): Judge a dataset using a model.

Examples

Training a model:

>>> from oumi import train
>>> from oumi.core.configs import TrainingConfig
>>> config = TrainingConfig(...)
>>> train(config)

Evaluating a model:

>>> from oumi import evaluate
>>> from oumi.core.configs import EvaluationConfig
>>> config = EvaluationConfig(...)
>>> results = evaluate(config)

Performing inference:

>>> from oumi import infer
>>> from oumi.core.configs import InferenceConfig
>>> config = InferenceConfig(...)
>>> outputs = infer(config)

Quantizing a model:

>>> from oumi import quantize
>>> from oumi.core.configs import QuantizationConfig
>>> config = QuantizationConfig(...)
>>> result = quantize(config)

Judging a dataset::

>>> from oumi import judge_dataset
>>> from oumi.core.configs import JudgeConfig
>>> config = JudgeConfig(...)
>>> judge_dataset(config, dataset)

oumi.evaluate(config: EvaluationConfig) → list[dict[str, Any]][source]#

Evaluates a model using the provided configuration.

Parameters:

config – The desired configuration for evaluation.

Returns:

A list of evaluation results (one for each task). Each evaluation result is a dictionary of metric names and their corresponding values.

oumi.evaluate_async(config: AsyncEvaluationConfig) → None[source]#

Runs an async evaluation for a model using the provided configuration.

Overview:

This is a utility method for running evaluations iteratively over a series of checkpoints. This method can be run in parallel with a training job to compute metrics per checkpoint without wasting valuable time in the main training loop.

Parameters:

config – The desired configuration for evaluation.

Returns:

None.

oumi.infer(config: InferenceConfig, inputs: list[str] | None = None, inference_engine: BaseInferenceEngine | None = None, *, input_image_bytes: list[bytes] | None = None) → list[Conversation][source]#

Runs batch inference for a model using the provided configuration.

Parameters:

• config – The configuration to use for inference.

• inputs – A list of inputs for inference.

• inference_engine – The engine to use for inference. If unspecified, the engine will be inferred from config.

• input_image_bytes – A list of input PNG image bytes to be used with image+text VLMs. Only used in interactive mode.

Returns:

A list of model responses.

Return type:

object

oumi.infer_interactive(config: InferenceConfig, *, input_image_bytes: list[bytes] | None = None, system_prompt: str | None = None) → None[source]#

Interactively provide the model response for a user-provided input.

oumi.quantize(config: QuantizationConfig) → QuantizationResult[source]#

Quantizes a model using the provided configuration.

Parameters:

config – Quantization configuration containing model parameters, method, output path, and other settings.

Returns:

• quantized_size_bytes: Size of the quantized model in bytes

• output_path: Path to the quantized model

• quantization_method: Quantization method used

• format_type: Format type of the quantized model

• additional_info: Additional method-specific information

Return type:

QuantizationResult containing

Raises:

• RuntimeError – If quantization fails for any reason

• ValueError – If configuration is invalid for this quantizer

oumi.synthesize(config: SynthesisConfig) → list[dict[str, Any]][source]#

Synthesize a dataset using the provided configuration.

oumi.train(config: TrainingConfig, additional_model_kwargs: dict[str, Any] | None = None, additional_trainer_kwargs: dict[str, Any] | None = None) → None[source]#

Trains a model using the provided configuration.