Fine-Tuning Methods Guide: Which One Is Right for You?
Compare fine-tuning methods: SFT, LoRA, QLoRA, DPO and GRPO. Learn what each one changes in the model, what it needs from your data and when to use it.
Choosing an approach to (custom) fine-tuning involves two decisions: the training method and the parameter update strategy. The training method determines how the model learns from training samples or feedback. Examples include supervised fine-tuning (SFT), preference tuning methods like direct preference optimization (DPO), and reinforcement learning (RL) methods like group relative policy optimization (GRPO). Different fine-tuning algorithms have different data requirements, but all of them update the parameters of a base model to improve performance on the task exemplified in the training data.
The parameter update strategy determines how the model itself is modified during training – full-rank fine-tuning (or full fine-tuning, FFT) updates all model parameters, while parameter-efficient fine-tuning (PEFT) techniques like low-rank adaptation (LoRA) train the weights of a small adapter alongside the original model, whose parameters remain frozen. Quantized LoRA (QLoRA) is an even more efficient variation of LoRA, in which the base model is quantized prior to training.
The question of what training method to use and what parameter update strategy to use are independent – one can apply LoRA or FFT to any of the methods mentioned above. The compute requirements and ideal hyperparameter ranges will vary across training methods and update strategies.
Start with the behavior you need to change and the examples or feedback you can provide. Choose a training method that can use them. Then choose how much of the model to update, based on the model, hardware, performance requirements, and deployment constraints. The software and product you use must support the combination. Evaluate the final updated model, using any adapter with the exact base model it was trained with.
First, choose the examples or feedback
Supervised fine-tuning: you have examples of the desired answer
Supervised fine-tuning (SFT) trains on examples that pair an input with the response you want. It is a reasonable first method when you can provide reliable examples of the desired behavior and response format: classify this request as “billing,” return information in a required structure, or answer this kind of question in a defined way.
The examples must express the intended behavior clearly. If they disagree, omit important edge cases, or encode a flawed policy, SFT can make those patterns easier for the model to reproduce. A lower training loss doesn’t prove that the model improved on the real task. Compare it with examples set aside before training, and inspect difficult cases.
When you can compare answers
Preference methods use judgments about which of two (or more) responses is better, rather than giving one target answer for each input. Direct Preference Optimization (DPO) is one such method. It learns from responses marked as preferred/chosen or rejected. The original DPO formulation uses a reference model but, unlike related methods such as Proximal Policy Optimization (PPO), doesn't require training a separate reward model.
Consider a preference method when you can judge which of two alternative responses is superior, such as which is more helpful or follows a policy more closely. You still need consistent judgments, a clear rubric, and tests of the behavior you care about. A response preferred by a reviewer is not automatically safe, correct, or useful in the real application.
When you can score outcomes
Reinforcement Learning (RL) methods such as Group Relative Policy Optimization (GRPO) use scores to guide training. In some setups, a program can score a checkable result, such as an exact answer or required format. Other setups use a learned reward model. In either case, the reward signal should track the intended outcome and resist gaming ("reward hacking").
In common GRPO setups, the model generates several responses to an input, and their scores help guide the update. This is not simply SFT with a different dataset – there are no supervised "reference" responses, and the model generates many alternative responses to each training prompt ("rollouts"). The score must meaningfully track success, and the model still needs separate testing. If the score measures something different from the real task, training can improve the score while missing the goal.
| What examples or feedback do you have? | Method to consider | What must you check? |
|---|---|---|
| Examples pairing inputs with desired answers | SFT | Do examples cover the behavior and its limits? |
| Clear choices about which response is better | DPO or another preference method | Do those choices reflect the outcome you want? |
| A score that reflects success | GRPO or another method that learns from scores | Does the score reflect success, and can it be gamed? |
This table helps you choose how the model learns from examples or feedback. It does not choose how much of the model to update.
Then choose how to update the model
Full fine-tuning
Full fine-tuning updates all of the model’s learned values. It offers broad freedom to change the model, but it can require more computing resources and storage than methods that update fewer values. It may make sense when the task, scale, available resources, or measured quality justify that cost. It is not automatically more accurate, and it still needs careful testing.
LoRA
Low-Rank Adaptation (LoRA) leaves the original model weights unchanged and trains small added components, called adapters, attached to selected parts of the model. Use the adapter with the base model and settings it was trained for. LoRA reduces the number of trained parameters and keeps task-specific updates small to store or compare. It does not guarantee that training will fit on a particular machine or match the quality of full fine-tuning.
QLoRA
QLoRA combines LoRA training with a compressed version of the unchanged base model, represented with fewer bits. The original method uses a four-bit representation to reduce memory use while training the adapter. Whether it will work well depends on the model, hardware, software, and settings. “Four-bit” describes part of the setup. It does not promise identical quality or compatibility with every machine.
| How much of the model is updated? | What changes during training? | Practical trade-off |
|---|---|---|
| Full fine-tuning | All of the model’s learned values | Broad freedom to change the model, often with higher computing and storage needs |
| LoRA | Small added adapter values; original model values stay unchanged | Smaller update to store, but it depends on the base model it was trained for |
| QLoRA | Adapter values; unchanged base model uses a compressed four-bit representation | Can reduce memory use, with extra software and setup requirements |
The choices in these two tables can be combined when the software and product support them. For example, a model can learn from example answers using LoRA. “SFT versus LoRA” is not an either-or comparison: SFT describes what examples the model learns from, while LoRA describes which values are updated. Don’t assume every training method works with every model or product.
Treat the base model as part of the adapter
An adapter is not a replacement for the model it was trained with. Use it with that exact base model and version. Record the model name and version, tokenizer, adapter settings, and how it was run. A similar name or newer model does not mean the adapter will work with it.
Don’t swap in a different base model under an adapter. If you want to use a different model, retrain the adapter for it or get explicit confirmation in the model or software documentation that the pairing is supported. Then test that exact combination on the same task and on other important cases. Never infer compatibility from similar model names.
Also test whether the model got worse at other tasks that matter. Training for one narrow task can improve those cases while weakening other behavior. Include examples of those other tasks and relevant safety checks. Don’t use one combined score as a substitute for reviewing them.
A decision path you can use
- Name the behavior to change. Describe what the model should do, not which algorithm to use.
- Choose the examples or feedback. Use example answers for SFT, comparisons between answers for DPO, or a reliable outcome score for GRPO.
- Choose how much of the model to update. Compare full fine-tuning with LoRA or QLoRA based on memory, model support, where you’ll run it, and test results.
- Test the current model first. Set aside examples for testing, separate from training. Include edge cases, required formats, and other important tasks. This gives you a starting result to compare against.
- Compare both models on the same test. Record the exact model and version, examples, settings, scoring rules, and actual responses. Review individual mistakes, not just one overall score.
- Move forward only if the test supports it. If the updated model meets the success measure, test it in a limited workflow and keep a way to switch back. If it doesn’t, find out why before changing methods.
Try this in Oumi
Take this decision path into Oumi. Bring one task, the examples or feedback you have, and test examples kept separate from training. Choose a method and model supported in the Oumi workflow you’ll use, then compare the current and updated models on the same cases.
Inspect mistakes and check where and how you’ll run the model before moving forward.
Bottom line
Choose how the model learns from the examples or feedback you have. Choose full fine-tuning, LoRA, or QLoRA based on the model and resources available. Then use the same task-specific test examples to decide whether the combination deserves further testing.
