Models learn in stages: first you show them correct answers (SFT), then you teach them what humans prefer (RLHF), then — for tasks with verifiable outputs — you let execution results do the teaching (RLEF).
Pre-training → SFT → Alignment (RLHF / DPO / RLEF) → Deployed model
These aren't competing approaches. They're sequential layers, each one refining what the previous one produced.
Before any of this, the base model is pre-trained on a massive corpus of text using next-token prediction — predict what word comes next, billions of times, across trillions of tokens.
Pre-training gives the model broad world knowledge and language ability. But it produces a model that completes text in arbitrary ways — it doesn't know how to be a helpful assistant, follow instructions, or avoid harmful outputs.
That's what the next stages fix.
You're teaching the model by example. You have a dataset of (instruction, ideal response) pairs. You train the model to produce outputs that look like those ideal responses.
Think of it like onboarding a new employee: you're giving them a library of "here's how we do things here" examples to learn from.
- Collect a dataset of high-quality prompt-response pairs (human-written, or generated and filtered)
- Fine-tune the pre-trained model on this dataset using standard supervised learning (cross-entropy loss on the target tokens)
- The model learns to follow the style, format, and approach of the training examples
| Advantage | Limitation |
|---|---|
| Simple, well-understood | Quality is bounded by your data quality |
| Fast to train relative to RL methods | Doesn't teach preference ranking — just imitation |
| Good foundation for further alignment | Model can overfit to training format |
| Low annotation cost at small scale | Doesn't handle "better vs. worse" — just right vs. wrong |
SFT is where data quality decisions matter most. The model will imitate your training data. Inconsistent, ambiguous, or low-quality annotations will be reproduced. Budget for annotation QA, not just annotation volume.
SFT datasets typically range from tens of thousands to hundreds of thousands of examples. Each example needs to be reviewed for quality — this is a significant ops undertaking.
- "More data is always better" — No. 10,000 high-quality, consistent examples often outperform 100,000 noisy ones.
- "SFT alone is enough for alignment" — SFT teaches imitation, not preference. The model doesn't know which of its possible outputs is better, only which looks most like the training data.
SFT taught the model to produce human-like responses. RLHF teaches it to produce responses that humans prefer.
The key insight: it's hard to write the perfect answer, but easy to say which of two answers is better. RLHF exploits this asymmetry.
Three stages:
Stage 1 — Collect preference data For a given prompt, generate several candidate responses. Ask human raters: which response do you prefer? Collect thousands of these pairwise comparisons.
Stage 2 — Train a reward model Train a separate neural network to predict human preference scores. Input: a prompt + response. Output: a scalar reward score. This model learns to approximate "what a human would rate highly."
Stage 3 — RL fine-tuning with PPO Use the reward model as the reward signal in a reinforcement learning loop. The policy (the language model being trained) generates responses, the reward model scores them, and the policy is updated to produce higher-scoring responses. PPO (Proximal Policy Optimisation) is the most common RL algorithm used here.
A KL divergence penalty prevents the model from drifting too far from the SFT baseline — without it, the model would quickly learn to "hack" the reward model with nonsensical but high-scoring outputs.
| Advantage | Limitation |
|---|---|
| Captures nuanced human preferences | Expensive — requires human raters at scale |
| Handles "better vs. worse" not just "right vs. wrong" | Reward model can be gamed (reward hacking) |
| Strong results for instruction-following and helpfulness | Training instability — PPO is finicky |
| Enables constitutional/values alignment | Rater subjectivity introduces noise |
RLHF is annotation-intensive. A typical RLHF run requires tens of thousands of preference comparisons. Each comparison requires a human rater to read two responses and make a judgment — typically 2–5 minutes each.
Budget considerations:
- Internal raters (team members): slower but more domain-consistent
- Crowdsourced raters: faster but higher variance, needs strong quality controls
- Expert raters (e.g. for medical or legal): expensive but necessary for high-stakes domains
The reward model itself becomes a bottleneck — if it misrepresents human preferences, the RL training amplifies that error. Spend disproportionate effort on reward model quality.
- "RLHF ensures the model is aligned with all humans" — No. It's aligned with the preferences of your rater population, which is a specific, non-representative group.
- "More RLHF is better" — Over-optimising against a reward model degrades performance on tasks the reward model doesn't cover (Goodhart's Law: when a measure becomes a target, it ceases to be a good measure).
DPO achieves the same goal as RLHF (aligning to human preferences) but without training a separate reward model or running a full RL loop. It's simpler, more stable, and increasingly popular.
Instead of using preference data to train a reward model and then using that reward model in a RL loop, DPO directly optimises the language model on the preference data using a mathematically equivalent but simpler objective.
You still need the same pairwise preference dataset. You just skip the reward model and PPO steps.
If your team is evaluating whether to use RLHF vs. DPO: DPO is generally preferred for new projects unless you have a specific reason to need a separate reward model (e.g., you want to use the reward model for ongoing evaluation, not just training).
DPO is faster to train, cheaper to run, and produces comparable results on most tasks.
For tasks with verifiable correct answers, you don't need human raters to assess quality. You can just run the output and check if it works.
Does the code execute without errors? Does it pass the test cases? Is the maths answer correct? Is the logical proof valid?
RLEF uses these execution results — pass/fail signals — as the reward instead of human preference scores.
- Model generates a candidate output (e.g., a code solution)
- Output is executed in a sandbox environment
- Execution result (pass/fail, or partial score) is used as the reward signal
- RL training updates the model toward outputs that pass execution
RLEF is responsible for a significant portion of recent gains in coding and mathematical reasoning models. It's scalable in a way RLHF is not — you can generate millions of (problem, solution, result) triples cheaply, without human raters.
| Advantage | Limitation |
|---|---|
| Extremely scalable — no human raters needed | Only works for verifiable tasks (code, maths, logic) |
| Ground truth is objective | Can't be used for open-ended tasks (writing, summarisation) |
| Enables very high sample efficiency via Pass@K | Requires robust execution environment (sandboxing, timeouts) |
| Self-improving: model generates its own training signal | Model may learn to game the verifier (e.g. hardcoded test outputs) |
If your team is working on an AI product that involves code generation, mathematical reasoning, or any task with verifiable outputs: ask whether RLEF is in the training loop. Models trained with RLEF on coding tasks substantially outperform those trained with RLHF alone for those use cases.
RLEF requires an execution infrastructure — sandboxed environments that can safely run arbitrary code at scale. This is a non-trivial engineering investment, but it unlocks a self-improving training signal.
- "RLEF replaces RLHF" — No. They're complementary. RLEF works for verifiable tasks; RLHF covers the rest.
- "If it passes the tests, it's correct" — Test suites are incomplete. A model can pass all provided tests with a brittle or wrong solution. Test quality matters as much as execution success.
| Problem Type | Recommended Approach | Reasoning |
|---|---|---|
| Teaching the model your task format and style | SFT | You have examples of correct outputs |
| Improving helpfulness, tone, safety | RLHF or DPO | Requires human preference judgments |
| Code generation quality | SFT + RLEF | Execution feedback is objective and scalable |
| Mathematical reasoning | SFT + RLEF | Same — verifiable outputs |
| Open-ended writing quality | SFT + RLHF/DPO | No execution verifier; needs human preference |
| Domain adaptation (e.g., medical, legal) | SFT on domain data | Inject domain knowledge before alignment |
- What training paradigm are you using, and why is it the right fit for this task?
- How is the reward signal defined? What does "better" mean in this context?
- How big is the preference/annotation dataset, and who created it?
- What's the plan for detecting and preventing reward hacking?
- How does the reward model performance get evaluated before it's used in RL training?
- Ziegler et al. (2019) — "Fine-Tuning Language Models from Human Preferences" (original RLHF paper)
- Rafailov et al. (2023) — "Direct Preference Optimization: Your Language Model is Secretly a Reward Model" (DPO paper)
- OpenAI (2022) — InstructGPT paper — best accessible description of RLHF in practice
- DeepSeek-R1 (2025) — demonstrates large-scale RLEF for reasoning