Training an LLM
Training of a LLM involves several phases:
-
Pre-training: The objective is to create a "foundation model" by training a Transformer on enormous amounts of unlabeled text (trillions of tokens) using self-supervised learning, specifically next-token prediction. The aim is to build broad world knowledge, language competence, and general reasoning ability. The output is a "base model" that can complete text plausibly but isn't yet reliable at following instructions or producing relevant answers.
-
Supervised Fine-tuning (SFT): Continue training on a smaller, curated dataset of (instruction, ideal response) pairs, typically written or vetted by humans. The aim is to teach the model to follow user instructions and perform specific tasks (such as translation or summarization). This involves training the pre-trained model on a much smaller, high-quality dataset of labeled instruction-response pairs. This is the first point where the model starts behaving like a chatbot rather than a text-completion engine.
-
Reinforcement Learning from Human Feedback (RLHF): Often referred to as preference tuning,or alignment, this phase aims to further refine the model's behavior to match complex human values and preferences. The aim is to make outputs more helpful, harmless, and honest according to human judgment — optimizing for qualities (tone, tact, calibration, refusal behavior) that are hard to specify as a single "correct" target label the way SFT requires.
Newer "reasoning models" have an additional distinct phase:
- Reasoning-focused RL RL against verifiable and objective correctness signals (does the math answer match ground truth? does the code pass tests?) rather than subjective human preference. The aim is to improve the model's ability to reason step-by-step, solve harder multi-step problems and self-correct (see Self-Reflection below).
Weights are always updated with backpropagation.
The definition of the loss function in the RL steps is not based on the difference between observed output and expected output, as in pre-training and in SFT. It is instead based on a synthetically generated numerical reward given to each observed output in a way that approximates human preferences (RLHF) or a correctness signal (Reasoning-focused RL).
Capabilities developed with fine tuning
The training phases that follow pre-training are collectively denoted as fine-tuning. The capabilities developed with fine tuning can be summarized as follows
| Capability | Definition | SFT | RLHF |
|---|---|---|---|
| Instruction-following | Responding helpfully and accurately to a wide range of user instructions, rather than just completing text in a generic way (the core shift from a raw pretrained "base model" to an assistant-like model) | ✅ primary | – |
| Conversational coherence | Maintaining context, tone, and consistency across multi-turn dialogue rather than treating each turn in isolation | ✅ primary | ✅ secondary |
| Task-specific expertise | Specializing the model for a narrow domain (e.g., legal document analysis, medical Q\&A, code generation) using domain-specific data | ✅ primary | – |
| Tool/function-calling | Learning to correctly decide when to invoke external tools, format tool calls per a schema, and integrate tool results into a coherent response | ✅ primary | ✅ secondary |
| Safety and harm avoidance | Refusing or appropriately handling requests involving harmful, illegal, or dangerous content | ✅ secondary | ✅ primary |
| Honesty and calibration | Expressing appropriate uncertainty, avoiding confident-sounding fabrications (hallucinations), and acknowledging the limits of its knowledge | – | ✅ primary |
| Alignment with human preferences | Producing outputs that humans (or AI raters, as a proxy) actually judge as helpful, relevant, and well-formatted, not just technically correct | – | ✅ (by definition) |
| Style/format adherence | Following formatting conventions (Markdown, code blocks, specific output structures like JSON) reliably and consistently | ✅ primary | – |
| Persona/brand consistency | Adopting a specific tone, personality, or identity consistent with a product's intended voice | ✅ primary | ✅ secondary |
| Multilingual/cross-lingual competence | Improving performance in specific languages or on translation tasks, especially lower-resource languages underrepresented in pretraining data | ✅ primary | ✅ secondary |
| Reduced toxicity/bias | Decreasing the likelihood of outputs that are offensive, discriminatory, or reflect harmful stereotypes present in web-scale pretraining data | ✅ secondary | ✅ primary |
| Reasoning (Self-Refinement and Reflection) | Ability to identify errors in its own initial outputs and refine them based on internal logic or specific feedback. | RL* (not RLHF) |
RL training steps (in a nutshell)
RLHF is used for aligning the model with preferences that are hard to express as a single "correct" target output (e.g., "which of these two responses is more helpful/harmless?" doesn't have one ground-truth answer the way "translate this sentence" does).
Step A — Train a reward model. Humans are shown pairs of model outputs for the same prompt and asked which is better. This preference data trains a separate reward model: a neural network that takes a (prompt, response) pair and outputs a scalar score estimating "how good is this response." This reward model is itself trained via ordinary backpropagation, with a loss that pushes it to score the preferred response higher than the rejected one. The resulting reward model will be used at the next step to imitate human preferences at scale.
Step B — Use the reward model to update the LLM. This is the actual Reinforcement Learning (RL) step:
- The LLM generates a response to a prompt.
- The reward model scores that response.
- The score is used as a reward signal in a policy-gradient RL update: weights are adjusted to make outputs that scored highly more probable in the future, and low-scoring outputs less probable. Specifically, the score multiplies the probability of the generated tokens and that product is used for backpropagation.
The specific RL algorithm used for computing the loss and thus updating the weights with backpropagation is called PPO (Proximal Policy Optimization). In a nutshell, this RL algorithm makes sure that weights are modified only a small step at a time, and never too far from how the original model behaved. The exploration ability of RL is a consequence of a sampling from the probability distribution of the predicted words.
Increasingly common as a replacement for PPO, DPO (Direct Preference Optimization) reformulates the same preference data (A is better than B) into a loss function that can be optimized directly via standard backpropagation, without needing a separate reward model or the RL sampling loop at all. It's mathematically derived to have the same optimal solution as RLHF, but is simpler and more stable to train. So in practice, some "RL-based" alignment today is actually done without explicit RL machinery, just backpropagation on a cleverly designed loss.
| Method | Data type | Loss/reward source | Update mechanism |
|---|---|---|---|
| Pretraining | Raw text | Next-token prediction accuracy | Backprop |
| SFT | (instruction, ideal response) pairs | Cross-entropy vs. target | Backprop |
| RLHF (PPO) | (prompt, response, reward score) | Reward model's scalar score | Backprop, but gradient comes from an RL objective (policy gradient), not a fixed target |
| DPO | (prompt, preferred response, rejected response) | Preference comparison, reformulated as a direct loss | Backprop, no separate RL loop |
Reasoning (Self-Refinement and Reflection)
This is the ability to identify errors in its own initial outputs and refine them based on internal logic or specific feedback.
Even a model with no special reflection training can be prompted to critique and revise its own output — e.g., "Here's your answer. Now critique it for errors, then produce an improved version." This works to some degree purely from the base model's/SFT model's general competence, exploiting the same capability that lets it evaluate other text for errors.
If you fine-tune on examples that are written to look like reflection (e.g., "Draft: ... Critique: ... Revised: ..."), SFT will teach the model to produce that structure. But this is largely imitating the surface pattern of self-correction, not necessarily instilling a robust ability to actually detect its own errors in novel cases. It works reasonably well within the distribution of the training examples but tends to generalize poorly — the model learns "reflection-shaped text follows this answer" not "genuinely re-examine whether this answer is right."
Self-refinement is most robustly instilled through RL against verifiable correctness signals. SFT can scaffold the format, but the actual skill of catching and correcting its own errors is something RL training incentivizes far more directly, precisely because RL rewards the outcome (being right) rather than the resemblance to a human-written example.
- When training uses outcome-based RL (reward \= did the final answer match ground truth, e.g., correct math answer or passing test suite), the model isn't told how to reason — it's only rewarded for arriving at the correct answer.
- Through many rounds of trial and error, the model discovers that certain reasoning behaviors — checking its own intermediate steps, noticing an inconsistency, backtracking and trying a different approach — causally increase the reward, because they lead to more correct final answers.
- Crucially, this is learned as a strategy that pays off, not imitated from human-written examples of what reflection "should look like." This is why models trained this way (e.g., OpenAI's o1/o3, DeepSeek-R1) exhibit long, sometimes messy-looking reasoning traces with genuine self-correction ("
wait, that's not right, let me reconsider...") — behavior that emerged from the RL objective rather than being explicitly written into training examples by a human.
| Mechanism | What it produces | Genuine or imitative? |
|---|---|---|
| Prompting alone | Surface-level self-critique, quality bounded by base capability | Neither trained — relies on existing competence |
| SFT on reflection-formatted examples | Model learns to produce reflection-shaped output | Largely imitative — mimics the pattern |
| RL with outcome-based reward | Model learns self-correction because it improves accuracy | Genuine — reflection emerges as an instrumentally useful strategy |