How to read this lesson

This lesson teaches training runs. A training run is one execution of a model-training experiment: one model, one dataset version, one set of settings, one stream of logs, and one set of saved checkpoints.

The goal is not to make you an expert in large-scale distributed training on day one. The goal is to make training legible. When you see a chart of loss going down, a checkpoint folder, a learning rate, or a failed fine-tuning job, you should know what is happening and what questions to ask.

We will move from intuition to professional practice:

1. What a training run is.
2. How one batch produces loss, gradients, and an optimizer update.
3. Why learning rate, batch size, epochs, steps, seeds, and checkpoints matter.
4. How to read training and validation curves.
5. How teams track experiments so results are reproducible.
6. How compute budgets shape model size, data size, and training tokens.
7. What breaks in real training work and how engineers diagnose it.

Explain it in 5 minutes

A model starts with learned numbers called parameters: numbers inside a neural network that are adjusted during training. Training is the process of changing those numbers so the model makes better predictions on examples.

In one simple training step:

1. The training system loads a batch: a small group of examples processed together.
2. The model makes predictions for that batch.
3. A loss function: a formula that turns prediction error into one number. measures how wrong the predictions were.
4. Backpropagation: the algorithm that computes how each trainable parameter contributed to the loss. computes gradients.
5. An optimizer: an algorithm that updates parameters using gradients. changes the parameters.
6. The system logs metrics, sometimes evaluates on a validation split, and sometimes saves a checkpoint.

The loop repeats many times. A run is the full record of that loop: data version, code version, model architecture, hyperparameters, hardware, logs, metrics, checkpoints, and final decision.

theta_next = theta - learning_rate * gradient

This equation is the simplest version. Real optimizers such as AdamW keep extra moving averages and add details such as weight decay. But the professional intuition stays the same: each step uses recent error signals to move trainable weights.

Training runs matter because many AI engineering decisions depend on them. A fine-tuning result is not just "we trained a model." It is "we trained model M on dataset version D with config C, watched metrics X, selected checkpoint K, compared against baseline B, and decided whether the improvement justified the cost and risk."

Learning objectives

By the end, you should be able to:

• Define training run, parameter, gradient, optimizer, learning rate, batch, step, epoch, checkpoint, validation split, hyperparameter, seed, reproducibility, overfitting, underfitting, and experiment tracking.
• Trace a batch through forward pass, loss, backpropagation, optimizer update, logging, validation, and checkpointing.
• Read the basic gradient-descent update equation and explain every symbol.
• Interpret common loss-curve patterns: healthy learning, underfitting, overfitting, instability, data leakage, and noisy validation.
• Explain why checkpoints must save more than model weights when a run needs to resume.
• Explain why model size, data size, tokens, and compute budget are linked.
• Design a beginner-friendly run record that a teammate can reproduce.
• Diagnose production-relevant failure modes before a trained model reaches users.

Prerequisites from zero

You need these ideas before going further:

• A model: a function with learned parameters that maps inputs to outputs. For a language model, the input is tokens and the output is probabilities for possible next tokens.
• A token: a small text unit processed by a model, such as a word piece, punctuation mark, or special symbol.
• A dataset: a collection of examples used for training, validation, testing, retrieval, or evaluation.
• A split: a named subset of a dataset. Training examples teach the model. Validation examples help tune decisions during development. Test examples estimate performance after development choices are made.
• A metric: a measured number used to judge behavior. Loss, accuracy, latency, cost, and human preference scores are all metrics in different contexts.
• A baseline: the current or simpler system used for comparison. A training run is only meaningful if you can compare it to something.

Glossary of essential terms

Section 1: What counts as a training run?

A training run is more than the command that starts training. It is the unit of evidence for a model-building claim.

If someone says, "Run 42 beat the baseline," a careful engineer asks:

• What exact model architecture or base model was used?
• What training data version was used?
• Were validation and test examples kept separate from training examples?
• What hyperparameters changed from the baseline?
• Which checkpoint was selected, and why?
• Did the model improve on the target metric without regressing on safety, latency, cost, or important edge cases?
• Can someone reproduce the result from the recorded config?

Official tools reflect this shape. Hugging Face Trainer exposes training arguments such as learning rate, batch size, logging strategy, evaluation strategy, save strategy, seeds, mixed precision, and checkpoint resume settings. MLflow and Weights & Biases organize work around runs, parameters, metrics, and artifacts. TensorBoard scalar plots show how metrics such as loss and learning rate change across training.

Section 2: One batch, step by step

The easiest way to understand training is to follow one batch.

Imagine a tiny classifier that predicts whether a support ticket is about billing or shipping. One batch might contain 32 tickets and 32 correct labels. The model reads the tickets, produces predicted probabilities, and the loss function scores the predictions.

For a language model, the same idea applies to tokens. The model sees earlier tokens and predicts the next token. The target is the actual next token from the training text or conversation example.

The training loop has a standard shape:

for batch in train_loader:
    predictions = model(batch.inputs)
    loss = loss_fn(predictions, batch.targets)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()

This code is intentionally simplified. Real code adds device placement, mixed precision, gradient clipping, distributed synchronization, learning-rate schedules, logging, evaluation, checkpointing, and error handling. But these five lines contain the core loop.

L_batch = (1 / B) * sum(loss_i for i = 1 to B)

PyTorch's autograd system builds a computation graph during the forward pass and computes gradients during loss.backward(). The optimizer then uses those gradients to update parameters.

Beginner warning: lower training loss is not the same as a better product. A model can memorize training examples and still fail on new examples. That is why professional training work separates training metrics from validation, test, and task-specific evaluations.

Section 3: Settings that shape the run

Training behavior depends heavily on hyperparameters. A hyperparameter: a setting chosen by the engineer rather than learned directly from data. Examples include learning rate, batch size, number of steps, optimizer, warmup schedule, weight decay, random seed, and validation frequency.

Large language-model papers show that run settings are part of the research claim. The GPT-3 paper reports model sizes, batch sizes in tokens, learning rates, training data mixture, compute estimates, and evaluation setup. Scaling-law work studies how loss changes as model size, dataset size, and compute change. Chinchilla later argued that many large models were trained with too few tokens for their size and that compute-optimal planning should scale model size and training data together.

The practical lesson is simple: "we trained longer" is not automatically better. The right question is what changed: more examples, more tokens, more steps, different learning rate, different data mix, larger model, smaller model, better labels, new optimizer, or new evaluation.

Section 4: Validation and loss curves

Training loss measures error on examples the model is learning from. Validation loss measures error on examples held out during development. The validation split is not used for parameter updates. It helps you decide whether the model is learning in a way that generalizes.

gap = validation_loss - training_loss

Loss curves are diagnostic tools.

For AI products, validation loss is rarely enough. A lower language-model loss may not mean the support assistant cites sources correctly, refuses unsafe requests, formats JSON reliably, or chooses tools correctly. Professional teams pair loss with task-specific evals and human review for high-risk behavior.

Section 5: Checkpoints and resuming

A checkpoint is a saved training state. Beginners often think a checkpoint is just model weights. For inference, saving weights may be enough. For resuming training, it usually is not.

A resumable checkpoint should preserve enough state for the next optimizer update to behave as if training had not stopped.

PyTorch's official saving guide makes this distinction: inference can often save a model state dictionary, while a general checkpoint for resuming training should include model state, optimizer state, epoch, loss, and other needed metadata.

Professional checkpoint decisions include:

• Save every fixed number of steps, every epoch, or whenever validation improves.
• Keep only the best few checkpoints to control storage cost.
• Record which checkpoint was evaluated and why it was selected.
• Test loading before trusting the checkpoint.
• Separate "best validation checkpoint" from "final checkpoint" when they differ.

Section 6: Experiment tracking and reproducibility

Reproducibility means a teammate can rerun the experiment and get meaningfully comparable results. It does not always mean byte-for-byte identical results, especially with parallel hardware and nondeterministic kernels. But the run record should make differences explainable.

A useful run name is not enough. Record the facts that determine behavior:

{
  "run_name": "support-sft-2026-06-06-lr-2e-5",
  "code_commit": "abc1234",
  "base_model": "example-model-7b",
  "dataset_version": "support_examples_v3",
  "train_examples": 4200,
  "validation_examples": 600,
  "optimizer": "AdamW",
  "learning_rate": 0.00002,
  "batch_size": 64,
  "max_steps": 1200,
  "seed": 42,
  "selected_checkpoint": "checkpoint-950",
  "ship_decision": "do not ship: regression on refund edge cases"
}

This is not a perfect schema. It is the minimum mindset: record what changed, what was measured, what artifact was selected, and what decision followed.

Experiment tracking tools help compare runs. MLflow Tracking records parameters, metrics, and artifacts. Weights & Biases tracks experiment configs and metrics over time. TensorBoard visualizes scalar metrics such as loss and learning rate. Hugging Face Trainer can report to tracking tools and handles common training-loop concerns.

candidate_wins only if metric_gain - regression_cost - added_operational_cost > 0

Professional context: reproducibility is not bureaucracy. It is how teams avoid arguing from memory. When a model improves, the run record helps you know why. When a model breaks, the run record narrows the search.

Section 7: Compute budgets and scaling laws

Training consumes compute. Compute is the amount of numerical work performed by hardware such as graphics processing units (GPUs) or tensor processing units (TPUs). For language models, teams often plan around model parameters, training tokens, sequence length, batch size, number of steps, and hardware time.

Large-model training is shaped by tradeoffs:

• More parameters can increase model capacity but raise memory and inference cost.
• More training tokens can improve learning but require more data processing and compute.
• Longer context increases the amount of token information per example but can raise attention cost.
• Larger batch sizes can improve hardware utilization but may change optimization behavior.
• More steps can improve loss until the run plateaus or overfits.

Scaling-law papers are empirical studies of how model performance changes with model size, data size, and compute. Kaplan et al. reported power-law relationships between loss and scale across model size, dataset size, and compute. Hoffmann et al. later argued that compute-optimal training should balance model size and data more evenly, and showed Chinchilla, a smaller model trained on more tokens than larger comparison models, outperforming them across many benchmarks under a comparable compute budget.

The beginner lesson is not "memorize one ratio forever." The lesson is that training budget is an experimental design variable. Data, model size, and compute must be planned together, and conclusions depend on the task, architecture, data quality, hardware, and inference economics.

training_cost grows with parameters * training_tokens

For most product teams, the lesson appears at a smaller scale. A fine-tuning run with bad data is still bad if it is cheap. A run with excellent labels but no baseline cannot prove improvement. A run that improves one benchmark but doubles latency may be a poor product decision. Compute is one budget; evaluation quality is another.

Section 8: What can go wrong?

Training failures are often ordinary engineering failures wearing machine-learning clothes. The model may be fine; the data, split, metric, logging, checkpoint, or comparison may be broken.

Section 9: A beginner checklist before starting a run

Before spending training budget, write down:

• Goal: What behavior should improve?
• Baseline: What current system must the run beat?
• Data: What exact dataset version will train the model?
• Splits: Which examples are train, validation, and test?
• Config: What model, optimizer, learning rate, batch size, steps, seed, and schedule will be used?
• Metrics: Which loss, task metrics, safety checks, and regression checks matter?
• Logging: Where will parameters, metrics, curves, notes, and artifacts be recorded?
• Checkpoints: How often will they be saved, and how will the best checkpoint be selected?
• Stop rule: When should the run stop early, resume, or be declared failed?
• Decision rule: What result is good enough to ship, and what result is only interesting?

This checklist is intentionally plain. It prevents the most common beginner mistake: starting a run before the team knows what the run is supposed to prove.

Common misconceptions

Practice checks

1. You start a run and loss becomes NaN after 50 steps. Name three things to inspect before changing the model architecture.
2. A run has better validation loss but worse JSON formatting reliability. Should it ship? Explain what additional evaluation is needed.
3. You want to resume a run after a power outage. What checkpoint fields matter beyond model weights?
4. A teammate claims a trained model improved because the final loss is lower than yesterday. What run-record fields do you need before trusting the claim?
5. A small model trained on more high-quality tokens beats a larger model trained briefly. Which scaling-law intuition does this illustrate?

Additional implementation resources

• PyTorch autograd tutorial: how gradients are computed and used in training.
• PyTorch saving and loading guide: how to save weights and resumable checkpoints.
• Hugging Face Trainer documentation: a widely used training loop abstraction for Transformer models.
• TensorBoard scalars tutorial: how to visualize loss, metrics, and learning rate over time.
• MLflow Tracking documentation: how to log parameters, metrics, and artifacts.
• Weights & Biases experiment tracking documentation: how to compare and inspect runs.

What you should be able to say now

A training run is a tracked experiment, not a magic event. Each batch creates predictions, loss, gradients, and an optimizer update. The run's configuration controls how those updates happen. Validation curves and task-specific evals show whether learning generalizes. Checkpoints preserve model and optimizer state so training can resume or the best candidate can be evaluated. Experiment tracking makes claims reproducible. Compute budgets connect model size, data size, training tokens, and cost.

Most importantly: a training run is only useful if it answers a decision question. Did this model improve the behavior we care about, against a strong baseline, without unacceptable regressions?