5 Pretraining on unlabeled data
ver. 1.0.0, 5_pretraining_on_unlabeled_data
Train, evaluate, sample from, persist, and load a GPT-2 style language model
This unit teaches how to turn an untrained GPT-style model into a usable language model: compute cross-entropy loss and perplexity, create training/validation loaders, implement a complete AdamW training loop and interpret loss curves (including memorization), replace greedy decoding with temperature-controlled multinomial sampling and top-k filtering, combine decoding options into a single generate function with early stopping, save and resume model + optimizer checkpoints, and map official GPT-2 weights into your PyTorch implementation to produce coherent text.
You learn to evaluate, train, decode from, and persist a GPT-style transformer so it becomes a usable LLM. Starting from model logits, you derive the cross-entropy loss (softmax → target probabilities → log → negative mean) and interpret model error as perplexity. You split a corpus into training and validation sets, build sliding-window data loaders, and compute average loss over an entire loader so you can monitor generalization.
You implement a complete pretraining loop that optimizes model weights with the AdamW optimizer, run multi-epoch training, and read training/validation loss curves to detect memorization when training loss drops while validation loss lags. For generation you replace deterministic argmax decoding with multinomial sampling and introduce temperature to control sampling sharpness. You also implement top-k filtering to cut off low-probability tails, and fold context truncation, top-k, temperature and early stopping on an end-of-sequence token into a single, reusable generate function.
You persist progress by saving and loading model state_dicts and extend checkpoints to include optimizer state so training can resume seamlessly. Finally, you import official GPT-2 weights into your PyTorch model by mapping released tensors into your modules and verify success by generating coherent English text. After this unit you can compute and interpret loss/perplexity, train and monitor a language model, sample flexibly from it, save and restore full checkpoints, and load pretrained GPT-2 weights for downstream use or fine-tuning.
Materials
Source document
- Building a Large Language Model (from scratch), Sebastian Raschka, 2024, Manning Books — Link — Page 150-190