LLMs Derived

What does a large language model actually compute? Not what it might do one day, but what it does, expressed in the mathematics of the original papers, derived from first principles and connected end to end.

LLMs Derived is a complete, derivation-first treatment of the mathematics and engineering behind modern large language models, for readers who want the real thing rather than a product tour. Across eighteen chapters it derives the transformer architecture, attention, tokenization, pretraining, scaling laws, distributed training systems, fine-tuning, RLHF and DPO, and inference-time decoding. Every claim is backed by a derivation, a citation, or working code.

Inside you will find:

Scaled dot-product attention, derived from first principles and implemented from raw tensors
The full parameter-counting formula for a transformer, checked against GPT-2 and GPT-3 configurations
Chinchilla’s compute-optimal scaling laws, re-derived and fit to synthetic data
Distributed training: data, tensor and pipeline parallelism, and the ZeRO memory-partitioning stages
LoRA and QLoRA, derived and implemented as working PyTorch modules
The Direct Preference Optimization (DPO) loss, derived step by step from the RLHF objective
Speculative decoding, with a proof that its output distribution matches the target model
A grounded, citation-backed discussion of hallucination, evaluation contamination and mechanistic interpretability

Every chapter includes complete, runnable Python (PyTorch and NumPy), a summary of key results, and exercises that mix hand derivations with coding tasks.

Who it is for: graduate students, software engineers moving into ML roles, and researchers in adjacent fields who already know linear algebra, calculus, probability and basic Python/PyTorch, and want the specific mathematics of large language models rather than a general introduction to machine learning.

Material portions of this book were generated using generative AI tools, then reviewed, edited, designed, and curated by Choral Software.

Companion source code: Download the examples and pseudocode from https://pub-e945d7c8d1d640c7be2864f060e0f048.r2.dev/llms-derived-companion-v1.zip