Train Your Own Language Model
An Independent, Hands-On Guide to Building Small and Large Language Models from Scratch with Python and PyTorch
-
- $17.99
-
- $17.99
Publisher Description
An independent, hands-on guide to building language models yourself, from an empty folder to a tested, documented model you can run on your own hardware. Following one fictional library consortium that trains its own family of small domain models, the book shows every step in original Python and PyTorch code: the math and tensor skills you need; collecting only openly licensed data while respecting dataset licenses, site terms and privacy law; cleaning, deduplicating and filtering a corpus; training a byte-level BPE tokenizer; token embeddings and rotary position encodings; causal attention, multi-head and grouped-query attention; and a complete decoder-only transformer. It then covers the training loop with AdamW, learning-rate schedules, mixed precision, gradient accumulation and checkpointing, a tiny model trained end to end on a laptop or a single GPU, scaling laws and compute budgets with worked illustrative costs, and distributed training with DDP and FSDP. The final part covers evaluation and contamination checks, decoding, supervised fine-tuning, LoRA and preference tuning, safety evaluation, licensing and model cards, quantization and local serving, and when fine-tuning an open-weight model is the better choice.
This book is an independent publication and is not authorized, sponsored or endorsed by any company whose products it mentions. All trademarks belong to their respective owners and are used only for identification. This book was drafted with generative AI tools and checked with automated tools. Tools, features and prices change often; confirm them in current documentation.