Build a Large Language Model (From Scratch) is a book by bestselling author Sebastian Raschka, designed to teach you step by step how to create your own large language model (LLM). Each stage is explained through clear text, diagrams, and examples. You will go through the complete process from initial design and creation, to pretraining on a general corpus, to fine-tuning for specific tasks.
Main purpose: This book is aimed at readers who want to deeply understand and build an LLM by hand. It takes you inside the AI black box, hands-on adjusting the internal systems that drive generative AI. By completing the key stages of LLM creation, you will gain a deep understanding of how LLMs work, their limitations, and how to customize them. The LLM you build can be developed on an ordinary laptop and used as your own personal assistant.
Main capabilities: The book teaches the following: planning and writing all parts of an LLM; preparing datasets suitable for LLM training; fine-tuning an LLM for text classification and custom data; using human feedback to ensure an LLM follows instructions; loading pretrained weights into an LLM. Specifically, you will learn to plan and write an LLM comparable to GPT-2, load pretrained weights, build a complete training pipeline, fine-tune an LLM for text classification, and develop an LLM that follows human instructions.
How to use: The book is available in print (black-and-white printing), eBook (PDF, ePub, online), and online + audio (readable and listenable) versions. It can be obtained through a Manning subscription: the Pro subscription is $24.99 per month, providing access to all Manning books, MEAP, liveVideos, liveProjects, and audiobooks, with the option to keep one free eBook each month, a 50% discount on all purchases, monthly renewal, and the ability to pause or cancel at any time; the Lite subscription is $19.99 per month, providing access to all Manning books (including MEAP). Teams can purchase 5, 10, or more than 20 seats. When renewing a subscription, you can choose free products, with twelve free products available each year.
Applicable scenarios: This book is suitable for developers, researchers, and enthusiasts who want to build an LLM from scratch. Readers should have some programming background. The LLM discussed in the book can be applied to tasks such as chatbots, machine translation, text summarization, sentiment analysis, and content creation, and may transform fields such as customer service, education, healthcare, and research. The book points out that with this book you can build capabilities that aresimilar to GPT-2 chatbots; GPT-3 is larger and more complex, requiring substantial computing resources and data, which is beyond the scope of this book, but the foundational knowledge learned is still applicable to understanding and working with GPT-3-like models.
Limitations and considerations: The book mentions that key challenges facing LLMs include handling bias, ensuring safety and ethical use, maintaining transparency and interpretability, and ensuring data privacy and security. Ethical issues include potential job displacement, the spread of misinformation, perpetuation of bias, and malicious misuse of LLMs. Training LLMs strongly recommends using GPUs, because their parallel computing capability can significantly accelerate the training process. The book emphasizes building models from scratch rather than merely fine-tuning existing models, covering the basic concepts and techniques needed to develop and train your own large language model.
Other information: ISBN 9781633437166. Available in complex Chinese, German, Italian, Japanese, Korean, Polish, Russian, Simplified Chinese, and Spanish versions. The print edition includes an eBook. The first chapter provides audio and a summary. This book is a bestseller, has received positive reader reviews, and has been praised as "truly inspiring," "the most accessible and comprehensive explanation of language models to date," and "authoritative and up-to-date coverage, highly recommended."
