"Foundations of Large Models" provides a comprehensive introduction to the fundamental knowledge, architecture design, training optimization, and application practices of large language models (LLMs). Starting from the basic theory of language models, it delves into model architectures based on statistics, RNNs, and Transformers, with a focus on analyzing the architecture types of large language models (such as Encoder-only, Encoder-Decoder, and Decoder-only) and their representative models (such as BERT, T5, and the GPT series). The book explains in detail key technologies such as Prompt engineering, parameter-efficient fine-tuning, model editing, and retrieval-augmented generation, and demonstrates the application of these technologies in different scenarios through practical cases. Through systematic learning and practical guidance, it helps readers deeply understand and effectively apply large language model technology.
Get the "Foundations of Large Models" report PDF original file. Scan the QR code to follow and reply: 20250915
Foundations of Language Models
- Language Models Based on Statistical Methods: Introduces the n-gram model and its statistical principles (Markov assumption, maximum likelihood estimation).
- RNN-Based Language Models: Explains the structure of recurrent neural networks (RNNs), training issues (vanishing/exploding gradients), and their application in language modeling.
- Transformer-Based Language Models: Provides a detailed analysis of the Transformer architecture (self-attention, FFN, layer normalization, residual connections) and its application in language models.
- Sampling Methods for Language Models: Includes decoding strategies such as greedy search, beam search, Top-K sampling, Top-P sampling, and the Temperature mechanism.
- Evaluation of Language Models: Introduces intrinsic evaluation (such as perplexity) and extrinsic evaluation (such as BLEU, ROUGE, BERTScore, G-EVAL).
Large Language Model Architectures
- Big Data + Large Models → New Intelligence: Analyzes the impact of model scale and data scale on model capabilities, and introduces Scaling Laws (Kaplan-McCandlish, Chinchilla).
- Overview of Large Language Model Architectures: Compares the attention mechanisms and applicable tasks of the three mainstream architectures: Encoder-only, Encoder-Decoder, and Decoder-only.
- Encoder-only Architecture: Represented by BERT, introduces its structure, pre-training tasks (MLM, NSP), and derivative models (RoBERTa, ALBERT, ELECTRA).
- Encoder-Decoder Architecture: Takes T5 and BART as examples to introduce their unified text generation framework and diverse pre-training tasks.
- Decoder-only Architecture: Provides a detailed introduction to the development and characteristics of the GPT series (GPT-1 to GPT-4) and the LLaMA series (LLaMA1/2/3).
- Non-Transformer Architectures: Introduces state space models (SSMs) such as RWKV and Mamba, as well as the test-time training (TTT) paradigm.
Prompt Engineering
- Introduction to Prompt Engineering: Defines Prompt and Prompt engineering, and introduces the tokenization and vectorization process (Tokenization, Embedding).
- In-Context Learning (ICL): Introduces zero-shot, one-shot, and few-shot learning, and example selection strategies (similarity, diversity).
- Chain-of-Thought (CoT): Explains the three modes of CoT: step-by-step (CoT, Zero-Shot CoT, Auto-CoT), think before acting (ToT, GoT), and pooling wisdom (Self-Consistency).
- Prompt Techniques: Introduces standardized prompt writing, reasonable summarization of questions, timely use of CoT, and skillful use of psychological suggestion (role-playing, scenario immersion).
- Related Applications: Introduces large-model-based agents (Agent), data synthesis, Text-to-SQL, GPTS, etc.
Parameter-Efficient Fine-Tuning
-
Introduction to Parameter-Efficient Fine-Tuning: Introduces the two mainstream methods for downstream task adaptation—in-context learning and instruction tuning—and leads to parameter-efficient fine-tuning (PEFT) technology, explaining its advantages in reducing cost and improving efficiency.
-
Parameter Addition Methods: Provides a detailed introduction to methods that achieve efficient fine-tuning by adding new, smaller trainable modules to the model structure, including implementations and advantages of adding to the input (such as Prompt-tuning), adding to the model (such as Prefix-tuning and Adapter-tuning), and adding to the output (such as Proxy-tuning).
-
Parameter Selection Methods: Introduces methods that select only a portion of the model parameters for fine-tuning, divided into rule-based methods (such as BitFit) and learning-based methods (such as Child-tuning), reducing computational burden and improving model performance through selective parameter updates.
-
Low-Rank Adaptation Methods: Provides a detailed introduction to methods that achieve efficient fine-tuning by approximating the original weight update matrix with low-rank matrices, focusing on LoRA and its variants (such as ReLoRA, AdaLoRA, and DoRA), and discusses LoRA's parameter efficiency and task generalization ability.
-
Practice and Applications: Introduces the usage methods and related techniques of the HF-PEFT framework, demonstrates application cases of PEFT technology in tabular data querying and tabular data analysis, and proves the effectiveness of PEFT in improving the performance of large models on specific tasks.
Model Editing
-
Introduction to Model Editing: Introduces the ideas, definitions, and properties of model editing, and explains the importance of model editing in correcting bias, toxicity, and knowledge errors in large language models.
-
Classic Methods of Model Editing: Divides model editing methods into external extension methods (such as knowledge caching and additional parameter methods) and internal modification methods (such as meta-learning and locate-then-edit methods), and introduces representative works in each category.
-
Additional Parameter Method: T-Patcher: Provides a detailed introduction to the T-Patcher method, which achieves precise control over model output by adding specific parameters to the model, suitable for scenarios requiring fast and accurate correction of specific knowledge points in the model.
-
Locate-then-Edit Method: ROME: Provides a detailed introduction to the ROME method, which achieves precise control over model output by locating and modifying specific layers or neurons inside the model, suitable for scenarios requiring deep modification of the model's internal knowledge structure.
-
Applications of Model Editing: Introduces practical applications of model editing in precise model updating, protecting the right to be forgotten, and improving model safety, demonstrating the application potential of model editing technology in different scenarios.
Retrieval-Augmented Generation
-
Introduction to Retrieval-Augmented Generation: Introduces the background and composition of retrieval-augmented generation, and explains the importance and application scenarios of improving model performance by combining retrieval and generation in natural language processing tasks.
-
Retrieval-Augmented Generation Architectures: Introduces RAG architecture classification, black-box enhancement architectures, and white-box enhancement architectures, and comparatively analyzes the characteristics and applicable scenarios of different architectures.
-
Knowledge Retrieval: Introduces methods for knowledge base construction, query enhancement, retrievers, and retrieval efficiency enhancement, and how to improve retrieval effectiveness through reranking of retrieval results.
-
Generation Enhancement: Introduces methods for when to enhance, where to enhance, multiple enhancements, and cost reduction and efficiency improvement, and discusses application strategies of generation enhancement in different tasks.
-
Practice and Application: Introduces the steps to build a simple RAG system, showcases cases of RAG in typical applications, and helps readers understand and apply retrieval-augmented generation technology.
Get the "Large Model Basics" report PDF original file. Scan the QR code to follow and reply: 20250915