How Do Ordinary Users Perform Post-Training? A Clear Guide

In the field of artificial intelligence, especially in the development and deployment of large language models (LLMs), although pre-training provides the model with broad language understanding and generation capabilities, post-training is the key step that truly transforms these foundational capabilities into specific tasks and practical application value.Post-training can significantly improve the model's performance, safety, controllability, and efficiency, making it more aligned with human expectations and the needs of specific scenarios.

Through post-training, a general-purpose language model can be tuned into a professional legal assistant, a medical consultant, or a lightweight application that can run efficiently on edge devices such as mobile phones.

What Is Post-Training?

Definition

Post-Training refers to a series of optimization and adjustment processes carried out after the model has completed initial pre-training.

Pre-training is conducted on massive amounts of general text data, with the goal of enabling the model to learn basic language patterns, world knowledge, and a certain level of reasoning ability. A pre-trained model is often like a "generalist": although it has broad knowledge, its performance on specific tasks or in specific domains may not be precise enough, or its behavioral patterns may not fully align with human expectations.

Post-training exists to solve these problems. By further training the pre-trained model on specific datasets, it is adapted to specific needs.

The process can be compared to this: although a university graduate has mastered professional knowledge, before entering a specific job position, they need to receive pre-job training to learn the company's rules and regulations, business processes, and the skill requirements of the specific position.

The goal of post-training is to improve the model's performance on specific tasks, produce outputs that better align with human preferences, or enable it to run efficiently in resource-constrained environments。

The core of post-training lies in "refinement" and "alignment".

Refinement:refers to strengthening the model's capability in a certain aspect through specific data, such as code generation, mathematical reasoning, or mastery of knowledge in a specific domain.

Alignment:focuses more on making the model's output consistent with human values, preferences, and instructions, ensuring that the model is safe, useful, and harmless.

Timing and Purpose of Post-Training

The timing of post-training is after the model has completed large-scale pre-training. When we need the model to improve in the following aspects, we need to consider post-training:

• Improving performance on specific tasks: When a pre-trained model performs poorly on the specific task you care about (such as text summarization, sentiment analysis, code generation, or question-answering systems), or when you want it to perform more professionally in a vertical domain (such as law, healthcare, or finance), post-training is needed to inject domain knowledge or optimize task-related capabilities.
• Aligning with human preferences and values: Pre-trained models may generate inaccurate, biased, or even harmful content. Post-training, especially reinforcement learning from human feedback (RLHF) or direct preference optimization (DPO), can help the model learn human preferences so that its outputs are safer, more helpful, and more ethically compliant. Llama3's post-training objectives include a reward model and a language model, training the reward model through human-annotated preference data, and then performing rejection sampling and DPO to align with human preferences.
• Improving model efficiency and deployment convenience: Large pre-trained models have a huge number of parameters and high requirements for computing resources and storage space, making them difficult to deploy on edge devices or in resource-constrained environments. Model compression techniques in post-training, such as quantization, pruning, and knowledge distillation, can significantly reduce model size, lower inference latency, and make deployment easier. For example, post-training quantization can reduce model size by 4x and improve speed by 2-3x.
• Adapting to specific data distributions or formats: If your application scenario involves specific types of data (such as tables or code in a particular format) or requires the model to follow a specific output format (such as JSON), post-training can help the model better understand and process this data. The post-training of the Qwen2.5 model significantly improved its ability to understand structured data (such as tables) and generate structured output (especially JSON).
• Iterative improvement and continuous learning: After a model goes live, it may encounter new problems or user feedback. Post-training can also serve as a continuous process, continuously optimizing the model based on new data and feedback to maintain optimal performance and adapt to changing requirements.

The purpose of post-training is to transform a general, raw pre-trained model into a customized model that is more specialized, safer, more efficient, and better able to meet specific application requirements. It is an indispensable part of the model development lifecycle and directly determines the model's performance and value in real applications.

• Supervised Fine-Tuning (SFT): This is the most basic and most commonly used post-training method. By training on high-quality, manually annotated (input, output) data pairs, it teaches the model how to perform specific tasks or follow specific instructions. For example, to teach a model to conduct multi-turn conversations, the SFT data will include the conversation history between the user and the assistant, as well as the expected assistant replies. The post-training of Llama3 includes multiple rounds of SFT, using human-annotated data and synthetic data. The goal of SFT is to teach the model to "imitate" the ideal answers in the annotated data.
• Preference Alignment: Although a model after SFT can answer questions, the quality, safety, or usefulness of the answers may still be insufficient. Preference alignment techniques aim to solve this problem by enabling the model to learn to distinguish between "good" answers and "bad" answers, making it more inclined to generate responses preferred by humans. Common techniques include:
  • Reinforcement Learning from Human Feedback (RLHF): This is a complex but highly effective technique. First, collect human rankings or ratings of different model outputs, then train a Reward Model (RM) to predict human preferences, and finally use reinforcement learning algorithms (such as PPO) to optimize the language model so that it generates text that can obtain higher rewards (that is, better aligns with human preferences). The post-training of Llama3 and GLM-4 both applied RLHF or similar RL techniques.
  • Direct Preference Optimization (DPO): DPO is a simpler and more efficient preference alignment method than RLHF. It does not require explicitly training a reward model, but directly uses preference data to optimize the language model, learning to distinguish between chosen and rejected responses. DPO is usually superior to PPO in training speed, cost, and tuning difficulty, so it is becoming increasingly popular in practice.
• Parameter-Efficient Fine-Tuning (PEFT): Due to the enormous number of parameters in LLMs, full fine-tuning requires huge computational resources and GPU memory. PEFT techniques achieve model adaptation by fine-tuning only a small portion of the model's parameters or adding a small number of trainable parameters, greatly reducing computational cost. Common PEFT methods include:
  • LoRA（Low-Rank Adaptation）: LoRA adds low-rank decomposition matrices beside the model's attention weight matrices for fine-tuning, rather than directly updating the original large weight matrices.
  • QLoRA（Quantized LoRA）: QLoRA combines quantization and LoRA. It first quantizes the base model to low precision (such as 4-bit), and then applies LoRA on the quantized model for fine-tuning, further reducing GPU memory requirements and making it possible to fine-tune large models on a single GPU.
  • Adapter: Adapter inserts small neural network modules in the middle of Transformer layers for fine-tuning, while keeping the backbone parameters of the pretrained model unchanged.
• Model Merging: It is a technique that combines the parameters of multiple trained models with the same structure and size (for example, multiple Llama 3 8B models fine-tuned for different tasks) in some way (such as weighted averaging) to obtain a new model that may have stronger capabilities. Commonly used model merging techniques include TIES, DARE, and SLERP, among others.

Post-training Techniques for Model Compression and Optimization

When large pretrained models need to be deployed in resource-constrained environments (such as mobile devices and embedded systems), or when inference costs need to be reduced,model compression and optimization techniques become particularly important. These techniques are usually applied after model training is completed and are part of post-training. They reduce model size, computational complexity, and memory usage while preserving model performance as much as possible. The main post-training techniques for model compression and optimization include:

• Post-Training Quantization (PTQ): Quantization is the process of converting the weights and/or activations in a model from a high-precision representation (such as 32-bit floating point, FP32) to a low-precision representation (such as 16-bit floating point FP16, 8-bit integer INT8, or even 4-bit integer INT4). PTQ is quantization performed directly after model training is complete, without retraining or requiring only a small amount of calibration data. The PTQ options provided by Google AI Edge include dynamic range quantization (4x smaller, 2-3x speedup, suitable for CPU), full integer quantization (4x smaller, more than 3x speedup, suitable for CPU, Edge TPU, microcontrollers), and Float16 quantization (2x smaller, GPU acceleration, suitable for CPU and GPU). Intel also provides the INCQuantizer class for PTQ. Quantization can significantly reduce model size and accelerate inference, but may cause some accuracy loss, requiring careful selection of quantization methods and calibration data to balance the trade-offs.
• Post-Training Pruning (PTP): Pruning is a technique that reduces model size and computation by removing unimportant weights or neurons in a model (or even entire layers or attention heads). Magnitude-based pruning is a common method, which holds that weights with smaller absolute values contribute less to the model output and can be removed. After pruning, the model usually needs fine-tuning (re-training) to recover the lost accuracy. Pruning can be divided into weight pruning (removing individual connections), neuron pruning (removing entire neurons), and structured pruning (removing larger structures, such as filters or layers). The goal of pruning is to obtain a sparser, more compact model.
• Knowledge Distillation (KD): Knowledge distillation is a model compression technique whose core idea is to have a small model (the student model) learn the behavior and knowledge of a large, high-performance model (the teacher model). In the post-training stage, the teacher model's "soft labels" (soft labels, i.e., output probability distributions) or intermediate-layer features (usually from a fully trained large LLM) are used to guide the training of the student model. By learning to imitate the teacher model's outputs, the student model can achieve performance close to or even surpassing the teacher model while having far fewer parameters than the teacher model. For example, the DeepSeek-RI model serves as a teacher model, transferring its advanced reasoning capabilities to smaller model architectures. Knowledge distillation can not only compress models, but sometimes also improve the student model's generalization ability.

Common Post-Training Techniques

Supervised Fine-Tuning (SFT)

Supervised Fine-Tuning (SFT) is the most fundamental and core part of post-training for large language models. The core idea is very simple: use high-quality, manually labeled (input, output) data pairs to further train a pretrained model so that it adapts to a specific task or learns to follow specific instructions. SFT can be thought of as providing the model with a "standard answers" manual, letting it learn what kind of response it should give in a specific situation. These labeled data are carefully constructed by domain experts or trained annotators to ensure their accuracy and usefulness. For example, when building a customer service chatbot, the SFT data may contain a large number of user questions and the corresponding ideal customer service responses. In the post-training of Llama3, the SFT data mainly consists of prompts collected by human annotation and their responses obtained through rejection sampling, synthetic data for specific capabilities, and a small amount of human-curated data.

The implementation process of SFT is as follows:

• Prepare the dataset: Collect or construct a high-quality labeled dataset relevant to the target task. Each sample contains an input (for example, a user's question or instruction) and a desired output (for example, the model's ideal response or execution result). The quality and diversity of the data are crucial to the effectiveness of SFT.
• Choose the base model: Choose a suitable pretrained language model as the starting point for SFT. This base model should already have relatively good language understanding and generation capabilities.
• Configure training parameters: Set the training hyperparameters, such as learning rate, batch size, number of training epochs, etc. The choice of these parameters affects the speed of training and the performance of the final model. For example, for Llama3's largest model, the learning rate during SFT was set to 1e-5, and the number of training steps was between 8.5k and 9k.
• Execute training: Fine-tune the base model on the prepared dataset. During training, the model tries to predict the output given the input, compares its prediction with the labeled true output, calculates the loss, and then updates the model parameters through the backpropagation algorithm to minimize the loss.
• Evaluate and iterate: After training is completed, evaluate the model's performance on an independent validation set or test set. If the results are not ideal, it may be necessary to adjust the dataset, hyperparameters, or training strategy and perform multiple rounds of iteration.

The advantage of SFT is that its concept is simple, it is easy to implement, and it is very effective at improving the model's performance on specific tasks . It can help the model learn new skills, such as code generation or text summarization, or help it better follow instruction formats, for example by learning a specific dialogue template.

SFT also has some limitations,it relies heavily on the quality of labeled data, if the data contains bias or errors, the model will also learn these bad patterns. Standard SFT does not directly optimize the model to align with broad human preferences (for example, generating more interesting or safer responses), and requires subsequent preference alignment steps (such as RLHF or DPO) to accomplish this. Full-parameter SFT has high computational cost, and for very large models, efficient fine-tuning techniques (such as LoRA) may be needed to reduce resource consumption.

Direct Preference Optimization (DPO)

Direct Preference Optimization (DPO) is an emerging and increasingly popular post-training technique used to align large language models with human preferences. Compared with reinforcement learning from human feedback (RLHF), DPO provides a simpler, computationally cheaper, and often easier-to-tune method to achieve similar goals. RLHF involves training a separate reward model to predict human preferences, and then using reinforcement learning algorithms such as PPO to optimize the language model to maximize this reward. The core idea of DPO, by contrast, is to directly use a dataset containing preference information to optimize the language model itself, without explicitly training a reward model.

The working principle of DPO can be summarized in the following steps:

• Prepare a preference dataset: construct a dataset in which each sample contains a prompt and at least two model-generated responses to that prompt: one response that is "preferred" or "chosen" by human annotators (chosen response), and another response that is "rejected" or considered worse (rejected response). When collecting preference data, Llama3 even introduced a third "edited response", with the preference order being edited > chosen > rejected.
• Define the loss function: DPO optimizes the language model through a carefully designed loss function. This loss function encourages the model to assign higher probability to the "preferred" response and lower probability to the "rejected" response. It cleverly draws on the theory of pairwise comparison models such as the Bradley-Terry model, transforming the preference learning problem into a supervised learning problem.
• Optimize the language model: use standard gradient descent methods to update the language model parameters by minimizing the DPO loss function. This process directly adjusts the model's policy so that it is more inclined to generate text preferred by humans.

The main advantages of DPO compared with RLHF are:

• Simpler: DPO avoids the steps in RLHF of training a reward model and using complex reinforcement learning algorithms, and is more direct to implement.
• More efficient: because it does not require an additional reward model and RL loop, DPO training is usually faster and consumes fewer computational resources. It may only need to load one language model (or two, if a reference model is used), whereas PPO may need to simultaneously load the original model, the model being trained, and the reward model.
• More stable and easier to tune: the PPO algorithm in RLHF is notoriously difficult to tune and is very sensitive to hyperparameters. As a supervised learning problem, DPO's training process is usually more stable, and hyperparameter tuning is relatively simple.
• Comparable or even better performance: in many experiments, DPO has achieved performance comparable to or even better than PPO on preference alignment tasks. Especially when resources are limited, DPO's simplicity makes it a more attractive choice.

Reinforcement Learning (RL)

Reinforcement Learning (RL) plays a crucial role in the post-training of large language models, especially in aligning model behavior with human preferences and improving model performance in complex interactive tasks. Unlike supervised learning, reinforcement learning does not train the model by providing "standard answers"; instead, itlets the model learn the optimal policy through interaction with the environment, so as to maximize cumulative reward. In the post-training context, the environment can be the question or instruction posed by the user, the model's action is generating text, and the reward reflects the quality of the generated text (for example, whether it is accurate, helpful, harmless, fluent, etc.).

Reinforcement Learning from Human Feedback (RLHF) is RL in post-training and includes the following key steps:

• Collect human preference data: First, a dataset needs to be collected that contains human preference rankings or ratings of responses generated by different models. For example, given a prompt, annotators may see several different model outputs and be asked to rank them from best to worst, or to score each output.
• Train a Reward Model (RM): Use the collected human preference data to train a reward model. The task of this reward model is, given a prompt and a response generated by the model, to predict the degree of human preference for that response (that is, to give a scalar reward value). When training the reward model, Llama3 removes the margin term and filters out similar response samples, while using all preference data for training.
• Use an RL algorithm to optimize the language model (Policy Model): After the reward model is trained, use it as the signal for optimizing the language model. The language model (policy model) is fine-tuned through a reinforcement learning algorithm (such as Proximal Policy Optimization, PPO), with the goal of generating responses that can obtain higher rewards from the reward model. To prevent the model from over-optimizing the reward model and deviating too far from the original language model (for example, generating meaningless text that can fool the reward model), a KL divergence penalty term is usually added to constrain the difference between the policy model and the original pretrained model.

The advantage of reinforcement learning is that it can handle complex, sequential decision-making problems, and can guide the model to learn specific behavioral patterns by designing an appropriate reward function. RL also has some challenges:

• Reward function design is difficult: Designing a reward function that can accurately and comprehensively reflect human preferences is very difficult. An imperfect reward function may cause the model to learn unintended behaviors (reward hacking).
• Low sample efficiency, unstable training: RL requires a large amount of interaction data to learn an effective policy, and the training process may be unstable and sensitive to hyperparameters.
• High computational cost: Methods such as RLHF that involve multiple models (policy model, reward model, reference model) and a complex optimization process usually have high computational costs.

Post-Training Quantization (PTQ)

Post-Training Quantization (PTQ) is an important model optimization technique that reduces the size of a model and accelerates inference speed while minimizing the negative impact on model accuracy as much as possible. It is performed after the model has finished training and requires no or only a small amount of unlabeled calibration data, so it is called "post-training" quantization. The core idea is to convert the weights and/or activations in the model from high-precision floating-point representations (for example, 32-bit floating point, FP32) to low-precision representations, such as 16-bit floating point (FP16 or BF16), 8-bit integer (INT8), or even 4-bit integer (INT4) or 2-bit integer (INT2).

Key benefits:The storage space occupied by the model is significantly reduced, memory bandwidth requirements are lowered, and faster computation speeds can be achieved on certain hardware (such as CPUs, GPUs, or specialized accelerators like Edge TPU that support low-precision operations)。

Google AI Edge provides some common PTQ options and their advantages:

• Dynamic Range Quantization: This is the simplest form of PTQ, which only converts weights from FP32 to INT8, while activations are dynamically quantized and dequantized during inference. This method can reduce model size by about 4 times, improve CPU latency by 2-3 times, and usually has little impact on accuracy.
• Full Integer Quantization: This method not only quantizes weights, but also converts activations to INT8 (or other integer formats). This requires a small representative calibration dataset to determine the dynamic range of activations. Full integer quantization can achieve about 4 times model compression and more than 3 times speedup, and the model can run entirely on integer-only hardware (such as microcontrollers, Edge TPU).
• Float16 Quantization: Converts weights and/or activations to 16-bit floating point numbers (FP16 or BF16). This can reduce model size by about 2 times, and can achieve significant acceleration on GPUs that support FP16 operations.

The PTQ process includes the following steps:

• Choose a quantization scheme: Determine the parts to quantize (weights only, or weights and activations), the target precision (such as INT8, FP16), and the granularity of quantization (such as per-layer quantization, per-channel quantization).
• Calibration (for full integer quantization): If activations are quantized, a small, representative calibration dataset is usually needed. When the model performs inference on this calibration data, it collects statistics of activations (such as minimum and maximum values) to determine quantization parameters (such as scale factor and zero point).
• Conversion and deployment: Based on the selected quantization scheme and the parameters obtained from calibration, convert the original FP32 model into a quantized model. The converted model can be deployed to run on the target hardware.

Post-Training Pruning (PTP)

Post-Training Pruning (PTP) is a model compression technique that aims to reduce model size, lower computational complexity, and accelerate inference by removing unimportant parameters (weights) or structural units (such as neurons, channels, or even entire layers) from a neural network, while preserving model performance as much as possible. Unlike quantization, pruning achieves compression by reducing the number of connections or units in the model. Pruning can be thought of as putting the model on a "diet," removing the "excess fat" that contributes little to the final output. Pruning is performed after model training is complete, so it is part of post-training optimization.

The PTP process includes the following steps:

• Importance evaluation: First, the importance of each parameter or structural unit in the model needs to be evaluated. Commonly used evaluation methods include:
  • Magnitude-based Pruning: This is the simplest and most commonly used method. It considers that parameters with smaller absolute weight values contribute less to the model output, and therefore can be pruned first.
  • Sensitivity-based Pruning: Analyze the degree of impact of removing a certain parameter or unit on the model loss function or accuracy; the smaller the impact, the less important it is considered to be.
  • Gradient-based pruning: Use gradient information to determine the importance of parameters.
• Pruning strategy: Based on the importance evaluation results, select the parameters or units to remove. Pruning can have different granularities:
  • Unstructured Pruning: Also known as weight pruning, it removes individual weights one by one. This method can achieve very high sparsity, but the pruned model is usually irregular and difficult to obtain actual acceleration on general-purpose hardware unless supported by specialized sparse computation libraries.
  • Structured Pruning: Removes entire structural units, such as neurons, channels (filters), attention heads, or even entire layers. The model produced by this method remains regular and can directly obtain acceleration on existing hardware, but it may be more difficult to maintain accuracy than unstructured pruning.
• Pruning Execution: Removes unimportant parameters or units according to the selected strategy and pruning ratio (for example, removing 20% of the weights).
• Fine-tuning / Re-training: Pruning inevitably leads to a decline in model performance. Therefore, after pruning, the model usually needs to be fine-tuned, retraining on the remaining parameters for a period of time to recover the lost accuracy. This fine-tuning process usually uses a smaller learning rate.

Pruning techniques are very effective for reducing model size and computation, especially when deploying models on resource-constrained devices. For example, in the field of computer vision, pruning can significantly reduce the size and computational requirements of deep neural networks. For large language models, pruning is also an important research direction, although LLM pruning faces greater challenges due to their huge number of parameters and complex structures.

For non-technical people, the key to understanding PTP is that it is a method of making a model smaller and faster by "removing unimportant parts of the model." Choosing an appropriate pruning strategy and pruning ratio, and performing sufficient fine-tuning after pruning, are key to ensuring model performance. Similar to quantization, pruning also requires a trade-off between model compression rate and performance preservation.

Knowledge Distillation

Knowledge Distillation (KD) is a model compression and performance improvement technique. Its core idea is to transfer the "knowledge" learned by a large, high-performance, complex model (called the Teacher Model) to a small, lightweight model (called the Student Model). This method enables the student model to achieve performance close to or sometimes exceeding that of the teacher model while having far fewer parameters than the teacher model. Knowledge distillation is performed after model training is completed and is part of post-training optimization, creating a smaller and more efficient model that retains the powerful capabilities of the large model. For example, the DeepSeek-RI model serves as a teacher model, transferring its advanced reasoning capabilities to smaller model architectures.

The process of knowledge distillation involves the following steps:

• Train the Teacher Model: First, a large teacher model that performs excellently on the target task is needed. This model usually has a large number of parameters and high training cost.
• Define the Student Model: Design a small student model with a simpler structure, fewer parameters, and lower computational complexity.
• Transfer Knowledge: The training objective of the student model is to imitate the behavior of the teacher model. This is achieved in two ways:
  • Soft Target Learning: The teacher model produces "soft labels" on the training data, i.e., output probability distributions (for example, for an image of a cat, the teacher model may output "cat: 0.9, dog: 0.05, fox: 0.05"). These soft labels contain richer information than "hard labels" (for example, "cat: 1"), reflecting the similarities between categories. While learning the true labels, the student model also learns to fit the teacher model's soft labels.
  • Feature Representation Learning: The student model can also learn to imitate the feature representations of the teacher model's intermediate layers, acquiring more robust feature extraction capabilities.
• Student Model Training: The student model is trained by minimizing a combined loss function. This loss function usually consists of two parts: one is the standard loss between the student model's predictions and the true hard labels (such as cross-entropy loss), and the other is the distillation loss between the student model's predictions and the teacher model's soft labels (or features) (such as KL divergence).

The advantage of knowledge distillation is that it can compress the model and sometimes also improve the student model's generalization ability and robustness. This is because the student model learns smoother and richer knowledge representations from the teacher model. For non-technical people, knowledge distillation can be understood as a process of "master-apprentice inheritance," in which a small model (the student) quickly masters complex skills by learning from a large model (the teacher).

How to determine whether post-training is needed?

To determine whether a model needs post-training, it is necessary to comprehensively consider the model's performance in the current application scenario and the constraints of the deployment environment. The following are some common signals indicating that a model may need post-training:

• The model performs poorly in a new application or domain and needs to adapt to new data/tasks: If a pre-trained model is directly applied to a new domain or task whose data distribution differs from that of the pre-training data, its performance may decline significantly. For example, a language model trained on general corpora may fail to accurately understand technical terminology or industry norms when processing professional texts in a specific industry (such as legal documents or medical reports), resulting in low-quality or erroneous generated content. In this case, post-training (such as supervised fine-tuning, SFT) is needed to optimize the model using data from that specific domain so that it adapts to the new data characteristics and task requirements.
• The model needs to be deployed on resource-constrained mobile devices or embedded systems, requiring it to be smaller and more efficient: Many advanced deep learning models have a huge number of parameters and high computational complexity, making it difficult to deploy them directly on mobile devices with limited storage space and weak computing power (such as smartphones and tablets) or embedded systems (such as smart cameras and wearable devices). Direct deployment may cause applications to run slowly, consume a lot of power, or even fail to run. In this case, post-training techniques for model compression, such as post-training quantization (PTQ) or post-training pruning (PTP), are needed to reduce the model size and computational load so that it meets the resource constraints of the deployment environment.
• Model inference takes too long, affecting user experience, and the model needs to be accelerated: In certain application scenarios with high real-time requirements, such as online translation, real-time speech recognition, and interactive dialogue systems, the inference speed of the model is crucial. If model inference takes too long, users may need to wait a long time to get results, which can seriously affect the user experience. Through post-training techniques such as quantization, pruning, or knowledge distillation, the model structure can be optimized, reducing the amount of computation and thereby significantly improving the model's inference speed to meet real-time requirements.
• The model output does not meet specific requirements or preferences: Sometimes, although the model can complete the task, its output style, tone, or content may not meet specific application needs or user preferences. For example, a general-purpose text generation model may produce content that is too generic and lacks expertise in a specific domain, or its tone may not be friendly or formal enough. In such cases, post-training techniques such as reinforcement learning from human feedback (RLHF) or direct preference optimization (DPO) can be used to adjust the model with human preference data, making its output more in line with expectations.
• The model has bias or safety issues: A pretrained model may learn some social biases from the training data (such as gender or racial bias) or produce unsafe, unethical outputs. If such problems are found in the model, post-training can be used to fine-tune the model with filtered and corrected data, or specific constraints and penalty terms can be introduced to reduce bias and improve the model's safety and reliability.

How to choose an appropriate post-training technique?

Consider the optimization objective and computational resources: First, it is necessary to clarify what the main objective of post-training is.

If the objective is to improve the model's accuracy on a specific task or adapt it to a new domain, then supervised fine-tuning (SFT) or direct preference optimization (DPO) can be chosen.

If the objective is to reduce model size and accelerate inference for deployment on resource-constrained devices, then post-training quantization (PTQ), post-training pruning (PTP), or knowledge distillation is a more suitable choice.

It is necessary to evaluate the available computational resources. For example, SFT and knowledge distillation usually require more computational resources and time for training, while the implementation cost of PTQ and PTP (especially PTQ) is relatively low.

• Want a Chinese model to adapt to the legal domain: In this case, the objective is to improve the model's understanding and generation capabilities in a specific professional domain. You can choosesupervised fine-tuning (SFT). First, it is necessary to collect or construct a dataset containing a large amount of legal-related text (such as laws and regulations, cases, contracts) and their corresponding labels or expected outputs. Then, use this dataset to perform SFT on the pretrained Chinese language model. Through fine-tuning, the model can learn professional terminology, expression habits, and logical structures in the legal domain, thereby performing more professionally in tasks such as answering legal consultations and analyzing legal documents.
• Deploying a CNN image classification model on mobile devices: Deployment on mobile devices has strict requirements for model size and inference speed. Post-training quantization (PTQ) is a very effective and relatively simple compression method. PTQ can convert the weights and activations of a CNN model from FP32 to INT8, significantly reducing model size (by about 4 times) and accelerating inference (on hardware that supports INT8 operations). It requires only a small calibration dataset, does not require retraining, and has low deployment cost.
• There is already a pretrained model but no GPU resources for large-scale training: If computational resources are limited, but you still want to compress and optimize the model, you can considerpost-training pruning (PTP). PTP can be completed on a CPU, reducing the number of parameters by removing unimportant weights or neurons from the model. Although the pruned model may need fine-tuning to restore performance, the computation required for fine-tuning is usually much less than training from scratch. Choosing an appropriate pruning rate and pruning strategy can achieve a balance among model size, speed, and accuracy.
• Deploying a small model to an edge device while hoping to maintain relatively high performance: When a model needs to be deployed on an edge device with very limited computing power and storage space,knowledge distillationis a very good choice. You can train a large teacher model (in the cloud or on a high-performance server), and then use knowledge distillation techniques to transfer its knowledge to a small student model. The student model has a simple structure and few parameters, making it suitable for running on edge devices, while also being able to learn rich knowledge from the teacher model, thereby maintaining relatively high performance.
• Hoping that model outputs better align with human preferences or a specific style: If there are specific requirements for the model's output style, creativity, or safety, you can consider usingdirect preference optimization (DPO) or reinforcement learning from human feedback (RLHF). These techniques use human preference data over different model outputs to guide model training and generate text that better meets expectations. For example, they can be used to optimize the conversation quality of a chatbot, making its responses more natural and more helpful.

Implementation steps of post-training

• Evaluate the model's current performance and determine whether post-training is needed: This is the starting point of the post-training process. First, a comprehensive evaluation of the performance of the pretrained model or existing model needs to be conducted on the target dataset or in the actual application scenario. Evaluation metrics should depend on the specific task; for example, for classification tasks they can be accuracy, precision, and recall; for generation tasks they can be BLEU, ROUGE, etc.; for deployment scenarios, attention needs to be paid to the model's inference speed, memory usage, etc. Through evaluation, clearly identify the model's current problems, such as inference speed being too slow, accuracy not being high enough, model size being too large, etc. Only after the problems are clarified can it be decided whether and how to perform post-training.
• According to the optimization goal and model type, choose an appropriate post-training technique: After determining that post-training is needed, the next step is to choose an appropriate post-training technique based on the problems found in the first step's evaluation (that is, the optimization goals) and the type of model (for example, whether it is an LLM or a CNN). If the goal is to improve an LLM's performance on a specific task, SFT or DPO can be considered. If the goal is to compress a CNN model for deployment on mobile devices, PTQ or PTP may be a better choice. When choosing a technique, its complexity, required resources, and expected effects also need to be considered.
• Prepare the necessary resources, including data (such as fine-tuning data and calibration data) and computing resources: Different post-training techniques have different data requirements. For example, SFT requires labeled data for a specific task; PTQ requires a small amount of unlabeled calibration data to collect statistics of activation values; knowledge distillation requires training data and soft labels generated by the teacher model. The corresponding dataset needs to be prepared according to the selected technique. The post-training process (especially SFT, RL, and knowledge distillation) may require certain computing resources, such as GPUs, which also need to be planned and prepared in advance.
• Apply the selected post-training technique and adjust parameters to observe the effect: After the data and resources are ready, you can begin applying the selected post-training technique. For example, if SFT is chosen, you need to set hyperparameters such as the learning rate, batch size, and number of training epochs, and train the model on the fine-tuning dataset. If PTQ is chosen, you need to select an appropriate quantization algorithm and calibration method. During implementation, you may need to try different parameter configurations and observe the model's performance on the validation set to find the optimal settings.
• Evaluate the performance of the post-trained model on the validation set (accuracy, speed, size, etc.): After completing one round of post-training, the optimized model must be rigorously evaluated. Use an independent validation set (non-overlapping with the training set and test set) to evaluate the model's performance on the target task, including the main evaluation metrics (such as accuracy) as well as related efficiency metrics (such as inference speed and model size). Compare it with the original model to determine whether post-training has achieved the expected effect.
• Analyze the results; if expectations are not met, adjust the post-training method or parameters and repeat the process: If the effect of post-training does not meet expectations, for example, if the performance improvement is not obvious, or the accuracy loss is too large, then the cause needs to be analyzed. It may be that the selected technique is not applicable, the parameter settings are improper, or the data quality is not high. Based on the analysis results, adjust the post-training method (for example, switching from PTQ to PTP) or parameters (for example, adjusting the learning rate or pruning rate), and then repeat steps 4 and 5 for iterative optimization. This process may require multiple attempts to find the best solution.
• After optimization is complete, deploy the post-trained model to the target environment and enjoy the improved performance: When the post-trained model achieves satisfactory performance on the validation set, it can be deployed to the actual target environment for application. After deployment, it is still necessary to continuously monitor the model's performance in actual operation and collect user feedback so that further adjustments and optimizations can be made when necessary. After successful deployment, users will be able to experience the benefits brought by improvements in model performance or efficiency.

Application cases of post-training

Case 1: A large language model in a medical field improved its accuracy in understanding domain terminology by 20% after SFT, and its question-answering responses became more professional.

In this case, a general-purpose large language model was applied in the healthcare field to answer user inquiries about diseases, drugs, treatment plans, and other aspects. However, the initial model performed poorly in handling medical terminology, understanding complex medical record descriptions, and providing answers that conform to medical standards. To improve the model's professionalism in this field, the R&D team collected a large amount of medical literature, clinical guidelines, electronic medical records, and doctor-patient question-answer data, and cleaned and annotated these data. Using the supervised fine-tuning (SFT) method, the pretrained language model was further trained with these medical domain data.After SFT, the model's accuracy in understanding medical terminology improved by about 20%, and it can more accurately grasp the intent of user inquiries and generate more professional and more reliable answers.

Case 2: A deep CNN model trained in the cloud, after PTQ compression, was reduced in size by 4 times and improved in inference speed by 3 times, and was successfully deployed in a mobile application.

A company developed an image recognition application based on a deep convolutional neural network (CNN). The model was trained on cloud servers and can recognize thousands of objects with high accuracy. To integrate this functionality into its mobile application for offline use by users, it faced the challenges of an excessively large model size and overly slow inference speed. The original FP32 model was about 200MB in size, and inference on a single image on a high-end phone took several seconds, failing to meet the real-time and storage constraints of a mobile application. To solve this problem, post-training quantization (PTQ) was used. Through PTQ, they converted the model's weights and activations from FP32 to INT8 precision. This process required only a small calibration dataset containing a few representative images and did not require retraining the model.The quantized model size was reduced to about 50MB (a 4x reduction), and on mobile processors that support INT8 operations, inference speed increased by nearly 3x, reaching the level of processing multiple images per second.

The Value and Outlook of Post-Training

As an important part of the model development process, post-training is valuable because it can significantly improve a model's performance and efficiency in specific scenarios and maximize the model's practical value. Through post-training, a general pretrained model can be "refined" into a "sharp tool" adapted to a specific task,

As deep learning models continue to grow larger and their applications become increasingly widespread, post-training techniques will continue to play a key role. On the one hand, post-training techniques for large language models will place greater emphasis on aligning with human intent, improving the safety of generated content, reducing bias, and exploring more efficient and more stable training methods. On the other hand, model compression techniques will develop toward higher compression ratios, lower precision, and stronger hardware adaptability to meet the growing demand for on-device intelligence. Automated post-training workflows, interpretable evaluation of post-training effects, and post-training optimization for specific hardware platforms will also be important research directions. Understanding the principles and techniques of post-training and reasonably selecting and applying these methods according to actual needs is key for developers and enterprises to enhance the competitiveness of AI products and promote the practical implementation of AI technology.