What is KuaiFormer
KuaiFormer is a Transformer-based retrieval framework launched by the Kuaishou technology team, used in large-scale content recommendation systems. By redefining the retrieval process, it shifts from the traditional score estimation task to a Transformer-driven "next action prediction" paradigm, effectively performing real-time interest acquisition and multi-interest extraction, and significantly improving retrieval performance. KuaiFormer uses multi-interest query tokens and an adaptive sequence compression mechanism to achieve stable training on a candidate set of hundreds of millions. It was integrated into the Kuaishou App's short video recommendation system in May 2024, serving more than 400 million daily active users and significantly increasing users' average daily usage time.
KuaiFormer's main features
- Multi-interest extraction: Introduces multiple query tokens to capture users' diverse interests, enabling the model to better understand and predict users' complex interests.
- Adaptive sequence compression: To improve the efficiency of long sequence modeling, KuaiFormer designs an adaptive sequence compression mechanism that reduces the input sequence length by compressing earlier watched video sequences while retaining the latest video information.
- Stable training techniques: KuaiFormer uses a customized softmax learning objective and LogQ correction method to stably train the model, maintaining performance when facing a candidate set of hundreds of millions.
- Real-time recommendation: Responds to user requests in real time, quickly selecting candidate items related to users' real-time interests from billions of options.
KuaiFormer's technical principles
- Transformer architecture: Captures complex dependencies in user behavior sequences based on the self-attention mechanism.
- Next Action Prediction: KuaiFormer transforms the retrieval process into predicting the user's next action, which helps more accurately capture users' real-time interests.
- Multi-interest Query Token: Inspired by the [CLS] Token in BERT, it introduces multiple learnable Query Tokens, combined with a multi-interest training strategy, to extract different user interest representations from historical item sequences.
- Adaptive Item compression mechanism: KuaiFormer groups and compresses early item sequences to reduce the input sequence length, while performing fine-grained modeling of the latest items.
- Smooth In-Batch Softmax Loss: KuaiFormer uses In-Batch Softmax as the learning objective and applies the LogQ correction method to correct sampling bias.
KuaiFormer's project address
- arXiv technical paper：https://arxiv.org/pdf/2411.10057
KuaiFormer's application scenarios
- Short video recommendation: The Kuaishou App's short video recommendation system analyzes users' historical viewing behavior and interests to recommend new videos they may be interested in.
- Content discovery: In content discovery, it helps users explore and discover new content, increases the diversity and novelty of content, and improves user experience.
- Personalized recommendation: Captures users' multi-dimensional interests and provides more personalized recommendations to meet users' personalized needs.
- Real-time recommendation system: Responds in real time to changes in user behavior and quickly updates recommendation lists, suitable for scenarios that require real-time recommendation updates.
- Large-scale data processing: When processing video libraries at the scale of billions, KuaiFormer's high efficiency and scalability make it an ideal choice for large-scale data processing.