What is LIMO
LIMO (Less Is More for Reasoning) is an efficient reasoning method proposed by a research team at Shanghai Jiao Tong University. It activates the complex reasoning capabilities of large language models (LLMs) through a very small number of high-quality training samples. The core hypothesis is that in models that already possess a rich knowledge foundation during the pretraining stage, complex reasoning capabilities can be effectively activated through a small number of carefully designed training samples. LIMO uses only 817 training samples and achieves significant performance improvements on multiple mathematical reasoning benchmarks.
Main Features of LIMO
- Efficient reasoning activation: LIMO uses only 817 carefully curated training samples and achieves significant performance improvements on multiple mathematical reasoning benchmarks. On the AIME benchmark, LIMO's accuracy reached 57.1%, and on the MATH benchmark it reached 94.8%, improving by 50.6 percentage points and 35.6 percentage points respectively compared with previous models.
- Excellent generalization ability: LIMO demonstrated outstanding out-of-distribution generalization ability on 10 different benchmarks, with an average accuracy of 72.8%. Compared with models trained on 100 times more data, LIMO achieved an absolute performance improvement of 40.5%.
- Validating the "Less Is More" hypothesis: LIMO's research proposes the "Less Is More for Reasoning Hypothesis" (LIMO Hypothesis), namely that in base models, when domain knowledge has already been comprehensively encoded during the pretraining stage, complex reasoning capabilities can emerge through minimal but precisely orchestrated demonstrations of cognitive processes.
- Data efficiency: LIMO used only 1% of the training data required by previous methods, challenging the traditional view that complex reasoning tasks require large amounts of training data.
Technical Principles of LIMO
- Core hypothesis: LIMO's core hypothesis is the "Less Is More for Reasoning Hypothesis" (LIMO Hypothesis): in base models, when domain knowledge has already been comprehensively encoded during pretraining, it can emerge through minimal but precisely organized examples of cognitive processes. The hypothesis holds that the threshold for eliciting complex reasoning is not inherently limited by the complexity of the target reasoning task, but is determined by two key factors:
- The completeness of the knowledge foundation encoded by the model during pretraining: Modern foundation models have integrated an unprecedented amount of mathematical content during pretraining. For example, Llama 2 and Llama 3 reached 1.8T and 3.7T tokens of training data for mathematical reasoning, respectively.
- The effectiveness of the post-training examples: These examples serve as "cognitive templates" that demonstrate how the model can effectively use its existing knowledge foundation to solve complex reasoning tasks.
- Problem definition: LIMO focuses on reasoning tasks with verifiable answers, with the goal of generating an answer and a reasoning chain.
- Problem selection: Select challenging problems that can promote complex reasoning chains, diverse thinking processes, and knowledge integration.
- Reasoning chain construction: Collect official solutions, supplement them with solutions written by human experts and AI experts, and generate diverse solution methods based on state-of-the-art reasoning models.
- Training protocol: Use supervised fine-tuning to fine-tune large language models on the LIMO dataset, adopting full-parameter fine-tuning, using DeepSpeed ZeRO-3 optimization and FlashAttention-2.
LIMO Project Address
- Github repository：https://github.com/GAIR-NLP/LIMO
- HuggingFace model library：https://huggingface.co/GAIR/LIMO
- arXiv technical paper：https://arxiv.org/pdf/2502.03387
Application Scenarios of LIMO
- Education: LIMO enhances students' logical thinking and complex problem-solving abilities. Through a small number of high-quality reasoning training samples, LIMO can help students better understand and master complex mathematical and logical reasoning processes.
- Scientific Research: In scientific research, LIMO can be used for modeling and analysis of complex problems. In the fields of mathematics and physics, LIMO can help researchers quickly validate and optimize complex theoretical models.
- Industrial Applications: LIMO can be used to optimize production processes and quality control. LIMO can help enterprises quickly identify and solve complex problems in production, improving production efficiency and product quality.
- Medical Field: LIMO can be used to assist in diagnosis and optimization of treatment plans. Through a small number of high-quality medical cases, LIMO can help doctors quickly identify complex disease patterns and provide more accurate diagnostic recommendations.