DataChef - An open-source AI data recipe generation model by Shanghai AI Lab and Fudan University

What is DataChef

DataChef is an open-source AI data recipe generation model jointly developed by Shanghai Artificial Intelligence Laboratory and Fudan University. The model uses reinforcement learning to automatically generate complete data processing pipelines for large model adaptation tasks, including executable code for steps such as data selection, cleaning, synthesis, and mixing. DataChef only requires the input of the target task and available data sources, and the AI can automatically "alchemize" data. The 32B parameter version approaches the performance ofGemini-3-Proin tests across 6 domains including mathematics, code, and finance, and surpasses industrial-grade expert recipes on some tasks, marking a new paradigm in which data engineering moves from manual experience to automation.

DataChef's main features

• Automatic recipe generation: DataChef can automatically generate a complete data recipe based on the target task and available data sources.
• Code output: The model outputs executable Python data processing code, building an automated pipeline from raw data to training set.
• Multi-step processing: Supports various processing operations such as data selection, cleaning, enhancement, mixing, and deduplication.
• Train and use immediately: The generated data recipe can be directly used for fine-tuning large models to adapt to specific domains.
• Quality verification: DataChef has a built-in data verifier that can evaluate data quality and guide the optimization process.

DataChef's technical principles

• Task modeling: DataChef models data recipe generation as an end-to-end reinforcement learning task. The policy model receives task instructions and generates a data recipe containing a natural language plan and Python code.
• Proxy reward mechanism: To solve the problem of high training feedback costs, the research team designed Data Verifier as a proxy reward mechanism. The verifier classifies samples into five quality levels and scores based on sampled subsets, enabling low-cost real-time prediction of data quality.
• Training optimization: The training process adopts cold-start supervised fine-tuning initialization, combined with the GRPO algorithm for online reinforcement learning optimization. By decoupling reasoning and code generation, it improves policy stability, enabling the model to efficiently explore optimal data solutions in a vast code combination space.

DataChef's key information and usage requirements

• R&D team: Shanghai Artificial Intelligence Laboratory (InternLM team) in collaboration with Fudan University
• Model scale: Open-source 32B parameter version (DataChef-32B)
• Performance level: Approaches the top closed-source model Gemini-3-Pro on 6 held-out test tasks, surpassing industrial-grade expert recipes on some tasks
• Core innovation: For the first time, end-to-end data recipe generation is modeled as a global decision problem, achieving an automatic optimization loop through online reinforcement learning
• Data foundation: Covers 19 domains, 31 evaluation sets, and 257 source datasets
• Environment configuration: Python 3.12, install dependencies via pip
• Hardware resources: Running the 32B model requires sufficient GPU memory or API access capability
• Model endpoints: API endpoints for Planner and Coder models need to be configured (support OpenAI-compatible interfaces)

DataChef's core advantages

• End-to-end automation: Breaking through traditional local heuristic rules, elevating data recipe generation to an end-to-end task, where AI directly outputs a complete executable data processing pipeline without the need for manual orchestration of operation steps.
• Reinforcement Learning Driven: Through online reinforcement learning, an automatic optimization loop is formed, enabling the model to self-evolve in a vast code combination space, continuously explore better data solutions, and bid farewell to manual repeated trial and error.
• Low-Cost Validation Mechanism: The model innovatively proposes the Data Verifier proxy reward signal, which can predict data quality in real time without full model training, greatly reducing the effectiveness validation cost of traditional data engineering.
• Leapfrog Performance: The 32B-parameter open-source model approaches Gemini-3-Pro in multiple tests and surpasses industrial-grade expert recipes on some complex tasks, proving that AI-generated solutions are superior to manually designed experience-based ones.
• Open Infrastructure: A large-scale task pool covering 19 domains has been built and fully open-sourced, providing systematic support for frontier research such as automated data engineering and self-evolving AI.

How to Use DataChef

• Environment Installation: Create a Python 3.12 virtual environment, and complete the DataChef installation via thepip install -e .command.
• Configuration File: Copy the example configuration file and name itdatachef.config.json, then fill in the model endpoint address, API key, and the names of the Planner and Coder models.
• Prepare Input: Prepare the task file in JSONL format, with each line containing the task ID, task description and evaluation benchmark, and a list of candidate data sources.
• Run Generation: Execute thedatachef-eval --config testcommand to start the process; the system automatically generates the data recipe plan and Python code, and performs validation.
• Get Output: In thedata/code/anddata/data-verifier/directories, obtain the generated data processing code, execution report, and data quality validation scores.
• Adjust Parameters: Via--timeoutset the code execution timeout,--max_workerscontrol the number of parallel workers,--parse_reasoningoutput the model reasoning process.

DataChef Project Addresses

• GitHub Repository：https://github.com/yichengchen24/DataChe
• HuggingFace Model Hub：https://huggingface.co/yichengchen24/DataChef-32B
• arXiv Technical Paper：https://arxiv.org/pdf/2602.11089
• Online Demo：https://huggingface.co/spaces/yichengchen24/DataChef

Comparison of DataChef with Similar Competitors

Application scenarios of DataChef

• Domain large model training: Automatically generate adapted data recipes for vertical domains such as mathematics, code, finance, healthcare, and meteorology, to quickly build domain-specific models.
• Data engineering automation: Replace the traditional manual data filtering and mixing process that relies on expert experience, and realize full-process automation from raw data to training set.
• Model post-training optimization: Generate high-quality fine-tuning data for existing base models to improve model performance on specific tasks.
• Data augmentation for low-resource scenarios: Automatically synthesize training samples in data-scarce domains, and expand the scale of effective training data through data augmentation techniques.
• AI research automation: As infrastructure for Automated AI Research, support the data self-improvement loop of self-evolving AI systems.