diagnosis_zero is an open-source project that reproduces the DeepSeek R1 Zero training method in the field of medical rare disease diagnosis, published on GitHub by wizardlancet. The core goal of the project is to explore applying R1 Zero-style training to disease diagnosis tasks, especially rare disease diagnosis, and to observe behavioral changes in models of different scales during training.
In terms of use, the project is aimed at researchers and developers interested in reinforcement learning training of reasoning models and medical diagnostic AI, and provides a reference reproduction pipeline and experimental findings. The project is implemented based on the Verl framework, using the RareArena dataset (RDC setting) and the GRPO algorithm for training. During training, given a patient's medical record, chain-of-thought prompting guides the model to reason first and then give a diagnosis. The reward function uses a sequence matching algorithm to compare model predictions with ground-truth labels, in order to alleviate special character and formatting issues, but the project also points out that this method cannot fully solve the challenge posed by synonyms of disease names.
Main capabilities and exploration contents include: reproducing DeepSeek R1 Zero training in the field of medical diagnosis; conducting multi-scale exploration on three model sizes, 1.5B, 3B, and 7B; and implementing a custom reward function for diagnosis tasks. The project records training progress and publishes several findings. In terms of model performance improvement, Top-1 accuracy increases with model scale: the 1.5B model improves from 4.2% to 8.1%, the 3B model from 9.8% to 12.1%, and the 7B model from 15.9% to 21.6%. After training, the models analyze medical history more comprehensively, have enhanced recognition of systemic diseases, better integrate disease features, and significantly improve diagnostic accuracy.
In terms of model output behavior and format adherence, after 1.5K training steps, many test cases for the 1.5B model output only the <answer> tag without a <think> part, which may mean that the model has abandoned the reasoning chain of thought. The 1.5B model can maintain the output format fairly well after RL training, while the 7B model, although having a higher overall diagnosis success rate, has an increased failure rate in adhering to the output format. For tasks such as MedQA, MMLU (mmlu-college-medicine), and MedMCQA, no significant changes were observed after training.
Regarding the mixed-language phenomenon, after training, the 3B and 7B models produced outputs mixing multiple languages, and the 7B model showed a clear increase in mixed-language outputs at 1K steps. Mixed-language percentage statisticsDisplay: 1.5B is 0.4% at 0 steps, 0.0% at 500 steps, 0.2% at 1000 steps, 0.2% at 1500 steps; 3B is 0.0% at 0 steps, 0.0% at 500 steps, 0.3% at 1000 steps, 0.1% at 1500 steps; 7B is 0.0% at 0 steps, 0.1% at 500 steps, 10.6% at 1000 steps, 0.2% at 1500 steps. When mixed-language output occurs, the overall matching probability for all models is about 14.16%, while it rises to about 28.93% in the mixed-language case; specifically for the 7B model, the accuracy under mixed-language conditions is 31.19%, while the overall accuracy is 18.90%. The project points out that this phenomenon is worth further study and gives two examples of the mixed-language problem in the 7B model at 1K steps, manifested as a second language appended after the <answer> tag.
The project summarizes some conclusions: larger models achieve higher Top-1 diagnostic accuracy gains in this attempt. There is a problem between format and accuracy; the 1.5B model tends to maintain the output format, and sometimes even discards the <think> tag and the chain of thought; although the 7B model has a higher format-following failure rate, its diagnostic performance is better, which may stem from a conflict between the format reward and the accuracy reward. The mixed-language effect is triggered by the 7B model at 1K steps, appending a second language after the <answer> tag. Although these cases seem to bring a higher diagnostic matching probability, their root cause needs further study.
Regarding usage, the project recommends installing the latest version of Verl from the official repository, because Verl is updated frequently. Data preparation requires obtaining data from the original RareArena dataset and preprocessing it, and the data template is aimed at instruct models. The core implementation involves the above training process and custom reward function. The project also provides a Wandb report link and acknowledges related resources.
The applicable scenarios are mainly reinforcement learning training research for medical rare disease diagnosis, reproduction and exploration of R1 Zero-style training, and comparative experiments on the impact of model scale on diagnostic tasks.
Regarding limitations, the project clearly points out that the experimental setup is preliminary, and the reward function and parameters have not been fully optimized; the Qwen series models do not perform best on the Rare Arena benchmark, and the results may not be easy to generalize to other models; training stability is still a problem, and models usually collapse after long training (although the KL loss issue has been fixed from the official Verl repository); the accuracy improvement in disease diagnosis tasks is smaller than in reasoning tasks, which may be because diagnosis requiresmore comprehensive and integrated information processing and stronger attention to key symptoms, so the generality and cost-effectiveness of this method need further study. The project also reminds that the results should be interpreted cautiously, and some results (the conflict between format reward and accuracy reward) may support the view that "there may be no Aha Moment in R1-Zero-like training."