SuperAPI AI Directory
Partner with us中文AI newsFavorites
All toolsAI healthcareCareGPT

CareGPT

CareGPT is an open-source medical large language model project, formerly named CareLlama, completed by Wang Rongsheng, Zhou Ruizhe, and Chen Haoming, master's students at the Faculty of Applied Sciences of Macao Polytechnic University, under the supervision of Associate Professors Tan Tao and Wang Yapeng. The project integrates dozens of publicly available medical fine-tuning datasets and open medical large language models, covering the entire process of LLM training, evaluation, and deployment. The project supports training of the entire LLaMA and LLaMA-2 model series, supports LoRA, QLoRA, and subsequent PPO and DPO reinforcement learning training, and supports ChatGPT fine-tuning, ChatGPT-Next-Web, and Gradio deployment. The project has open-sourced guidance material information for more than 60 hospital departments and has developed a tool that supports distilling medical data with GPT-4/ChatGPT models. In the CMB benchmark evaluation, its IvyGPT leads ChatGPT and a number of open-source medical LLMs. Based on its own datasets, it has trained and open-sourced multiple medical LLMs for direct download and experience. The project follows the MIT License, and the related resources are for academic research use only.

Open website

About this tool

CareGPT: Medical LLM, open-source driven, co-creating a healthy future. This project is a medical large language model and also brings together dozens of publicly available medical fine-tuning datasets and openly available medical large language models, including LLM training, evaluation, deployment, and more, to promote the rapid development of medical LLMs. The project was formerly named CareLlama and is a branch of MPU's medical large language model IvyGPT. Its significance lies in exploring work related to medical data, medical LLM training, and deployment.

Purpose This work was completed by Wang Rongsheng, Zhou Ruizhe, and Chen Haoming, master's students at the Faculty of Applied Sciences of Macao Polytechnic University, under the supervision of Associate Professors Tan Tao and Wang Yapeng. The work (IvyGPT) has been cited by the CMB paper, and the related paper has been submitted to NAACL. The project participates in the CMB benchmark evaluation of medical LLMs - IvyGPT, and in testing it leads ChatGPT and a number of open-source medical LLMs. The project has been included in the Machine Heart SOTA repository, received a weekly hot list placement, and participated in the Backdrop Build V2 event; it participated in the hackathon jointly held by Baichuan Intelligence and Amazon Cloud Technology and won third place in the national finals.

Main capabilities

  1. Training support: supports training of the entire LLaMA and LLaMA-2 model series; supports LoRA, QLoRA, including subsequent PPO and DPO reinforcement learning training; adds a ChatGPT fine-tuning implementation, and friends with quota are recommended to conduct fine-tuning experiments on ChatGPT.
  2. Deployment support: supports ChatGPT-Next-Web deployment of fine-tuned models; supports Gradio deployment of fine-tuned models.
  3. Datasets and resources: aggregates abundant open-source medical LLMs, medical data for LLM training, LLM deployment materials, LLM evaluation, and related LLM resource organization. It has open-sourced guidance material information for more than 60 hospital departments; it has developed a tool that supports distilling medical data with GPT-4/ChatGPT models, capable of batch-generating various data used for building knowledge bases and fine-tuning.
  4. Open-source models: based on its own datasets, it has trained and open-sourced multiple medical LLMs on different base LLMs, which can be downloaded and experienced directly.

Datasets The project lists a large number of medical training datasets, including: LLM-Pretrain-FineTune/data_pretrain, MedicalGPT/pretrain, TCM-Ancient-Books (nearly 700 items of ancient Chinese medicine text), epfl-llm/guidelines, icliniq-10k(en), HealthCareMagic-100k(en), ShenNong_TCM_Dataset, ChatMed_Consult_Dataset, Chinese-medical-dialogue-data, LLM-Pretrain-FineTune/data_sft, Medical-Dialogue-System, HuatuoGPT-sft-data-v1, MedicalGPT/finetune, shibing624/medical, Zhongjing/sft, huatuo_encyclopedia_qa, Med-ChatGLM/data, GenMedGPT-5k(en), Alpaca-CoT(general), HuatuoGPT2_sft_instruct, FreedomIntelligence/Medbase_data, openmedlab/Awesome-Medical-Dataset, MedicalGPT/reward, comparison_gpt4_data, etc.

Usage To use a custom dataset, a dataset definition must be provided in the dataset_info.json file in the specified format, where the prompt and response columns should be non-empty strings, the content of the query column will be concatenated with the prompt column as model input, and the history column should be a list, where each element is a string pair representing the user request and the model response respectively. The project provides example formats such as PT example data (.txt format, one unsupervised data item per line), SFT example data 1, SFT example data 2, and RW example data. ChatGPT-Next-Web deployment method: install and open Next, then open settings, modify the API address tohttp://127.0.0.1:8000/（即API接口地址），然后即可使用。LLaMA模型需转为HF格式，LLaMA-2若下载为HF格式则不需要转化。需查看显卡是否为NVLINK连接，NVLINK连接才能有效使用accelerate进行并行加速训练。实验评估使用BLEU和ROUGE_CHINESE；在4/8-bit评估时，推荐使用--per_device_eval_batch_size=1和--max_target_length 128.

Applicable Scenarios The project-related resources are for academic research only and are strictly prohibited for commercial use. When using parts involving third-party code, please strictly follow the corresponding open source licenses.

Limitations and Notes The project clearly states: in CareGPT, no Chinese word segmentation was added or retrained for the tokenization model, but the results are still encouraging; the full LLM training process includes pretraining, supervised fine-tuning, reward modeling, and reinforcement learning, and in most cases supervised fine-tuning can meet one's needs; when computing power is sufficient, it is recommended to train with medical data and general corpus data; do not expect a single medical LLM to meet all needs. A reasonable approach may be a real-time updated knowledge base + a fine-tuned medical LLM (such as ChatLaw). The content generated by the model is affected by factors such as model computation, randomness, and quantization precision loss, and this project cannot guarantee its accuracy. Even if the model output of this project conforms to medical facts, it cannot be used as a basis for actual medical diagnosis. This project assumes no legal responsibility for any content output by the model, nor does it assume responsibility for any losses that may arise from the use of related resources and output results. This repository follows the MIT License.

The project also compiles empirical methodologies regarding pretraining, supervised fine-tuning, data quality, etc., for example: data quality > quantity; large-scale pretraining + small-scale supervised fine-tuning = a super strong LLM model; usually lora and full-tuning have non-negligible performance differences (such as LoRA results in 4-6% lower performance compared to full fine-tuning); for the 7B series of models, please prioritize full-parameter fine-tuning, and for models with 13B parameters and above, methods such as LoRA and QLoRA can be used; QLoRA saves 33% of memory at the cost of a 39% increase in runtime, etc. According to the recommendations of the NeurIPS workshop, as of December 18, 2023, the suggested model choices for fine-tuning are Mistral-7B for English below 10B, Yi-6B for Chinese below 10B, and Qwen-14B and Yi-34B for above 10B.

Related tools

Browse category