Purpose
document.ai is a general-purpose local knowledge base solution based on a vector database and GPT-3.5, with the goal of providing question-answering capabilities for local knowledge bases. The project author uses the MSD case as an example to explore feasibility in vertical domains, and explains that the solution can be transferred to any familiar domain without being limited to the medical field.
Main capabilities
- Convert the local answer dataset into vectors and store them in a vector database.
- When a user enters a query question, convert the question into a vector and query the topK similar answers from the vector database.
- Without GPT, directly returning the relevant answers is enough to complete the question-answering query.
- After introducing GPT, the overall structure of the answer content can be optimized; in a pure search scenario, the optimization is of little significance, but in chat scenarios such as customer service, when citing content from the relevant domain in a reply, it will appear less abrupt.
Usage
- The repository contains a code directory, which stores the relevant example code for this MSD example; the docs directory mainly contains some of the author's thoughts and summaries in this direction. The materials do not provide specific installation commands, dependency configuration, or running steps, so they are not provided for now.
Applicable scenarios
- Local knowledge base question answering.
- Citing content from the relevant domain in chat scenarios such as customer service for replies.
- Feasibility exploration of vertical-domain knowledge bases, such as the MSD case demonstrated by the author.
Limitations and notes
- Please refer to OpenAI's usage policies:https://openai.com/policies/usage-policies
- When directly matching questions and answers, the ambiguity of the question may cause matching to irrelevant answers.
- When there is already a large amount of question-answer mapping data, you can first search the question set, and then return the answer from the question set matched by the current question based on the existing mapping, to improve question accuracy to a certain extent.
- When the answer contains a large amount of non-answer content, you can extract the answer topic and then combine it to generate vector data, improving similarity to a certain extent; topic algorithms include LDA, LSA, etc.
- OpenAI's Embedding model data is trained more on general data. If the question-answering domain is too specialized, inaccurate query data may occur. The solution is to train the Embedding model yourself. The author recommends the project text2vec, and mentions that shibing624 has provided the model shibing624/text2vec-base-chinese based on CoSENT + MacBERT + STS-B. The author also trained two models based on CoSENT + LERT + STS-B: text2vec-large-chinese with a hidden layer size of 1024, and text2vec-base-chinese with 768. For the Demo, the author also trained two medical question-answering related models based on the cMedQQ dataset: text2vec-cmedqq-lert-large and text2vec-cmedqq-lert-base.
- The author's tests found that after Fine-tune of the GPT model using the question-answering dataset, the accuracy for this type of question improved greatly; it can be understood that after GPT is trained on a large amount of specialized domain data, when asking questions it feels more like talking with an expert in that domain, and by adjusting the temperature parameter in the interface to a smaller value, more deterministic results can be obtained.
- However, the cost of Fine-tune training and use is still too high, and new data appears every day, so it is impossible to perform Fine-tune at high frequency. The author's idea is to Fine-tune the data every long period, and then use similarity queries with an external vector database to supplement the data accuracy issues of the Fine-tune model itself.
Other
The repository page contains a Buy me a coffee link. The materials do not provide charging, pricing, platform support scope, or specific performance data, so they are not provided for now.
