What is LongWriter
LongWriter is a long-text generation model launched jointly by Tsinghua University andZhipu AI. It can generate coherent text of over 10,000 words, and the project has been open-sourced. By analyzing the output limitations of existing large language models, the "LongWriter-6k" dataset was created, successfully expanding the output capability of AI models. LongWriter also adopts Direct Preference Optimization (DPO) technology to improve the model's output quality and its ability to follow length constraints in instructions.
LongWriter's main features
- Ultra-long text generation: LongWriter can generate coherent text of over 10,000 words, breaking through the previous limits of AI models on text length.
- Dataset construction: By creating the "LongWriter-6k" dataset, which contains writing samples ranging from 2,000 to 32,000 words, it provides rich long-text data for model training.
- AgentWrite method: A method that uses existing LLMs to automatically construct ultra-long output SFT data, adopting a divide-and-conquer strategy and effectively improving the model's long-text generation capability.
- Direct Preference Optimization (DPO): Further optimizes the model through DPO technology to improve output quality and the ability to follow length constraints.
LongWriter's technical principles
- Long-context processing capability: LongWriter is based on long-context large language models (LLMs) with significantly increased memory capacity, and the model can process historical records of over 100,000 tokens.
- Output length limit analysis: By analyzing the maximum output length of existing models under different queries, LongWriter identified that the model's output length limit mainly originates from the characteristics of the supervised fine-tuning (SFT) dataset.
- Supervised fine-tuning (SFT): LongWriter uses the "LongWriter-6k" dataset in the SFT stage. In this way, the model learns how to generate longer text.
LongWriter's project addresses
- GitHub repository：https://github.com/THUDM/LongWriter
- HuggingFace model library：https://huggingface.co/THUDM/LongWriter-glm4-9b
- arXiv technical paper：https://arxiv.org/pdf/2408.07055
How to use LongWriter
- Environment configuration: Ensure there are sufficient computing resources to run the LongWriter model, including a high-performance GPU and sufficient memory.
- Obtain the model: Visit GitHub to obtain LongWriter's open-source code and model.
- Install dependencies: Install the required dependency libraries and tools according to the project documentation, including deep learning frameworks, data processing libraries, etc.
- Data preparation: Prepare long-text data suitable for LongWriter processing. Preprocess the data to meet the model's input requirements.
- Model loading: Load the pre-trained LongWriter model, or further fine-tune it based on your own data.
- Write prompts: According to the text content to be generated, write clear prompts or instructions. The prompts will guide the model to generate specific text.
- Generate text: Use the interface or API provided by the model, input the prompt, and start the text generation process.
LongWriter's application scenarios
- Academic research: LongWriter can assist scholars and researchers in writing long academic papers, research reports, or literature reviews.
- Content creation: Writers and content creators can use LongWriter to generate drafts of novels, scripts, or other creative writing.
- Publishing Industry: Publishers can use LongWriter to assist with editing and proofreading, or to automatically generate book content.
- Education Sector: Educators can use LongWriter to generate teaching materials, course content, or study guides.
- News Media: News organizations can use LongWriter to quickly generate news reports, in-depth analysis articles, or feature stories.