Purpose
open-strawberry is an open-source proof-of-concept project whose goal is to build an open-source alternative version of OpenAI o1 by constructing "reasoning traces." Inspired by OpenAI's Strawberry algorithm, the project attempts to use open-source tools and methods to reproduce a similar search-generate system in order to generate and verify training data.
Main capabilities
- Supports multiple model services: Groq, ollama, Anthropic, Gemini, OpenAI, Azure.
- Generates reasoning traces through multi-turn dialogue, guiding the model to gradually approach the answer.
- Supports prompt caching (Anthropic, Google), reducing operating costs; vLLM supports prefix caching.
- Provides a Streamlit app for convenient output monitoring.
- Supports asking the model step by step whether it is confident in the answer; if confident, asks it to place the final answer in the <final_answer> tag and terminates reasoning trace generation.
- Generates multiple reasoning traces and checks errors in historical records through a verification system.
- Filters correct and incorrect reasoning traces based on true labels, applies DPO or NLHF for fine-tuning, and adjusts preference weights according to the number of steps.
- Through progressive learning and repeated fine-tuning, gradually handles more difficult problems.
Usage
- Requires Python >=3.10.
- Fill in the required API keys in the .env file or set environment variables.
- For ollama, you can use the OpenAI service, setting OLLAMA_OPENAI_BASE_URL=http://localhost:11434/v1/ and OLLAMA_OPENAI_MODEL_NAME=ollama:mistral:v0.3 or a model list, then run the CLI or select the model in the UI.
- Open in the browser http://localhost:8501（应自动弹出），选择提示词。
- Demo address:https://huggingface.co/spaces/pseudotensor/open-strawberry
Applicable scenarios
- Research and experiments exploring reasoning trace generation and verification methods.
- Training reasoning ability for problems with verifiable true labels.
- As a starting point for the open-source community to reproduce the o1 reasoning mechanism.
Limitations
- The project is in an early stage, and results and comparisons will be added gradually.
- The demo prompts are aimed at simple models; even sonnet3.5 and gpt-4o cannot find solutions under standard CoT, and only o1-mini or o1-preview can sometimes solve it, while coding agents can solve it easily.
- Harder problems are still difficult to solve; o1-preview has only about a 50% success rate (coding agents about 90%).
- Users need to configure API keys themselves; some model services may require additional fees (the original text does not provide specific prices).
- The project is speculative work, based on OpenAI public information, and is not affiliated with OpenAI.
- Using the Continue button will retain grayed-out old conversations; it is recommended to start cleanly; token counting is only displayed after clicking Continue, and it is best to display it every round.
