awesome-o1 is a GitHub repository positioned as a bibliography and survey of papers around OpenAI o1. According to the repository description, this is a bibliography of 'papers considered related to OpenAI o1,' and it provides o1 tutorial slides as well as a YouTube video link (https://youtu.be/6PEJ96k1kiw）。
In terms of content organization, the repository lists related literature grouped by topic, with brief explanations. These topics include: methods that are hoped to actually work, such as Self-Consistency (X. Wang et al. 2022, noting that majority voting over LLM outputs yields slight improvements), Scratchpad (Nye et al. 2021) / Chain-of-Thought (Wei et al. 2022, asking whether LLMs can get better by 'talking to themselves'), Tree-of-Thought (Yao et al. 2023, asking whether this can be extended into a tree structure). Regarding why it may be achievable, it lists AlphaGo (Silver et al. 2016, quantifying the value of self-play training and test-time search), AlphaZero (Silver et al. 2017, showing that training based on guided self-trajectories can generalize and scale), Libratus (N. Brown and Sandholm 2017, a poker bot built by scaling search), Scaling Laws for Board Games (Jones 2021, experiments comparing training/test FLOPs in a controlled setting), Noam Lecture (Paul G. Allen School 2024, Noam Brown's talk on the power of search).
Regarding whether reasoning can be viewed as a verifiable game, it lists WebGPT (Nakano et al. 2021, showing that test-time rejection sampling against a reward model is a very strong model), GSM8K (Cobbe et al. 2021, exploring why mathematical reasoning is difficult and introducing ORM models for verification), Process Reward (Uesato et al. 2022, introducing the distinction between process reward and outcome reward models, and using expert iteration RL), Let's Verify (Lightman et al. 2023, demonstrating that PRM can be very effective in rejection sampling), Math-Shepard (P. Wang et al. 2023, experiments using rollouts for automatic value function learning).
Regarding whether verifiers can create better LLMs, it lists Expert Iteration (Anthony, Tian, and Barber 2017,Search, collection, training, methods for RL self-improvement), Self-Training (Yarowsky 1995, a classic unsupervised method: generate, prune, retrain), STaR (Zelikman et al. 2022, formulates LLM improvement as retraining on rationales that lead to correct answers, and argues it as approximate policy gradient), ReST (Gulcehre et al. 2023, models improvement as offline RL: sample trajectories, augment the corpus, retrain), ReST-EM (Singh et al. 2023, formalizes similar methods as EM for RL and applies them to reasoning).
Regarding whether LLMs can learn to plan (the repository notes this part is the most speculative), it lists Stream of Search (Gandhi et al. 2024, training on linearized, non-optimal search trajectories can induce better search), DualFormer (Su et al. 2024, training on optimal reasoning trajectories with masked steps can improve reasoning ability), AlphaZero-like (Feng et al. 2023) / MCTS-DPO (Xie et al. 2024) / Agent Q (Putta et al. 2024, outlines MCTS-style expert iteration for LLM planning), PAVs (Setlur et al. 2024, advocates using advantage functions rather than value-function PRMs to learn search, and shows improved search efficiency), SCoRE (Self-Correct) (Kumar et al. 2024). On whether this brings test-time scaling, it lists Optimal test scaling (Snell et al. 2024), Large Language Monkeys (B. Brown et al. 2024), Inference Scaling (Y. Wu et al. 2024).
The repository also contains a full bibliography, listing the complete citation information for each paper in alphabetical order by author, including authors, year, title, publication venue, and arXiv or conference links. For example, Anthony, Thomas, Zheng Tian, and David Barber 2017's "Thinking Fast and Slow with Deep Learning and Tree Search"; Brown, Bradley et al. 2024's "Large Language Monkeys: Scaling Inference Compute with Repeated Sampling"; Brown, Noam, and Tuomas Sandholm 2017's "Libratus: The Superhuman AI for No-Limit Poker"; and entries for authors such as Chen, Cobbe, Feng, Gandhi, Gulcehre, Jones, Kazemnejad, Kirchner, Kumar, Lightman, Nakano, Nye, Putta, Setlur, Silver, Singh, Snell, Su, Uesato, Wang, Wei, Welleck, Wu, Xie, Yao, Yarowsky, Yoshida, Zelikman, and Zhao.
In terms of usage, this repository is a public GitHub project. Users can navigate through the folders and files in the repository to browse topic groupings and the full bibliography, and can access the provided tutorial slides and video links for learning. Applicable scenarios include: researching o1-related reasoning and reinforcement learning literature, finding key papers by topic, and tracking work in the directions of test-time compute and verifiers. In terms of limitations, the repository itself is a literature catalog and survey; it does not provide directly installable or runnable software or models. Its content is a paper list and brief descriptions; specific features, fees, prices, platforms, effects, or usage steps are not currently provided in the materials.
