Stanford University's CS336 course "Language Modeling from Scratch" is a deep learning course focused on language model development. Through hands-on practice, the course enables students to fully master the language model development process, from data collection and model building to training and evaluation. Drawing on the model of operating systems courses, it requires students to build a complete language model from scratch, including implementing key components such as the tokenizer, Transformer architecture, and optimizer.
The course requires students to have strong Python programming skills, experience in deep learning and system optimization, and foundational knowledge of linear algebra, probability and statistics, and machine learning. The assignments are rich in content, covering everything from basic model training to system optimization, data processing, and model alignment. The course provides recommendations for GPU computing resources to help students efficiently complete the hands-on tasks. Through this course, students will gain a deep understanding of the internal mechanisms of language models and master how to optimize models to meet the challenges of large-scale training and real-world applications.
Course Introduction
-
Course Content: Stanford University's CS336 course aims to give students a comprehensive understanding of the language model development process, including data collection and cleaning, Transformer model construction, model training and evaluation, and more. Inspired by operating systems courses, the course requires students to build a complete language model from scratch.
-
Course Objectives: Through hands-on practice, students will gain a deep understanding of the various components of language models and master how to optimize models to improve efficiency and performance.
Course Requirements
-
Prerequisite Knowledge：
-
Python Proficiency: Most assignments need to be completed using Python, so students need to have strong Python programming skills.
-
Deep Learning and System Optimization Experience: Familiarity with PyTorch and basic system concepts such as the memory hierarchy is required.
-
Mathematical Foundations: A grasp of the basics of linear algebra, probability theory, and statistics is required.
-
Machine Learning Foundations: An understanding of the basic concepts of machine learning and deep learning is required.
-
-
Course Difficulty: This is a 5-credit course with a strong emphasis on practice, and students need to invest a significant amount of time.
Course Assignments
-
Assignment 1: Implement the basic components of a Transformer language model (tokenizer, model architecture, optimizer) and train a minimal language model.
-
Assignment 2: Use advanced tools to profile and optimize the model's performance, implement a Triton version of FlashAttention2, and build distributed training code.
-
Assignment 3: Understand each component of the Transformer and fit model scaling laws through a training API.
-
Assignment 4: Convert raw Common Crawl data into data usable for pretraining, and perform filtering and deduplication to improve model performance.
-
Assignment 5: Apply supervised fine-tuning and reinforcement learning to train a language model to solve math problems, and optionally implement safety alignment methods.
Official Course Website Address
- Official Website Address：https://stanford-cs336.github.io/spring2025/
Other Information
-
GPU Computing Resources: Students can use GPU resources from cloud service providers to complete assignments, and the course provides several recommended cloud service options.
-
Academic Integrity: Students may use AI tools to ask questions about low-level programming problems or high-level conceptual questions, but are prohibited from directly using AI tools to solve problems.
-
Assignment Submission: All assignments are submitted through Gradescope, and late days are allowed, up to a maximum of 3 days.