autoresearch - Karpathy's open-source AI autonomous research experimentation framework

What is autoresearch

autoresearch is an AI autonomous research experimentation framework open-sourced by Andrej Karpathy. The framework lets an AI Agent automatically run nanochat training experiments on a single GPU, autonomously tuning parameters, executing training, analyzing validation metrics, and deciding the next optimization direction, fully automating the traditional manual loop of tune parameters - run experiments - read logs - tune parameters again. autoresearch has an extremely low barrier to entry and can run on a single card, making AI no longer merely an object of study, but an executor that independently conducts research.

Main features of autoresearch

• Autonomous code iteration: The AI Agent directly edits the train.py file, modifying the model architecture, optimizer, hyperparameters, and other configurations across the entire training pipeline.
• Fixed time budget experiments: Each training run is strictly limited to 5 minutes of wall-clock time, ensuring experiment results are comparable across platforms.
• Automatic improvement screening: Using val_bpb(validation bits per byte) as the unified metric, automatically retaining effective modifications that lower the metric and discarding ineffective attempts.
• Human-controllable research organization: Through program.md defining agent behavior and research strategy, humans iteratively optimize the "research organization code" rather than directly modifying code.
• Single-file focused design: The Agent only modifies train.py a single file, keeping the experiment scope controllable and the diff reviewable.

Technical principles of autoresearch

• Simplified nanochat training: A lightweight GPT model implemented on a single GPU, integrating the Muon optimizer with AdamW, supporting a complete training loop.
• BPE tokenization and data pipeline：prepare.py handles one-time data preparation and BPE tokenizer training, providing standardized data loading and evaluation tools.
• Unified evaluation metric: Adopts the vocab-size-independent val_bpb metric, making experiment results fairly comparable before and after architecture changes.
• Agent-environment closed loop：program.md serves as the agent's "skill file": the agent reads instructions → modifies code → runs training → evaluates metrics → decides to keep/discard → iterates in a loop.
• Time standardization mechanism: Regardless of changes in model size or batch size, training always runs for a fixed duration, eliminating the impact of hardware differences on experiment comparability.

How to use autoresearch

• Install uv: Execute curl -LsSf https://astral.sh/uv/install.sh | sh to install the uv project manager.
• Install dependencies: After entering the repository directory, run uv sync to complete project dependency installation.
• Data preparation: Execute uv run prepare.py to download training data and train the BPE tokenizer (one-time, about 2 minutes).
• Verify environment: Manually run uv run train.py to confirm a single training pipeline executes normally (about 5 minutes).
• Launch the agent: Load an AI agent such as Claude/Codex in the IDE, pointing it to the program.md file.
• Start research: Enter a prompt such as “Hi have a look at program.md and let’s kick off a new experiment” to start automatic iteration.
• View results: Check the experiment logs and train.py modification history, obtain the optimized model and improvement path.

Core advantages of autoresearch

• Minimalist design: only three core files (prepare.py、train.py、program.md), zero external dependencies, runnable on a single GPU.
• Fair comparison: fixed 5-minute time budget and vocab-independent metrics ensure that any architectural modification can be evaluated under the same benchmark.
• Human-machine collaboration: humans use program.md to set research strategy, and the Agent is responsible for executing code experiments, realizing the division of labor where humans set the direction and AI conducts experiments.
• High throughput: about 12 experiments per hour, nearly 100 automatic iterations can be completed overnight, far exceeding the efficiency of manual experiments by humans.
• Auditability: the single-file modification mode makes the diff of each experiment clear and traceable, making it easy to understand the agent's improvement path.

autoresearch project address

• GitHub repository：https://github.com/karpathy/autoresearch

Comparison of autoresearch with similar competitors

Application scenarios of autoresearch

• Automatic hyperparameter search for LLM training: automatically explores combinations of learning rate, batch size, model depth, etc., to find the optimal configuration.
• Architecture innovation validation: quickly validates the actual effects of new attention mechanisms, positional encodings, or optimizer variants.
• Low-cost model tuning: automatically optimizes small-model performance in a single-GPU environment, suitable for individual researchers or teams with limited resources.
• Exploration of AI research methodology: serves as a proof-of-concept platform for autonomous AI research agents, exploring the feasibility boundaries of "AI scientists."