What is Circuit Tracer
Circuit Tracer is an open-source tool launched by Anthropic, used to study the internal working mechanisms of large language models. Circuit Tracer reveals the internal steps a model goes through when generating specific outputs based on attribution graphs. Attribution graphs can help researchers track the model's decision-making process, visualize the relationships between features, and test different hypotheses. Circuit Tracer supports a variety of popular open-source models, such as Gemma and Llama, and provides an interactive visualization interface based on Neuronpedia, making it convenient for users to explore and analyze model behavior.
Main Features of Circuit Tracer
- Generate Attribution Graphs: Reveal the model's decision path and show the influence relationships between features and nodes.
- Visualization and Interaction: Based on an interactive interface, intuitively view and operate attribution graphs, making them easy to understand and share.
- Model Intervention: Modify feature values to observe output changes and verify model behavior.
- Support for Multiple Models: Compatible with mainstream models such as Gemma and Llama, facilitating comparative research.
Technical Principles of Circuit Tracer
- Transcoders: Use pre-trained transcoders to generate attribution graphs. A transcoder is a neural network component that can convert the model's internal features into a form that is easier to understand and interpret. Based on transcoders, Circuit Tracer can capture the relationships between the model's internal features and nodes.
- Direct Effect Computation: Circuit Tracer calculates the direct influence of each non-zero transcoder feature, transcoder error node, and input token on other non-zero transcoder features and output logits.
- Graph Pruning: Prune the generated graph. The pruning process removes nodes and edges with smaller influence, retaining only the parts that have a significant impact on the model's decisions. Pruning parameters (such as node threshold and edge threshold) are customized by the user to control the complexity and clarity of the graph.
- Interactive Visualization Interface: Provide a web-based interactive visualization interface, allowing users to directly view and operate attribution graphs in the browser. The interface supports labeling, grouping, and annotating nodes, enabling users to more intuitively understand and analyze the model's internal mechanisms.
Project Address of Circuit Tracer
- Project Official Website：https://www.anthropic.com/research/open-source-circuit-tracing
- GitHub Repository：https://github.com/safety-research/circuit-tracer
Application Scenarios of Circuit Tracer
- Model Behavior Research: Analyze the model's decision-making process based on attribution graphs, and understand the internal logic when generating specific outputs.
- Multilingual Model Analysis: Study the internal representations of multilingual models (such as Llama), and explore cross-language processing mechanisms.
- Multi-step Reasoning Research: Analyze the model's behavior in multi-step reasoning tasks, and reveal the process and logic of step-by-step reasoning.
- Model Optimization and Improvement: Test different hypotheses based on intervention functions, verify whether certain behaviors of the model meet expectations, and optimize the model structure.
- Education and Sharing: Based on the interactive visualization interface, intuitively display complex model decision-making processes to others, facilitating teaching and communication.