Qwen-Scope - Alibaba Tongyi's Open-Source Large Model Interpretability Toolkit

What is Qwen-Scope

Qwen-Scope is an open-source large model interpretability toolkit from the Alibaba Tongyi Qianwen team, based on Sparse Autoencoder (SAE) technology, extracting interpretable features from the hidden layers of the Qwen3/Qwen3.5 series models. The model can transform the complex parameter computations inside the model into human-understandable concepts and patterns, and can be used for post-hoc analysis, enabling inference control, data processing, and model optimization through feature-level intervention, becoming a practical interface connecting the model's internals with downstream development.

Main Features of Qwen-Scope

Steering: Without explicit natural language instructions, by enabling or disabling specific SAE features, achieve targeted modification of language, entities, and style, as well as badcase repair.
Data Classification and Synthesis: Based on a small amount of seed data, discover toxicity/safety-related features to achieve classification without an additional trainer; identify inactive features and synthesize supplementary samples in a targeted manner to cover long-tail capabilities.
Model Training Optimization: Locate abnormal activation features such as language mixing and repetitive generation, assisting in optimizing model behavior during supervised fine-tuning (SFT) and reinforcement learning (RL) stages.
Evaluation Redundancy Analysis: Compute feature activation patterns across different evaluation sets, determine the degree of evaluation set redundancy and capability coverage, and guide the selection of high-coverage, low-cost test samples.

Technical Principles of Qwen-Scope

Sparse Autoencoder (SAE): Insert SAE into the residual stream of each Transformer layer of Qwen, and by imposing sparsity constraints, decompose high-dimensional activation vectors into sparse, interpretable feature dictionaries.
Top-k Activation and Reconstruction: Train SAE separately for each layer; the encoder maps activations into an overcomplete latent representation, retaining only the largest k activations (k=50/100) for reconstruction, ensuring highly disentangled features.
Contrastive Feature Identification: Construct positive and negative sample sets, compare their average SAE activation differences, and identify the feature directions most relevant to target attributes (such as toxicity, Chinese, classical writing style).
Feature Intervention Formula: During inference, modify the residual stream via h′ ← h + αd, where d is the SAE feature direction and α controls the intervention strength; a positive value enhances and a negative value suppresses that feature.

How to Use Qwen-Scope

Visit the Experience Platform: Visit the Hugging Face online space.
Select Model Weights: Load the corresponding SAE weights according to the target model (such as Qwen3-8B, Qwen3.5-27B).
Input Prompts to Observe Activations: Input prompt words to view the SAE feature activation heatmap and rankings.
Identify Target Features: Locate abnormal or target feature IDs (such as Chinese feature 6159, classical Chinese feature 36398).
Adjust Intervention Strength: Set the feature intervention coefficient α to positively enhance or negatively suppress specific features.
Verify Control Effect: Compare the model output before and after intervention to confirm successful badcase repair or style transfer.
Integrate Training Process: Connect SAE signals to the SFT/RL loss function to enable targeted model optimization.

Key information and usage requirements of Qwen-Scope

Publisher: Alibaba / Tongyi Qianwen Team
Covered models: Qwen3-1.7B/8B, Qwen3-30B-A3B, Qwen3.5-2B/9B/27B/35B-A3B (7 in total)
Model types: Dense models + Mixture-of-Experts (MoE) architecture
SAE weights: 14 sets, covering all Transformer layers
Training data: 0.5B tokens sampled from each model's pretraining data
Feature dimensions：32K / 64K / 80K / 128K
Number of reconstruction features: 50 or 100
Online experience: Available on both Hugging Face and ModelScope

Core advantages of Qwen-Scope

From "understanding" to "improvement": Goes beyond traditional post-hoc analysis, turning interpretability into a core engine that drives model evolution.
Zero-weight-modification intervention: Directly manipulates feature directions during inference, changing output behavior without fine-tuning or updating model parameters.
Data-efficient and low-dependency: Only a small amount of seed data (about 200 pairs) is needed to discover highly discriminative features, with classification F1 reaching 0.90+, significantly reducing annotation costs.
Targeted and precise optimization: Precisely locates abnormal features for low-frequency bad cases such as language mixing and repetitive generation. During SFT, it can reduce the Chinese mixing rate from 0.81% to 0.22%.
Evaluation cost optimization: Identifies redundancy in evaluation sets through feature coverage analysis, helping select high-coverage samples and reducing evaluation overhead.

Qwen-Scope project addresses

HuggingFace model repository：https://huggingface.co/collections/Qwen/qwen-scope
Technical paper：https://qianwen-res.oss-accelerate.aliyuncs.com/qwen-scope/Qwen_Scope.pdf

Comparison of Qwen-Scope with similar competitors

Comparison dimension	Qwen-Scope	Gemma Scope
Publisher	Alibaba / Tongyi Qianwen	Google DeepMind
Covered models	Qwen3 / Qwen3.5 series (7 models)	Gemma 2 / 3 series
Architecture support	Dense models + MoE	Dense models
SAE architecture	Top-k SAE	JumpReLU SAE
Open-source scale	14 sets of SAE weights	400+ SAEs, 30 million+ features
Core applications	Reasoning control, evaluation analysis, data synthesis, training optimization	Mechanistic interpretability, safety analysis, circuit tracing
Data synthesis	Feature-driven synthesis, improving data efficiency by about 15x	Mainly relies on traditional synthesis approaches
Evaluation analysis	Supports benchmark redundancy and coverage analysis	Focuses on feature visualization and interactive exploration
Chinese support	Native support, including distinctive stylistic features such as classical Chinese	Mainly oriented toward English scenarios
Interactive platform	Hugging Face / ModelScope	Neuronpedia

Application scenarios of Qwen-Scope

Reasoning control and repair: Fixing language-mixing issues such as unexpected mixing of Chinese under English prompts; enabling style transfer such as converting modern Chinese into classical Chinese.
Safety data governance: Feature-based discovery for multilingual toxic content classification; targeted synthesis of safety training data, where 4k synthetic data can approach the safety alignment effect of 120k real data.
Model training assistance: In the SFT stage, SAE auxiliary loss helps suppress anomalous activations; in the RL stage, manipulating repetition-related features increases the sampling frequency of anomalous responses and accelerates convergence.
Evaluation set optimization: Analyzing the feature overlap matrix among benchmarks such as GSM8K, MATH, and MMLU-Pro, removing redundant evaluation sets, and improving evaluation efficiency.
Model interpretability research: Providing an open-source SAE foundation for academia and industry, supporting mechanistic interpretability, circuit tracing, and root cause analysis of hallucinations and bias.