In the data-driven era, charts have become a core medium for information delivery. Enabling machines to truly "read" charts—not only recognizing graphical elements, but also understanding the logic of the data, discerning trends and patterns, and answering complex questions—has always been a major challenge in the field of artificial intelligence. Traditional methods rely on OCR (Optical Character Recognition) and rule engines, with cumbersome processes and poor robustness. Since 2023, with the rise of multimodal large models such as GPT-4V and Gemini, chart understanding technology has ushered in a revolutionary breakthrough.
As of 2025,Vision-Language Models(VLMs) can already achieve end-to-end chart parsing, from pixel-level visual perception to semantic-level logical reasoning, demonstrating comprehension abilities approaching those of human experts. This article will deeply deconstruct the engineering practices and scientific principles behind this technological miracle.
Core Technology Evolution: From Multi-Stage Pipelines to End-to-End Intelligence
Limitations of Traditional Methods
Early chart understanding systems used modular pipelines: first extracting text via OCR, then using computer vision to detect graphical elements (such as the bars of a bar chart or the lines of a line chart), and finally matching logical relationships with a rule engine. This architecture has three fatal flaws:
- Error Accumulation: A single OCR mistake causes the entire subsequent reasoning chain to collapse
- Poor Generalization: New chart styles require rewriting rules, resulting in extremely high maintenance costs
- Inability to Understand Deep Semantics: It can extract numbers, but cannot understand the business meaning behind "significant year-over-year growth"
The Paradigm Revolution of End-to-End Large Models
The mainstream approaches of 2024-2025 adopta unified neural network architecture, deeply integrating visual encoders with language models to achieve end-to-end understanding of "image in, answer out." Key technological breakthroughs include:
- Vision-Language Alignment: Through large-scale image-text pair pretraining, the model automatically learns the correspondence between chart elements and data concepts
- Instruction-Following Capability: Users ask questions in natural language, and the model directly generates answers or code without the need for intermediate structured data
- Contextual Reasoning: Combining multimodal information such as chart titles, axis labels, and legends to perform causal inference and trend prediction
Mainstream Chart Understanding Large Models
First Tier: Closed-Source Commercial Giants
GPT-4V / GPT-4o (OpenAI)
As a benchmark for multimodal large models, GPT-4V adopts a Mixture of Experts (MoE) architecture , but its technical details have not been fully disclosed. Based on analysis of search results, its core advantages lie in:
- Technical Features：
- Visual Encoder: A CLIP-based variant that extracts 512-dimensional visual feature vectors and supports input resolutions up to 8192×8192 pixels
- Language Model: GPT-4 base, with an estimated parameter count of around 1.8 trillion (MoE activated parameters approximately 220 billion)
- Training Method: Two-stage training—first aligning visual and textual representations on billions of image-text pairs, then optimizing chart reasoning ability through Reinforcement Learning from Human Feedback (RLHF)
- Key Technologies：
- Chain-of-Thought : Automatically decomposes complex problems into multi-step reasoning of "read data → calculate → verify"
- Code Generation Capability: Can output Python code to reproduce charts and verify comprehension accuracy
- Cross-chart analysis: Supports advanced cognitive tasks such as multi-chart comparison and trend correlation
Performance: On the "cognitive tasks" subset of the ChartX benchmark, GPT-4V achieves an accuracy of78.3%, surpassing most open-source models, but slightly underperforming specialized models on structure extraction tasks.
Gemini 1.5 Pro / Gemini 2.5 Pro (Google)
The Gemini series adopts anative multimodal architecture, not late-stage stitching, enabling joint modeling of vision and language from the ground up:
- Technical Features：
- Vision Encoder: A custom ViT based on the Pathways architecture, supporting up to 1 hour of video or a 10 million token context; for chart understanding it adopts adynamic resolutionstrategy, allocating more computational resources to high-information-density regions
- Language Model: Gemini Ultra base, with a total parameter count of approximately 540 billion
- Training Method: OnGemini ChartCorpusspecialized training was conducted; this dataset contains 5 million synthetic charts and real business charts, covering 18 chart types
- Key Technologies：
- Spatially Aware Attention: Precisely captures the spatial relationships of chart elements through 2D positional encoding
- Program-of-Thought : Transforms chart questions into executable programs, verifies answers through a code executor, improving accuracy by 12%
- Incremental Understanding: Supports user follow-up questions, continuously deepening chart analysis based on historical dialogue
Performance: On the PlotQA dataset, Gemini 1.5 Pro achieves an accuracy of73.1%, and on ChartQA reaches87.2%, significantly outperforming GPT-4V on some structured tasks.
Second Tier: Open-Source Specialized Models
ChartVLM (Peking University & Microsoft Research)
An open-source SOTA model designed specifically for complex chart reasoning, released in 2025:
- Model Structure：
- Vision Encoder：InternViT-6B(Vision Transformer, 600 million parameters), supports 1024×1024 high-resolution input, adoptssliding window attentionto handle large-size charts
- Cross-Modal Connector：two-layer MLP projection(drawing on the LLaVA-1.5 design), mapping visual features into the language model embedding space
- Language Model：Qwen2.5-72B-Instruct, fine-tuned via LoRA to inject chart knowledge while preserving general language capabilities
- Instruction Adapter: A lightweight Transformer encoder that encodes user questions into query vectors, dynamically guiding visual attention
- Training Method：
- Two-Stage Instruction Fine-Tuning：
- Chart-to-Table Pretraining: Masked reconstruction on 1 million chart-table pairs, forcing the model to learn the mapping from visual elements to structured data
- Multi-Task Instruction Fine-Tuning: OnChartInstruct-500Ktraining is conducted on the dataset, covering 8 categories of tasks (QA, summarization, chart generation, data extraction, etc.)
- Data Augmentation: ThroughChartAugthe toolchain randomly changes chart colors, fonts, and layouts to improve robustness
- Two-Stage Instruction Fine-Tuning：
- Key Technologies：
- Structure-aware visual token merging: Merges adjacent same-color pixels into super tokens, reducing computation by 50% and lowering memory usage from 24GB to 12GB
- Dual-path training strategy: Simultaneously optimizes pixel-level reconstruction loss and semantic-level question-answering loss, balancing perception and cognition capabilities
Performance: On theChartXbenchmark's overall score, ChartVLM ranks first among open-source models with82.4 points, achieving accuracy of91.2%on "perception tasks" (such as legend recognition and data reading), but still lags behind GPT-4V by about 6 percentage points on "cognition tasks" (such as trend attribution and anomaly detection).
ChartScope (Tsinghua University)
A multimodal large model emphasizing "depth and breadth of understanding":
- Model Architecture：
- Visual encoder：SigLIP-SO-400M, with a resolution of 896×896, adopting adual-tower architectureto separately encode chart images and title text
- Cross-modal fusion：Q-Former variant(12-layer Transformer), extracting question-relevant features from the visual stream through learnable query tokens (32), enabling fine-grained cross-modal interaction
- Language model：LLaMA-3-70B, throughparameter-efficient fine-tuning(PEFT) to inject chart-domain knowledge
- Training Methods：
- Generated data pipeline: utilizesChartCodertools to automatically generate 5 million diverse charts, covering 18 types, with each chart accompanied by structured metadata (data tables, plotting code, descriptions)
- Curriculum learning: first trains on simple charts (bar charts, line charts), then gradually increases complexity (radar charts, Sankey diagrams, 3D charts)
- Contrastive learning: treats different visualization forms of the same data as positive sample pairs, enhancing semantic invariance
- Key Technologies：
- Breadth understanding: supports unseen chart types, quickly adapting to new layouts through meta-learning
- Depth understanding: introduceschart-specific vocabulary(such as "peak", "inflection point", "year-over-year"), adding 2,000 specialized terms to the word embedding layer
Performance: InMMC-Benchmark's "multi-chart understanding" task, ChartScope achieves accuracy of68.5%, significantly higher than ChartVLM's61.2%, demonstrating the advantage in cross-chart correlation analysis.
ChartAssistant (Zhejiang University & Alibaba Cloud)
Focusing on "lightweight and efficient":
- Model Architecture：
- Visual encoder：TinyCLIP-ViT-B/16(86 million parameters), specifically optimized for edge devices
- Language model：Qwen2.5-7B, balancing performance and cost
- Special module：ChartTokenizer, encoding chart elements (bar heights, line slopes) into discrete "chart tokens", similar to text tokenization, reducing sequence length by 60%
- Training Methods：
- Three-stage progressive training：
- onWebPlot-50Mfor pretraining, learning general chart grammar
- onChartReason-200KInstruction fine-tuning, strengthening logical reasoning
- onRLHF-Chart-10KAlignment with human preferences, improving answer readability
- Three-stage progressive training：
- Key Technologies：
- Program-of-Thought Learning: Forces the model to first output Python code to extract data, then answer questions based on the data, improving interpretability
- Visual Token Compression: AdoptsToken Mergingtechnology to merge redundant visual tokens, increasing inference speed by 3x
Performance: On theCharXivbenchmark (evaluating academic paper charts), ChartAssistant scores65.8, although lower than GPT-4V's78.9, but the model size is only 1/20, deployable on a single RTX 3090, with latency <1 second.
How to measure a model's "chart IQ"
Mainstream Benchmark Datasets
- ChartX (2025): The most comprehensive chart understanding benchmark, constructed by Peking University:
- Scale: 50,000 charts, 18 types (including complex types such as radar charts, heatmaps, and 3D charts)
- Task Classification：
- Perception Tasks(Perception): legend recognition, data reading, color matching
- Cognition Tasks(Cognition): trend analysis, anomaly detection, causal relation inference
- Generation Tasks(Generation): chart redrawing, description generation, code writing
- Evaluation Metrics：
- SCRM(Structured Chart-oriented Representation Metric): Customized structured information extraction evaluation, measuring the similarity between the predicted data table and the ground-truth data table
- Accuracy@K: Top-K answer accuracy
- RNSS(Relative Number Set Similarity): Evaluates the relative error of numerical sets
- MMC-Benchmark (Multimodal Chart Understanding Benchmark): A high-quality benchmark annotated by humans
- Scale: 10,000 real business charts, from finance, scientific research, and government open data
- Features: Includescontext understandingtasks, requiring the model to reason by combining text outside the chart (such as report paragraphs)
- Tasks: chart information extraction, reasoning, context understanding, chart type classification, stock chart analysis
- ChartBench: Focuses oncomplex visual reasoning
- Challenges: Charts contain misleading visualizations (such as truncated Y-axis, area illusions), testing the model's critical thinking
- Subtasks: Fact-checking, Chart-to-Table, Open-ended QA
- PlotQA & ChartQA: A classic question-answering dataset
- PlotQA: 280,000 charts; questions require multi-step logical reasoning (such as "find the quarter with the highest growth rate")
- ChartQA: 32,000 charts, including human-written and synthetic questions, evaluating the model's deep understanding of chart structure
Evaluation Metrics
- Accuracy: The most intuitive metric, but it has limitations in chart tasks. For example, when reading numerical values, an error within 5% is usually acceptable, but traditional accuracy would judge it as incorrect. Therefore,Relaxed Accuracy(relaxed accuracy) was derived, allowing numerical values to be within a certain error range.
- F1 score: Used to evaluate structured data extraction (such as legend names, data point coordinates), balancing precision and recall.
- BLEU-4 / ROUGE-L: Evaluates generation tasks, such as "Please describe the trend of this chart". BLEU measures n-gram overlap, and ROUGE-L measures the longest common subsequence.
- RNSS（Relative Number Set Similarity）: An innovative metric introduced by ChartX, it computes the normalized similarity between the predicted value set and the ground-truth set. It is insensitive to order and is suitable for evaluating data extraction tasks.
End-to-end technical pipeline: full-chain parsing from pixels to answers
Preprocessing: making charts "clearer"
Image enhancement
- Denoising and sharpening: Use the Non-local Means algorithm to remove noise from scanned charts
- Resolution normalization: Uniformly scale to 1024×1024, preserve the aspect ratio, and fill blank areas with the background color
- Color space conversion: Convert from RGB to HSV to enhance color segmentation, especially for pie charts and heatmaps
Element detection and cropping
- Object detection model: Use YOLOv8 to detect chart regions, titles, axes, and legends, crop out the core region, and reduce background interference
- Text region enhancement: Perform super-resolution reconstruction (ESRGAN) on detected text regions to improve OCR accuracy
Visual tokenization: how images become "text"
Core challenge: convert the pixel grid into a discrete token sequence that a language model can process.
ViT encoding
Taking ChartVLM as an example:
- Image patchification: Split the 1024×1024 chart into 16×16 patches, 4096 in total
- Linear embedding: Flatten each patch into a 768-dimensional vector and map it to the embedding space through a learnable projection
- Positional encoding: Add 2D sinusoidal positional encoding to preserve spatial relationships
- Transformer encoding: After 24 Transformer layers, output a 4096×768 visual feature matrix
Token compression
- Token Merging（ToMe） : ChartAssistant adopts this technique, merging tokens with similarity >0.9, reducing 4096→~1600 and cutting computation by 60%
- Q-Former query: ChartScope's Q-Former uses 32 learnable query vectors to "retrieve" key information from 4096 visual tokens and output a compact 32×768 representation
Multimodal fusion: letting vision and language "converse"
Early Fusion
The LLaVA series adopts this: concatenate visual tokens and text tokens at the input layer and feed them together into the Transformer. It is simple but computationally expensive, and the sequence length can reach 2000+.
Middle Fusion
ChartVLM adopts this: insert a cross-modal attention layer at layer 8 of the Transformer, where text queries can attend to all visual tokens, but visual tokens do not attend to text with one another, reducing computational complexity.
Attention mechanism details
- Cross-attention: Visualization of text→vision cross-attention weights shows that the model automatically focuses on the chart region corresponding to keywords in the question (such as "maximum value")
- Self-attention mask: During training, a causal mask is applied to visual tokens to prevent them from "peeking" at future text tokens, preserving autoregressive properties
Post-processing: making answers more "human-like"
Numerical calibration
The numerical values output by the model may have slight errors, and the post-processing module usescubic spline interpolationto refit the data curve, ensuring numerical accuracy within 1%.
Answer verification
- Code executor: executes the Python code generated by the model in a sandbox environment to verify whether the extracted data is consistent with the chart
- Logical consistency check: checks the internal logic of the answer (e.g., whether "growth rate > 0" contradicts "data rising")
Performance comparison: let the data speak, who understands charts better?
Summary of benchmark test results
| Model | ChartX (comprehensive) | ChartQA | PlotQA | MMC-Bench | CharXiv |
|---|---|---|---|---|---|
| Gemini 1.5 Pro | 85.7 | 87.2 | 73.1 | 79.3 | 81.4 |
| GPT-4V | 83.2 | 84.5 | 69.8 | 77.1 | 78.9 |
| ChartVLM | 82.4 | 82.3 | 67.4 | 74.6 | 65.8 |
| ChartScope | 79.8 | 81.7 | 66.2 | 82.1 | 68.5 |
| ChartAssistant | 75.3 | 78.9 | 63.5 | 71.2 | 65.8 |
| Qwen2.5-VL-Max | 80.5 | 83.1 | 68.9 | 76.4 | 72.3 |
| InternVL-Chat-V1.5 | 78.9 | 79.4 | 64.7 | 73.8 | 69.1 |
(core data as of November 2025)
Key findings
- Closed-source models lead overall: Gemini 1.5 Pro and GPT-4V are superior on most tasks, especially on cognitive tasks with a significant advantage (>5 percentage points)
- Specialized models surpass in some areas：
- ChartVLM onperception taskshas an accuracy of 91.2%, surpassing GPT-4V (89.7%), due to its specialized optimization of structure extraction
- ChartScope onmulti-chart associationtasks leads, because its dual-path training strategy strengthens cross-chart reasoning
- Cost-effectiveness of lightweight models：
- Although ChartAssistant has a lower comprehensive score, itsinference speed is 15 times that of GPT-4V, with a latency of only 0.8 seconds on an RTX 3090, making it suitable for edge deployment
- Misleading visualizations are a common weakness：
In the "critical thinking" subset of ChartBench, all models have an accuracy of <60%, indicating that the ability to identify visualization traps (such as truncated axes) urgently needs improvement
Deployment in practice: from the lab to the production environment
Software and hardware toolchain
Hardware configuration
- Cloud recommendation: NVIDIA A100 80GB × 8 cards, supports batch inference, throughput can reach 50 QPS
- Edge recommendation: NVIDIA Jetson AGX Orin 64GB, power consumption only 50W, supports ChartAssistant after INT4 quantization
- Cost comparison: A100 costs $12.24 per hour, Orin device has a one-time investment of $1999, with a daily cost of <$1 (electricity)
Software stack
# Typical deployment environment
– Inference framework: vLLM 0.6.1 (supports continuous batching, throughput increased by 3-5 times)
– Quantization tools: AWQ / GPTQ (achieve INT4 quantization, reducing VRAM usage by 70%)
– Serving: TensorRT Inference Server + FastAPI
– Monitoring: Prometheus + Grafana, tracking latency, throughput, and VRAM usage
– Containerization: Docker + Kubernetes, enabling elastic scaling
Inference acceleration techniques
Core optimizations of vLLM
- PagedAttention: manages the KV cache in blocks, increasing VRAM utilization from 50% to 90% and supporting larger batch sizes
- Continuous Batching: dynamically merges requests, eliminating pipeline bubbles, increasing throughput by 3.8 times (measured data)
Quantization techniques in practice
- AWQ INT4: applied to ChartVLM, reducing model size from 144GB to 36GB, with accuracy loss <2%
- KV Cache quantization: quantizes the cache to INT8, further reducing VRAM usage by 50%, supporting 100+ concurrent requests
Latency Optimization Case Study
A financial company deployed Gemini 1.5 Pro on Google Cloud TPU v5e:
- Before optimization: average latency 2.3 seconds, P99 latency 4.1 seconds
- After optimization: using Speculative Decoding, average latency dropped to 1.1 seconds, P99 dropped to 1.8 seconds
- Cost: through dynamic batching, QPS increased from 5 to 22, and cost per request decreased by 65%
Deployment Mode Selection
| Deployment Mode | Latency | Cost | Applicable Scenarios | Representative Solution |
|---|---|---|---|---|
| Public Cloud API | Medium (1-3s) | High ($0.01/request) | Low frequency, high quality requirements | GPT-4V API |
| Dedicated Cloud Instance | Medium (0.5-2s) | Medium ($5000/month) | Medium frequency, data-sensitive | AWS SageMaker |
| Edge Device | Low (<1s) | Low ($2000 one-time) | High frequency, real-time requirements | Jetson + TinyChart |
| Hybrid Deployment | Variable | Optimized | Complex business | Core charts on cloud, real-time charts on edge |
Future Trends and Challenges
Technology Evolution Directions
- Deep Multimodal Integration: From the simple concatenation of "visual encoder + language model" tonative multimodal architecture, such as Gemini's Pathways system, to achieve more efficient cross-modal reasoning
- Self-Supervised Learning Upgrade: Usingchart-to-codegeneration tasks as pretraining objectives, allowing the model to learn charting logic and inversely strengthen understanding capabilities
- Enhanced Interpretability: Throughattention visualizationdisplay the chart regions the model focuses on, combined withprogram thinkingto generate verifiable reasoning paths
Core Challenges
- Recognition of Misleading Visualizations: Existing models have insufficient sensitivity tovisual traps(such as non-zero baseline axes, area misleading). A 2025 study found that when the Y-axis is truncated, model accuracy drops sharply from 85% to 42%. Solutions include:
- Injectingadversarial examplesinto the training set, deliberately adding misleading designs
- IntroducingCritical Head(Critical Head), specifically to determine whether a chart contains visual manipulation
- Long-Sequence Chart Understanding: Complex charts containing 50+ data points, or financial report chart sets up to 10 pages long, have tens of thousands of visual tokens, resulting in high computational complexity. Latest progress:
- LongVLM Architecture: AdoptsRing Attention(Ring Attention), reducing computational complexity from O(n²) to O(n)
- Hierarchical Processing: First extract the chart outline (title, axis labels), then zoom into detail regions as needed, similar to the human observation method of "overall first, then local"
- Domain Adaptation: Charts in professional fields such as healthcare and finance contain a large number of terms and specific specifications. Fine-tuning solutions include:
- LoRA+Adapter hybrid fine-tuning: Freeze the backbone network and train only the domain adapter, completing financial domain adaptation in 1 hour
- Retrieval-Augmented Generation (RAG) : Combine knowledge base retrieval of professional term explanations to improve answer professionalism
Large models in the field of chart understanding have achieved a leap from "usable" to "easy to use." Closed-source models such as Gemini 1.5 Pro approach human expert level on professional tasks, while open-source models such as ChartVLM stand out in cost-effectiveness in specific scenarios. The technology stack has moved from the laboratory to the production line, and toolchains such as vLLM, quantization, and edge deployment are mature, with controllable costs.
However,critical thinkinganddomain depthremain weak points. In the next 2-3 years, with the upgrading of self-supervised learning paradigms and continuous optimization of computational efficiency, chart understanding will become as widespread as today's OCR and become a standard feature of every data analysis platform. By then, the vision that "everyone is a data analyst" will truly be realized—just upload a chart, and AI can become your intelligent data assistant, revealing the story behind the data.
How to choose a suitable solution
| Use case | Recommended model | Deployment method | Estimated cost | Difficulty to get started |
|---|---|---|---|---|
| Personal learning, lightweight analysis | ChartAssistant | Local Docker | Free | ⭐⭐ |
| Small and medium-sized enterprises, data dashboards | ChartVLM | Cloud virtual machine | $500/month | ⭐⭐⭐ |
| Financial risk control, scientific research analysis | Gemini 1.5 Pro | API calls | Pay-as-you-go | ⭐ |
| Industrial quality inspection, real-time monitoring | TinyChart | Edge devices | $2000 one-time | ⭐⭐⭐⭐ |
Quick start recommendation: Start with the ChartAssistant Hugging Face Demo to experience the basic functions; if higher accuracy is needed, then consider fine-tuning ChartVLM or integrating a commercial API. The era of chart understanding has arrived, and the key is to find the technical path that suits your needs.