GLM-4.1V-Thinking - Zhipu AI's Open-Source Vision-Language Model Series

What is GLM-4.1V-Thinking

GLM-4.1V-Thinking is an open-source vision-language model launched by Zhipu AI, designed specifically for complex cognitive tasks and supporting multimodal inputs such as images, videos, and documents. Based on the GLM-4V architecture, the model introduces a chain-of-thought reasoning mechanism and, based on a curriculum sampling reinforcement learning strategy, systematically improves cross-modal causal reasoning capability and stability. The lightweight version of the model, GLM-4.1V-9B-Thinking (the GLM-4.1V-9B-Base foundation model and GLM-4.1V-9B-Thinking have deep thinking and reasoning capabilities), has a parameter count controlled at the 10B level. Among 28 authoritative evaluations, 23 achieved the best results for 10B-level models, of which 18 matched or surpassed Qwen-2.5-VL, whose parameter count is as high as 72B, demonstrating the extreme performance potential of a small-sized model.

Main Features of GLM-4.1V-Thinking

• Image Understanding: Accurately recognizes and analyzes image content, supports complex visual tasks such as object detection, image classification, and visual question answering.
• Video Processing: Has temporal analysis and event logic modeling capabilities, supports processing video input, and performs video understanding, video description, and video question answering.
• Document Parsing: Supports processing image and text content in documents, and supports long document understanding, chart understanding, and document question answering.
• Mathematical and Scientific Reasoning: Supports complex mathematical problem solving, multi-step deduction, and formula understanding, and can handle reasoning tasks in the STEM field.
• Logical Reasoning: Supports logical reasoning and causal analysis, and supports complex reasoning tasks such as multi-step reasoning and logical judgment.
• Cross-Modal Reasoning: Combines visual and linguistic information for reasoning, and supports tasks such as image-text understanding, visual question answering, and visual grounding.

Technical Principles of GLM-4.1V-Thinking

• Architecture Design: Uses AIMv2Huge as the visual encoder to process and encode image and video inputs. An MLP adapter aligns visual features to the token space of the language model. The language decoder uses GLM as the language model to process multimodal tokens and generate output.
• Training Method: Pretrains on large-scale image-text pairs, academic literature, and knowledge-intensive data to build a powerful vision-language foundation model. Uses long chain-of-thought (CoT) data for supervised fine-tuning to improve the model's reasoning ability and human alignment. Based on curriculum sampling reinforcement learning (RLCS), dynamically selects the most informative samples for training to improve the model's performance on various tasks.
• Technical Innovation: Introduces a chain-of-thought reasoning mechanism, allowing the model to think step by step and generate detailed reasoning processes. Based on a curriculum sampling strategy, dynamically adjusts the difficulty of training samples to ensure that the model receives the most effective training at different stages. Based on 2D-RoPE and 3D-RoPE technologies, supports image input of arbitrary resolution and aspect ratio, enhancing the model's spatiotemporal understanding capability.

Performance of GLM-4.1V-Thinking

Among 28 authoritative evaluations such as MMStar, MMMU-Pro, ChartQAPro, and OSWorld, the model achieved the best results for 10B-level models in 23 of them, of which 18 matched or surpassed Qwen-2.5-VL, whose parameter count is as high as 72B.

Project Address of GLM-4.1V-Thinking

• GitHub Repository：https://github.com/THUDM/GLM-4.1V-Thinking
• HuggingFace Model Library：https://huggingface.co/collections/THUDM/glm-41v-thinking-6862bbfc44593a8601c2578d
• arXiv Technical Paper：https://arxiv.org/pdf/2507.01006v1
• Online Experience Demo：https://huggingface.co/spaces/THUDM/GLM-4.1V-9B-Thinking-Demo

How to Use GLM-4.1V-Thinking

• API Interface：
  • Register an Account: VisitZhipu AI Open Platform, register an account and log in.
  • Obtain an API Key: Create an application on the platform to obtain your exclusive API Key.
  • Call the API: According to the API documentation, use HTTP requests to call the model interface, send input data, and obtain the model's output results.For example, here is a code example for calling the API using Python:

import requests
import json

# 设置API接口地址和API Key
api_url = "https://api.zhipuopen.com/v1/glm-4.1v-thinking"
api_key = "your_api_key"

# 准备输入数据
input_data = {
    "image": "image_url_or_base64_encoded_data",
    "text": "your_input_text"
}

# 设置请求头
headers = {
    "Authorization": f"Bearer {api_key}",
    "Content-Type": "application/json"
}

# 发送请求
response = requests.post(api_url, headers=headers, data=json.dumps(input_data))

# 获取结果
result = response.json()
print(result)

• Open-Source Model：
  • Download the Model: Visit Hugging Face, find the GLM-4.1V-Thinking model page, and download the model files.
  • Load the Model: Use the corresponding deep learning framework (such as PyTorch) to load the model.
  • Perform Inference: After preprocessing the input data, input it into the model to obtain the model's output results. For example, here is a code example for loading the model and performing inference using PyTorch:

from transformers import AutoModelForVision2Seq, AutoProcessor
import torch

# 加载模型和处理器
model_name = "THUDM/glm-4.1v-thinking"
model = AutoModelForVision2Seq.from_pretrained(model_name)
processor = AutoProcessor.from_pretrained(model_name)

# 准备输入数据
image_url = "image_url_or_image_path"
text = "your_input_text"
inputs = processor(images=image_url, text=text, return_tensors="pt")

# 进行推理
with torch.no_grad():
    outputs = model(**inputs)

# 获取结果
result = processor.decode(outputs.logits[0], skip_special_tokens=True)
print(result)

• Online Experience Platform：
  • Visit the Experience Link: Visit the GLM-4.1V-Thinking experience page on the Hugging Face platform.
  • Input Data: Upload an image or enter text on the webpage.
  • Get Results: Click the "Run" button, wait for the model to process, and view the results.

Application Scenarios of GLM-4.1V-Thinking

• Educational Tutoring: Assist students in solving complex problems in subjects such as mathematics and science, provide detailed problem-solving steps and reasoning processes, and help students better understand and master knowledge.
• Content Creation: Combine images and text to generate creative content, such as advertising copy, social media posts, news reports, etc., improving the efficiency and quality of content creation.
• Intelligent Interaction: As an intelligent customer service or virtual assistant, understand users' questions and needs, provide accurate and timely answers and solutions, and support multimodal input.
• Industry Applications: In fields such as healthcare, finance, and industry, assist professionals in tasks such as data analysis, report generation, and equipment monitoring, improving work efficiency and accuracy.
• Entertainment and Daily Life: Provide travel guides and attraction introductions for tourism, recommend dishes and cooking methods for food, generate plots and quest designs for games, and enrich users' entertainment experience.