AutoFigure-Edit is an upgraded version of AutoFigure, used to generate editable scientific illustrations from the text of a paper's Methods section. It can transform the original generated results into fully editable SVG vector graphics, and allows users to refine and adjust them in an embedded SVG editor. The project has been accepted at ICLR 2026, the paper is available on arXiv, and it was featured on Hugging Face Daily Papers. The online platform deepscientist.cc is now live and open free of charge to all scholars.
Main capabilities and workflow: The AutoFigure-Edit pipeline starts from the paper's method text and converts the content into an editable SVG in four stages.
- Generation (figure.png): the LLM generates a raster draft figure based on the method text.
- Segmentation (sam.png): SAM3 is used to detect and segment different icons and text regions.
- Templating (template.svg): the system uses placeholders to build a structured SVG wireframe.
- Assembly (final.svg): high-quality cropped icons and vectorized text are injected into the template to generate the final SVG.
Specifically, the system first calls a text-to-image LLM to render a journal-style schematic diagram and saves it as figure.png; then it uses SAM3 to segment according to one or more text prompts (such as "icon, diagram, arrow"), merges overlapping detections using an IoU-like threshold, generates annotated boxes with gray fill and black outlines, and outputs samed.png and boxlib.json containing coordinates, scores, and prompt sources. Next, each box is cropped from the original image and background removal is performed with RMBG-2.0 to obtain transparent icon assets. The LLM combines figure.png, samed.png, and boxlib.json to generate a placeholder-style SVG template. Optionally, an LLM optimizer iteratively optimizes the SVG alignment, layout, and style, then compares the SVG dimensions with the original image to calculate a scaling factor and align the coordinate system, and finally replaces the placeholders with the corresponding transparent icons to assemble final.svg.
In addition, AutoFigure-Edit supports two breakthrough capabilities: fully editable SVG (implemented purely in code, with text, shapes, and layout all losslessly modifiable) and style transfer (the system can imitate the artistic style of a reference image provided by the user).
How to use: The project provides two deployment methods.
Option 0: Docker deployment (recommended). Use Docker's reproducible one-click environment, with no need to install Python/SAM3 locally. Requires Docker Desktop (including Docker Compose v2), host port 8000 available, and HuggingFace access to briaai/RMBG-2.0. At minimum, the relevant variables must be set in .env; optionally configure the build image when the network is restricted. After running, visit http://localhost:8000，健康检查返回 {"status":"ok"}. Outputs are persisted in ./outputs and ./uploads, and the HuggingFace cache is stored in the Docker volume hf_cache.
Option 2: Web interface. On the configuration page, paste the paper method text on the left and set generation parameters on the right, including Provider (Bianxie AI, OpenRouter, Custom, Gemini, or OpenAI Responses), Image Provider (optional, overrides only the first step using OpenAI GPT-Image), Optimize (number of SVG template refinement iterations; standard use recommends 0), Image Size (optional 1K/2K/4K when the first-step image provider is Gemini), Auto Upscale (enabled by default, upscaling the long side of figure.png to 4K, i.e., 3840px, while preserving the aspect ratio), Reference Image (upload a target image to enable style transfer), SAM3 Backend (local SAM3 or fal.ai API; API key optional). If you already have a stage-1 raster image, you can use the button in the upper-right corner, "I already have the stage-1 figure", to enter the import page, upload an existing academic illustration, and continue directly from SAM+SVG reconstruction. The generated result is loaded into the integrated SVG-Edit canvas, supporting full vector editing. The interface also provides status and logs, an Artifacts drawer (containing intermediate products, which can be dragged and dropped onto the canvas), and History (browse historical runs in outputs/ and reopen them).
SAM3 installation instructions: AutoFigure-Edit depends on SAM3 but does not bundle it. Please follow the official SAM3 installation guide; the upstream repository targets Python 3.12+, PyTorch 2.7+, and CUDA 12.6 (GPU build). SAM3 checkpoints are hosted on Hugging Face and may require requesting access and authenticating (e.g., huggingface-cli login) before downloading. If you do not want to install it locally, you can use the API backend (also supported by the Web demo); Roboflow (free) is recommended. fal.ai is also an option.
Configuration: Supported LLM providers include openrouter, bianxie, custom, gemini, openai_response. Common CLI parameters include --method_text, --method_file, --input_figure_path, --provider, --image_provider, --image_api_key, --image_base_url, --image_model, --svg_model, --image_size, --disable_auto_upscale, --sam_prompt, --sam_backend, --sam_api_key, --sam_max_masks, --merge_threshold, --optimize_iterations, --reference_image_path.
Custom Provider supports self-hosted or third-party OpenAI-compatible endpoints, using --provider custom, --base_url (must be the /v1 root path), --image_model, --svg_model; AUTOFIGURE_CUSTOM_BASE_URL can also be set. Text and SVG reconstruction use the OpenAI-compatible chat shape, and multimodal SVG reconstruction requires support for image_url data URIs, returning the standard content shape, with SVG either bare or wrapped in a markdown code block. The built-in bianxie route also uses the OpenAI-compatible chat shape, and the first-step image generation uses the OpenAI Images-compatible path.
OpenAI GPT-Image for the first step: As of April 23, 2026, the official OpenAI Images API supports images.generate and images.edit. --image_provider openai uses the Images API only for the first step; use generate when there is no reference image and edit when there is one, with the default model gpt-image-2. After the first step by default, the image is upscaled to a long side of 3840px.
OpenAI Responses Provider: --provider openai_response indicates that text invocation uses client.responses.create(...), and multimodal SVG reconstruction also uses this interface; the first step, image generation, falls back to the official Images API, with the default SVG model being gpt-5.5.
Applicable scenarios and limitations: AutoFigure-Edit is aimed at researchers, and is especially suitable for users who need to quickly convert the methods section of a paper into editable vector illustrations and who wish to perform style transfer. It can be used online for free, and can also be deployed locally or with Docker. In terms of limitations, the materials do not provide detailed performance metrics or specific guarantees regarding generation quality; SAM3 must be installed separately and may require Hugging Face authentication; some models and API routes require users to configure keys or compatible endpoints themselves.
