LocateAnything - NVIDIA's vision-language localization model

What is LocateAnything

LocateAnything is a vision-language localization model launched by NVIDIA, based on parallel box decoding technology, allowing users to precisely box-select targets in images by entering natural language. The model supports tasks such as multi-target detection, GUI localization, OCR text detection, and point-level pointing, with an inference speed of 12.7 BPS (H100), which isQwen3-VL10 times faster, achieving SOTA on benchmarks such as LVIS, and is suitable for robotics, document intelligence, and autonomous driving scenarios.

Main features of LocateAnything

• Referring expression localization: Locates and boxes specific targets in an image based on natural language descriptions.
• Multi-target dense detection: Supports simultaneous detection of multiple object categories in an image, outputting a dense set of bounding boxes.
• GUI element localization: Can recognize and locate interactive elements such as buttons and input boxes in interfaces, supporting point-level output.
• Text detection and OCR: Locates text regions in images, supporting recognition of documents, scene text, and table structures.
• Point-level localization: Supports outputting precise point coordinates for fine-grained pointing tasks.
• Hybrid inference mode: Uses fast parallel mode by default, and automatically falls back to stable serial mode in complex scenarios.

Technical principles of LocateAnything

• Parallel Box Decoding (PBD): The model treats each bounding box or point as an atomic unit and predicts the complete coordinate set in parallel in a single forward pass (x1,y1,x2,y2), rather than traditional token-by-token serial generation.
• Model architecture: The model uses the Moon-ViT vision encoder to extract native-resolution visual features, which are mapped through a two-layer MLP projector and then input into the Qwen2.5 language decoder for localization inference.
• Structured output: The model generates semantic blocks, box blocks, negative sample blocks, and end blocks, organizing structured localization results through special tokens such as <box>、<ref> .
• Hybrid inference mode: Uses fast parallel mode (MTP) by default to maximize throughput, and automatically and seamlessly falls back to stable serial mode (NTP) when encountering format anomalies or spatial ambiguity.
• Corrective re-decoding: When parallel decoding detects malformed box structures or coordinate conflicts, it automatically discards the problematic blocks and falls back to the verified prefix, regenerating corrections through NTP.

Follow WeChat and reply "open source" to join theAI open source project exchange group

How to use LocateAnything

• Environment preparation: Clone the Eagle repository from GitHub, enter the Embodied directory and run pip install -e . to complete dependency installation.
• Model loading: Import the LocateAnythingWorker class and instantiate it, loading the nvidia/LocateAnything-3B pretrained weights from Hugging Face.
• Object detection: Call the detect() method, passing in an image and a list of category names (such as ["person", "car"]), and the model returns the bounding box coordinates of all matching targets.
• Phrase localization: Use the ground_multi() method to input an image and description text (such as "people wearing red shirts") to locate the specific target matching the semantic description.
• Text detection: Call the detect_text() method; the model automatically identifies text regions in the image and outputs the corresponding set of bounding boxes.
• GUI localization: Locate UI elements via the ground_gui() method; setting output_type="point" outputs point coordinates instead of bounding boxes.
• Point-level pointing: Use the point() method to input an image and a description, and obtain the precise single-point coordinates of the specified target for fine-grained pointing.
• Batch inference: Configure the la_flash runtime and add batch_utils/ to PYTHONPATH, then call detect_batch() to process multiple image-query pairs simultaneously.
• Result parsing: Extract the integer coordinates (range 0-1000) within the <box> tags in the output, divide by 1000, and map them to the pixel values of the original image's width and height.
• Fine-tuning adaptation: Prepare a JSONL data recipe, use torchrun to launch the full fine-tuning script, or run the LoRA script to freeze the backbone and train only the projector and low-rank adaptation layers.

LocateAnything's core advantages

• Leading speed: Reaches 12.7 BPS on a single H100, about 10 times faster than Qwen3-VL and about 2.5 times faster than Rex-Omni.
• SOTA accuracy: Achieves best-in-class performance on multiple benchmarks such as LVIS, M6Doc, ScreenSpot-Pro, and DocLayNet.
• General-purpose unification: A single model covers diverse tasks such as detection, localization, GUI, OCR, and layout, with no need to switch specialized models for different scenarios.
• Geometric consistency: Parallel decoding preserves the geometric coupling relationships among coordinates within a box, avoiding structural deformities caused by serial generation.
• VRAM optimization: Through the la_flash runtime, peak VRAM can be reduced from 35GB to 11GB on consumer/data center GPUs such as the A100.

LocateAnything's project addresses

• Project website：https://research.nvidia.com/labs/lpr/locate-anything/
• GitHub repository：https://github.com/NVlabs/Eagle/tree/main/Embodied
• HuggingFace model library：https://huggingface.co/nvidia/LocateAnything-3B
• Technical paper：https://research.nvidia.com/labs/lpr/locate-anything/LocateAnything.pdf

Comparison of LocateAnything with similar competitors

LocateAnything's application scenarios

• Embodied intelligence and robotics: Provides robots with visual perception capabilities, localizing operation targets through language instructions.
• Intelligent document processing: Automatically identify and box structured elements in documents such as titles, tables, paragraphs, and seals.
• GUI Automation Testing: Locate buttons, input fields, and icons in the interface, supporting automated testing and RPA workflows.
• Autonomous Driving Perception: Densely detect vehicles, pedestrians, and traffic signs in street view images, and output precise bounding boxes.
• Retail and Warehousing: Quickly locate specific products on shelves or target packages in warehouses through descriptions.