VisionAgent is a "Visual AI pilot" tool developed by LandingAI. Users only need to provide a single prompt and an image or video, and it will automatically select the appropriate vision model and output directly runnable code, allowing developers to quickly build applications with visual capabilities.
In terms of use, VisionAgent is mainly used to generate vision task code based on natural language prompts, such as counting objects in an image, detecting specific targets, tracking people in a video, etc. It can also be used as a set of independent tool libraries, directly calling specific vision functions in your own scripts.
Its main capabilities include: being able to generate a task plan for code generation based on a prompt (numbered steps are shown when detailed output is enabled); generating code and test cases based on the plan; validating the generated code with test cases, and if it fails, iterating the generation process until it passes. The VisionAgent library has built-in multiple tools (independent models or functions) that can complete specific tasks. For example, for the prompt "count the number of dogs in the image," it may first use florence2_object_detection to detect all dogs, and then use countgd_object_detection to count the number. The tools are located in the vision_agent.tools API, and after installation can be directly called in your own scripts, such as using countgd_object_detection to count the number of objects in an image, or using countgd_sam2_video_tracking to track people in a video and, together with extract_frames_and_timestamps, return the frames and timestamps where people appear.
How to use: Python 3.9 or higher is required, and you need to prepare an Anthropic API key, a Google API key, and a VisionAgent API key (you need to register an account first to obtain it). Before running, these three keys need to be set as environment variables. Installation can be done via uv or pip. The quick start process is: obtain the keys, set the environment variables, install VisionAgent, create a quickstart folder, put in the image to be analyzed, copy the example script as source.py and run it, and then a generated_code.py file will be generated to save the generated code. You can also experience it through the local web app (examples/chat). If you want to change or use only one LLM provider, you can do so in vision_agent/configs/config.py, and add the corresponding API key. VisionAgent uses Anthropic Claude 3.7 Sonnet and Gemini Flash 2.0 Experimental (gemini-2.0-flash-exp) to respond to prompts and generate code; v1.0.2 and earlier versions were powered by Anthropic Claude-3.5 and OpenAI o1, and using these versions requires obtaining an OpenAI API key and setting it as an environment variable.
Applicable scenarios: suitable for developers who need to quickly build visual AI applications, such as image object counting, object detection, video person tracking, etc. It can also be integrated as a tool library into your own projects.
Limitations and notes: This tool has been deprecated, and the official recommendation is to switch to Agentic Document Extraction. Before use, you must obtain and set the Anthropic, Google, and VisionAgent API keys, otherwise it cannot run; the Anthropic and Google APIs have their own rate limits and paid tiers. Specific information such as VisionAgent's own charges, pricing, and platform is not currently provided.
