What is Ferret-UI 2
Ferret-UI 2 is a multimodal large language model introduced by Apple, used for understanding and interacting with mobile user interfaces. Ferret-UI 2 can recognize and understand UI elements on the screens of various mobile devices, execute complex user instructions, observe the user's operations on the mobile device screen in real time, and be ready at any time to provide assistance and perform tasks. Compared with earlier versions, Ferret-UI 2 has undergone substantial improvements and updates. Based on high-resolution image encoding and advanced data training methods, it improves the recognition accuracy and interaction capability of UI elements, allowing users to interact with smart devices more naturally and efficiently.
Main Features of Ferret-UI 2
- Multi-platform support: Ferret-UI 2 can handle user interfaces on multiple platforms, including iPhone, Android, iPad, Webpage, and AppleTV.
- High-resolution image perception: Based on adaptive scaling technology, Ferret-UI 2 can achieve more accurate visual element recognition while maintaining the original resolution of UI screenshots.
- Advanced task training data generation: Based on GPT-4o and set-of-mark visual prompts, Ferret-UI 2 generates training data for complex tasks, improving the model's understanding of the spatial relationships of UI elements.
- User-centered interaction: Ferret-UI 2 can understand and execute user-centered interaction tasks, such as confirming submissions and clicking buttons, rather than merely mechanical clicks.
- Cross-platform transfer capability: Ferret-UI 2 demonstrates strong cross-platform transfer capability and can transfer and adapt between different platforms.
Technical Principles of Ferret-UI 2
- Multimodal Large Language Model (MLLM): Combines visual perception and language processing capabilities to understand and generate complex interactions with UIs.
- Adaptive N-grid mechanism: An algorithm determines the optimal grid size to encode each part of a UI screenshot with minimal resolution distortion and pixel changes.
- Dynamic high-resolution image encoding: Uses a CLIP image encoder to extract global and local features and feeds the features into a Large Language Model (LLM).
- Visual sampler: Identifies and selects relevant UI regions according to user instructions, and outputs perceptual or interactive descriptions of UI elements.
- set-of-mark (SoM) visual prompting: When generating training data, SoM prompts are used to enhance the model's understanding of the spatial relationships of UI elements, especially in multi-turn perception and interaction question-answering tasks.
- End-to-end training: The model learns from raw data annotations through an end-to-end training process, generates high-quality training data, and optimizes model performance.
Project Address of Ferret-UI 2
- arXiv technical paper：https://arxiv.org/pdf/2410.18967
Application Scenarios of Ferret-UI 2
- Smartphones and tablets: Ferret-UI 2 can understand and execute various user instructions on iOS and Android devices, such as navigating applications, sending messages, and setting reminders.
- Web browsing: In web browsing, it helps users interact more effectively with webpage elements, such as clicking buttons, filling out forms, and navigating links.
- Smart TVs: For smart TV platforms such as Apple TV, it provides voice control and other interaction methods to enhance the user experience.
- Multitasking environments: In scenarios where multiple applications or windows need to be handled simultaneously, help users manage and switch between different tasks more efficiently.
- Assistive technology: Integrate into assistive technology to help people with disabilities interact with devices through voice commands or other input methods.