What is AutoConsis
AutoConsis is an intelligent UI content consistency detection tool jointly launched by the Meituan technical team and Fudan University. The tool is based on deep learning and large language models to automatically identify and extract key data in the interface, and detect and identify inconsistency issues between data. AutoConsis can improve user experience, reduce user confusion caused by data display errors, and ease the workload of testers. In practical applications, AutoConsis shows the advantages of low cost, high generalization, and high confidence, and is of great significance to the quality assurance of large front-end UIs.
Main features of AutoConsis
- Target area recognition: First identify the key areas related to detection in the UI interface, and locate the UI parts containing important information based on image processing and pattern recognition technology.
- Target information extraction: After the target area is confirmed, the tool uses OCR technology and UI component analysis to extract the text and elements in the target area, and uses a large language model (LLM) to reason and extract the key information required for consistency verification.
- Consistency verification: Perform consistency verification on the extracted information to ensure the accuracy and consistency of UI information, including verification of numerical logic types and semantic types.
- Automated intelligent detection process: Implement a set of automated intelligent detection processes that can detect UI content consistency with low cost, high generalization, and high confidence.
- Adaptation to multiple business scenarios: The tool can adapt to different page templates, technology stacks, and Apps, automatically perform adaptation, and cover multiple types of businesses and diverse layouts.
Technical principles of AutoConsis
- Multimodal deep learning models: Use multimodal deep learning models to analyze GUI pages, combining image and text information to improve recognition accuracy.
- Large language model (LLM): LLMs extract deep semantic information from text to identify and parse key data.
- Object detection and content understanding: Transform the UI page analysis task into a combination of object detection and content understanding, and use the capabilities of large models to adapt to pages of different technology stacks.
- OCR and UI component analysis: Use OCR technology to extract text information, combine UI component analysis tools to extract elements, fill in the preset CoT Prompt (Chain of Thought Prompt), and extract key information based on large model reasoning.
- Consistency verification rules: According to predefined numerical logic and semantic rules, determine the consistency of the extracted UI content, directly check the consistency of numerical logic types with rules, and for complex semantic rules, use the understanding ability of LLMs to achieve verification.
- Information extraction prompt design: To address the common "hallucination" problem of large language models, design prompts that include contextual learning (CoT) to improve the accuracy of information extraction.
- Multimodal UI area recognition: Perform target area recognition based on the visual semantic recognition model CLIP, and use the semantic mapping of images and text to match in a high-dimensional vector space, improving the generalization and accuracy of recognition.
AutoConsis project address
- Technical paper：https://dl.acm.org/doi/10.1145/3639477.3639748
Application scenarios of AutoConsis
- E-commerce platforms: On e-commerce platforms, detect whether the price and inventory information is consistent across product detail pages, shopping cart pages, and checkout pages, ensuring that the information users see on different pages is accurate and synchronized.
- Marketing campaign validation: In promotions and marketing campaigns, verify whether the descriptions of discounts, offers, and campaign rules are consistent across different pages, avoiding user confusion caused by inconsistent information.
- Financial applications: In financial applications, detect whether key financial data such as account balances, transaction records, and investment returns are displayed consistently across different pages.
- Travel and hotel booking: In travel and hotel booking applications, check whether prices, availability, and booking terms are consistent across different pages to improve the user experience.
- Social media and content platforms: On social media and content platforms, detect whether the data and information in user profiles, posts, and comments are consistent.