GPT-4o

What is GPT-4o

GPT-4o is an advanced artificial intelligence model newly launched by OpenAI, featuring powerful multimodal reasoning capabilities and able to process voice, text, and visual information. The model can respond to user input in real time, detect and express emotions in audio interactions, and provide a more natural and expressive communication experience. The design of GPT-4o focuses on improving computing speed and reducing cost, with twice the speed of previous models and only half the cost. GPT-4o performs outstandingly in multilingual processing, audio, and visual understanding, while also strengthening safety design to ensure the safety of interactions. It supports generating and editing images in ChatGPT and Sora, can generate high-quality, realistic images through text instructions, and can make multi-turn dialogue modifications to gradually optimize images. It can handle complex instructions containing as many as 10 to 20 different objects. The model combines chat context and a knowledge base to generate images, ensuring consistency of characters and elements in multi-turn generation. It has been opened to ChatGPT Plus, Pro, Team, and free users.

Main features of GPT-4o

• Native image generation feature: GPT-4o can now directly generate and edit images in ChatGPT and Sora, replacing the previous DALL-E 3 model. Users can generate high-quality, realistic images through text instructions and make multi-turn dialogue modifications to gradually optimize images.
• Accurate presentation of text content: It can accurately render text in images and generate images containing text such as menus and invitations, solving the previous difficulty of text processing in AI image generation.
• Understanding and execution of complex instructions: GPT-4o can handle complex instructions containing as many as 10 to 20 different objects, far exceeding the 5 to 8 object limit of other models.
• Contextual association and consistency maintenance: The model combines chat context and a knowledge base to generate images, ensuring consistency of characters and elements in multi-turn generation.
• Content safety: All generated images carry C2PA metadata identifiers. OpenAI has built an internal search tool to verify content sources and block image requests that violate content policies.
• Restrictions on real-person images: There are stricter restrictions on generating real-person images to prevent offensive content.
• Multimodal interaction: GPT-4o can process text, voice, and visual information, and can understand and respond to a broader range of user input, including real-time video analysis.
• Real-time dialogue feedback: The model can provide immediate responses and quickly give feedback in text dialogue, voice interaction, or video content analysis. Its response time to audio input is extremely short, averaging 320 milliseconds, similar to human conversation response time.
• Emotion recognition and simulation: GPT-4o can recognize the user's emotional state and simulate the corresponding emotion in voice output, making conversations closer to natural human-to-human communication.
• Programming code assistance: GPT-4o can analyze and understand code snippets in programming languages, helping users understand the function and logic of the code. Users can ask GPT-4o questions about code by voice, and the model will respond in voice, explaining how the code works or pointing out potential issues.
• Multilingual support: GPT-4o supports more than 50 languages, meeting the needs of different language environments. It supports real-time simultaneous interpretation in multiple languages, such as interpreting spoken English into Italian.

Technical principles of GPT-4o

• Autoregressive model: Unlike DALL-E's diffusion model, GPT-4o uses an autoregressive model, which enables it to better understand and generate images.
• Training data: To support the new image capabilities, OpenAI used public data as well as proprietary data obtained through partnerships with companies such as Shutterstock to train GPT-4o.

GPT-4o performance

• Text performance evaluation: GPT-4o set the highest score of 87.2% on the Multilingual Understanding Mean (MMLU) benchmark, ranked first with a GPQA score of 53.6%, first with a MATH score of 76.6%, first with a HumanEval score of 90.2%, and second with an MGSM score of 90.5% (slightly below Claude 3 Opus). This demonstrates its strong reasoning ability on common-sense questions and text processing capability.
  
• Audio ASR performance: GPT-4o significantly improves speech recognition performance across all languages compared with Whisper-v3, especially for low-resource languages.
  
• Audio translation performance: GPT-4o has achieved a new state of the art in speech translation and outperforms Whisper-v3 on the MLS benchmark, surpassing Meta's SeamlessM4T-v2 and Google's Gemini.
  
• Visual understanding evaluation: GPT-4o achieved SOTA state-of-the-art performance on visual perception benchmarks, surpassing Gemini 1.0 Ultra, Gemini 1.5 Pro, and Claude 3 Opus.
  

Comparison between GPT-4o and GPT-4 Turbo

• Price: GPT-4o is 50% cheaper than GPT-4 Turbo. Specifically, the prices for input and output tokens are $5 per million (M) input and $15 per million output, respectively.
• Rate limits: GPT-4o's rate limit is 5 times that of GPT-4 Turbo, and it can process up to 10 million tokens per minute.
• Visual capabilities: In evaluations and tests related to visual capabilities, GPT-4o performs better than GPT-4 Turbo.
• Multilingual support: GPT-4o has improved support for non-English languages and provides better performance than GPT-4 Turbo.

Currently, GPT-4o has a context window of 128k, and its knowledge cutoff date is October 2023.

How to use GPT-4o

GPT-4o's text and image capabilities have begun to be gradually rolled out in ChatGPT. Users can experience GPT-4o's related features for free on the ChatGPT platform, but the free version has usage limits. The message limit for Plus users will be 5 times higher than that for free users.

At the same time, OpenAI also plans to launch a new version of Voice Mode based on GPT-4o within the next few weeks, which will be provided to Plus users as an alpha version of ChatGPT Plus. In addition, GPT-4o will also be provided to developers through the API as a text and vision model. Developers can use the API to integrate GPT-4o into their own applications, and GPT-4o is faster, cheaper, and has higher rate limits in the API compared with GPT-4 Turbo.

As for GPT-4o's audio and video capabilities, OpenAI will continue to develop the technical infrastructure, improve usability through post-training, and ensure safety in the coming weeks and months before releasing these capabilities and gradually making them available to the public.

Official blog introduction:Hello GPT-4o

GPT-4o application scenarios

• Creative design: Designers and artists can quickly generate creative images and improve creative efficiency.
• Education and presentations: Educators can generate teaching materials, such as scientific diagrams and historical scene recreations.
• Game Development: Game developers can generate game assets, such as character designs and environmental backgrounds.