SuperAPI AI Directory
Partner with us中文AI newsFavorites
DirectoryNews and tutorials

Step-Audio-AQAA - StepFun's End-to-End Large Audio Language Model

Author: AI小集 · Original source: AI工具集

What is Step-Audio-AQAA

Step-Audio-AQAA is an end-to-end large audio language model launched by the StepFun team, specifically designed for audio query-audio answer (AQAA) tasks. It can directly process audio input to generate natural and accurate speech responses, without relying on traditional automatic speech recognition (ASR) and text-to-speech (TTS) modules, simplifying the system architecture and eliminating cascading errors. The training process of Step-Audio-AQAA includes multimodal pre-training, supervised fine-tuning (SFT), direct preference optimization (DPO), and model merging. Through these methods, the model performs excellently in complex tasks such as speech emotion control, role-playing, and logical reasoning. In the StepEval-Audio-360 benchmark test, Step-Audio-AQAA surpasses existing LALM models in multiple key dimensions, demonstrating strong potential in end-to-end speech interaction.

Step-Audio-AQAA

Main Features of Step-Audio-AQAA

  • Direct Audio Input Processing: Can directly generate speech responses from raw audio input, without relying on traditional automatic speech recognition (ASR) and text-to-speech (TTS) modules.
  • Seamless Speech Interaction: Supports speech-to-speech interaction, users can ask questions by voice, and the model responds directly by voice, improving the naturalness and fluency of interaction.
  • Emotional Tone Adjustment: Supports adjusting the emotional tone of speech at the sentence level, such as expressing happiness, sadness, or seriousness.
  • Speech Rate Control: Users can adjust the speed of speech responses as needed, making them more suitable for the scenario requirements.
  • Timbre and Pitch Control: Can adjust the timbre and pitch of speech according to user instructions, adapting to different roles or scenarios.
  • Multilingual Interaction: Supports multiple languages such as Chinese, English, and Japanese, meeting the language needs of different users.
  • Dialect Support: Covers Chinese dialects such as Sichuanese and Cantonese, enhancing the model's applicability in specific regions.
  • Speech Emotion Control: Can generate speech responses with specific emotions based on context and user instructions.
  • Role-playing: Supports playing specific roles in conversations, such as customer service, teacher, friend, etc., generating speech responses that match the role's characteristics.
  • Logical Reasoning and Knowledge Q&A: Can handle complex logical reasoning tasks and knowledge Q&A, generating accurate speech responses.
  • High-Quality Speech Output: Generates high-fidelity, natural and fluent speech waveforms through a neural vocoder, improving user experience.
  • Speech Coherence: Maintains speech coherence and consistency in long sentence or paragraph generation, avoiding speech interruptions or abrupt changes.
  • Interleaved Text and Speech Output: Supports interleaved output of text and speech, users can choose speech or text responses as needed.
  • Multimodal Input Understanding: Can understand mixed inputs containing speech and text, and generate corresponding speech responses.

Technical Principles of Step-Audio-AQAA

  • Dual-Codebook Audio Tokenizer: Converts the input audio signal into a structured token sequence. It contains two tokenizers: the linguistic tokenizer extracts the phonemes and linguistic attributes of speech, sampled at 16.7 Hz with a codebook size of 1024; the semantic tokenizer captures the acoustic features of speech, such as emotion and intonation, sampled at 25 Hz with a codebook size of 4096. It can better capture complex information in speech.
  • Backbone LLM: Uses a pre-trained 130-billion-parameter multimodal LLM (Step-Omni), whose pre-training data covers three modalities: text, speech, and images. It embeds the dual-codebook audio tokens into a unified vector space and performs deep semantic understanding and feature extraction through multiple Transformer blocks.
  • Neural Vocoder: Synthesizes the generated audio tokens into natural, high-quality speech waveforms. It adopts a U-Net architecture combined with ResNet-1D layers and Transformer blocks, and can efficiently convert discrete audio tokens into continuous speech waveforms.

Project address of Step-Audio-AQAA

  • HuggingFace Model Repository：https://huggingface.co/stepfun-ai/Step-Audio-AQAA
  • arXiv Technical Paper：https://arxiv.org/pdf/2506.08967

Application scenarios of Step-Audio-AQAA

  • Emotional companion robot: Automatically adjusts the tone of responses according to the user's emotions, providing emotional support and companionship.
  • Multilingual intelligent customer service: Directly handles dialect speech queries and supports multiple languages (such as Chinese, English, Japanese, etc.).
  • Game NPC interaction: Generates speech feedback with emotional changes in real time. Enables dynamic switching of emotion and speaking rate within a single-turn conversation.
  • Intelligent voice assistant: Supports voice queries and voice answers, providing services such as real-time information queries and schedule reminders.
  • Education and entertainment: Used for voice teaching, story creation, poetry recitation, etc. in educational scenarios. It can flexibly switch between voice or text responses according to user needs.