SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillsrealtime-voice-chat

realtime-voice-chat

Coding & development

View repository

Updated 10-15 22:19

About this skill

Real-time AI Voice Dialogue Skill

Overview

Based on xiaozhi-esp32-server Pure API real-time voice dialogue capability extracted.

Two Usage Modes

  1. Deployment Mode — Deploy a complete voice dialogue service (server + client)
  2. Generation Mode — Embed voice dialogue client code into a new project

Core Flow

麦克风 → Opus编码 → WebSocket → 服务端[VAD→ASR→LLM→TTS] → Opus音频流 → 扬声器

Full-chain interruption support: automatically interrupts the AI reply when the user speaks.

Quickly Deploy the Server

Prerequisites

  • Python 3.10+
  • macOS: brew install opus portaudio ffmpeg
  • Linux: apt install libopus-dev portaudio19-dev ffmpeg

One-Click Installation

# 克隆 xiaozhi-esp32-server
git clone https://github.com/78/xiaozhi-esp32-server.git
cd xiaozhi-esp32-server/main/xiaozhi-server

# 运行安装脚本（或使用 skill 自带脚本）
bash ~/.claude/skills/realtime-voice-chat/scripts/setup_server.sh

Configuration

Edit data/.config.yaml, choose a configuration preset:

Preset File Suitable Scenario
Domestic Free config_presets/china_free.yaml ChatGLM + FunASR + EdgeTTS, zero cost
Domestic High Quality config_presets/china_quality.yaml Qwen ASR + SiliconFlow + Qwen-TTS voice cloning
Overseas OpenAI config_presets/global_openai.yaml Whisper + GPT-4 + OpenAI TTS
Fully Local config_presets/local_ollama.yaml FunASR + Ollama + GPT-SoVITS, fully offline

Preset file path: ~/.claude/skills/realtime-voice-chat/templates/config_presets/

Start the Service

cd xiaozhi-esp32-server/main/xiaozhi-server
source venv/bin/activate
python app.py
# 服务端监听: ws://0.0.0.0:8000/xiaozhi/v1/

Quickly Generate Client Code

Method 1: Complete Client (Recommended)

Copy templates/voice_chat_client.py(285 lines), ready to use out of the box:

cp ~/.claude/skills/realtime-voice-chat/templates/voice_chat_client.py ./
pip install websockets pyaudio opuslib
python voice_chat_client.py --server ws://localhost:8000/xiaozhi/v1/

Features: dialogue upon connection, automatic reply after speaking, automatically continues listening after replying.

Method 2: API Library Embedding

Copy templates/voice_chat_api.py(303 lines), embed into your own project:

from voice_chat_api import XiaozhiVoiceAPI

api = XiaozhiVoiceAPI("ws://localhost:8000/xiaozhi/v1/")
api.on_stt = lambda text: print(f"识别: {text}")
api.on_tts_text = lambda text: print(f"AI: {text}")
api.on_tts_audio = lambda data: play_opus(data)

session = await api.connect()
await api.start_listening(mode="auto")
# 持续发送 Opus 编码的麦克风数据
await api.send_audio(opus_frame)
await api.close()

Method 3: Minimal Example

Refer to templates/minimal_client.py(~50 lines), demonstrating the most core connection logic.

Architecture Overview

┌──────────────────────────────────────────────────┐
│                    客户端                         │
│  麦克风 → Opus编码 → WebSocket → Opus解码 → 扬声器  │
└──────────────┬───────────────────┬───────────────┘
               │ Opus音频帧        │ Opus音频帧
               ▼                   ▲
┌──────────────────────────────────────────────────┐
│                    服务端                         │
│                                                  │
│  ┌─────┐   ┌─────┐   ┌─────┐   ┌─────┐        │
│  │ VAD │ → │ ASR │ → │ LLM │ → │ TTS │        │
│  └─────┘   └─────┘   └─────┘   └─────┘        │
│  静音检测   语音转文字  生成回复   文字转语音      │
│                                                  │
│  ┌────────┐  ┌──────────┐                       │
│  │ Memory │  │ Intent   │                       │
│  └────────┘  └──────────┘                       │
│  对话记忆     意图识别(播放音乐/天气等)            │
└──────────────────────────────────────────────────┘

The server uses a queue-driven multithreaded model:

  • asr_audio_queue — Audio input buffer
  • tts_text_queue — Text output buffer
  • tts_audio_queue — Audio output buffer

Detailed architecture documentation: references/architecture.md

Configuration Guide

config.yaml Structure

server:                    # 服务端地址和端口
  ip: 0.0.0.0
  port: 8000

selected_module:           # 选择使用的 Provider
  VAD: SileroVAD
  ASR: FunASR              # 可选: FunASR, DoubaoASR, AliyunASR, OpenaiASR, ...
  LLM: ChatGLMLLM          # 可选: ChatGLMLLM, DeepSeekLLM, OllamaLLM, ...
  TTS: EdgeTTS             # 可选: EdgeTTS, DoubaoTTS, OpenAITTS, ...
  Memory: nomem            # 可选: nomem, mem0ai, powermem, mem_local_short
  Intent: function_call    # 可选: nointent, intent_llm, function_call

ASR:                       # 各 ASR Provider 配置
  FunASR: { type: fun_local, ... }

LLM:                       # 各 LLM Provider 配置
  ChatGLMLLM: { type: openai, ... }

TTS:                       # 各 TTS Provider 配置
  EdgeTTS: { type: edge, ... }

Memory:                    # 记忆 Provider 配置
  nomem: { type: nomem }

prompt: |                  # 系统提示词（角色设定）
  你是...

Complete configuration template: templates/config.yaml

Provider Quick Reference

ASR (Speech Recognition) — 11

Name type Deployment Method Features
FunASR fun_local Local model Free, requires downloading the model
FunASRServer fun_server Docker Independent deployment, high concurrency
SherpaASR sherpa_onnx_local Local Lightweight, multilingual
DoubaoASR doubao Cloud Volcano Engine, billed per use
DoubaoStreamASR doubao_stream Cloud Volcano Engine, real-time streaming
TencentASR tencent Cloud Tencent Cloud
AliyunASR aliyun Cloud Alibaba Cloud, batch
AliyunStreamASR aliyun_stream Cloud Alibaba Cloud, streaming
BaiduASR baidu Cloud Baidu
OpenaiASR openai Cloud Whisper/GPT-4o
Qwen3ASRFlash qwen3_asr_flash Cloud Tongyi Qianwen, multilingual

LLM (Large Language Model) — 9+

Name type Features
ChatGLMLLM openai Zhipu glm-4-flash,free
AliLLM openai Tongyi Qianwen qwen-turbo
DoubaoLLM openai Doubao doubao-1.5-pro
DeepSeekLLM openai DeepSeek-chat
OllamaLLM ollama Local deployment, fully offline
GeminiLLM gemini Google Gemini
DifyLLM dify Dify workflow
CozeLLM coze Coze Bot
FastgptLLM fastgpt FastGPT

TTS (Speech Synthesis) — 21

Name type Features
EdgeTTS edge Free, Microsoft Edge online synthesis
DoubaoTTS doubao Volcano Engine, rich voice tones
HuoshanDoubleStreamTTS huoshan_double_stream Volcano Engine bidirectional streaming
CosyVoiceSiliconflow siliconflow SiliconFlow CosyVoice
AliyunTTS aliyun Alibaba Cloud traditional TTS
AliyunStreamTTS aliyun_stream Alibaba Cloud CosyVoice streaming
AliBLTTS alibl_stream Alibaba Bailian CosyVoice
TencentTTS tencent Tencent Cloud
OpenAITTS openai OpenAI TTS
FishSpeech fishspeech Local, supports reference audio
GPT_SOVITS_V2 gpt_sovits_v2 Local, voice cloning
GPT_SOVITS_V3 gpt_sovits_v3 Local, v3 version
MinimaxTTSHTTPStream minimax_httpstream Minimax streaming
XunFeiTTS xunfei_stream iFlytek ultra-realistic
QwenTTSClone qwen_tts_clone Voice replication，DashScope
CustomTTS custom Custom HTTP interface
LinkeraiTTS linkerai Linkerai, supports cloning
PaddleSpeechTTS paddle_speech Baidu PaddlePaddle, offline
IndexStreamTTS index_stream Index-TTS streaming
ACGNTTS ttson Anime voice tones
CozeCnTTS cozecn Coze TTS

Memory — 4

Name type Features
nomem nomem No memory used
mem0ai mem0ai Mem0 cloud, 1000 times/month free
powermem powermem OceanBase open-source memory component
mem_local_short mem_local_short Local memory, privacy-first

Detailed Provider documentation: references/providers-*.md

WebSocket Protocol Summary

Connection

ws://{host}:{port}/xiaozhi/v1/
Headers: Device-Id, Client-Id, Protocol-Version: 1
Optional: Authorization: Bearer {token}

Handshake

// 客户端 → 服务端
{"type":"hello","version":1,"transport":"websocket",
 "audio_params":{"format":"opus","sample_rate":24000,"channels":1,"frame_duration":60}}

// 服务端 → 客户端
{"type":"hello","session_id":"uuid-..."}

Core Messages

Direction Type Description
C→S listen start/stop Start/stop receiving audio
C→S abort Interrupt AI speaking
C→S Binary Opus audio frames
S→C stt Speech recognition result
S→C llm LLM emotion/state
S→C tts start/sentence_start/stop TTS state
S→C Binary TTS Opus audio frames

Listen mode

  • auto — VAD automatically detects end of speech (recommended)
  • manual — manually call stop
  • realtime — full-duplex (requires AEC)

Detailed protocol documentation: references/websocket-protocol.md

Voice Cloning

Use DashScope Qwen-TTS for voice replication:

  1. Prepare 10~20 seconds of clear human voice audio (mp3/wav)
  2. Configure TTS.QwenTTSClone： 
    TTS:
  QwenTTSClone:
    type: qwen_tts_clone
    api_key: <DashScope API Key>
    voice_audio_path: /path/to/voice.mp3
    voice_name: my_voice
  3. Automatically call the replication API on service startup to obtain voice_id

Detailed voice cloning documentation: references/voice-cloning.md Standalone example: examples/voice_clone_example.py

Code Generation Guidelines

When generating voice conversation-related code for users, follow these guidelines:

Audio Parameters

# 麦克风输入
SAMPLE_RATE_INPUT = 16000   # 16kHz
CHANNELS = 1                # 单声道
FRAME_DURATION_MS = 60      # 60ms
FRAME_SIZE_INPUT = 960      # 16000 * 60 / 1000

# TTS 输出
SAMPLE_RATE_OUTPUT = 24000  # 24kHz
FRAME_SIZE_OUTPUT = 1440    # 24000 * 60 / 1000

# 编码
FORMAT = "opus"             # Opus 编码

Required Dependencies

websockets>=14.0    # WebSocket 客户端
opuslib             # Opus 编解码
pyaudio             # 音频 I/O（CLI 客户端需要）

Callback Mode

All asynchronous events are handled through callbacks:

api.on_connected    # 连接建立
api.on_stt          # 语音识别结果 (text)
api.on_llm          # LLM 情感 (emotion, text)
api.on_tts_start    # TTS 开始
api.on_tts_text     # TTS 句子文本 (text)
api.on_tts_audio    # TTS 音频帧 (opus_data: bytes)
api.on_tts_stop     # TTS 结束
api.on_disconnected # 连接断开

State Machine

IDLE → CONNECTING → CONNECTED ⇄ LISTENING ⇄ SPEAKING → CLOSED

Common Tasks

"Help me set up a real-time voice conversation service"

→ Use the quick deployment server process + select an appropriate configuration preset

"Add voice conversation functionality to my project"

→ Copy voice_chat_api.py + refer to the minimal example

"The voice doesn't sound similar/poor replication effect"

→ Check the audio sample quality, see references/voice-cloning.md

"Inaccurate recognition"

→ Try changing the ASR Provider, see references/providers-asr.md

"Latency too high"

→ Check the Provider combination + network, see references/troubleshooting.md

"How to run fully offline"

→ Use the local_ollama.yaml preset: FunASR + Ollama + GPT-SoVITS