SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillsfunasr-transcribe

funasr-transcribe

Office & productivity

View repository

Updated 10-15 20:10

About this skill

FunASR Speech-to-Text

This skill provides a local speech recognition service that converts audio or video files into structured Markdown documents.

Feature Overview

  • Supports multiple audio and video formats (mp4, mov, mp3, wav, m4a, flac, etc.)
  • Automatically generates timestamps
  • Supports speaker diarization (enabled by default)
  • ONNX acceleration mode: supports paraformer-onnx and the experimental SenseVoice-Small ONNX
  • Single-speaker fast mode：--fast / "fast": true disables diarization, and by default still uses paraformer
  • Paraformer ONNX post-processing optimization：paraformer-onnx Both the single-speaker and multi-speaker paths first perform VAD segmentation, then clean the text output, restore punctuation, and output sentence-level timestamps; the single-speaker path uses global punctuation restoration, while the multi-speaker path uses per-segment punctuation to preserve speaker alignment
  • Video keyframe screenshot extraction: automatically detects and extracts PPT slides and inserts them at the corresponding positions in the transcript (automatically enabled for video files)
  • After transcription, an AI summary prompt is automatically included, allowing the Agent to complete the summary in one step
  • Outputs in Markdown format, making it easy to read and edit

Dependencies

System Dependencies

Dependency Installation Method
Python 3.8+ macOS: brew install python@3.14
curl macOS usually includes it; if missing, you can run brew install curl

Python Packages

Package Name Purpose Installation Command
funasr FunASR native inference and CAM++ diarization pip install -r assets/requirements.txt
funasr-onnx Paraformer / SenseVoice ONNX acceleration pip install -r assets/requirements.txt
scenedetect[opencv]、imagehash Video keyframe extraction pip install -r assets/requirements.txt

When ONNX mode needs to be run for the first time, simply execute:

python3 scripts/setup.py

This will install both funasr-onnx and its dependencies;SenseVoiceSmall is downloaded on demand only when model=sensevoice is explicitly specified.

ONNX Quality Parameter Tuning

paraformer-onnx By default, a quality-first parameter combination is used; the single-speaker path reuses the multi-speaker path's ONNX VAD segmented ASR but does not perform CAM++ speaker clustering:

Parameter Default Value Description
FUNASR_ONNX_TEXT_SOURCE preds Use the cleaned ONNX preds text; if anomalies occur, it can be set to raw_tokens to fall back
FUNASR_SERVER_ONNX_THREADS 4 Number of ONNX Runtime inference threads, mainly affects speed and does not directly improve recognition quality
FUNASR_ONNX_COMPAT_CACHE ~/.cache/funasr-onnx-compat ONNX-compatible export cache directory; compatible export will copy the model directory, and the cache can be deleted and regenerated

Single-speaker paraformer-onnx first concatenates the recognized text of each VAD segment, then performs global punctuation restoration once; this is closer to native paraformerthan restoring punctuation segment by segment, and can also reduce the time spent repeatedly calling the punctuation model.

ONNX sentence-level timestamps are approximate mappings based on character positions and token timestamps. They are suitable for locating paragraphs and speaking turns, and should not be regarded as word-by-word forced alignment results.

The following tuning directions have been verified and are not recommended as defaults:

  • Increasing the VAD silence threshold will reduce segmentation and speed things up, but on a 90-second multi-speaker sample, text similarity drops noticeably.
  • Merging adjacent VAD segments or transcribing the entire segment is more prone to typos, repetition, and long-audio collapse, so neither single-speaker nor multi-speaker ONNX transcribes the entire segment by default.
  • Adding extra padding to VAD segments introduces boundary repetition, and the overall quality is not as good as the default segmentation.

Agent default workflow (transcription + automatic summarization)

When the user requests transcription of audio/video, the following process should be followed,complete transcription and AI summarization in one go：

Prerequisite step (must be executed first): set PATH. In some execution environments (such as agent-executor headless mode), PATH is restricted to only the plugin directory,curl、python3 and other system commands cannot be found. You must first execute:

export PATH="/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:$PATH"

After that, all bash commands must be preceded in the same command block by export PATH=... , or this line must be added at the beginning of every command block.

Step 0: Environment detection (automatic)

Before performing transcription, check whether assets/skill-env.json exists. If it does not exist, first run environment detection:

export PATH="/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:$PATH" && cd <skill目录> && python3 scripts/init_env.py

If detection fails (non-zero exit code), run the installation script as prompted:

export PATH="/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:$PATH" && cd <skill目录> && python3 scripts/setup.py

After installation completes, it will automatically re-detect and generate skill-env.json。

Step 1: Start/check the service

export PATH="/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:$PATH" && curl -s http://127.0.0.1:8765/health

If the service is not running, start it in the background:

export PATH="/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:$PATH" && cd <skill目录> && python3 scripts/server.py --idle-timeout 600 &

Wait for the service to be ready (poll /health until it returns 200).

Step 2: Transcribe the file

export PATH="/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:$PATH" && curl -s -X POST http://127.0.0.1:8765/transcribe \
  -H "Content-Type: application/json" \
  -d '{"file_path": "/path/to/audio.aac"}'

Note:diarize defaults to true, no need to pass it explicitly. To disable, pass "diarize": false. Video files (mp4, mov, etc.) automatically enable keyframe screenshot extraction (extract_slides), no need to pass it manually. To disable, explicitly pass "extract_slides": false. For single-speaker lectures/voice, you can pass "fast": true to turn off speaker diarization; by default it still uses paraformer；"model": "sensevoice" only as an experimental explicit option.

The response includes the following key fields:

  • output_path: path to the Markdown file output by transcription
  • text: full transcription text
  • summary_prompt: AI summarization prompt (is automatically attached, no need to additionally call /summary）
  • text_preview: preview of the first 500 characters of the transcribed text

Step 3: Generate AI summary

Based on summary_prompt(or directly based on the content of text ), the Agent generates a structured JSON summary:

{
  "full_summary": "至少400字，分成2-3段，交代背景、问题、关键事实、数据、风险与行动建议",
  "speaker_summary": [
    {
      "speaker_order": "发言人1",
      "speaker_name": "如能识别请写姓名，否则写未知",
      "summary": "至少180字，涵盖该发言人的观点、依据、数据、态度与潜在影响"
    }
  ],
  "highlights": ["6-10条重点，每条60-100字"],
  "keywords": ["5-8个关键词"]
}

Step 4: Inject the summary into the file

Important: do not just describe the injection operation; you must actually execute the following command.

Write the JSON generated in step 3 to a temporary file, then call the script to inject it (more reliable than curl injection, no JSON escaping needed):

export PATH="/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:$PATH" && cat > /tmp/summary_<文件名>.json << 'JSONEOF'
{步骤3生成的JSON内容}
JSONEOF
python3 <skill目录>/scripts/summary.py inject "<output_path>" /tmp/summary_<文件名>.json

The script will automatically:

  • parse the JSON and format it as Markdown
  • inject it into the correct position in the Markdown file
  • add the <!-- AI-SUMMARY:START --> / <!-- AI-SUMMARY:END --> marker

Step 5: Verify the injection result (must be executed)

After injection, verification must be performed to confirm that the summary was actually written to the file. If verification fails, step 4 must be retried.

export PATH="/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:$PATH" && python3 <skill目录>/scripts/summary.py verify "<output_path>"
  • If it outputs ✅ 摘要已存在 → success, report completion to the user
  • If it outputs ❌ 摘要不存在 → failure, return to step 4 and retry

Complete workflow example

用户：转录这个音频
  ↓
Agent：
  1. 检查/启动服务
  2. POST /transcribe {"file_path": "xxx.aac"}  ← 一次调用拿到转录+提示词
  3. 根据转录内容直接生成总结 JSON
  4. 写 JSON 到临时文件 → python3 summary.py inject 注入
  5. python3 summary.py verify 验证 → 失败则重试步骤 4
  ↓
用户：收到带 AI 总结的 Markdown 文件

Usage workflow

First use: environment detection and dependency installation

Important: before first use, you must first check whether the environment meets the requirements.

Run environment detection:

python3 scripts/check_env.py

The detection script checks the following environment requirements:

Required Item Requirement Detection Command
Python >= 3.8，python3 command available python3 --version
curl HTTP client (for API calls) curl --version
Basic commands ls, ps, grep shell built-in

If environment detection fails:

  1. Python3 command unavailable：

    # macOS 使用 homebrew 安装 Python
brew install python@3.14

  2. curl unavailable：

    # macOS 确保 curl 已安装
brew install curl

  3. After verifying the environment is fixed, rerun the detection:

    python3 scripts/check_env.py

First use: install dependencies and download models

Run the installation script to complete environment setup:

python3 scripts/setup.py

The installation script automatically:

  1. Checks the Python version (requires >= 3.8)
  2. Installs dependency packages (FastAPI, Uvicorn, FunASR, funasr-onnx, PyTorch)
  3. Downloads the ASR model to ~/.cache/modelscope/hub/models/

Verify installation status:

python3 scripts/setup.py --verify

Start the transcription service

python3 scripts/server.py

To enable ONNX acceleration and INT8 quantization by default, use:

python3 scripts/server-onnx.py --preload

The service runs by default on http://127.0.0.1:8765

Smart features:

  • Auto-start: automatically loads the model on the first request
  • Idle shutdown: by default, automatically shuts down after 10 minutes of inactivity to save resources
  • Configurable timeout: use the --idle-timeout parameter to customize the idle timeout (seconds)

Service lifecycle:

  1. After startup, enters idle monitoring state
  2. Automatically loads the model and performs transcription when a request is received
  3. Each request resets the idle timer
  4. Automatically shuts down after 10 consecutive minutes without requests
  5. Restarts on the next request

Important note:

  • ⚠️ Do not manually shut down the service - after transcription is complete, let the service keep running; it will automatically shut down after 10 minutes of inactivity
  • This allows you to transcribe multiple files consecutively without repeatedly starting the service
  • If you need to shut down the service immediately, press Ctrl+C or wait for the 10-minute idle timeout

Example: custom 30-minute idle timeout

python3 scripts/server.py --idle-timeout 1800

Run transcription

Use the client script to transcribe files:

# 转录单个文件
python3 scripts/transcribe.py /path/to/audio.mp3

# 指定输出路径
python3 scripts/transcribe.py /path/to/video.mp4 -o transcript.md

# 启用说话人分离
python3 scripts/transcribe.py /path/to/meeting.m4a --diarize

# Paraformer ONNX（更快；默认仍支持 diarization）
python3 scripts/transcribe.py /path/to/meeting.m4a --model paraformer-onnx

# Paraformer ONNX 单人路径（VAD 分段 ASR，不做说话人聚类）
python3 scripts/transcribe.py /path/to/course.m4a --model paraformer-onnx --no-diarize

# 单人讲课快速模式（关闭说话人分离，保留默认 Paraformer）
python3 scripts/transcribe.py /path/to/course.m4a --fast

# 批量转录目录
python3 scripts/transcribe.py /path/to/media_folder/

# 提取视频关键帧截图（PPT幻灯片）
python3 scripts/transcribe.py /path/to/video.mp4 --slides

# 自定义场景检测阈值（值越低越灵敏，默认20.0）
python3 scripts/transcribe.py /path/to/video.mp4 --slides --slide-threshold 15.0

AI Smart Summary (Claude Code Environment)

After transcription is complete, you can generate an AI smart summary, making full use of Claude Code's native AI capabilities.

Automatic mode (recommended):

Use the --auto-summary parameter to automatically generate and inject a summary after transcription is complete:

# 转录并自动生成总结（Claude Code 原生环境，无需配置 API Key）
python3 scripts/transcribe.py /path/to/audio.m4a --auto-summary

# 完整流程：说话人分离 + 自动总结
python3 scripts/transcribe.py /path/to/meeting.m4a --diarize --auto-summary

How it works:

  • The script outputs a structured summary request (AI_SUMMARY_REQUEST）
  • Claude Code automatically recognizes it and uses built-in AI capabilities to generate the summary
  • No external API Key configuration is required

Manual mode:

  1. After running transcription, the script automatically prepares the summary prompt
  2. Send the prompt to Claude AI to generate a structured summary
  3. Paste the JSON result returned by Claude back into the script
  4. Automatically injects the summary into the Markdown file
# 转录单个文件（输出提示词供手动调用）
python3 scripts/transcribe.py /path/to/audio.mp3

# 禁用自动总结（只输出提示词）
python3 scripts/transcribe.py /path/to/audio.m4a --no-summary

Summary content structure:

  • Full-text summary - 400+ characters, including background, issues, and key facts
  • Speaker summary - each speaker's views, attitude, and contributions
  • Key content - 6-10 core key points
  • Keywords - 5-8 key terms

Prompt features:

  • Specifically optimized for Chinese colloquial conversations
  • Preserves speaker context and conversation flow
  • Structured JSON output for easy parsing and formatting

For detailed documentation, see: <references/api-reference.md>

Call via HTTP API

Check service status：

curl http://127.0.0.1:8765/health

Use curl to call the API directly:

curl -X POST http://127.0.0.1:8765/transcribe \
  -H "Content-Type: application/json" \
  -d '{"file_path": "/path/to/audio.mp3"}'

# 单人快速模式（关闭说话人分离，保留默认 Paraformer）
curl -X POST http://127.0.0.1:8765/transcribe \
  -H "Content-Type: application/json" \
  -d '{"file_path": "/path/to/course.m4a", "fast": true}'

# 指定 Paraformer ONNX（默认启用 diarization）
curl -X POST http://127.0.0.1:8765/transcribe \
  -H "Content-Type: application/json" \
  -d '{"file_path": "/path/to/meeting.m4a", "model": "paraformer-onnx"}'

# Paraformer ONNX 单人路径（VAD 分段 ASR，不做说话人聚类）
curl -X POST http://127.0.0.1:8765/transcribe \
  -H "Content-Type: application/json" \
  -d '{"file_path": "/path/to/course.m4a", "model": "paraformer-onnx", "diarize": false}'

# 提取视频关键帧截图
curl -X POST http://127.0.0.1:8765/transcribe \
  -H "Content-Type: application/json" \
  -d '{"file_path": "/path/to/video.mp4", "extract_slides": true}'

API Documentation (Swagger UI)：

FastAPI automatically generates interactive API documentation, visit:http://127.0.0.1:8765/docs

On this page you can:

  • View all API endpoints
  • Test the API online (no curl needed)
  • View request/response formats
  • View detailed parameter descriptions

Response example(health check):

{
  "status": "ok",
  "service": "FunASR Transcribe",
  "uptime": 300,
  "idle_time": 120
}

Returned field descriptions:

  • uptime: Service uptime (seconds)
  • idle_time: Current idle time (seconds)

Complete API documentation

For detailed API reference documentation, see: <references/api-reference.md>

Includes:

  • Complete specification of all API endpoints
  • Detailed explanation of request/response formats
  • Parameter descriptions and examples
  • Complete curl command examples

Script descriptions

Script Purpose
scripts/init_env.py Environment detection + generate skill-env.json
scripts/check_env.py Environment detection (simplified version)
scripts/setup.py One-click dependency installation and model download
scripts/server.py Start the HTTP API service
scripts/server-onnx.py Start the default ONNX acceleration service
scripts/transcribe.py Command-line client
scripts/auto_transcribe.py Automated transcription script (recommended)

Automated transcription + summarization workflow

This skill supports automatic completion on any Agent platformtranscription + summarizationfull workflow.

Option 1: Use the automation script (recommended)

# 自动转录 + 获取总结提示词（说话人分离默认启用）
python3 scripts/auto_transcribe.py /path/to/audio.aac

# 禁用说话人分离
python3 scripts/auto_transcribe.py /path/to/audio.aac --no-diarize

# 单人快速模式（关闭说话人分离，保留默认 Paraformer）
python3 scripts/auto_transcribe.py /path/to/course.m4a --fast

# 只获取总结提示词，不生成总结
python3 scripts/auto_transcribe.py /path/to/audio.aac --prompt-only

Option 2: HTTP API call

1. Transcribe audio (the response automatically includes the summarization prompt)

curl -X POST http://127.0.0.1:8765/transcribe \
  -H "Content-Type: application/json" \
  -d '{"file_path": "/path/to/audio.aac"}'

The response includes the summary_prompt field, which can be used directly to generate a summary without an additional call to /summary。

2. Inject AI summary

After generating the summary, call:

curl -X POST http://127.0.0.1:8765/inject_summary \
  -H "Content-Type: application/json" \
  -d '{
    "md_path": "/path/to/audio.md",
    "summary_content": "## AI 摘要\n\n### 全文总结\n...\n\n### 重点内容\n- ...\n\n### 关键词\n..."
  }'

API endpoint summary

Endpoint Method Function
/health GET Health check
/transcribe POST Transcribe audio/video
/batch_transcribe POST Batch transcribe directory
/summary POST Generate AI summarization prompt
/inject_summary POST Inject summary into Markdown file
/verify_summary POST Verify whether the summary has been injected

Configuration files

File Description
assets/models.json ASR model configuration manifest
assets/requirements.txt Python dependency manifest

Output format

Transcription results are saved as Markdown files, including:

  1. Title - Filename (without transcription timestamp)
  2. Transcription content - Format:发言人N HH:MM:SS newline 内容
  3. AI summary(optional) - includes full-text summary, speaker summary, key content, keywords

Example format (video with screenshots):

# 转录：视频.mp4

## 转录内容

发言人1 00:02:49
![](slides/slide_001_02m49s.jpg)
各位好，今天我们来讲...

发言人1 00:03:30
![](slides/slide_002_03m30s.jpg)
这是第二段的内容...

Model information

Models are stored in the ModelScope default cache directory ~/.cache/modelscope/hub/models/：

  • ASR main model (Paraformer) - 867MB
  • SenseVoice-Small (experimental single-speaker ONNX path) - downloaded on demand when explicitly specified
  • VAD model - 4MB
  • Punctuation model - 283MB
  • Speaker diarization model - 28MB

STT transcription priority (important)

Correct order: FunASR (preferred) → Whisper CLI (fallback)

  • FunASR is the primary choice: higher Chinese recognition quality, supports timestamps, speaker diarization, and video keyframes
  • Whisper CLI is the fallback: only used when the FunASR service is unavailable (for example, funasr-onnx installation failure, service error 500)
  • Never: use Whisper directly without first trying FunASR

Troubleshooting steps when FunASR fails

  1. Run python3 scripts/setup.py --verify to check whether funasr-onnx is available
  2. View the service process logs:process_log view proc_<session_id>
  3. If funasr-onnx cannot be installed, use Whisper CLI as a temporary fallback (see below)

Whisper CLI Fallback (only when FunASR is unavailable)

# 提取音频（16kHz 单声道）
ffmpeg -i "/path/to/video.mp4" -vn -acodec pcm_s16le -ar 16000 -ac 1 -y "/tmp/audio.wav"

# Whisper 转录（tiny 模型最快，medium 质量更好）
/opt/homebrew/bin/whisper "/tmp/audio.wav" \
  --model tiny \
  --language Chinese \
  --output_dir /tmp/transcript \
  --output_format all

Performance reference: 19 minutes of audio, tiny model takes about 3-5 minutes (Mac CPU).

Troubleshooting

cv2 / opencv import failure

Symptom：POST /transcribe returns {"detail":"No module named 'cv2'"}, but pip show opencv-python-headless shows it is installed.

Root cause: the Python environment used by the service process is different from the pip installation target. This commonly occurs in macOS Homebrew Python 3.14 environments, where pip installs to the system site-packages, but the service process loads from the Homebrew path.

Troubleshooting steps：

  1. Verify in the terminal whether cv2 can be imported:python3 -c "import cv2; print('ok')"
  2. If the import fails, run:python3 -m pip install opencv-python-headless --break-system-packages
  3. Confirm the Python path of the service process:lsof -p <server_pid> | grep python

Correct startup procedure：

# 确认 cv2 可用后再启动服务
python3 -c "import cv2; print('cv2 ok')"

# 如服务已在运行，先杀掉再重启
lsof -ti:8765 | xargs kill -9 2>/dev/null; sleep 1

# 重启服务
python3 scripts/server.py --idle-timeout 600 &

Service port is occupied

Symptom：Address already in use（Errno 48）

# 杀掉占用端口的进程
lsof -ti:8765 | xargs kill -9 2>/dev/null
sleep 1

FunASR service not responding / slow model loading

The first transcription requires downloading the model (about 1-2GB), so wait patiently. For subsequent requests the model is already cached, so it will be much faster.

Video screenshot feature:

Keyframe extraction is automatically enabled when transcribing video files (mp4, mov, avi, mkv, wmv, webm). Dependencies scenedetect[opencv] and imagehash are already included in requirements.txt,setup.py they will be installed together during installation. If these dependencies are not installed, the server will output a prompt but this will not affect normal transcription functionality.

When service startup fails, run the verification command to check the installation status:

python3 scripts/setup.py --verify

Re-download the model:

python3 scripts/setup.py --skip-deps