SuperAPI AI Directory
Partner with us中文AI newsFavorites
Skillslocal-asr

local-asr

Office & productivity

View repository

Updated 10-15 20:10

About this skill

Local ASR speech-to-text (MOSS by default)

This skill provides a local speech recognition service that converts audio or video files into structured Markdown documents. This skill was formerly named funasr-transcribe; it now uses local-asr as the unified entry point, with MOSS-MLX as the default engine and FunASR as a retained backend that can be explicitly selected.

Feature overview

  • Supports multiple audio and video formats (mp4, mov, mp3, wav, m4a, flac, etc.)
  • Automatically generates timestamps
  • Supports speaker diarization (enabled by default)
  • ONNX acceleration mode: supports paraformer-onnx and the experimental SenseVoice-Small ONNX
  • MOSS-MLX default mode: on Apple Silicon, uses MOSS to simultaneously transcribe, mark timestamps, and distinguish anonymous speakers; after long recordings are segmented, CAM++ is used to link labels across segments
  • FunASR retained mode: explicitly specify --model paraformer or --model paraformer-onnx to invoke the original pipeline
  • Single-speaker fast mode：--fast / "fast": true disables diarization without changing the selected model
  • Paraformer ONNX post-processing optimization：paraformer-onnx Both the single-speaker and multi-speaker paths first perform VAD segmentation, then clean the text output, restore punctuation, and output sentence-level timestamps; the single-speaker path uses global punctuation restoration, while the multi-speaker path uses per-segment punctuation to preserve speaker alignment
  • Video keyframe screenshot extraction: enabled automatically by default for videos,--no-slides can skip the time-consuming screenshot step
  • Voiceprint enrollment and speaker recognition: after your own voiceprint is enrolled via claiming, subsequent transcriptions automatically recognize and directly label the enrolled name; see references/speaker-registry.md
  • After transcription, an AI summary prompt is automatically attached, allowing the Agent to complete the summary in one step
  • The API returns elapsed time for stages such as transcription, screenshots, archiving, and summarization; include_summary_prompt=false can skip summary preparation
  • Outputs Markdown format for easy reading and editing

Dependencies

System dependencies

Dependency Installation method
Native Apple Silicon Python 3.10–3.12 (3.11 recommended; the full legacy pipeline also requires this range) macOS: brew install python@3.11
curl Usually included with macOS; if missing, run brew install curl
ffmpeg (required for MOSS audio normalization, also used for video processing) macOS: brew install ffmpeg

Python packages

Package name Purpose Installation command
funasr Native FunASR inference and CAM++ diarization pip install -r assets/requirements.txt
funasr-onnx Paraformer / SenseVoice ONNX acceleration pip install -r assets/requirements.txt
opencv-python、imagehash Video keyframe extraction pip install -r assets/requirements.txt
mlx-audio[stt] Default MOSS backend on Apple Silicon python3 -m pip install -r assets/requirements-moss-mlx.txt

The default installation on Apple Silicon retains FunASR/CAM++ dependencies while also installing MOSS-MLX;python3 scripts/setup.py --legacy installs only the original FunASR pipeline. When using the legacy pipeline on non-Apple Silicon, set FUNASR_SERVER_DEFAULT_MODEL=paraformerbefore starting the service, or run server-onnx.py; the CLI must still explicitly pass --model paraformer(or --model paraformer-onnx）。 funasr-onnx 0.4.1 requires NumPy ≤1.26.4, so installing both backends together requires Python 3.10–3.12; a Python 3.13+ environment with NumPy 2 cannot be guaranteed to make the legacy ONNX route usable.

On first install or when replacing the backend, follow references/model-backends.md to verify the device, dependencies, weight IDs, download timing, and enablement commands; below is the shortest installation path for the default backend.

Use a virtual environment inside the skill directory that is ignored by Git, to avoid installing MLX-Audio dependencies into the system Python:

cd <skill目录>
/opt/homebrew/bin/python3.11 -m venv venv
source venv/bin/activate
python3 scripts/setup.py

From now on, run python3 scripts/server.py and python3 scripts/auto_transcribe.pyfrom the same environment. Users who have already configured an old environment can install the MOSS dependencies in that environment; if this machine already uses venv/, you can run venv/bin/python。

If you already have an old environment, before using the default MOSS for the first time, startin the same Python environment server.py to launchand install the MOSS dependencies:

python3 -m pip install -r assets/requirements-moss-mlx.txt

setup.py downloads the MOSS weights and CAM++ from ModelScope by default; if model download is skipped, the first call will preferentially fetch MOSS from Hugging Face, and fall back to ModelScope on failure. Explicitly selecting the old FunASR pipeline does not require the MOSS dependencies. The MLX backend only supports Apple Silicon macOS; long recordings require the existing CAM++ model to link segmented speakers. MOSS's [S01] labels such as these only indicate anonymous speakers within the same recording and do not identify real names.

For offline operation or fixed model paths, you can pre-download models via the installed modelscope and pass the actual output directory to --model-id：

python3 -c "from modelscope.hub.snapshot_download import snapshot_download; print(snapshot_download('OpenMOSS/MOSS-Transcribe-Diarize'))"
python3 scripts/transcribe.py /path/to/meeting.m4a --model-id /path/to/downloaded/model

Long MOSS recordings are split by default at about 600 seconds, choosing continuous silence near the target point;FUNASR_MOSS_CHUNK_SECONDS=300 can shorten it to about 5 minutes (service restart required). Segmented speakers are linked by CAM++ voiceprint heuristics, and short interjections may still be split into multiple labels; speakers with less than 3 seconds of total speech will be flagged in warnings for manual review. Errors are raised when output timestamps are out of range, labels are missing, or the generation limit is reached; entire segments with near-zero waveform energy skip inference, and hallucinated sentences are removed with a warning. No speaker accuracy is guaranteed for actual meetings; real samples require manual review.

MOSS Performance and Optional Quantization

The original MOSS weights are used by default. MLX-Audio can convert local weights into 4-bit or 8-bit models; quantization mainly reduces weight and runtime memory,does not guarantee being faster or maintaining the same speaker labels. On an M1 Max, with about 4.5 minutes of repeated synthetic two-speaker speech, the original weights took 13.4/14.7 seconds across two runs, 4-bit took 18.4/19.4 seconds, and 8-bit took 21.6/20.3 seconds; the quantized models were slower. Quantized output on short real call clips also produced extra short speaker labels, without manual ground-truth verification. This test cannot replace acceptance testing on real long recordings.

If device memory is insufficient, you can generatea separate quantized copy from the downloaded original model directory, then explicitly select it (the directory should be placed outside the skill repository; the first conversion requires sufficient temporary disk space):

python3 -m mlx_audio.convert --hf-path /path/to/original-moss-model --mlx-path /path/to/moss-q8 --quantize --q-bits 8 --model-domain stt
python3 scripts/transcribe.py /path/to/meeting.m4a --model-id /path/to/moss-q8

For the resident service, you can set FUNASR_MOSS_MODEL_ID=/path/to/moss-q8 and then restart; to restore the default, remove that environment variable. In API requests, quantize only controls the old ONNX INT8 route,will notquantize MOSS; the CLI has no automatic MOSS quantization switch.FUNASR_MOSS_CHUNK_SECONDS can adjust the output budget and split points between 60–600 seconds, but shorter segments increase the risk of CAM++ cross-segment linking and split points, and the default should not be changed based on speed alone.

Required Permissions and Security Notes

Reads user-specified local audio and video, writes Markdown, screenshots, and archive/under the skill directory. The service by default only listens on 127.0.0.1:8765; the CLI calls this service via local HTTP and does not upload audio to the cloud. The first time dependencies or models are installed, it will access the package index, ModelScope, or Hugging Face over the network; afterward, the local cache can be reused. The script will call ffmpeg, Python subprocesses, and local file operations; no ASR API Key is required. The archive/ containing customer recordings and the locally generated assets/skill-env.json are for local use only and are not sharing materials.

When ONNX mode needs to be run for the first time, simply execute:

python3 scripts/setup.py --legacy

to install funasr-onnx and its dependencies at the same time;SenseVoiceSmall is downloaded on demand only when model=sensevoice is explicitly specified.

ONNX quality tuning

paraformer-onnx uses a quality-first parameter combination by default; the single-speaker path reuses the multi-speaker path's ONNX VAD segmented ASR, but does not perform CAM++ speaker clustering:

Parameter Default value Description
FUNASR_ONNX_TEXT_SOURCE preds Use the cleaned ONNX preds text; if anomalies occur, it can be set to raw_tokens to fall back
FUNASR_SERVER_ONNX_THREADS 4 Number of ONNX Runtime inference threads, mainly affects speed and does not directly improve recognition quality
FUNASR_ONNX_COMPAT_CACHE ~/.cache/funasr-onnx-compat ONNX-compatible export cache directory; compatible export copies the model directory, and the cache can be deleted and regenerated

Single-speaker paraformer-onnx first concatenates the recognized text of each VAD segment, then performs one global punctuation restoration; this is closer to native paraformerthan restoring punctuation segment by segment, and also reduces the time spent repeatedly calling the punctuation model.

ONNX sentence-level timestamps are an approximate mapping based on character positions and token timestamps. They are suitable for locating paragraphs and speaker turns and should not be regarded as word-by-word forced alignment results.

The following tuning directions have been verified and are not recommended as defaults:

  • Increasing the VAD silence threshold reduces segmentation and speeds things up, but on a 90-second multi-speaker sample, text similarity drops noticeably.
  • Merging adjacent VAD segments or transcribing the entire segment is more prone to typos, repetition, and long-audio collapse, so neither single-speaker nor multi-speaker ONNX defaults to whole-segment transcription anymore.
  • Adding extra padding to VAD segments introduces boundary repetition, and the overall quality is worse than the default segmentation.

Agent default workflow (transcription + automatic summarization)

When the user requests audio/video transcription, the following process should be followed,completing transcription and AI summarization in one go：

Prerequisite step (must be executed first): enter the skill directory and set PATH. In some execution environments (such as agent-executor headless mode), PATH is restricted to only the plugin directory, andcurl、python3 system commands such as these cannot be found. You must first execute:

cd <skill目录> && export PATH="$PWD/venv/bin:/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:$PATH"

Afterward, all bash commands must follow this line in the same command block, or be repeated at the beginning of each command block.venv/ If it does not exist, first create and install it as described above.

Step 0: Environment detection (automatic)

Before executing transcription, check whether assets/skill-env.json exists, and whether FUNASR_SERVER_DEFAULT_MODEL in it is moss-mlx、FUNASR_PYTHON pointing to a Python with MLX-Audio installed. Old configurations need to add --force to re-detect:

cd <skill目录> && export PATH="$PWD/venv/bin:/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:$PATH" && python3 scripts/init_env.py

If the configuration still points to the old environment, run python3 scripts/init_env.py --force, and confirm that FUNASR_PYTHON in it points to venv/bin/python。

If detection fails (non-zero exit code), run the installation script as prompted:

cd <skill目录> && export PATH="$PWD/venv/bin:/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:$PATH" && python3 scripts/setup.py

After installation completes, it will automatically re-detect and generate skill-env.json。

Step 1: Start/check the service

cd <skill目录> && export PATH="$PWD/venv/bin:/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:$PATH" && curl -s http://127.0.0.1:8765/health

If the service is not running, start it in the background:

cd <skill目录> && export PATH="$PWD/venv/bin:/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:$PATH" && python3 scripts/server.py --idle-timeout 600 &

Wait for the service to be ready (poll /health until it returns 200).

Step 2: Transcribe the file

cd <skill目录> && export PATH="$PWD/venv/bin:/opt/homebrew/bin:/usr/bin:/bin:/usr/sbin:/sbin:$PATH" && curl -s -X POST http://127.0.0.1:8765/transcribe \
  -H "Content-Type: application/json" \
  -d '{"file_path": "/path/to/audio.aac"}'

Note:diarize defaults to true, no need to pass it explicitly. To disable it, pass "diarize": false. Video files (mp4, mov, etc.) automatically enable keyframe screenshot extraction (extract_slides), no need to pass it manually. To disable it, explicitly pass "extract_slides": false. For single-speaker lectures/voice, you can pass "fast": true to turn off speaker diarization, still using the default MOSS;"model": "paraformer" or "model": "paraformer-onnx" can invoke the original FunASR pipeline.

The response includes the following key fields:

  • output_path: Path to the Markdown file of the transcription output
  • text: Full transcription text
  • summary_prompt: AI summary prompt (is automatically attached, no need to additionally call /summary）
  • text_preview: Preview of the first 500 characters of the transcription text

Step 3: Speaker identification and claiming (default MOSS path)

In the transcription response, speaker_identification shows per person whether each speaker has been identified by the voiceprint library ({"S01": {"name": "杨律师", "score": 0.74}, "S02": null}; when the library is empty for the first time, all are null). Speakers that have been matched are shown directly with their registered names in the Markdown.

speaker_states provides the full status for each speaker; decide the next action based on the status:

  • matched: already matched to a registered voiceprint (name is the registered name), no need to ask;
  • unknown: has a voiceprint vector but has not reached the threshold,can be claimed and registered—ask the user according to the process below;
  • insufficient_audio: total speech is less than 3 seconds, no voiceprint vector,can only be annotated for this transcript in the Markdown based on the user's explanation, and cannot be registered；
  • extraction_failed: voiceprint extraction/recognition failed, cannot be identified or claimed this time, you can prompt the user to retry;
  • disabled: speaker voiceprint recognition is not enabled (fast/explicitly disabled/CAM++ unavailable), skip the claiming process.

When there are speakers in the unknown status, you must first confirm their identities with the user one by one before continuing.Before asking, first use the response segments(each segment contains speaker and text) togenerate a brief narrative for each unidentified speaker, and present it together with the question so the user can determine the identity from the content—this is the main basis for labeling, do not just report numbers:

Narrative format (2-3 sentences per speaker, summarized based on all of their speech):

  • Speaking share: the proportion of this speaker's speaking duration/segment count in the whole meeting;
  • Content summary: what they said—demands, positions, information given, to whom they asked questions;
  • Identity clues: self-reference ("I am Lawyer Yang"), how the other party addresses them ("Ms. Zhang"), role tone (consulting party/answering party/leading party).

Example presentation: "Transcription complete, 2 speakers in total.Speaker 1(35% of speech): self-identified as the landlord, asked about the process for suing over unpaid rent, and was repeatedly called 'Ms. Zhang' by the other party.Speaker 2(65% of speech): analyzed the evidence and litigation strategy in a lawyer's tone, and started with 'Hey, Ms. Zhang.' Who are these two people?"

After the user answers "it's me," "client Zhang San," or "no need to label," perform claim registration for the named speakers:

curl -s -X POST http://127.0.0.1:8765/speaker/register \
  -H "Content-Type: application/json" \
  -d '{"name": "<用户确认的名字>", "embedding": <响应 speaker_embeddings 中该标签的数组>, "source_file": "<文件名>", "source_label": "S01"}'
  • After claiming is complete, the current Markdown does not need to be rerun; in the next recording, the same person will be automatically recognized and annotated.
  • The speaker whom the user says "no need to label" remains anonymous and is not registered. When the user is only explaining a certain speaker'sidentity in this transcriptand does not want long-term registration, annotate directly in the Markdown without invoking registration.
  • Registration of a customer's or third party's voiceprint requires obtaining their separate consent first (voiceprints are sensitive personal information); this requirement does not apply to one's own voiceprint. For the complete process, thresholds, and error correction, see references/speaker-registry.md。
  • Recommended entry point (auto_transcribe.py) can use --json to get the complete response (including recognition results and vectors),--save-result to save the result file; when claiming, use python3 scripts/speaker_registry.py claim <名字> --result <结果JSON> --label S01 to deterministically retrieve the vector by label, avoiding manually copying 192 floating-point numbers.

Step 4: Generate AI summary

Based on summary_prompt(or directly based on the text content), the Agent generates a structured JSON summary:

{
  "full_summary": "至少400字，分成2-3段，交代背景、问题、关键事实、数据、风险与行动建议",
  "speaker_summary": [
    {
      "speaker_order": "发言人1",
      "speaker_name": "如能识别请写姓名，否则写未知",
      "summary": "至少180字，涵盖该发言人的观点、依据、数据、态度与潜在影响"
    }
  ],
  "highlights": ["6-10条重点，每条60-100字"],
  "keywords": ["5-8个关键词"]
}

Step 5: Inject the summary into the file

Important: do not merely describe the injection operation; you must actually execute the following command.

Write the JSON generated in step 3 to a temporary file, then call the script to inject it (more reliable than curl injection, no JSON escaping needed):

export PATH="/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:$PATH" && cat > /tmp/summary_<文件名>.json << 'JSONEOF'
{步骤3生成的JSON内容}
JSONEOF
python3 <skill目录>/scripts/summary.py inject "<output_path>" /tmp/summary_<文件名>.json

The script will automatically:

  • Parse the JSON and format it as Markdown
  • Inject it into the correct position in the Markdown file
  • Add <!-- AI-SUMMARY:START --> / <!-- AI-SUMMARY:END --> markers

Step 6: Verify the injection result (must be executed)

After injection, verification must be executed to confirm that the summary was actually written to the file. If verification fails, step 5 must be retried.

export PATH="/opt/homebrew/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:$PATH" && python3 <skill目录>/scripts/summary.py verify "<output_path>"
  • If the output is ✅ 摘要已存在 and ✅ 发言人覆盖完整 → success, report completion to the user
  • If the output is ❌ 摘要不存在 → failure, return to step 5 and retry
  • If the output is ❌ 摘要遗漏发言人 / ❌ 摘要出现稿件中不存在的发言人 / ❌ 缺少必需章节(non-zero exit code) → it means the summary omitted/fabricated speakers or is missing sections; return to step 4 and regenerate the summary according to the speaker_states/body tags, then inject again;A summary with incomplete coverage or missing sections must not be accepted as delivered
  • The inject command itself will also exit non-zero when coverage is incomplete or required sections are missing(prints "summary quality failed", and the injected content is retained as a draft): successful writing ≠ quality passed; see non-zero exit, you must return to step 4 to regenerate, and must not treat it as delivered
  • When the same speaker appears multiple times in the summary JSON (padding the headcount with duplicates, including cases where one of the summaries is empty), inject directly refuses to write and points out the duplicate labels → regenerate a summary containing all speakers
  • Output ⚠️ 发言人覆盖无法验证(fast/no speaker-line structured transcript) → as long as the sections are complete, it can be delivered, and truthfully state that coverage was not verified
  • HTTP /inject_summary、/verify_summary of success only indicates that the call executed successfully; quality is based on the quality_passed / speaker_coverage field in the response, and you must not announce delivery based only on success Announced as delivered

Complete process example

用户：转录这个音频
  ↓
Agent：
  1. 检查/启动服务
  2. POST /transcribe {"file_path": "xxx.aac"}  ← 一次调用拿到转录+识别结果+提示词
  3. 按 speaker_states 处理：unknown 说话人询问用户身份 → POST /speaker/register 认领注册
     （insufficient_audio 只能本稿标注，不能注册）
  4. 根据转录内容直接生成总结 JSON（speaker_order 逐字用稿件发言标签）
  5. 写 JSON 到临时文件 → python3 summary.py inject 注入
  6. python3 summary.py verify 验证（含发言人覆盖）→ 失败则重试步骤 4-5
  ↓
用户：收到带 AI 总结的 Markdown 文件

Usage process

First use: environment detection and dependency installation

Important: before first use, you must first check whether the environment meets the requirements.

Run environment detection:

python3 scripts/check_env.py

The detection script will check the following environment requirements:

Required item Requirement Detection command
Python >= 3.8，python3 command is available python3 --version
curl HTTP client (for API calls) curl --version
Basic commands ls, ps, grep shell built-in

If environment detection fails:

  1. Python3 command unavailable：

    # macOS 使用 homebrew 安装 Python
brew install python@3.14

  2. curl unavailable：

    # macOS 确保 curl 已安装
brew install curl

  3. After verifying the environment is fixed, rerun detection:

    python3 scripts/check_env.py

First use: install dependencies and download models

Run the installation script to complete environment configuration:

/opt/homebrew/bin/python3.11 -m venv venv
source venv/bin/activate
python3 scripts/setup.py

The installation script will automatically:

  1. Check Apple Silicon and Python version (default dual-backend installation requires 3.10–3.12)
  2. Install dependency packages (FastAPI, FunASR/CAM++, MLX-Audio, etc.; legacy pipeline dependencies are also retained)
  3. Download the MOSS and CAM++ models to the ModelScope cache;--legacy Install only the legacy FunASR models

Verify installation status:

python3 scripts/setup.py --verify

Start the transcription service

python3 scripts/server.py

To run the retained legacy ONNX pipeline, use:

python3 scripts/server-onnx.py --preload

The service runs by default on http://127.0.0.1:8765

Smart features:

  • Auto-start: automatically load the model on the first request
  • Idle shutdown: by default, automatically shut down after 10 minutes of inactivity to save resources
  • Configurable timeout: use the --idle-timeout parameter to customize the idle timeout (seconds)

Service lifecycle:

  1. After startup, enter the idle monitoring state
  2. When a request is received, automatically load the model and perform transcription
  3. Each request resets the idle timer
  4. Automatically shut down after 10 consecutive minutes without requests
  5. Restart on the next request

Important note:

  • ⚠️ Do not manually shut down the service - after transcription is complete, let the service continue running; it will automatically shut down after 10 minutes of inactivity
  • This allows continuous transcription of multiple files without repeatedly starting the service
  • If you need to shut down the service immediately, press Ctrl+C or wait for the 10-minute idle timeout

Example: customize a 30-minute idle timeout

python3 scripts/server.py --idle-timeout 1800

Perform transcription

Use the client script to transcribe files:

# 转录单个文件
python3 scripts/transcribe.py /path/to/audio.mp3

# 指定输出路径
python3 scripts/transcribe.py /path/to/video.mp4 -o transcript.md

# 启用说话人分离
python3 scripts/transcribe.py /path/to/meeting.m4a --diarize

# 保留的 FunASR 原生管线
python3 scripts/transcribe.py /path/to/meeting.m4a --model paraformer

# 保留的 Paraformer ONNX 管线
python3 scripts/transcribe.py /path/to/meeting.m4a --model paraformer-onnx

# Paraformer ONNX 单人路径（VAD 分段 ASR，不做说话人聚类）
python3 scripts/transcribe.py /path/to/course.m4a --model paraformer-onnx --no-diarize

# 单人讲课快速模式（关闭说话人分离，仍使用默认 MOSS）
python3 scripts/transcribe.py /path/to/course.m4a --fast

# 默认 MOSS 说话人转录与法律术语热词
python3 scripts/transcribe.py /path/to/meeting.m4a --hotword 请求权基础 --hotword DeepSeek

# 批量转录目录
python3 scripts/transcribe.py /path/to/media_folder/

# 提取视频关键帧截图（PPT幻灯片）
python3 scripts/transcribe.py /path/to/video.mp4 --slides

# 视频只转录，不提取截图
python3 scripts/transcribe.py /path/to/video.mp4 --no-slides

# 自定义场景检测阈值（值越低越灵敏，默认20.0）
python3 scripts/transcribe.py /path/to/video.mp4 --slides --slide-threshold 15.0

AI smart summary (Claude Code environment)

After transcription is complete, you can generate an AI smart summary, making full use of Claude Code's native AI capabilities.

Automatic mode (recommended):

Use the --auto-summary parameter to automatically generate and inject the summary after transcription is complete:

# 转录并自动生成总结（Claude Code 原生环境，无需配置 API Key）
python3 scripts/transcribe.py /path/to/audio.m4a --auto-summary

# 完整流程：说话人分离 + 自动总结
python3 scripts/transcribe.py /path/to/meeting.m4a --diarize --auto-summary

How it works:

  • The script outputs a structured summary request (AI_SUMMARY_REQUEST）
  • Claude Code automatically recognizes it and uses its built-in AI capabilities to generate the summary
  • No external API Key configuration is required

Manual mode:

  1. After performing transcription, the script will automatically prepare a summary prompt
  2. Send the prompt to Claude AI to generate a structured summary
  3. Paste the JSON result returned by Claude back into the script
  4. Automatically inject the summary into the Markdown file
# 转录单个文件（输出提示词供手动调用）
python3 scripts/transcribe.py /path/to/audio.mp3

# 禁用自动总结（只输出提示词）
python3 scripts/transcribe.py /path/to/audio.m4a --no-summary

Summary content structure:

  • Full-text summary - 400+ characters, including background, issues, and key facts
  • Speaker summary - each speaker's views, attitude, and contributions
  • Key content - 6-10 core points
  • Keywords - 5-8 key terms

Prompt features:

  • Specifically optimized for Chinese colloquial conversations
  • Preserves speaker context and conversation flow
  • Structured JSON output for easy parsing and formatting

For detailed documentation, see: <references/api-reference.md>

Call via HTTP API

Check service status：

curl http://127.0.0.1:8765/health

Call the API directly with curl:

curl -X POST http://127.0.0.1:8765/transcribe \
  -H "Content-Type: application/json" \
  -d '{"file_path": "/path/to/audio.mp3"}'

# 单人快速模式（关闭说话人分离，仍使用默认 MOSS）
curl -X POST http://127.0.0.1:8765/transcribe \
  -H "Content-Type: application/json" \
  -d '{"file_path": "/path/to/course.m4a", "fast": true}'

# 指定 Paraformer ONNX（默认启用 diarization）
curl -X POST http://127.0.0.1:8765/transcribe \
  -H "Content-Type: application/json" \
  -d '{"file_path": "/path/to/meeting.m4a", "model": "paraformer-onnx"}'

# Paraformer ONNX 单人路径（VAD 分段 ASR，不做说话人聚类）
curl -X POST http://127.0.0.1:8765/transcribe \
  -H "Content-Type: application/json" \
  -d '{"file_path": "/path/to/course.m4a", "model": "paraformer-onnx", "diarize": false}'

# 提取视频关键帧截图
curl -X POST http://127.0.0.1:8765/transcribe \
  -H "Content-Type: application/json" \
  -d '{"file_path": "/path/to/video.mp4", "extract_slides": true}'

API Documentation (Swagger UI)：

FastAPI automatically generates interactive API documentation, accessible at:http://127.0.0.1:8765/docs

On this page you can:

  • View all API endpoints
  • Test the API online (no curl needed)
  • View request/response formats
  • View detailed parameter descriptions

Response example(health check):

{
  "status": "ok",
  "service": "Local ASR",
  "uptime": 300,
  "idle_time": 120
}

Returned field descriptions:

  • uptime: service uptime (seconds)
  • idle_time: current idle time (seconds)

Complete API documentation

For detailed API reference documentation, see: <references/api-reference.md>

Includes:

  • Complete specifications for all API endpoints
  • Detailed explanation of request/response formats
  • Parameter descriptions and examples
  • Complete curl command examples

Script description

Script Purpose
scripts/init_env.py Environment detection + generate skill-env.json
scripts/check_env.py Environment detection (simplified version)
scripts/setup.py One-click dependency installation and model download
scripts/server.py Start the HTTP API service
scripts/server-onnx.py Start the default ONNX acceleration service
scripts/transcribe.py Command-line client
scripts/auto_transcribe.py Automated transcription script (recommended)
scripts/speaker_registry.py Voiceprint library management (list / remove / claim / test)
scripts/align_words.py Word-level alignment: reference transcription + audio → character-level words (in seconds, normalized axis consistent with MOSS; requires this skill's venv funasr and cached Paraformer)

Automated transcription + summarization workflow

This skill supports automatic completion in any Agent platformTranscription + summarizationfull workflow.

Option 1: Use the automated script (recommended)

# 自动转录 + 获取总结提示词（说话人分离默认启用）
python3 scripts/auto_transcribe.py /path/to/audio.aac

# 禁用说话人分离
python3 scripts/auto_transcribe.py /path/to/audio.aac --no-diarize

# 单人快速模式（关闭说话人分离，仍使用默认 MOSS）
python3 scripts/auto_transcribe.py /path/to/course.m4a --fast

# 只获取总结提示词，不生成总结
python3 scripts/auto_transcribe.py /path/to/audio.aac --prompt-only

# 机器模式：stdout 仅输出完整响应 JSON（含说话人识别/向量/段落），日志走 stderr
python3 scripts/auto_transcribe.py /path/to/audio.aac --json

# 保存完整响应 JSON 到文件（新建与覆盖均保持 0600 私密权限；含声纹向量，注意保密）
python3 scripts/auto_transcribe.py /path/to/audio.aac --json --save-result /tmp/asr-result.json

# 认领注册：从结果 JSON 按标签取声纹向量（需用户确认身份且同意长期注册）
python3 scripts/speaker_registry.py claim <用户确认的名字> --result /tmp/asr-result.json --label S01

Option 2: HTTP API call

1. Transcribe audio (the response automatically includes the summary prompt)

curl -X POST http://127.0.0.1:8765/transcribe \
  -H "Content-Type: application/json" \
  -d '{"file_path": "/path/to/audio.aac"}'

The response includes the summary_prompt field, which can be used directly to generate a summary without an extra call to /summary。

2. Inject AI summary

After generating the summary, call:

curl -X POST http://127.0.0.1:8765/inject_summary \
  -H "Content-Type: application/json" \
  -d '{
    "md_path": "/path/to/audio.md",
    "summary_content": "## AI 摘要\n\n### 全文总结\n...\n\n### 重点内容\n- ...\n\n### 关键词\n..."
  }'

API endpoint summary

Endpoint Method Function
/health GET Health check
/transcribe POST Transcribe audio/video
/batch_transcribe POST Batch transcribe directory
/summary POST Generate AI summary prompt
/inject_summary POST Inject summary into Markdown file
/verify_summary POST Verify whether the summary has been injected
/speaker/register POST Claim a registered speaker voiceprint
/speaker/list GET List registered speakers
/speaker/remove POST Delete a registered speaker
/speaker/test POST Compare audio voiceprint with the library

Configuration file

File Description
assets/models.json ASR model configuration manifest
assets/requirements.txt Python dependency manifest

Output format

The transcription result is saved as a Markdown file, containing:

  1. Title - File name (no transcription timestamp)
  2. Transcription content - Format:发言人N HH:MM:SS Line break 内容
  3. AI summary(optional) - includes full-text summary, speaker summary, key content, keywords

Example format (video with screenshots):

# 转录：视频.mp4

## 转录内容

发言人1 00:02:49
![](slides/slide_001_02m49s.jpg)
各位好，今天我们来讲...

发言人1 00:03:30
![](slides/slide_002_03m30s.jpg)
这是第二段的内容...

Word-level timestamps (align_words.py)

/transcribe By default, segment-level is returned segments(MOSS path), with no word-level boundaries. When character-level timestamps are needed (such as deleting speech and synchronously deleting subtitle text when editing subtitles), use the standalone entry point for alignment—without changing the default transcription behavior:

./venv/bin/python scripts/align_words.py <音视频> \
    --transcript work/transcript.json --output work/transcript-words.json
  • The input reference transcription is in the cut standard segments(or Volcano result.utterances); the output is preserved as-is and each segment is given words(character-level, in seconds, normalized audio axis, on the same axis as the MOSS transcription).
  • Alignment strategy: segment the audio one by one → recognize with cached Paraformer → align and map with reference text chunks using difflib; segments with similarity below --min-similarity(default 0.6) have words left empty and are recorded in word_alignment_warnings — use only when trustworthy; do not create false alignments.
  • asr.word_timestamps Metadata declares available/granularity=character/aligned_segments and other real states.
  • Dependencies: this skill's venv funasr(cached in ~/.cache/modelscope/hub/models/iic/, separate from the MOSS weights); if the cache is missing on first use, a clear prompt will be given. Reference: about 25-30s for a 2-minute original video.

Model information

Models are stored in the ModelScope default cache directory ~/.cache/modelscope/hub/models/：

  • ASR main model (Paraformer) - 867MB
  • SenseVoice-Small (experimental single-speaker ONNX path) - downloaded on demand when explicitly specified
  • VAD model - 4MB
  • Punctuation model - 283MB
  • Speaker diarization model - 28MB
  • MOSS-Transcribe-Diarize (default) - model weights about 1.8GB, runtime memory higher than the weight size

STT transcription priority (important)

Default order: use MOSS-MLX first; if MOSS fails, check the error, dependencies, and model path. When processing needs to continue, you may explicitly switch to --model paraformer or --model paraformer-onnx, and explain to the user that the pipeline has been switched; do not silently fall back. When using the old pipeline on non-Apple Silicon, first set FUNASR_SERVER_DEFAULT_MODEL=paraformer to start the service, then explicitly specify the old model in the CLI.

Troubleshooting steps when MOSS fails

  1. Run this in the same Python environment used to start the service python3 scripts/setup.py --verifyand check MLX-Audio, ffmpeg, and the model cache.
  2. Check the service logs; if the Hugging Face download fails, you can set FUNASR_MOSS_MODEL_ID to a local model directory and restart.
  3. When timestamps are missing, output is truncated, or short-speaker labels are unstable, keep the errors/warnings and review manually; if necessary, explicitly call the original FunASR pipeline for comparison.

Whisper CLI emergency path (requires explicit selection)

# 提取音频（16kHz 单声道）
ffmpeg -i "/path/to/video.mp4" -vn -acodec pcm_s16le -ar 16000 -ac 1 -y "/tmp/audio.wav"

# Whisper 转录（tiny 模型最快，medium 质量更好）
/opt/homebrew/bin/whisper "/tmp/audio.wav" \
  --model tiny \
  --language Chinese \
  --output_dir /tmp/transcript \
  --output_format all

Performance reference: for 19 minutes of audio, the tiny model takes about 3-5 minutes (Mac CPU).

Troubleshooting

cv2 / opencv import failure

Symptoms：POST /transcribe returns {"detail":"No module named 'cv2'"}, but pip show opencv-python-headless shows it is installed.

Root cause: The Python environment used by the service process differs from the pip installation target. This is common in the macOS Homebrew Python 3.14 environment, where pip installs to the system site-packages, but the service process loads from the Homebrew path.

Troubleshooting steps：

  1. Verify in the terminal whether cv2 can be imported:python3 -c "import cv2; print('ok')"
  2. If the import fails, run:python3 -m pip install opencv-python-headless --break-system-packages
  3. Confirm the Python path of the service process:lsof -p <server_pid> | grep python

Correct startup procedure：

# 确认 cv2 可用后再启动服务
python3 -c "import cv2; print('cv2 ok')"

# 如需重启，先核对占用进程确为本技能的 server.py，再正常停止该进程
lsof -nP -iTCP:8765 -sTCP:LISTEN

# 重启服务
python3 scripts/server.py --idle-timeout 600 &

Service port is already in use

Symptoms：Address already in use（Errno 48）

First run lsof -nP -iTCP:8765 -sTCP:LISTEN to identify the occupant. If it is another service, keep that process and specify another free port for this skill:

python3 scripts/server.py --port 8766
python3 scripts/transcribe.py /path/to/audio.mp3 --server http://127.0.0.1:8766

The automatic transcription entry point can be changed to python3 scripts/auto_transcribe.py /path/to/audio.mp3 --api http://127.0.0.1:8766, and its automatically started service will also use the specified port. Both clients will verify /health 's service identifier to avoid mistaking other services for the local ASR.

FunASR service not responding / model loading is slow

The first transcription requires downloading the model (about 1-2GB), so please wait patiently. For subsequent requests, the model is already cached and will be much faster.

Video screenshot feature:

Keyframe extraction is automatically enabled when transcribing video files (mp4, mov, avi, mkv, wmv, webm). Dependencies opencv-python and imagehash are already included in requirements.txt,setup.py will be installed together during installation. If these dependencies are not installed, the server will output a prompt but this will not affect normal transcription functionality.

When the service fails to start, run the verification command to check the installation status:

python3 scripts/setup.py --verify

Re-download the model:

python3 scripts/setup.py --skip-deps