SuperAPI AI Directory
Partner中文AI newsFavorites
Skillscarocut-media-audio

carocut-media-audio

Design & creation

View repository

Updated 10-27 20:02

About this skill

Audio Assets

Generates voiceovers via TTS, retrieves background music and sound effects. Creates durations.json for timing synchronization in Remotion.

Audio Types

Type Source Output Directory
Voiceover (VO) Edge TTS {project_dir}/raws/audio/vo/
Background Music (BGM) Freesound {project_dir}/raws/audio/bgm/
Sound Effects (SFX) Freesound {project_dir}/raws/audio/sfx/

Voiceover Generation

Generate speech audio from script using Edge TTS.

TTS Batch Processing Workflow

Processes entire script.md and generates all voiceover files:

  1. Parse {project_dir}/manifests/script.md for all [VO_XXX] markers and their text content
  2. For each VO segment, invoke Edge TTS with the configured voice and speed
  3. Save each result as VO_XXX.wav in {project_dir}/raws/audio/vo/
  4. After all segments complete, extract durations to create {project_dir}/manifests/durations.json

Voice Selection

For the complete voice list and selection recommendations, see references/edge-tts.md。

Input Meaning Recommendation
default Default alias Maps to zh-CN-XiaoxiaoNeural
any valid edge-tts voice ID Direct pass-through According to the project style, select from references/edge-tts.md Select in

Speed Adjustment Rules

  • The default TTS voice may sound slow. Consider using speed 1.2 for more natural pacing.
  • Speed range: 0.5 (half speed) to 2.0 (double speed)
  • Recommended: 1.0 - 1.3 for narration, 0.8 - 1.0 for emotional/dramatic content

TTS Speed and Storyboard Pacing (Speech Rate and Rhythm Correspondence)

When generating TTS, read the corresponding shot's pacing field from the storyboard to determine speed:

Pacing TTS Speed Description
slow 1.0 Slow down the speech rate to create immersion and emotional weight
medium 1.2 Default speech rate, natural narrative rhythm (unchanged)
fast 1.3 Speed up the speech rate to create a sense of urgency and excitement
pause N/A Breathing shot, no narration, no TTS generated

Rules:

  • If the storyboard does not specify pacing for a shot, default to medium (speed 1.2)
  • For pause pacing shots, skip TTS generation entirely — these are breathing shots with no voiceover
  • The pacing value is per-shot; different VO segments in the same video may have different speeds
  • After applying pacing-based speed, duration extraction must reflect the actual playback length at that speed

Duration Extraction

Extract audio durations for timing synchronization. This step is mandatory.

Output Format (durations.json)

{
  "VO_001": 3450,
  "VO_002": 2800,
  "VO_003": 4200,
  "VO_004": 3100
}

All durations are in milliseconds. This file is critical for shot timing in Remotion.

Duration Rules

  • All values MUST be in milliseconds (integer)
  • Every [VO_XXX] marker in script.md must have a corresponding entry
  • Duration reflects the actual audio length at the configured speed (e.g., 1.2x)
  • Always re-extract durations after any audio modification

durations.json and Audio-Visual Relation

The durations.json format itself remains unchanged (simple VO_XXX: milliseconds mapping). However, the Builder's timing.ts module uses the storyboard's audio_visual_relation field in combination with durations to calculate actual audio start frames:

audio_visual_relation Builder Behavior
sync Audio starts at the same frame as the visual shot
lead-visual Visual appears slightly before audio (audio offset positive)
lead-audio Audio starts slightly before the visual cut (audio offset negative)
counterpoint Audio timing is sync, but visual content is intentionally mismatched

The audioOffsetFrames function in timing.ts reads these values to apply frame-level offsets. The Audio agent does not need to modify durations.json for this — it is handled downstream by the Builder.

Important dependency note: Although the durations.json format is simple (VO_XXX: milliseconds), when the Builder calculates the actual audio start frame, it will also read the corresponding shot's audio_visual_relation field in storyboard.yaml. The Audio agent only needs to ensure that the durations in durations.json are accurate; the offset is automatically calculated by the Builder's timing.ts → audioOffsetFrames() function.

BGM and SFX Retrieval

Download royalty-free audio from Freesound.

BGM Search Strategy

  • Search for loops or long-form tracks matching the video mood
  • Minimum duration: 60 seconds (to cover full video length)
  • Keywords should describe mood, genre, and instrumentation

BGM Rhythm Matching (BGM Rhythm Matching)

Select BGM tempo based on the dominant pacing of each chapter/section in the storyboard:

Chapter Pacing BPM Range Genre / Style
slow 60-80 BPM ambient, piano, soft strings, lo-fi
medium 90-110 BPM corporate, light electronic, acoustic
fast 120-140 BPM upbeat electronic, driving percussion, energetic
breathing shot N/A Ambient sound/atmosphere sound, or fade-in/fade-out transitions with the BGM of the preceding and following chapters

Rules:

  • Determine a chapter's dominant pacing by the most frequent pacing value among its shots
  • If a video has distinct chapters with different pacing, consider using multiple BGM tracks with crossfade transitions
  • For breathing shots between chapters, use ambient/atmospheric sounds or fade the BGM to bridge the transition
  • When searching Freesound, include BPM range in keywords (e.g., "ambient piano 70bpm", "upbeat electronic 130bpm")
  • BGM should complement, not compete with, the voiceover. Lower BPM tracks naturally sit better under narration.

SFX Search Strategy

  • Search for short, punchy sound effects for transitions and UI feedback
  • Maximum duration: typically 3-5 seconds
  • Keywords should describe the exact sound needed

Search Keywords Tips

Need Keywords
BGM - Tech ambient electronic, minimal background, corporate
BGM - Emotional piano emotional, cinematic orchestra, inspiring
SFX - Transitions whoosh, swish, swoosh, transition
SFX - UI notification, alert, click, pop
SFX - Success success, complete, achievement, ding

License Information

License Attribution Commercial Use
CC0 (Public Domain) Not required Yes
CC-BY (Attribution) Required Yes

For CC-BY sounds, record attribution in raws/audio/LICENSES.txt.

Output Structure

raws/
  audio/
    vo/
      VO_001.wav
      VO_002.wav
      ...
      durations.json       # Critical timing data
    bgm/
      ambient_loop.mp3
    sfx/
      whoosh.mp3
      notification.mp3
    LICENSES.txt           # Attribution for CC-BY sounds

Workflow

  1. Generate Voiceovers

    • Process manifests/script.md for all [VO_XXX] markers
    • All [VO_XXX] markers become VO_XXX.wav files
    • Extract durations to create durations.json

  2. Retrieve BGM

    • Search for appropriate background music
    • Download to raws/audio/bgm/

  3. Retrieve SFX

    • Search for transition sounds, UI sounds
    • Download to raws/audio/sfx/

  4. Verify and Report

    • Check all files generated correctly
    • Report results to user

Incremental Mode

When mode: "incremental" in the dispatch context:

Input

  • The list of VO segment IDs specified in amendment.affected_resources (speech clips that need to be regenerated)

Execution rules

  1. Read script.md and locate only the specified VO segments
  2. Regenerate TTS audio only for the specified VO segments
  3. Use the same character, tone, and speed configuration as the original generation
  4. Replace the corresponding WAV files in raws/audio/vo/
  5. The complete durations.json must be re-extracted(because any audio change will affect the timing)
  6. Return a processing summary (which VOs were regenerated, comparison of old and new durations)

Do not execute

  • Do not regenerate unspecified VO audio
  • Do not re-retrieve BGM/SFX (unless explicitly required in the amendment)

Key note

  • Even if only one VO segment is modified, the entire durations.json must be re-extracted
  • Duration changes may affect downstream Remotion timing calculations

Audio Timing Rules

These rules are critical for correct Remotion synchronization:

Rule Detail
Unit All durations in milliseconds
Speed factor Default 1.2x recommended; durations reflect actual playback length
Re-extraction Mandatory after ANY audio file change
Completeness Every [VO_XXX] in script.md must have a durations.json entry
Integer values No floating point; round to nearest millisecond

Network Requirements

Edge TTS requires outbound internet access and the Python package edge-tts.

User Communication

Progress Report

音频素材生成进行中...

TTS 生成:
  进度: 15/45 条语音
  当前: VO_016
  音色: zh-CN-XiaoxiaoNeural

Completion Report

音频素材生成完成。

语音配音:
  - 45 条语音文件
  - 总时长: 4分23秒
  - 音色: zh-CN-XiaoxiaoNeural
  - 时长数据: raws/audio/vo/durations.json

背景音乐:
  - 1 个循环音频 (2分30秒)

音效:
  - 3 个过渡音效
  - 2 个提示音

输出目录: raws/audio/
请确认后继续下一步。

Error Handling

Issue Cause Solution
TTS timeout Text too long Split into shorter segments (max 200 chars)
Missing durations Duration extraction not run Run extraction after TTS generation
Audio quality poor Wrong Edge voice for content Try a different Edge voice ID
Freesound API error Missing API key Set FREESOUND_API_KEY
Network error Internet unavailable or Edge TTS request failed Check internet access and retry

Critical Notes

Duration File

durations.json is mandatory for Remotion implementation. It contains timing data in milliseconds that drives:

  • Shot duration calculations
  • Audio layer positioning
  • Voiceover synchronization

Always re-extract durations after any audio modification.

Default Voice Speed

The default TTS voice may sound slow. Consider using speed 1.2 for more natural pacing. When the storyboard provides a pacing field, use the TTS Speed and Storyboard Pacing table above to determine the appropriate speed for each VO segment.