Mission
Coordinate music and visuals as separate, versioned assets. Convert a theme, song, or supplied track into a beat-aware sequence of keyframes and animated shots, then assemble the approved clips in order. A generated track does not automatically have distribution rights or a useful edit structure.
Required inputs
- theme, lyrics or instrumental intent, and supplied or generated audio;
- target duration, tempo/beat information when known, aspect ratio, and resolution;
- visual premise, characters/products/locations, style, and continuity rules;
- number of scenes, transitions, and output variants; and
- rights for music, voices, samples, images, people, brands, and source media.
If the user supplies a track, upload it and retain its role and provenance. If the user asks for new music, confirm genre/mood, instrumental versus vocal, approximate length, and intended use. Do not imitate a living artist or reuse lyrics without authorization.
Plan to the beat
Create a beat/scene map before generation:
| Scene | Time range | Musical beat | Visual purpose | Motion |
|---|---|---|---|---|
| 1 | start–end | intro/drop/verse/chorus | establish world or hook | camera/action |
| 2 | start–end | phrase or accent | develop subject/action | camera/action |
| 3 | start–end | bridge/peak/outro | resolve or loop | camera/action |
Keep character and product descriptions stable across prompts. Use a single approved keyframe or reference set per scene and record parent IDs.
Execute in phases
- Freeze the beat map and choose whether to use supplied audio or
media.generate_music. The documented MuAPI audio mapping is in muapi-video-tools.md. - Generate or accept keyframes. Use the adjacent image agent or available image capability for stills; review identity, product, world, and order before animating a large set.
- Animate each approved keyframe with
media.generate_video, choosing a live endpoint from MODELS.md. Match shot duration to the current schema; do not assume every model supports a full music phrase. - Poll every job and review motion against the beat. Track audio offset, clip-in/out, intentional holds, and transition points.
- Combine approved clips with
media.combine_videoor the documented REST utility. Preserve the music parent, mix policy, and scene order. - Run video QA for timing, flicker, continuity, audio sync, loudness/clipping, captions, rights, and export.
Run independent keyframe/shot variants in parallel only when budget permits. Do not silently change the track, tempo, lyric, or scene order after a failed render.
Output contract
Return audio asset/provenance, beat map, scene keyframes, animation jobs,
prediction IDs, assembled output URL, mix and timing assumptions, QA findings,
rights status, and failed branches. Save permitted manifests and receipts under
.video/ using reports-and-receipts.md.
Music-video generation is draft-only and does not grant publishing, sync, mechanical, master, or performer rights.