Skill by ara.so — Devtools Skills collection.
Clipify is a Claude Code skill that automatically turns long-form videos into social-ready clips. It transcribes video, identifies clip-worthy moments, reframes 16:9 to 9:16 with face-tracking pans, and burns opus-style word-by-word captions.
Key capabilities:
- Auto-detect punchlines, reversals, and awkward pauses via Whisper transcription
- Face-tracking pan for 9:16 vertical clips (no ML models — uses motion energy)
- Opus/karaoke/minimal subtitle styles with word-level highlighting
- Hardware-accelerated rendering (VideoToolbox on macOS)
- ~20s render time for 20s clips on Apple Silicon
Installation
# Clone to Claude Code skills directory
git clone https://github.com/louisedesadeleer/clipify.git ~/.claude/skills/clipify
brew install ffmpeg pip install openai-whisper numpy
**Requirements:**
- macOS (or Linux/Windows with `-hwaccel videotoolbox` removed from SKILL.md)
- ffmpeg with libx264
- Python 3 with numpy
- Whisper (openai-whisper)
Restart Claude Code after installation. The `/clipify` slash command will be available.
## Usage Workflow
### 1. Invoke the skill
In Claude Code:
/clipify
Provide the path to your source video when prompted:
/path/to/long-interview.mp4
### 2. Review proposed clips
Clipify transcribes the video and proposes 3-5 candidates with:
- Timestamp range
- Title/description
- Reason (punchline, reversal, audio peak, awkward pause)
Example output:
Clip 1: "The worst product advice" (02:34 - 02:51) Reason: Reversal after awkward pause
Clip 2: "We burned $2M on this" (08:12 - 08:29) Reason: Audio peak + punchline
Clip 3: "My co-founder quit on Zoom" (15:03 - 15:24) Reason: Punchline
### 3. Select clip and format
Choose which clip to cut, then specify:
- **Aspect ratio:** 9:16 (vertical), 16:9 (horizontal), 1:1 (square)
- **Reframe style** (if 9:16 from 16:9 with two speakers): pan (follow speaker) or split-screen
- **Subtitle style:** opus (bold white + yellow highlight), karaoke (word-by-word), minimal, or paste reference image
### 4. Output
Final clips are saved to:
/clipify_out/clip_.mp4
## Scripts Reference
Clipify uses standalone Python scripts for each processing step. You can call these directly for custom workflows.
### analyze.py — Speaker timeline from motion energy
```python
# Generate motion energy files for two face regions
ffmpeg -i video.mp4 -vf "crop=300:200:100:50,format=gray,tblend=all_mode=difference" \
-f rawvideo -pix_fmt gray - | python scripts/analyze.py motion_left.bin
ffmpeg -i video.mp4 -vf "crop=300:200:1000:50,format=gray,tblend=all_mode=difference" \
-f rawvideo -pix_fmt gray - | python scripts/analyze.py motion_right.bin
# Analyze both to generate speaker timeline
python scripts/analyze.py motion_left.bin motion_right.bin --fps 30 > timeline.txt
Output format (timeline.txt):
0.00-2.34:left
2.34-5.67:right
5.67-8.12:left
build_pan.py — Generate ffmpeg crop expression
# From speaker timeline, build hard-cut pan expression
python scripts/build_pan.py timeline.txt --left-x 100 --right-x 1000 --width 608 > pan_expr.txt
# Use in ffmpeg crop filter
ffmpeg -i source.mp4 -vf "crop=608:1080:'$(cat pan_expr.txt)':0" output.mp4
Arguments:
--left-x: X coordinate of left speaker's face center--right-x: X coordinate of right speaker's face center--width: Width of the 9:16 crop window (e.g., 608 for 1080p)
Output: ffmpeg expression string like:
if(between(t,0,2.34),100,if(between(t,2.34,5.67),1000,if(between(t,5.67,8.12),100,1000)))
build_ass.py — Generate ASS subtitle file
# From Whisper JSON output, create opus-style captions
python scripts/build_ass.py whisper_output.json --style opus > captions.ass
# Burn into video
ffmpeg -i video.mp4 -vf "ass=captions.ass" output.mp4
Whisper JSON format (input):
{
"segments": [
{
"start": 0.5,
"end": 2.3,
"text": "This is the worst advice",
"words": [
{"word": "This", "start": 0.5, "end": 0.7},
{"word": "is", "start": 0.7, "end": 0.85},
{"word": "the", "start": 0.85, "end": 1.0},
{"word": "worst", "start": 1.0, "end": 1.4},
{"word": "advice", "start": 1.4, "end": 2.3}
]
}
]
}
Styles:
opus: Bold white text, yellow active-word highlight, centered topkaraoke: Word-by-word color change, bottom positionedminimal: Clean white text, no highlights
audio_align.py — Find clip offset in source
# Find where a 20s clip appears in a 2-hour source video
python scripts/audio_align.py source.mp4 clip.mp4
# Output: 00:15:34.2 (offset timestamp)
Uses audio cross-correlation. Useful for re-linking edited clips to source timestamps.
Common Patterns
Extract clip manually (without auto-detection)
# 1. Transcribe with Whisper
whisper source.mp4 --model base --output_format json --output_dir ./
# 2. Cut segment (03:15 to 03:42)
ffmpeg -i source.mp4 -ss 00:03:15 -to 00:03:42 -c copy raw_clip.mp4
# 3. Generate captions for this segment
python scripts/build_ass.py source.json --start 195 --end 222 --style opus > clip.ass
# 4. Reframe to 9:16 with center crop (no pan)
ffmpeg -i raw_clip.mp4 -vf "crop=608:1080:656:0,ass=clip.ass" final_clip.mp4
Two-speaker pan with manual face coordinates
# 1. Identify face regions on a sample frame
ffplay -ss 00:01:00 source.mp4 # visual inspection
# Left face: x=200, y=100, width=300, height=200
# Right face: x=1100, y=100, width=300, height=200
# 2. Generate motion energy files
ffmpeg -i source.mp4 -ss 00:03:15 -to 00:03:42 \
-vf "crop=300:200:200:100,format=gray,tblend=all_mode=difference" \
-f rawvideo -pix_fmt gray motion_left.bin
ffmpeg -i source.mp4 -ss 00:03:15 -to 00:03:42 \
-vf "crop=300:200:1100:100,format=gray,tblend=all_mode=difference" \
-f rawvideo -pix_fmt gray motion_right.bin
# 3. Analyze speaker timeline
python scripts/analyze.py motion_left.bin motion_right.bin --fps 30 > timeline.txt
# 4. Build pan expression (faces centered at x=350 and x=1250)
python scripts/build_pan.py timeline.txt --left-x 350 --right-x 1250 --width 608 > pan.txt
# 5. Apply crop with pan
ffmpeg -i source.mp4 -ss 00:03:15 -to 00:03:42 \
-vf "crop=608:1080:'$(cat pan.txt)':0" panned_clip.mp4
Custom subtitle styling
Edit ASS file generated by build_ass.py:
[V4+ Styles]
Format: Name, Fontname, Fontsize, PrimaryColour, SecondaryColour, OutlineColour, BackColour, Bold, Italic, Underline, StrikeOut, ScaleX, ScaleY, Spacing, Angle, BorderStyle, Outline, Shadow, Alignment, MarginL, MarginR, MarginV, Encoding
Style: Default,Impact,68,&H00FFFFFF,&H0000FFFF,&H00000000,&H00000000,-1,0,0,0,100,100,0,0,1,3,0,2,10,10,120,1
[Events]
Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text
Dialogue: 0,0:00:00.50,0:00:02.30,Default,,0,0,0,,{\k70}This {\k15}is {\k15}the {\k40}worst {\k90}advice
Customize:
Fontname: Impact, Arial, MontserratFontsize: 68 for 1080p verticalPrimaryColour:&H00FFFFFF(white in BGR hex)SecondaryColour:&H0000FFFF(yellow highlight)Outline: Border thickness (3 = thick black outline)Alignment: 2=bottom center, 8=top centerMarginV: Vertical margin from edge
Batch processing multiple clips
import subprocess
import json
# Load Whisper transcript
with open("source.json") as f:
data = json.load(f)
# Define clip ranges
clips = [
{"start": 154, "end": 171, "title": "clip1"},
{"start": 492, "end": 509, "title": "clip2"},
{"start": 903, "end": 924, "title": "clip3"}
]
for clip in clips:
# Cut raw clip
subprocess.run([
"ffmpeg", "-i", "source.mp4",
"-ss", str(clip["start"]),
"-to", str(clip["end"]),
"-c", "copy",
f"raw_{clip['title']}.mp4"
])
# Generate captions
subprocess.run([
"python", "scripts/build_ass.py", "source.json",
"--start", str(clip["start"]),
"--end", str(clip["end"]),
"--style", "opus"
], stdout=open(f"{clip['title']}.ass", "w"))
# Reframe and burn captions
subprocess.run([
"ffmpeg", "-i", f"raw_{clip['title']}.mp4",
"-vf", f"crop=608:1080:656:0,ass={clip['title']}.ass",
f"{clip['title']}_final.mp4"
])
Configuration
Hardware acceleration
macOS (default):
ffmpeg -hwaccel videotoolbox -i input.mp4 ...
Linux with NVIDIA:
ffmpeg -hwaccel cuda -hwaccel_output_format cuda -i input.mp4 ...
Windows:
ffmpeg -hwaccel dxva2 -i input.mp4 ...
Disable (CPU only):
Remove -hwaccel flags from SKILL.md ffmpeg commands.
Whisper model size
Faster but less accurate:
whisper video.mp4 --model tiny # ~1GB, 10x faster
More accurate but slower:
whisper video.mp4 --model medium # ~1.5GB, 2x slower
whisper video.mp4 --model large # ~3GB, 4x slower
Default in SKILL.md: base (good balance for dialogue).
Output quality settings
High quality (larger file):
ffmpeg -i input.mp4 -c:v libx264 -preset slow -crf 18 -c:a aac -b:a 192k output.mp4
Fast encode (lower quality):
ffmpeg -i input.mp4 -c:v libx264 -preset veryfast -crf 23 -c:a aac -b:a 128k output.mp4
Social media optimized (SKILL.md default):
ffmpeg -i input.mp4 -c:v libx264 -preset medium -crf 20 -c:a aac -b:a 160k output.mp4
Troubleshooting
"No motion detected in face regions"
Face crop coordinates are wrong. Verify on a sample frame:
# Extract frame at 1 minute mark
ffmpeg -ss 00:01:00 -i source.mp4 -frames:v 1 sample.png
# Overlay crop rectangles (adjust x,y,w,h)
ffmpeg -i source.mp4 -ss 00:01:00 -frames:v 1 \
-vf "drawbox=x=200:y=100:w=300:h=200:color=red:t=5,drawbox=x=1100:y=100:w=300:h=200:color=blue:t=5" \
sample_boxes.png
Red = left face, blue = right face. Adjust coordinates until boxes frame each person's mouth/chin.
"Captions out of sync"
Whisper timestamps drift on long videos. Use smaller segments:
# Transcribe only the relevant 5-minute section
ffmpeg -i source.mp4 -ss 00:15:00 -to 00:20:00 -c copy segment.mp4
whisper segment.mp4 --model base --output_format json
Or enable Whisper's word-level timestamps:
whisper video.mp4 --model base --word_timestamps True
"Pan cuts too frequently"
Increase motion threshold in analyze.py:
# Default threshold
MOTION_THRESHOLD = 0.1
# Higher = less sensitive (fewer speaker changes)
MOTION_THRESHOLD = 0.3
Or add minimum duration between cuts in build_pan.py:
# Ignore speaker changes shorter than 2 seconds
MIN_SEGMENT_DURATION = 2.0
"ffmpeg not found"
Ensure ffmpeg is in PATH:
which ffmpeg
# Should print: /usr/local/bin/ffmpeg or similar
# If not installed:
brew install ffmpeg # macOS
sudo apt install ffmpeg # Ubuntu
"Whisper import error"
# Uninstall conflicting whisper packages
pip uninstall whisper openai-whisper
# Reinstall correct package
pip install openai-whisper
# Verify
python -c "import whisper; print(whisper.__version__)"
"VideoToolbox acceleration failed"
macOS-specific. Fallback to CPU:
# Remove -hwaccel videotoolbox from all ffmpeg commands
ffmpeg -i input.mp4 ... # (no -hwaccel flag)
Or use software decode explicitly:
ffmpeg -hwaccel none -i input.mp4 ...
Advanced: Integration with Custom Workflows
Use Clipify detection with external editor
# 1. Run Clipify's detection logic (without cutting)
# This would be in your custom script:
import subprocess
import json
# Transcribe
subprocess.run(["whisper", "source.mp4", "--model", "base", "--output_format", "json"])
# Load transcript
with open("source.json") as f:
transcript = json.load(f)
# Simple punchline detector (look for laughter indicators)
candidates = []
for segment in transcript["segments"]:
text = segment["text"].lower()
if any(word in text for word in ["haha", "lol", "crazy", "worst", "insane"]):
candidates.append({
"start": segment["start"],
"end": segment["end"],
"text": segment["text"]
})
print(json.dumps(candidates, indent=2))
# Output to DaVinci Resolve, Premiere, etc.
Export timeline for manual editing
# Generate EDL (Edit Decision List) from Clipify candidates
def to_edl(clips, fps=30):
edl = ["TITLE: Clipify Export", "FCM: NON-DROP FRAME", ""]
for i, clip in enumerate(clips, 1):
start_tc = frames_to_tc(int(clip["start"] * fps), fps)
end_tc = frames_to_tc(int(clip["end"] * fps), fps)
edl.append(f"{i:03d} AX V C {start_tc} {end_tc} 00:00:00:00 {end_tc}")
return "\n".join(edl)
def frames_to_tc(frames, fps):
h = frames // (fps * 3600)
m = (frames % (fps * 3600)) // (fps * 60)
s = (frames % (fps * 60)) // fps
f = frames % fps
return f"{h:02d}:{m:02d}:{s:02d}:{f:02d}"
# Usage
clips = [{"start": 154.2, "end": 171.8}, {"start": 492.5, "end": 509.1}]
print(to_edl(clips))
Custom caption animations
Modify ASS file for animated entrances:
Dialogue: 0,0:00:00.50,0:00:02.30,Default,,0,0,0,,{\fad(200,200)\move(640,1000,640,900)}This is animated text
\fad(200,200): 200ms fade in/out\move(x1,y1,x2,y2): Slide from bottom to center\t(0,500,\fscx120\fscy120): Scale animation over 500ms
Performance Tips
-
Use proxy files for preview: Transcode to lower resolution before clipify analysis
ffmpeg -i source.mp4 -vf scale=960:540 -c:v libx264 -crf 28 proxy.mp4 -
Skip transcription on re-runs: Cache Whisper JSON output
if [ ! -f source.json ]; then whisper source.mp4 --model base --output_format json fi -
Parallel clip rendering: Process multiple clips simultaneously
for clip in clip1 clip2 clip3; do ffmpeg -i $clip.mp4 -vf "..." ${clip}_out.mp4 & done wait -
GPU acceleration: Use NVIDIA NVENC for faster encoding
ffmpeg -hwaccel cuda -i input.mp4 -c:v h264_nvenc -preset p4 -crf 20 output.mp4