Clips AI is an open-source Python library designed to help developers automatically convert long videos into clips. It identifies and creates clips by analyzing the transcribed text of the video, and can dynamically reframe the video from 16:9 to different aspect ratios such as 9:16, focusing on the current speaker.
Main capabilities:
- Automatically split long videos into multiple clips.
- Adjust the video aspect ratio from 16:9 to 9:16.
- Clip identification algorithm based on transcribed text.
- Dynamic reframing, focusing on the current speaker.
How to use: Clips AI is a Python library that can be installed via pip. It is recommended to use a virtual environment (such as venv) to avoid dependency conflicts. Installation command:
pip install clipsai
pip install whisperx@git+https://github.com/m-bain/whisperx.git
libmagic also needs to be installed. Since clips are found based on the video transcription text, the video must first be transcribed. Transcription is done using WhisperX, an open-source wrapper based on Whisper with the added ability to detect the start and stop time of each word. To crop the original video into the selected clips, refer to the cropping reference documentation. Resizing the video requires a Hugging Face access token because Pyannote is used for speaker diarization. Using Pyannote does not incur a charge, and related instructions can be found on the Pyannote HuggingFace page. To adjust the original video to the desired aspect ratio, refer to the resizing reference documentation.
Use cases: Clips AI is designed for audio-centric, narrative-based videos, such as podcasts, interviews, talks, and sermons.
Limitations: Resizing the video requires a Hugging Face access token. Other limitations are not provided at this time.
