V-JEPA: A visual model launched by Meta that can learn to understand the physical world by watching videos

What is V-JEPA?

V-JEPA (Video Joint-Embedding Predictive Architecture) is a new video self-supervised learning method launched by researchers at Meta, focusing on learning visual representations of videos through feature prediction. The core idea of this method is to enable the model to predict the feature representation of one region in a video (called the target region y), based on the feature representation of another region (called the source region x). This prediction process is carried out without external supervision (such as annotations or pretrained image encoders) and relies entirely on the structure and content of the video data itself.

What makes V-JEPA different is its self-supervised learning method, which can predict the missing parts of a video within an abstract feature space, rather than generative methods that fill in missing pixels. This technique builds a conceptual understanding of video clips through passive observation, much like humans, rather than through manual annotation.

• Project introduction:https://ai.meta.com/blog/v-jepa-yann-lecun-ai-model-video-joint-embedding-predictive-architecture/
• Research paper:https://scontent-hkg1-2.xx.fbcdn.net/v/t39.2365-6/427986745_768441298640104_1604906292521363076_n.pdf
• GitHub code repository:https://github.com/facebookresearch/jepa

Main features of V-JEPA

• Self-supervised learning: V-JEPA does not rely on pretrained image encoders, text, negative examples, pixel-level reconstruction, or other forms of external supervision. It learns visual representations entirely through feature prediction from the video data itself.
• Feature prediction objective: The core objective of V-JEPA is to predict feature representations between video frames. This objective allows the model to learn temporal continuity and spatial structure in videos, not just pixel-level information.
• Joint-embedding architecture: V-JEPA adopts a special network architecture that includes an encoder (x-encoder) and a predictor. The encoder is responsible for extracting feature representations of video frames, while the predictor predicts the features of the target frame based on these features.
• Multi-block masking strategy: During training, V-JEPA uses a multi-block masking strategy to process video frames. This method forces the model to learn more robust and comprehensive video representations by masking different regions at different time points in the video.
• Efficient pretraining dataset: V-JEPA is pretrained on a large-scale dataset consisting of 2 million videos, which come from multiple public datasets such as HowTo100M, Kinetics-400/600/700, and Something-Something-v2.
• No model parameter tuning required: Models trained by V-JEPA perform well on a variety of downstream tasks without the need to tune model parameters. This means the model can be evaluated directly in a frozen state, or adapted to new tasks with a small amount of fine-tuning.
• Label efficiency: V-JEPA can also achieve good performance with limited labeled data, which indicates that it has an advantage in label efficiency, and this is especially important in scenarios where data annotation costs are high.
• Cross-modal performance: V-JEPA not only performs well on video tasks, such as action recognition and motion classification, but is also competitive on image tasks, such as ImageNet image classification.
• Fast training: V-JEPA shows high efficiency during training and can learn effective visual representations in a relatively short time, which makes its application on large-scale video datasets possible.

How V-JEPA works

V-JEPA works based on self-supervised learning, and it trains the model by predicting feature representations between video frames.

The following are the detailed steps of the V-JEPA workflow:

1. Video preprocessing: First, randomly sample a series of frames from the input video (for example, 16 frames), and convert these frames into a format suitable for model processing. This usually includes adjusting the spatial resolution of the video frames to the size required by the model (such as 224×224 pixels), and converting the frame sequence into a series of spatio-temporal "tokens".
2. Encoder: The video frames pass through a vision Transformer (ViT) encoder, which converts the video frames into a series of feature representations. The encoder usually consists of multiple Transformer layers and can capture the spatial and temporal relationships between video frames.
3. Masking: In the feature representations of the video frames, randomly select a portion of regions to mask, and these masked regions will serve as prediction targets. The masking can be short-range or long-range to ensure that the model can learn features at different scales of the video content.
4. Predictor: The predictor network receives the feature representations output by the encoder and attempts to predict the features of the masked regions. The predictor is usually a narrower Transformer, which generates predictions by learning the feature relationships between video frames.
5. Loss function: V-JEPA uses a loss function to measure the difference between predicted features and actual features. This loss function is usually L1 loss, which calculates the mean absolute error between the predicted features and the target features.
6. Training process: During training, the model adjusts the weights of the encoder and predictor through the backpropagation algorithm to minimize the loss function. At the same time, the encoder weights are updated using exponential moving average (EMA) to maintain the stability of the model.
7. Downstream task evaluation: After pretraining is completed, the V-JEPA model can be evaluated on various downstream tasks, such as action recognition, motion classification, etc. In these tasks, the model can extract feature representations of videos through a lightweight network called an "attentive probe", and then use them for classification or other tasks.
8. Fine-tuning: If needed, the V-JEPA model can be fine-tuned on specific tasks. This usually involves adding one or more fully connected layers on top of the pretrained model and training on a labeled dataset.