Meta AI has developed the Perception Encoder Audiovisual (PE AV), a sophisticated model designed for integrated audio and video understanding. By employing large-scale contrastive training on approximately 100 million audio-video pairs with text captions, PE AV aligns audio, video, and text representations within a unified embedding space. This model architecture includes separate encoders for video and audio, an audio-video fusion encoder, and a text encoder, enabling versatile retrieval and classification tasks across multiple domains. PE AV achieves state-of-the-art performance on various benchmarks, significantly enhancing the accuracy and efficiency of cross-modal retrieval and understanding, which is crucial for advancing multimedia AI applications.
