EmbeddingGemma 2: How natively multimodal embeddings work
Summary
EmbeddingGemma 2 is a lightweight, natively multimodal embedding model designed to map diverse data types—including text, code, images, video, and audio—into a single, unified embedding space. The model achieves this through modality-specific encoders feeding into a bidirectional transformer, which is then refined using a two-stage training process involving distillation from Gemini Embedding 2 and finetuning with contrastive and spread-out loss objectives. Key efficiency features include Matryoshka Representation Learning and quantization down to INT4, enabling fast deployment on edge devices.
Key takeaways
-
Unified Multimodal Representation
EmbeddingGemma 2 natively understands interleaved sequences of different modalities (text, code, images, video, audio) to create a single unified representation, allowing, for example, a video to be searched using its audio clip.
-
Two-Stage Training Regime
4:34
The model is initialized from a pre-trained Gemma 4 decoder model, then learns from the larger teacher, Gemini Embedding 2, via distillation, followed by finetuning using contrastive and spread-out loss objectives.
-
Efficiency and Deployment
3:37
The model supports Matryoshka Representation Learning to nest embeddings of decreasing size, and utilizes quantization aware training to compress the footprint down to INT4, making it suitable for resource-constrained environments.
Technical details
-
Input Processing Pipeline
103s
Raw inputs pass through modality-specific encoders. The vision encoder processes images/video frames into patches, and the audio encoder converts sound waveforms into spectrograms, which are then sliced into 40 millisecond segments before being projected into a shared space.
-
Core Architecture
141s
A bidirectional transformer contextualizes and blends information across modalities. The intermediate output is collapsed into a single embedding via three steps: Mean pooling, a dense layer projection, and normalization onto the unit sphere.
-
Loss Functions
224s
Fineluning employs contrastive loss (where positive pairs are pulled closer and negative pairs are pushed apart) and spread out loss (to ensure embeddings are dispersed and robust to quantization). Same tower negatives are also used to prevent modality bias.
Mentioned resources
Channel & topics
Watch on YouTube · Back to latest
This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.