# EmbeddingGemma 2: How natively multimodal embeddings work

## Executive summary

EmbeddingGemma 2 is a lightweight, natively multimodal embedding model designed to map diverse data types—including text, code, images, video, and audio—into a single, unified embedding space. The model achieves this through modality-specific encoders feeding into a bidirectional transformer, which is then refined using a two-stage training process involving distillation from Gemini Embedding 2 and finetuning with contrastive and spread-out loss objectives. Key efficiency features include Matryoshka Representation Learning and quantization down to INT4, enabling fast deployment on edge devices.

## Key takeaways

- Unified Multimodal Representation: EmbeddingGemma 2 natively understands interleaved sequences of different modalities (text, code, images, video, audio) to create a single unified representation, allowing, for example, a video to be searched using its audio clip.
- Two-Stage Training Regime: The model is initialized from a pre-trained Gemma 4 decoder model, then learns from the larger teacher, Gemini Embedding 2, via distillation, followed by finetuning using contrastive and spread-out loss objectives.
- Efficiency and Deployment: The model supports Matryoshka Representation Learning to nest embeddings of decreasing size, and utilizes quantization aware training to compress the footprint down to INT4, making it suitable for resource-constrained environments.

## Technical details

- Input Processing Pipeline: Raw inputs pass through modality-specific encoders. The vision encoder processes images/video frames into patches, and the audio encoder converts sound waveforms into spectrograms, which are then sliced into 40 millisecond segments before being projected into a shared space.
- Core Architecture: A bidirectional transformer contextualizes and blends information across modalities. The intermediate output is collapsed into a single embedding via three steps: Mean pooling, a dense layer projection, and normalization onto the unit sphere.
- Loss Functions: Fineluning employs contrastive loss (where positive pairs are pulled closer and negative pairs are pushed apart) and spread out loss (to ensure embeddings are dispersed and robust to quantization). Same tower negatives are also used to prevent modality bias.

## Practical implications

- Enables multimodal retrieval (e.g., searching a video library using audio clips or retrieving audio effects using text prompts) in Python.
- Allows developers to choose a quality/performance trade-off using Matryoshka Representation Learning without retraining.
- Facilitates deployment on edge devices due to INT4 quantization and reduced model footprint.

## Topics

Multimodal AI, Embeddings, Generative AI, Resource Optimization, Vector Databases, EmbeddingGemma 2 tech report, Model Weights

Source: https://www.youtube.com/watch?v=aRIFzzr_Mi0
