Paper Reading: jina-embeddings-v5-omni & the GELATO Approach to Multimodal Embeddings

Description

In this paper reading, we walk through GELATO (Geometry-preserving Embeddings via Locked Aligned Towers), a novel architecture from Jina AI that extends text embedding models to handle images, audio and video in a single unified embedding space. The core insight: rather than training a new multimodal model from scratch, GELATO keeps the backbone text model and modality encoders frozen and trains only the small connecting projectors, just 0.35% of total model weights. The result is the jina-embeddings-v5-omni suite, which achieves competitive performance with larger multimodal embedding models. Paper: https://arxiv.org/abs/2605.08384 0:00 Introduction — the GELATO paper 0:33 Background: how vision-language models work 3:10 Adding more modalities (audio) 4:07 GELATO's twist: a frozen embedding model, train only the projectors 5:42 Architecture: the frozen jina-embeddings-v5 text model 6:43 Inside the text embedding / tokenization layer 8:11 The problem: telling image and text tokens apart 10:32 The fix: learned vision start/end tokens 12:20 Training setup: projectors, frozen encoders, four LoRA variants 14:27 Where the projectors come from (Qwen VL & Omni) + the dimension fix 17:43 Vision projector: which layers to tune 19:37 Two-stage training: align first, then train jointly 22:31 Audio projector & the generalization trade-off 24:46 Wrap-up & feedback --- What we cover: The GELATO architecture and why frozen encoders make sense How image and audio encoders (from Qwen3.5 and Qwen2.5-Omni) are integrated Geometry preservation: keeping text embedding space intact while adding modalities Benchmark results and comparisons to other multimodal embedding approaches Matryoshka truncation and task LoRA adapters --- This is part of our community paper reading series: unscripted, technical and designed for ML engineers and researchers who want to go deeper than the abstract. Subscribe for more community paper readings, workshops and deep dives: https://www.youtube.com/@OfficialElasticCommunity --- #embeddings #multimodalAI #RAG #vectorsearch #machinelearning #NLP #elasticsearchad