# Stop Chunking Like It's 2022 — Yuval Belfer, AI21 Labs

## Executive summary

This talk challenges the fixed-chunking paradigm in Retrieval-Augmented Generation (RAG) systems. The core claim is that there is no single optimal chunk size; the correct size is query-dependent. The speaker proposes 'multiscale indexing,' which involves duplicating the corpus and chunking it at multiple window sizes. At retrieval time, all resulting indices are queried, and the results are merged using Reciprocal Rank Fusion (RRF), achieving significant recall improvements (20-40%) over fixed-size methods. This approach increases memory usage but maintains low latency overhead.

## Key takeaways

- Chunking is Query-Dependent, Not Fixed: The optimal chunk size varies drastically based on the query type. For example, a highly focused question (e.g., 'Jerry's favorite church') performs best with smaller chunks, while a broad, narrative question (e.g., 'who does Jerry describe as his nemesis') requires larger context windows to capture the answer spread across a scene. (7:07)
- Multiscale Indexing is the Solution: Instead of committing to one chunk size, the system should index the corpus at multiple window sizes (multiscale indexing). At query time, all these indices are queried, and the resulting rankings are combined using Reciprocal Rank Fusion (RRF). (9:59)
- Significant Performance Gains: The gap between the performance of the optimal 'oracle' chunk size and any single fixed choice (e.g., 512 tokens) can cost 20% to 40% of recall. The proposed method closes this gap. (9:59)

## Technical details

- Chunking as Lossy Compression: The speaker argues that chunking is inherently a lossy compression process, meaning some information is inevitably lost regardless of the strategy. (4:17)
- The Oracle Experiment: The experiment demonstrated that for a given query, selecting the best chunk size (the 'oracle') yields significantly higher recall than any fixed chunk size. (8:12)
- Multiscale Indexing and RRF: The proposed architecture involves: 1) Indexing the corpus multiple times (N duplicates) with different chunk sizes. 2) Querying all N indices per query. 3) Combining the N rankings of the full documents using Reciprocal Rank Fusion (RRF). (9:59)

## Practical implications

- System architects must account for increased memory usage (2 to 5 times the original memory) to store the multiple indexed copies.
- The proposed method adds minimal latency overhead because the RRF combination step is computationally simple and fast.
- The implementation is described as a 'simple script' rather than a complex model, making it relatively straightforward to integrate into existing RAG pipelines.

## Topics

Retrieval-Augmented Generation (RAG), Vector Databases, Information Retrieval, Natural Language Processing (NLP), Indexing Strategies, Seinfeld Dataset, Blog/Code Example

Source: https://www.youtube.com/watch?v=r9OwPx_HoV0