# How to compress text to 1 bit per character | Compression is Intelligence Part 3

## Executive summary

The video explores advanced data compression techniques, arguing that the ability to predict data (intelligence) is fundamentally linked to the efficiency of compression. It details how modern algorithms, particularly those leveraging Language Models (LMs), can compress text far below theoretical limits like Huffman encoding. The core mechanism involves encoding text into a unique real number between 0 and 1, which is then represented in binary. The discussion highlights that the final compression rate depends critically on the quality of the underlying predictive model and its ability to minimize the cross-entropy loss function.

## Key takeaways

- Compression is tied to predictability (Shannon's Limit): Claude Shannon established that optimal compression requires the number of bits to be proportional to the negative log base two of the message's probability. The video demonstrates that highly predictable text can achieve compression rates far better than simple methods like ASCII or Huffman encoding.
- Arithmetic Coding and Real Number Encoding: A text string is encoded into a unique real number in the unit interval [0, 1]. The decoding process uses a shared Language Model (LM) to iteratively narrow down the possible characters based on the number's position in the interval, requiring the LM to be identical for both sender and receiver.
- Cross-Entropy and Model Training: The compression size is mathematically linked to the cross-entropy of the model's distribution of language against the true distribution. This suggests that optimizing an LM to minimize cross-entropy is equivalent to optimizing it as a text compressor.
- Advanced Compression Techniques: The video introduces Arithmetic Coding (developed by Rissanen and Pascoe) and Asymmetric Numeral Systems (developed by Yarik Duda) as practical methods that solve the issue of ever-growing numerical precision required for real number encoding.

## Technical details

- Information Theory/Compression Limits: Shannon's theorem states that the optimal number of bits for compression is approximately $-\log_2(P)$, where $P$ is the probability of the message. The goal is lossless compression, requiring the output to be an exact replica of the input.
- Encoding Mechanism: Text is encoded into a real number in [0, 1]. The probability of a sequence of characters (e.g., M-A-T-H) is found by multiplying the conditional probabilities (chain rule): $P(M, A, T, H) = P(M) \cdot P(A|M) \cdot P(T|M, A) \cdot P(H|M, A, T)$.
- Cross-Entropy Loss: The average cost of prediction is measured by cross-entropy, which acts as the loss function during pre-training. Minimizing this value is mathematically equivalent to maximizing the compression ratio.
- Arithmetic Coding: A practical algorithm (developed in 1976) that avoids the need for ever-growing numerical precision by renormalizing the interval at each step, offering a nearly perfect way to turn a predictor into a compressor.

## Practical implications

- The compression size of a data set is highly dependent on the quality and predictive capacity of the underlying Language Model (LM).
- For practical implementation, algorithms like Arithmetic Coding are necessary to manage numerical precision issues inherent in real number encoding.
- The concept suggests that a smaller model with equivalent predictive capacity to a larger model is considered superior in terms of overall compression efficiency (Hutter Prize metric).

## Topics

Data Compression, Information Theory, Language Models (LLMs), Cross-Entropy, Machine Learning Optimization, Hutter Prize, Arithmetic Codes, Asymmetric Numeral System

Source: https://www.youtube.com/watch?v=qbsb9DpdZ6s
