How much does a language model actually remember? It sounds like a simple question, but it has resisted clean answers for years. Extraction attacks, membership inference, differential privacy bounds , all useful tools, but none of them directly answer the core question: how many bits of specific training data are locked inside a model's weights? A new ICML paper from researchers at Meta, Google DeepMind, Cornell, and NVIDIA finally puts a number on it.

The measurement problem nobody solved

The field has long struggled with a fundamental confusion: when a model outputs something that was in its training set, is that memorization or generalization? Language models can be coerced to output almost any string, so the fact that a model outputs something is not necessarily a sign of memorization. A model that correctly answers a math problem it was trained on might simply be doing math , not recalling a specific example.

Prior studies of language model memorization struggled to disentangle memorization from generalization. The paper formally separates memorization into two components: unintended memorization (the information a model contains about a specific dataset) and generalization (the information a model contains about the true data-generation process). This distinction is what makes the measurement possible.

Compression as the measuring stick

The key insight is borrowed from information theory: if a model helps you compress a dataset, it must contain information about that dataset. The authors use Kolmogorov complexity , the theoretical minimum description length of a piece of data , as the formal backbone, then approximate it in practice using the model's own log-likelihoods. Concretely, unintended memorization of a sample is measured as how many fewer bits you need to encode it when the trained model is available as a reference, compared to using only a baseline model trained on the full distribution.

To isolate pure memorization capacity, the team first eliminated generalization entirely by training on datasets of