When a large language model gets a fact wrong, the instinctive fix is to train it on more data or scale it up. A new paper from Google Research challenges that assumption. The real culprit is often a lost key, not an empty shelf.

When LLMs get facts wrong, the failure could stem from never learning them or from being unable to recall what's already encoded. The knowledge profiling framework introduced in this work points to the latter: frontier LLMs encode nearly all facts, yet struggle to retrieve many of them. That distinction reshapes how the field should approach factuality.

The diagnosis problem hiding in plain sight

Factuality is essential for making LLMs reliable. When a model answers incorrectly, the fact may have never been encoded, or it may be encoded but inaccessible. Standard accuracy metrics collapse these cases together, even though they imply very different limitations and very different interventions.

Encoding failures call for scaling model size or expanding data coverage. Recall failures point toward post-training and inference-time methods that help LLMs better use what they already store. Treating both as the same problem means applying the wrong fix, like buying more groceries when the issue is that you can't find what's already in the fridge.

A new lens: knowledge profiling

The paper introduces knowledge profiling, a behavioral framework that analyzes factual behavior at the level of individual facts rather than individual questions. Instead of asking "did the model answer this question correctly?", it asks "what is the state of this fact inside the model?" Each fact gets classified into one of five profiles:

• Encoding failure: the fact was never stored in the model's weights
• Recall failure: the fact is encoded but the model can't retrieve it
• Direct recall: the model answers correctly without any extra prompting
• Recall with thinking: the model needs chain-of-thought or extended reasoning to surface the fact