Empty shelves or lost keys? Recall is the bottleneck for parametric factuality
Frontier LLMs often fail at factual questions not because they lack the information, but because they cannot reliably retrieve information already encoded in their parameters. The authors’ knowledge-profiling framework finds that models such as Gemini 3 and GPT-5 encode 95–98% of tested facts, yet still fail to directly recall 26–34% of them. This suggests that improving factuality increasingly depends on better knowledge utilization, not simply larger models or more training data. ## Knowledge Profiling - The framework evaluates facts rather than isolated question-answer pairs. - Each fact is assigned one of five profiles: - Encoding failure - Recall failure - Direct recall - Recall with thinking - Inference without encoding - It distinguishes three behaviors: - **Encoding:** reproducing a fact in a pre-training-like context. - **Knowledge:** answering semantically equivalent direct and reverse questions. - **Recall:** retrieving an encoded fact without external cues. - “Thinking” can help models recover facts through intermediate reasoning, multi-hop inference, or educated guesses. ## The WikiProfile Benchmark - WikiProfile contains 2,150 facts extracted from Wikipedia. - Each fact is tested with 10 tasks: - Two encoding tasks - Four knowledge questions - Four multiple-choice recognition questions - Questions include different phrasings and relational directions, such as asking either what entity *B* is associated with or what entity *A* is associated with. - An automated pipeline using Gemini-2.5-Pro generated, refined, searched, and filtered questions, followed by manual validation. - The study evaluated 13 LLMs, with and without thinking. - Eight responses were sampled for every model, fact, and task, producing roughly 4.5 million responses graded by LLM-based autoraters. ## Recall, Not Encoding, Is the Main Bottleneck - Frontier models show near-saturated factual encoding: - Gemini 3 and GPT-5 encode approximately 95–98% of the benchmark’s facts. - They nevertheless fail to directly recall 26–34% of those facts. - Thinking reduces, but does not eliminate, the failures; 11–12% remain inaccessible. - Scaling improves encoding more strongly than recall. - In the Gemma 3 family, larger models have substantially fewer encoding failures, while recall failures remain significant and account for a larger share of residual errors. - The results indicate a shift from knowledge acquisition toward knowledge accessibility and utilization. ## Why Recall Fails - Recall appears sensitive to the conditions under which a fact was learned. - Retrieval becomes more difficult when a question differs from the fact’s training-time context, wording, or ordering. - Rare, long-tail facts illustrate this distinction: - Their encoding rates are relatively close to those of popular facts. - Their recall rates are substantially worse. - This suggests that many rare facts are present in model parameters but are difficult to access, rather than completely absent. The practical implication is that future factuality improvements should target retrieval, post-training, prompting, and inference-time reasoning—not only model scale or broader data coverage.
Read original(opens in new tab)