Skip to content
Quantoros

Article

Hallucinations and Discoveries Grow from the Same Place

Why LLM hallucinations and discoveries are linked: monofacts, the Good-Turing estimate, experiments with gpt-oss-20b and gemma-3-12b, and a parallel with human memory.

There's a well-known result by Kalai and Vempala: even a perfectly calibrated language model is bound to hallucinate. The hallucination rate is bounded from below by the share of "monofacts", facts that appear in the training data exactly once. The logic is simple: if the model has seen a fact only once, it can't reliably tell it apart from a plausible fabrication of the same kind.

A recent paper looks at the same picture from the other side. If monofacts give rise to hallucinations, could they also give rise to discoveries?

A bit of theory

The idea goes back to the Good-Turing estimate (the very one from breaking Enigma): the fraction of items seen exactly once in a sample estimates the probability that the next item will be new. In other words, monofacts tell you how much "unseen" probability mass is left in the distribution. Hallucinations come from that mass, but discoveries, true statements that weren't in the data, can only come from there as well.

The authors introduce a "frontier": a zone of unseen statements that aren't obvious yet but aren't pure noise either. Within it, discoveries and hallucinations occur in a fixed ratio, and the capacity for discovery is bounded from above and below by the monofact rate. To put it simply: a model with no uncertainty left won't find anything new.

The experiment

They took gpt-oss-20b and gemma-3-12b and fine-tuned them on datasets where the monofact rate was varied from 0 to 1, all else being equal. The tasks came from two domains:

  • materials science: propose a chemical formula with given properties. Answers count as correct if the material appeared in Materials Project after the training snapshot;
  • genomics: edit a DNA sequence to achieve the desired activity in three cell types.

The result: the more monofacts, the more discoveries. The trend is monotonic for both models and both domains, with and without reasoning traces, with a single attempt and with sixteen.

The most interesting ablation is about diversity. The obvious objection is that monofacts simply make the data more diverse. To test this, the authors trained for 3 and 5 epochs. Same data, but now each fact is seen several times. The effect disappeared, and at high monofact rates a single epoch beat multiple epochs. What matters is not just diversity, but the fact that the model saw each fact exactly once.

The cost is clear too: within the training distribution, as the monofact rate grows, both discoveries and hallucinations go up. You can't separate one from the other for free; it's the same mechanism.

What to keep in mind

These are mid-sized models, this is fine-tuning rather than pretraining, and a discovery is counted as an exact match with a held-out answer, with no lab validation. The theoretical bounds are elegant, but the frontier's parameters are hard to estimate in practice.

A parallel with human memory

But the most interesting part here is the parallel with human memory. Schacter and Addis at Harvard have long shown that memory doesn't replay the past but reassembles it from fragments every time. Remembering the past and imagining the future rely largely on the same processes and brain regions. The price of this flexibility is false memories, when we confidently "remember" things that never happened.

And it shows up in the data: in a 2021 study, people who came up with more alternative uses for an everyday object were more likely to falsely recall words that weren't on the list. Creativity and memory errors grow from the same ability to recombine what we've seen.

It looks like the same story holds for LLMs. A model that only reproduces what it has learned exactly is safe, but sterile. We're used to treating the long tail of the data as noise and a candidate for filtering, but it may well be exactly where the room for something new remains. Then hallucinations aren't a bug to be scrubbed down to zero, but a tax on the capacity for novelty. The question isn't how to eliminate them, but how to filter them at the output.

Sources

New articles by email