Skip to content
Quantoros

Article

The answer is already there. Why does the model keep thinking?

A look at the paper When EOS Tokens Disagree: how mismatched teacher and student EOS tokens in on-policy distillation bloat outputs, and what that means for model evaluation.

In one experiment, Qwen3 reached the correct answer in 1,094 tokens. It then generated another 7,098 tokens of repetition and hit the length limit. More than 86% of the output came after the solution had already been found. Not extra depth of reasoning, just an inability to stop.

On September 17, the paper When EOS Tokens Disagree came out, examining one of the causes of this behavior. It's about on-policy distillation. The student model generates its own solutions, and the teacher model provides a training signal for the tokens the student picked. The idea is to train the student on its own attempts rather than on someone else's finished answers.

When teacher and student end differently

The problem turned up in a surprisingly mundane place. The teacher and the student may mark the end of a response with different special tokens, known as EOS. Semantically, both mean "that's it, we're done," but at the probability level they prefer different options. Even an identical list of allowed stop tokens doesn't guarantee that the models use them the same way.

Now for the unpleasant part.

The student picks its usual end-of-sequence token, and the teacher assigns exactly that token a low probability because it prefers a different one. Training suppresses the student's usual way of stopping. Meanwhile, the student almost never generates the alternative token, so it gets very little direct training signal for it either. It's a peculiar kind of pedagogy: you get punished for trying to finish, but nobody really teaches you how to finish properly.

How they fixed it

Simply adding both tokens to the generation config turned out not to be enough. The authors proposed treating the probabilities of functionally equivalent EOS tokens as the probability of a single "stop" action. In other words, fixing not just decoding but the training signal itself. In experiments with Qwen3, Llama, and Gemma, this substantially reduced output bloat and the rate of truncation at the length limit.

What the paper doesn't explain

But it's important not to turn an interesting finding into a universal explanation. The study covers a specific variant of distillation on single-answer math problems. In additional experiments with K2-Horizon, the late growth in length persisted even after the EOS fix. And stopping more correctly didn't guarantee higher math accuracy. They found a significant mechanism, but didn't explain the whole behavior.

The practical takeaway

I'd put the practical takeaway like this: when evaluating a model, it's not enough to ask whether it got the right answer. You should check when it got there, what it did afterwards, and whether it stopped on its own. Otherwise a working solution and thousands of useless repetitions can end up hidden behind a good metric.

Every additional token has to prove it's worth generating.

Further reading

New articles by email