Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?

0:00Hey there, paper enthusiasts. Welcome to another episode of the Daily Papercast, where we explore some of the latest and most intriguing research papers in the world of AI, NLP, CV, and all things tech. And I'm Nova. Today, we have an exciting episode lined up for you. Packed into four parts, just like a well-structured research paper, introduction, methods, experiments, and related work. We're going to dive deep and unpack the contents of an insightful paper that we're pretty excited about. Totally Nova, so grab your coffee, settle in, and let's dive right in. Today's paper is titled, Why Does Self-Distillation Sometimes Degrade the Reasoning Capability of LLMs. By John Hay Kim, Tsufang Luo, Minbom Kim, Sangmuk Lee, Donghyong Kim, Jiwon John, Dongsheng Lee, and Yuching Yang, from Microsoft Research, KIST, and Soul National University. That's quite the team. Now, let's start with the introduction.

1:02So, what's this paper all about, Echo? Well, Nova, this paper tackles a growing area of interest in the post-training of large language models. Self-Distillation, it's a method where two instances of the same model are utilized. One conditioned on ground truth solutions serves as the teacher, and another instance learns from it without having direct access to these solutions. Interesting. So, basically, it's like the model is teaching itself. How neat is that? Exactly. They've highlighted how earlier research demonstrated that combining self-distillation with methods like reinforcement learning from verifiable rewards, RLVR, can lead to significant performance gains. But, there's a twist when it comes to mathematical reasoning tasks. Oh, a twist? Do tell. Sure thing, while self-distillation often results in more concise and effective reasoning in many domains, in the realm of mathematical reasoning, it may actually hamper performance.

2:02The researchers found that it can lead to shorter responses and a drop in effectiveness, particularly because it suppresses epistemic verbalization. The model's expression of uncertainty during reasoning. Ah, so by cutting down on those unsure moments, the model might miss out on important reflective pauses that could lead to better reasoning. Makes sense. What did they aim to uncover with this research? Good question. They conducted a comprehensive empirical study to pinpoint exactly when and why self-distillation seems to suppress this crucial aspect of reasoning. They wanted to understand how richer information contexts and broader task coverage influence this suppression. So, they're digging into the nitty gritty of how information transparency and task diversity play roles here. What were their key findings, Echo? They found two main factors at play. Information richness and task coverage. When the teacher model has access to richer information, it results in reasoning trajectories with less uncertainty. This makes the student model imitate a confident reasoning

3:05style that's less suited for generalization, especially for out of distribution problems. So, richer isn't always better, interesting. And that comprehensive empirical study, they use this to essentially show that limiting epistemic verbalization can reduce performance on new or challenging tasks. Exactly, Nova. Their results reveal that even if the model is guided correctly towards the right answers during training, suppressing those little signals of uncertainty can quietly shift its reasoning style and hurt its ability to generalize well. Wow. That's a pretty significant insight for improving LLM's robustness and versatility. Anything else noteworthy from the introduction? Yes, another cool insight is that the authors emphasize the need to not only optimize for answer correctness, but also to encourage uncertainty-aware reasoning behaviors in post-training objectives. This is, they believe, a step towards deeper understanding and better application of self-distillation and other post-training methods for LLMs.

4:06Makes total sense. Encouraging more nuanced reasoning rather than just correct, confident answers could be a game changer. And with that, we've come to the end of the introduction section. Stay tuned as we jump into the methods used in this intriguing research right after this short break. All right, Nova. So let's dive into the method section of this paper. This part is where things get really meaty. Are you ready for the nitty-gritty details? Absolutely, Echo. Let's get into the heart of the methodology. From what I understand, the paper focuses on self-distillation for large language models, right? Yes, exactly. The authors start by providing a bit of context on self-distillation. Essentially, self-distillation involves the model acting as both the student and the teacher. The student generates a sequence based on input, and the teacher is tasked with predicting this sequence, but relies on a richer context. Interesting. So it's like the model is learning from itself, but with a little extra guidance.

5:08How's this richer context defined? Like, what kind of additional information are we talking about here? Good question, Nova. The context can include various forms of auxiliary signals, like solutions to problems, environment feedback, or any additional information that can help the model. The goal is to make the student model match the predictions of the teacher model, which is working with this extra context. Got it. That makes a lot of sense. The richer the context, the more information the teacher has to give precise guidance. So how do they actually measure this? Is there a specific metric or something? Yes. They quantify the informativeness of the context, using something called conditional mutual information. It's basically the reduction in uncertainty about the output sequence given this additional context. For example, in one of their experiments, they used a data set called DAPO-MAS-17K and compared the model's responses under different settings of context richness.

6:09Conditional mutual information sounds fancy. So in these experiments, they have different levels of context richness. Can you break down the different settings they tested? Sure thing. They have four main settings, unguided generation, solution-guided generation with full context, solution-guided without specific parts called think tags and regeneration condition generation, where the context is a previously generated response. Each of these levels increases in richness of the context, which directly impacts the model's performance. I see. So the richest context would be the full solution, and the least would be unguided generation. How do these settings impact the model's response length and epistemic verbalization? You know, those markers of uncertainty, like maybe, or perhaps. Great question, Nova. As the context gets richer, the responses become more concise and confident with fewer epistemic tokens.

7:11For instance, unguided generation led to much longer responses and higher counts of epistemic tokens. On the flip side, solution-guided generation with full context had the shortest and most confident responses with minimal epistemic verbalization. Ah, that makes sense. So the richer the context, the more the model can rely on it to make confident predictions. Therefore, reducing those um and maybe tokens. But that also means the model might miss out on exploring different hypotheses, right? Exactly. That's one of the key takeaways. While richer context can compress the reasoning process making the model more concise, it risks missing out on valuable exploration and error correction. So there's a fine balance to strike. Fascinating. So they also evaluated the impact of different training methods, right? Specifically, GRPO and SDPO. What's the difference between these two? Yes. GRPO, or generative reward proximal policy optimization

8:13and SDPO, or self distilled proximal policy optimization, are two different approaches they tested. GRPO tends to increase response length by promoting epistemic verbalization and achieve strong training gains. SDPO, on the other hand, reduces response length more aggressively, but struggles to improve, often showing slight out-of-distribution degradation. So, while SDPO might lead to more concise responses, it also risks underperformance in new unseen scenarios. It's like it's too aggressive in suppressing uncertainty, which sometimes is necessary for flexible reasoning. That's a bit of a double-edged sword, huh? Indeed. The balance between being concise and retaining useful epistemic signals is crucial. They actually conducted a comparison between fixed and moving target teachers in their study. A moving target where the teacher's policy updates continuously can introduce instability. They found that a fixed teacher policy yielded better performance overall.

9:14Interesting. What were some of the key metrics or benchmarks they used to evaluate these methods? They used metrics like response length, training score, and out-of-distribution performance on benchmarks such as AM24 and AMC23. For instance, they noted that while GRPO maintains slightly longer responses with stable performance, SDPO's more aggressive reduction in response length often leads to performance drops on challenging benchmarks like AM24. AM24 and AMC23, those are pretty tough benchmarks. It sounds like a really thorough evaluation. Did they also delve into how the different models handle epistemic tokens across various settings? Yes, they did. They analyzed how models like DeepSeek R1 is still QN7B and QN38B with and without thinking mode handle these tokens. Interestingly, thinking-enabled models generally produce more epistemic tokens.

10:15However, solution-guided generation drastically reduces these tokens across all models, showing how powerful context richness can be in shaping model behavior. Wow, that's a lot of depth. So what about the final takeaways from their oblation studies on different hyperbarameters, like learning rate and top-k distillation? They conducted additional experiments to see the effects of different training hyperparameters, like varying the top-k distillation parameter and the learning rate. They found that increasing top-k from 100 to 256 didn't significantly affect the training dynamics or final performance. However, reducing the learning rate from 1e-5 to 1e-6 only slowed the rate of degradation, but didn't prevent it. It suggests that these changes failed to mitigate the aggressive suppression of epistemic verbalization in SDPO. It sounds like one of those, you win some, you lose some situations. Enhance in certain aspects can sometimes

11:15lead to losses in others. Balancing these factors is key to creating robust models. Absolutely. It's all about finding that sweet spot where the model can be both confident and flexible in its reasoning, retaining useful epistemic expressions while compressing the verbose ones. And that wraps up the method section. All right, Nova. So we've got some pretty compelling findings from the experiments and results section of the paper. Shall we dive into how this paper tackled the experiments? Absolutely, Echo. So the paper conducts a comprehensive set of experiments to understand the impact of different guiding methods on reasoning behaviors in language models. They primarily compare four different conditions. Four conditions, huh? That sounds like a lot of detail. What are these conditions? Right. So they tested unguided generation, solution-guided generation, solution-guided without the think contents, and regeneration-conditioned generation. Essentially, these conditions range

12:16from having no guidance to having full solution paths handed to the model. Interesting. And we know that the richness of information provided in these guidance contexts is crucial. The paper mentions that richer conditioning contexts make the model's answers more confident and concise. But how did they measure this? Exactly. To quantify this, they analyzed three key metrics. The average response length, the model score, and the count of epistemic tokens. Epistemic tokens, as you might know, are those markers that indicate uncertainty in the response, like maybe, perhaps, and likely. Ah, got it. So more epistemic tokens generally indicate higher uncertainty in the model's response. And they found that with richer context, both the length and the count of these tokens decrease, right? That's correct. They observed a clear pattern where unguided generation had the longest responses and the highest number

13:18of epistemic tokens. On the other hand, when the full solution was provided, responses became much shorter and more confident. So it seems like the model compresses information more effectively with detailed guidance. But what happens if we strip some of that guidance away like removing the think portions? Interesting observation. When they removed the think tags, the model outputs were less concise than with the full solutions that's still more confident than the unguided responses. This confirms that those think components carry significant information. It makes sense. Those think segments really help the model navigate its reasoning process. What about the regeneration condition generation? How did that play out? Regeneration conditioning, where the model used responses that had previously generated led to intermediate values in both response length and epistemic token count. These outputs were more concise and confident than unguided responses, but not as much so as solution-guided ones.

14:20Sounds like a balancing act. Two little information and the model struggles, too much, and it might be overconfident. What about performance metrics outside of these characteristics? Good point. They also looked at the downstream performance across several benchmarks like AME24 and AMC23. Interestingly, during training, GRPO slightly increased response length with a modest score improvement, while SDPO caused an initial drop in both response length and score with gradual recovery. Ah, so GRPO and SDPO exhibit different dynamics. Could you elaborate a bit more on how these methods compare in terms of performance? Certainly, Echo. So GRPO, which stands for Guided Reinforcement Policy Optimization, yielded modest out-of-distribution OOD gains in both benchmarks. Meanwhile, SDPO or self-destilled policy optimization showed a significant drop, especially when conditioned on full solutions.

15:20Although reducing the conditioning slightly improved the outcome. It's fascinating how these different methods can lead to such varied outcomes. They also mentioned something about reasoning patterns. How did this factor into the results? Indeed, both methods actually reduce epistemic verbalization compared to the base model, but SDPO suppresses it more aggressively. This means that while GRPO encourages some level of uncertainty expression, SDPO aims to minimize it entirely, which could explain the observed drop in performance. That makes sense. Expressing uncertainty could be crucial, especially for out-of-domain tasks that require more flexible reasoning. Did they look into how the models performed on various data sets during these experiments? Exactly. They specifically used data sets like DEPO-Math 17K for benchmarking and found that SDPO's aggressive suppression of uncertainty led to poorer performance, especially on data sets like Aim24,

16:21where the model's ability to navigate uncertainty is crucial. So that's a key takeaway. Overly suppressing uncertainty can actually backfire, making the model less able to handle new or difficult tasks. It's a bit counterintuitive, but crucial for designing training methods. Did they conduct any ablation studies to support these findings? Yes, they did. Ablation studies showed that a fixed target teacher policy performed better than a moving target one. Narrowing the scope of the students learning by focusing only on prefixes, rather than full responses, also helped gauge the effectiveness of each method more precisely. A lot of nuanced dynamics here, Nova. This definitely sheds light on how we can better tailor self-distillation methods for different types of reasoning tasks. And all these empirical insights are really invaluable for advancing the state of the art. Absolutely, Echo. It's clear that achieving the right balance in self-distillation is not just about getting the correct answers,

17:23but also about preserving useful patterns of reasoning, including when to express uncertainty. Well, that wraps up our dive into the experiment and results section. Quite a lot to unpack, but all absolutely fascinating stuff. All right, Nova, let's dive into the related work section. This part really sets the stage by showing how the current study builds on existing research. Any standout points for you in the related work landscape? Definitely, Echo. The authors start by positioning their work within the domain of self-distillation in large language models or LLMs. They highlight several studies that have successfully leveraged self-distillation to improve model performance, such as those combining it with reinforcement learning from verifiable rewards, RLVR. Oh, RLVR. That's quite a mouthful. So this approach, combining self-distillation with RLVR has been shown to yield highly efficient performance gains, right? Exactly.

18:23They reference a variety of works, including recent studies by Zhu et al in 2025 and Ye et al in 2026, which have shown particularly strong improvements in domains like scientific question answering and reasoning tasks by distilling information available during training. Got it. And how do they connect these improvements to the current study on mathematical reasoning? Well, they argue that while self-distillation has improved performance in many areas, its effects on mathematical reasoning are more nuanced. They emphasize that mathematical tasks require not just performance, but also particular care in maintaining the reasoning processes integrity. They suggest that overly aggressive suppression of epistemic verbalization, essentially expressions of uncertainty can degrade performance in mathematical tasks. Interesting. So basically, they're saying that while self-distillation can make models more confident and concise, it can also strip away valuable signals of uncertainty needed for solving complex math problems.

19:25Exactly. Echo, they point to their own findings that suppressing such epistemic verbalization can lead to performance drops of up to 40% in some models. That's a significant hit, especially in areas requiring robust reasoning. 4-1, source. Wow. That's quite a drop. So what other studies do they compare their work to? I'm guessing they looked at different training scenarios and task types. Yes, they did. They compare their results against a range of existing models and data sets, like the live code bench V6 and science Q&A data sets, which cover fields such as chemistry and reinforced learning via text feedback. They found that in these domains, which tend to have lower task variation, self-distillation led to reduced verbosity and improved performance. Science Q&A and live code bench V6, those sound familiar, but it's intriguing that they show such different dynamics in response length and performance. Did the paper mention why these areas see improvements while math tasks suffer?

20:27Great question, Echo. They hypothesize that the differences come down to task coverage. Domains like chemistry and science Q&A often involve more structured problems and repeated exposure to similar types of questions, which can benefit from shorter, more confident responses. Meanwhile, mathematical problems are more diverse and complex, and thus they benefit from maintaining those signals of uncertainty to navigate different reasoning paths effectively. Makes sense. So it boils down to self-distillation's ability to compress reasoning, which is good for straightforward tasks, but potentially harmful for more intricate ones that need those epistemic cues. Exactly. This alignment shows where self-distillation shines and where it needs a more nuanced approach. They emphasize that post-training objectives need to balance correctness with uncertainty to foster robust reasoning behaviors throughout. It's amazing how such a tiny tweak in approach can swing performance so drastically.

21:28Imagine if we could apply this nuanced understanding to other fields. The impact on AI development could be massive. Absolutely echo. By identifying the pros and cons of self-distillation in different domains, they paved the way for more targeted improvements. Ensuring the AI systems are not just accurate, but also resilient and adaptable. All right, folks, that wraps up our deep dive into the related works of this paper. It's clear that understanding the subtle dynamics of self-distillation can make all the difference in optimizing AI for complex reasoning tasks. All right, Nova, we've covered quite a bit about this paper. Let's summarize its key contributions and takeaways for our listeners. Absolutely echo. So the authors aim to provide an information theoretic perspective on self-distillation for large language models, particularly focusing on why it can sometimes degrade reasoning capabilities instead of improving them. You know, it's a pretty comprehensive study.

22:29Yeah, they delve deep into the mechanics, right? They found that self-distillation reshapes a model's reasoning behavior by encouraging it to produce answers with higher confidence. Now, this sounds good at first glance, but there's a catch. Exactly. The catch is that while this method can improve performance on in-domain tasks with limited coverage quickly, it struggles with tasks that have broader coverage. It can even harm the model's performance on out-of-distribution tasks. That's a pretty big deal, right? Oh, for sure. It's like training a student to be really good at a few specific types of problems, but then falling apart when facing new unseen problems. The model essentially gets too confident and doesn't handle uncertainty well, which is crucial for reasoning tasks. Right, and the key takeaway here is how this suppression of epistemic verbalization, basically, the model's ability to express uncertainty plays a role. By imitating a confident teacher

23:30that generates reasoning with little expressed uncertainty, the model loses out on those valuable signals that would otherwise help it navigate new challenges. To wrap it up, their findings recommend that post-training strategies should not only target answer correctness, but also ensure the model retains uncertainty-aware reasoning behaviors. This might mean less aggressive suppression of those, hmm, and maybe tokens that signal epistemic uncertainty. That's an insightful point, Echo. And with that, we hope their analysis contributes to a deeper understanding of self-distillation. And by extension, makes our models better and more robust in the long run. Fingers crossed for future advancements, right? Absolutely, Nova. And with that, let's bring this episode to a close. We hope you all enjoyed our deep dive into self-distillation in large language models. Yes. And if you did, make sure to subscribe, leave us a review, and share this episode with your fellow researchers and AI enthusiasts.

24:33We have a lot more exciting papers to discuss in upcoming episodes. Thanks for tuning into daily papercast. We'll be back again soon to bring you more insights from the latest research. Until next time. Stay curious and keep learning everyone. Bye for now.