In Figure 19, is the maximum adv computed within the student's Top-K vocabulary scope or across the full vocabulary?

[vxfla]

Thank you for your excellent and insightful work. I have a small clarification question regarding Figure 19: is the maximum adv computed within the student's Top-K vocabulary scope, or is it evaluated across the full vocabulary? Your clarification would be greatly appreciated.

[Turingzero0]

Thank you for your question. In our code implementation, the maximum adv is computed within the student model’s Top-K vocabulary scope.

In our paper, adv is defined as:
$$A_t(v) = \bar{p}_t(v) \left( \log \bar{q}_t(v) - \log \bar{p}_t(v) \right)$$
However, we observed that both the student and teacher Top-K probability masses are greater than 0.97. Therefore, according to the definition above, I believe this maximum adv should effectively correspond to the maximum adv over the full vocabulary.