[1/5] feat: add TokenAligner and cross-tokenizer projection utilities#2347

Closed

avenkateshha wants to merge 2 commits into

NVIDIA-NeMo:mainfrom

avenkateshha:avenkateshha/xtoken-off-policy-distillation/01-tokenaligner

Commits on Apr 27, 2026

feat: add TokenAligner and cross-tokenizer projection utilities

avenkateshha
and
claude
committed

Commits on Apr 28, 2026

refactor(x_token): drop align_fast and numba code paths from TokenAligner

avenkateshha
and
claude
committed