Sparse MoE with top-k routing: router gradients + expert selection create dynamic computation graph. Hooks need to handle conditional execution paths. See validate_blackswans.py.
Sparse MoE with top-k routing: router gradients + expert selection create dynamic computation graph. Hooks need to handle conditional execution paths. See validate_blackswans.py.