Fix uniform-initializer docstrings to use the range limit, not sigma#3835
Fix uniform-initializer docstrings to use the range limit, not sigma#3835vineethsaivs wants to merge 1 commit into
Conversation
glorot_uniform and he_uniform sample from a uniform distribution on [-limit, limit], but their docstring math labeled that bound as sigma, the standard-deviation symbol. The standard deviation of U(-limit, limit) is limit / sqrt(3), so the symbol was misleading. The code already names the value 'limit'; use the same name in the math so the docs match the implementation. glorot_normal and he_normal keep sigma, which is correct for a normal distribution.
Pablosinyores
left a comment
There was a problem hiding this comment.
The relabel is correct. glorot_uniform / he_uniform sample mx.random.uniform(-limit, limit, ...) where limit = gain * math.sqrt(6.0 / (fan_in + fan_out)) (resp. sqrt(3.0 / fan)), so that value is the uniform bound, not a standard deviation — \sigma was wrong here (for U[-a, a], σ is a/√3). Using \text{limit} is a nice touch since it matches the code's actual variable name.
This overlaps with your own #3831, which makes the same \sigma-is-wrong fix and additionally rewrites the prose to state the [-a, a] interval explicitly. Since both edit the same glorot/he docstring lines they'll conflict, so it's worth consolidating into one — I'd lean toward #3831 for the extra interval clarity, ideally keeping this PR's \text{limit} symbol. Flagging so one doesn't end up rotting behind the other.
Proposed changes
nn.init.glorot_uniformandnn.init.he_uniformsample from a uniform distribution on[-limit, limit]: the code computeslimit = gain * math.sqrt(...)and callsmx.random.uniform(-limit, limit, ...). Their docstring math, however, labels that bound as\sigma, the standard-deviation symbol:The standard deviation of
U(-limit, limit)islimit / sqrt(3), notlimit, so using\sigmafor the bound is misleading. This renames the symbol to\text{limit}in both docstrings so the documented quantity matches the code variable and its true meaning.glorot_normalandhe_normalare left unchanged: they sample from a normal distribution, where\sigmais the correct symbol for the standard deviation.Documentation-only change; no code paths are affected.
Checklist
.. math::block), soblackformatting is unchanged