Found while implementing a SPAN-family SR forward on magetypes 0.9.28 (zensr).
f32xN::recip() is hardware-rcp + Newton-Raphson refinement, so recip(inf) = NaN (the refinement computes inf * 0). exp_midp/exp2_midp correctly saturate to inf for large inputs, which makes the natural sigmoid formulation
let s = ((-v).exp_midp() + one).recip(); // NaN wherever v <= ~-88
produce NaN lanes on real data (conv pre-activations reach ±100 in practice; 97% of the output tensor went NaN before root-causing). IEEE division is exact at the rails (1/inf = 0), so one / (exp + one) is the correct spelling.
Suggestions (any subset):
- Document the inf/NaN domain of
recip() (and rsqrt if refined the same way) loudly in the doc comment.
- Consider a
recip_precise() that blends recip(±inf) -> ±0 and recip(±0) -> ±inf post-refinement.
- A magetypes-level
sigmoid_midp()/silu_midp() helper would remove the footgun entirely for NN work.
Bonus observation from the same session: f32::max-based diff checks mask NaN (max ignores NaN), which let this slip past a golden comparison until RMSE went NaN — maybe worth a note in the testing docs.
Repro + fix in imazen zensr repo (crates/zensr-micro/src/simd.rs, silu/gate kernels), 2026-07-22 session.
Found while implementing a SPAN-family SR forward on magetypes 0.9.28 (zensr).
f32xN::recip()is hardware-rcp + Newton-Raphson refinement, sorecip(inf) = NaN(the refinement computesinf * 0).exp_midp/exp2_midpcorrectly saturate toinffor large inputs, which makes the natural sigmoid formulationproduce NaN lanes on real data (conv pre-activations reach ±100 in practice; 97% of the output tensor went NaN before root-causing). IEEE division is exact at the rails (
1/inf = 0), soone / (exp + one)is the correct spelling.Suggestions (any subset):
recip()(andrsqrtif refined the same way) loudly in the doc comment.recip_precise()that blendsrecip(±inf) -> ±0andrecip(±0) -> ±infpost-refinement.sigmoid_midp()/silu_midp()helper would remove the footgun entirely for NN work.Bonus observation from the same session:
f32::max-based diff checks mask NaN (max ignores NaN), which let this slip past a golden comparison until RMSE went NaN — maybe worth a note in the testing docs.Repro + fix in imazen zensr repo (crates/zensr-micro/src/simd.rs, silu/gate kernels), 2026-07-22 session.