Audit layers.py and cnn.py: fix BatchNorm eps and inference leak, add input validation and gradient guards

[vmspereira]

An audit of layers.py and cnn.py, the last unaudited code in the library.

PYTHONPATH=src python -m pytest tests/ -q
343 passed        (with cvxopt; 340 passed, 3 skipped without)
python -m flake8 src tests setup.py
0 violations
coverage: 92%     (was 91%)

Every commit is independently green and lint-clean.

First, what is not broken

The gradients are correct, and that is worth stating because it was the main thing the audit set out to check:

• BatchNorm dE/dX and dE/dgamma match finite differences to ~1e-9.
• Conv2D dE/dX holds for padding 0/1/2 and stride 1/2 — combinations the existing check never covered, and both change the im2col index arithmetic.
• All ten activation prime2 second derivatives match a finite difference of prime.

No fixes were needed in any of that. What follows is the six defects surrounding it.

BatchNormalization.eps was 1000x too large

It was hardcoded at 0.01 against a conventional 1e-5, and it sits inside the square root — the applied scale is 1/sqrt(var + eps). Once eps is comparable to a feature's variance it dominates, and the layer stops normalising the features that most need it:

Feature variance	Under-scaled by
1.0	0.5%
0.01	29.3%
0.0001	90.0%

Measured end to end, a feature with standard deviation 0.01 left the layer at 0.095 instead of ~1 — a normalisation layer not normalising. Now 1e-5 and configurable, so it can still be raised for very small batches where the variance estimate is itself noisy.

I checked the obvious risk of lowering it: at batch size 1, and for a constant feature, X_centered is exactly zero, so the output is exactly beta regardless of eps. No amplification in practice.

This is a numerical behaviour change — networks using BatchNorm will produce slightly different values than before.

BatchNorm leaked test statistics at inference

Running statistics were initialised on any first forward pass, so forward(training=False) on an untrained layer seeded them from the inference batch and then normalised that batch by its own mean and variance — precisely the leakage running estimates exist to prevent, and silently: two test samples came back as -0.995 and +0.995, as though centred on themselves. That path now raises with an explanation. Train-then-infer is unchanged.

Input validation that was missing entirely

Call	Before	After
Reshape.backward before forward	returned a wrongly-shaped gradient in silence (arr.reshape(None) does not raise)	AssertionError
Conv2D(stride=0)	ZeroDivisionError from the output-size formula	ValueError
Conv2D channel mismatch	matmul: Input operand 1 has a mismatch in its core dimension	names both channel counts
Conv2D non-batched input	opaque downstream failure	ValueError
Dense.set_weights wrong shapes	accepted; failed later inside forward	ValueError naming the expected shapes
Pooling2D window that doesn't tile	bare Exception("Invalid output dimension!")	reports window, stride, input size and the fractional result

Conv2D's declared spatial input_shape turns out to be unused — only input_shape[2] is read, and the convolution derives its output size from the input it is actually given, so a layer declared (5, 5, 2) processes a 7x7x2 input correctly. That is now documented as advisory rather than enforced, since the behaviour is right and enforcing it would break working code. A test pins it so the decision is explicit rather than accidental.

A flaw in my own test, caught by mutation

Worth flagging plainly. My first BatchNorm gradient check used an all-ones incoming error, and it passed with the variance-path term deleted from the implementation. That term is

X_centered * stddev_inv**2 * sum(output_error * X_centered)

and X_centered sums to zero by construction, so under a constant error the whole sum is zero and the term vanishes — the test looked rigorous while exercising nothing. Only the mutation run revealed it.

Fixed by using a random error, plus a companion test asserting the constant-error and random-error gradients genuinely differ, so the check cannot silently degenerate back to the weaker form. Both BatchNorm gradient terms now fail when removed.

Verification

Ten mutations, each reintroducing one defect; all are caught:

Reintroduced bug	Tests failing
eps back to 0.01	2
BatchNorm inference-before-training guard removed	1
BatchNorm variance-path gradient term dropped	1
BatchNorm mean-path gradient term dropped	1
Dense.set_weights validation removed	1
Pooling2D error back to bare Exception	1
Reshape.backward guard removed	1
Conv2D stride validation removed	1
Conv2D channel check removed	1
Conv2D ndim check removed	1
SoftPlus.prime2 corrupted	2
ELU.prime2 corrupted	1

Coverage: activation.py 91% → 98%, cnn.py 92% → 94%, layers.py 90% → 92%.

Still unaudited after this

Two areas remain, deliberately left for a separate pass rather than growing this branch: si/unsupervised/ (kmeans.py, pca.py — KMeans uses an unseeded RNG) and si/supervised/knn.py, linreg.py, logreg.py (high coverage, never read for bugs; linreg.py is the lowest at 84%).