lsuv

LSUV (Layer-Sequential Unit-Variance) is a data-driven initialization method that iteratively adjusts each layer's weights to produce unit-variance activations — using a mini-batch of real data to empirically calibrate the initialization, accounting for non-linearities and architectural specifics.

How Does LSUV Work?

1. Initialize: Start with orthogonal initialization.
2. Forward Pass: Pass a mini-batch through the network.
3. Per Layer: Measure the variance of each layer's activations.
4. Rescale: Multiply weights by $1/\sqrt{\text{Var}(output)}$ to achieve unit variance.
5. Iterate: Repeat until all layers have unit-variance activations.

Why It Matters

• Data-Driven: Accounts for the actual data distribution, not just theoretical assumptions.
• Architecture-Agnostic: Works for any architecture (CNNs, RNNs, exotic activations).
• Post-Init Calibration: Can be applied after any initialization to fix variance issues.

LSUV is empirical initialization calibration — using real data to tune each layer's scale for perfect signal propagation, regardless of the theoretical assumptions.