parseval networks

Parseval Networks are neural networks whose weight matrices are constrained to have spectral norm ≤ 1 using Parseval tight frame constraints — ensuring each layer is a contraction, resulting in a globally Lipschitz-constrained network with improved robustness.

How Parseval Networks Work

• Parseval Tight Frame: Weight matrices satisfy $WW^T = I$ (when the matrix is wide) or $W^TW = I$ (when tall).
• Regularization: Add a regularization term $\beta |WW^T - I|^2$ to the training loss.
• Projection: Periodically project weights onto the set of tight frames during training.
• Convex Combination: Blend the projected weights with current weights: $W \leftarrow (1+\beta)W - \beta WW^TW$.

Why It Matters

• Lipschitz-1: Each layer is a contraction — the full network has Lipschitz constant ≤ 1.
• Adversarial Robustness: Parseval networks show improved robustness to adversarial perturbations.
• Theoretical Foundation: Grounded in frame theory from signal processing.

Parseval Networks are contraction-constrained architectures — using tight frame theory to ensure each layer contracts rather than amplifies perturbations.