squeeze-and-excitation

Squeeze-and-Excitation (SE) is a channel attention mechanism that adaptively recalibrates channel-wise feature responses — by globally summarizing each channel (squeeze) and then learning inter-channel dependencies to produce per-channel importance weights (excitation).

How Does SE Work?

• Squeeze: Global average pooling across spatial dimensions: z_c = (1)/(HW)∑_h,w x_c,h,w.
• Excitation: Two FC layers with ReLU and sigmoid: s = sigma(W_2 · ReLU(W_1 · z)).
• Scale: Multiply each channel by its learned importance: hat(x)_c = s_c · x_c.
• Paper: Hu et al. (2018). Won ILSVRC 2017.

Why It Matters

• Universal: Can be inserted into any CNN (ResNet-SE, MobileNet-SE, EfficientNet uses SE).
• Minimal Cost: Only ~1% more parameters and ~0.5% more FLOPs for 1-2% accuracy improvement.
• Attention Pioneer: One of the first channel attention mechanisms, inspiring CBAM, ECA, and GE-Net.

SE is the channel importance learner — teaching the network to amplify useful channels and suppress uninformative ones.