attention rollout in vit
Attention rollout in ViT is the layer-wise aggregation method that composes attention matrices across depth to estimate end-to-end token influence on final predictions - instead of viewing one layer in isolation, rollout traces how information propagates from input patches to output tokens.
What Is Attention Rollout?
- Definition: Recursive multiplication of attention matrices with identity residual terms across transformer layers.
- Core Idea: Influence accumulates through many blocks, so global attribution must include the full chain.
- Output: A single influence map showing patch contribution to CLS or target token.
- Scope: Works for classification and can be adapted to dense token outputs.
Why Attention Rollout Matters
- Deeper Explainability: Captures cross-layer pathways missed by single-layer heatmaps.
- Consistency Checks: Detects if influence remains stable across augmentations and seeds.
- Bias Detection: Highlights unintended dependencies on background regions.
- Model Comparison: Enables fair explainability comparison across ViT variants.
- Debugging Efficiency: Reduces manual review time by summarizing layer dynamics.
How Rollout Is Computed
Step 1:
- Collect attention matrices A_l from each layer and average or select heads.
- Add identity matrix to model residual mixing, then normalize rows.
Step 2:
- Multiply adjusted matrices from shallow to deep layers to obtain cumulative influence matrix.
- Extract influence from output token to input patch tokens.
Step 3:
- Reshape influence vector to patch grid and overlay as saliency map.
- Validate map behavior against counterfactual image edits.
Implementation Notes
- Head Aggregation: Mean aggregation is stable baseline, max can overemphasize outliers.
- Numerical Stability: Use float32 for matrix products in long depth models.
- Residual Handling: Identity blending choice strongly affects attribution sharpness.
Attention rollout in ViT is a robust way to summarize multi-layer information flow and patch influence in one interpretable map - it turns raw attention tensors into actionable explainability signals for model governance.