funnel transformer

Funnel Transformer is an efficient transformer architecture that progressively reduces the sequence length through pooling layers — similar to how CNNs reduce spatial resolution, creating a funnel-shaped computation graph that saves FLOPs on long sequences.

How Does Funnel Transformer Work?

• Encoder: Standard transformer blocks with periodic sequence length reduction (mean pooling every few layers).
• Decoder: Upsamples back to full length for tasks requiring per-token predictions.
• Reduction: Sequence length is halved at each reduction stage (e.g., 512 → 256 → 128).
• Paper: Dai et al. (2020).

Why It Matters

• Efficiency: Processes long sequences with progressively fewer tokens -> significant FLOPs reduction.
• Classification: For classification tasks, only the final (shortest) representation is needed -> no upsampling needed.
• Pre-Training: Can be pre-trained like BERT but with lower compute cost for the same model quality.

Funnel Transformer is the CNN pyramid for transformers — progressively compressing sequence length to focus computation on the most important information.