seq2seq model
Sequence-to-Sequence (Seq2Seq) Models are the neural network architecture pattern where an encoder processes a variable-length input sequence into a fixed or variable-length representation, and a decoder generates a variable-length output sequence from that representation — the foundational architecture for machine translation, summarization, speech recognition, and any task that maps one sequence to another of potentially different length.
Seq2Seq Evolution
| Era | Architecture | Key Innovation | Example |
|---|---|---|---|
| 2014 | RNN Encoder-Decoder | Compress input to fixed vector | Sutskever et al. |
| 2015 | RNN + Attention | Attend to any input position | Bahdanau Attention |
| 2017 | Transformer Enc-Dec | Self-attention, parallelizable | "Attention Is All You Need" |
| 2019+ | Pre-trained Enc-Dec | Transfer learning + fine-tuning | T5, BART, mBART |
| 2020+ | Decoder-Only | Prompting, no explicit encoder | GPT-3, LLaMA |
Original RNN Seq2Seq
1. Encoder RNN: Processes input tokens x₁...xₙ → produces final hidden state hₙ (context vector).
2. Context vector: Fixed-size summary of entire input → bottleneck!
3. Decoder RNN: Initialized with context vector → generates output tokens y₁...yₘ autoregressively.
- Problem: Fixed-size context vector cannot capture all information from long sequences.
Attention Mechanism (Bahdanau, 2015)
- Instead of single context vector: Decoder attends to ALL encoder hidden states.
- At each decoder step t: Compute attention weights over encoder states → weighted sum = context.
- $c_t = \sum_i \alpha_{t,i} h_i$ where $\alpha_{t,i} = \text{softmax}(\text{score}(s_t, h_i))$
- Result: Decoder can focus on relevant parts of input → dramatically improved translation quality.
Transformer Encoder-Decoder
- Encoder: Stack of self-attention + FFN layers → contextual representations.
- Decoder: Masked self-attention + cross-attention to encoder + FFN.
- Cross-attention: Decoder queries attend to encoder outputs (keys and values from encoder).
- Fully parallelizable (no recurrence) → much faster training.
Pre-trained Seq2Seq Models
| Model | Pre-training Objective | Best For |
|---|---|---|
| T5 | Text-to-text (span corruption) | General NLP tasks |
| BART | Denoising autoencoder | Summarization, generation |
| mBART | Multilingual denoising | Multilingual translation |
| NLLB | Translation-specific pre-training | 200+ language translation |
| Flan-T5 | Instruction-tuned T5 | Following instructions |
Seq2Seq vs. Decoder-Only
| Aspect | Encoder-Decoder | Decoder-Only |
|---|---|---|
| Input processing | Bidirectional (encoder) | Causal (left-to-right) |
| Cross-attention | Yes (decoder→encoder) | No |
| Best for | Translation, summarization | Open-ended generation, chat |
| Efficiency | More parameters for same quality | Simpler, scales better |
Sequence-to-sequence models are the architectural foundation that enabled neural approaches to surpass traditional methods in machine translation and structured generation — while decoder-only models now dominate general-purpose language modeling, the encoder-decoder pattern remains the superior choice for tasks with distinct input and output sequences.