hat (hard attention to task)

HAT (Hard Attention to the Task) is a continual learning method that uses learnable binary masks to protect task-specific weights in a neural network, preventing catastrophic forgetting while allowing parameter sharing between tasks when beneficial.

How HAT Works

• Attention Masks: For each task, HAT learns a set of attention embeddings that produce near-binary gate values (0 or 1) for each unit in each layer.
• Mask Training: During forward passes, each unit's output is multiplied by its task-specific gate. Gates near 0 mean the unit is not used for this task; gates near 1 mean it is actively used.
• Gradient Masking: During backpropagation for task t, gradients are blocked for units that have high attention values for any previous task. This prevents updating weights that are important for old tasks.
• Annealing: The attention values are initially soft (sigmoid-like) during training, then progressively sharpened toward binary values through temperature annealing.

Key Properties

• Selective Protection: Only the units that are actually important for a previous task are protected — units unused by old tasks are fully available for new learning.
• Potential Sharing: If a unit is useful for both an old and new task, it can be shared (gated on for both tasks).
• No Buffer Required: HAT doesn't store any examples from previous tasks — protection is entirely through gradient masking.
• Task-Conditioned: At inference time, the model applies the mask for the relevant task, activating the appropriate subnetwork.

Advantages

• Near-Zero Forgetting: Very low forgetting due to hard gradient masking on important units.
• Better Capacity Utilization: More flexible than PackNet — units can be shared between tasks rather than exclusively allocated.
• No Replay: No memory buffer or generative model needed.

Limitations

• Task ID Required: Must know which task is active to select the correct mask.
• Capacity Saturation: Eventually most units are important for some task, limiting room for new learning.
• Optimization Complexity: The attention annealing process adds hyperparameters (temperature schedule) that need tuning.

HAT represents a sophisticated middle ground between rigid weight allocation (PackNet) and soft regularization (EWC) — offering strong forgetting prevention with more efficient parameter sharing.