relational knowledge distillation

Relational Knowledge Distillation (RKD) is a distillation method that transfers the geometric relationships between samples rather than individual sample representations — teaching the student to preserve the distance and angle structure of the teacher's feature space.

How Does RKD Work?

• Distance-Wise: Minimize $\sum_{(i,j)} l(\psi_D^T(x_i, x_j), \psi_D^S(x_i, x_j))$ where $\psi_D$ is the pairwise distance function.
• Angle-Wise: Preserve the angle formed by triplets of points in the embedding space.
• Representation: Instead of matching individual features, match the relational structure (distances, angles) between sample pairs/triplets.
• Paper: Park et al. (2019).

Why It Matters

• Structural Knowledge: Captures the manifold structure of the feature space, not just point-wise values.
• Robustness: Less sensitive to absolute scale differences between teacher and student representations.
• Metric Learning: Particularly effective for tasks where relative distances matter (face recognition, retrieval).

RKD is transferring the geometry of knowledge — teaching the student to arrange its representations in the same relative structure as the teacher, regardless of absolute coordinates.