Relational knowledge distillation transfers relations among data samples from a teacher network to a student network. Feature embeddings are used to model these relations as probability distributions, and knowledge is transferred by matching the student's relation distribution to the teacher's (Peng et al., 2019).

University of California, Berkeley

Google

Relation-based knowledge distillation methods train a student network to preserve relationships represented by a teacher network, such as pairwise similarities, correlations between instances, probabilistic relations among samples, or instance relationship graphs.

Relation-Based Knowledge Distillation Methods

Distilled knowledge contains instance-level information and correlations between instances. Student network learns correlation between instances using corelation congruence for distillation. (Peng et al. (2019a)). 


Correlation Congruence Knowledge Distillation 


Similar activations of input pairs from teacher networks are transferred into the student network. Pairwise similarities are preserved. (Chen et al., 2021) (Passalis and Tefas, 2019; Passalis et al., 2020) (Tung and Mori, 2019)

Similarity-Preserving Knowledge Distillation

In knowledge distillation, an instance relationship graph models transferred knowledge that contains instance features, instance relationships, and feature space transformations across layers.

Learn Before

Related