NAS-based knowledge distillation uses neural architecture search to design or select a student architecture under guidance from the teacher model. Architecture-aware approaches can coordinate the search for the student's structure with the knowledge-transfer scheme, helping address the teacher–student capacity gap while seeking an efficient student model.

University of California, Berkeley

Google

Directly matching the response-based knowledge, feature-based knowledge, or relation-based knowledge between the teacher and student models is a common way to transfer knowledge in knowledge distillation (KD). Different algorithms specialize in different mechanisms for this transfer, including: - Adversarial Distillation - Multi-Teacher Distillation - Cross-Model Distillation - Graph-Based Distillation - Attention-Based Distillation - NAS-Based Distillation - Quantized Distillation - Lifelong Distillation - Data-Free Distillation

Distillation Algorithms

Multi-Teacher Distillation 



Attention reflects the neuron activations of convolutional neural networks, and different attention transfer mechanisms are defined for distilling knowledge from the teacher network. The main task is defining the attention maps for feature embedding in neural network layers, because embedding knowledge is transferred using attention map functions. 

Attention-Based Distillation


Network quantization reduces a neural network’s computational complexity by converting high-precision networks to low precision. A high-precision teacher network, quantized on feature maps, distills knowledge to a low-precision network. This has been recently used with self-distillation training schemes, where the teacher and student parameters are the same.

Quantized Distillation



Lifelong learning, which includes continual, continuous, and meta learning, accumulates and transfers the previously learned knowledge  into future learning. KD does this effectively without catastrophic forgetting, and meta-learning has been used to determine what and where to make the transfer. Global distillation, knowledge distillation-based lifelong GAN, multi-model distillation, and other KD methods also address the problem of catastrophic forgetting. 

Lifelong Distillation



Adversarial distillation is a knowledge distillation algorithm that incorporates adversarial learning concepts, such as those used in Generative Adversarial Networks (GANs), to enhance knowledge transfer. This approach typically involves using a generator to create synthetic training data, or employing a discriminator to distinguish between the outputs or feature distributions of the teacher and student networks, compelling the student to more closely mimic the teacher.

Adversarial Distillation

Transferring knowledge between modalities is important because data/labels for every modality are not available during training/testing. An example is transferring the knowledge of a teacher model pre-trained on one modality to the student with a new unlabeled input modality, using pair-wise sample registration. Features obtained from the teacher are used for supervised training of the student. This performs well in visual recognition tasks in cross-modal scenarios.

Cross-Model Distillation

Graph-based distillation uses graphs to explore intra-data relationships, aligning with relation-based knowledge transfer. In this approach, graphs act either as the carrier of the teacher's knowledge or as a mechanism to control its transmission to the student model. While it effectively transfers the structural knowledge of data, constructing an accurate graph to model this knowledge remains a significant challenge.

Graph-Based Distillation

To solve the problem of unavailable data, data-free distillation uses no training data. Instead, it relies on synthetically generated data, which can be produced using a Generative Adversarial Network (GAN). For example, data can be reconstructed using the layer activations of the teacher network. However, the task of generating this synthetic data is challenging.

Learn Before

Related