Several techniques can mitigate the teacher-student training gap in knowledge distillation: introducing a teacher assistant and using residual learning so the assistant learns the residual error; reducing structural differences between the teacher and student by combining network quantization with knowledge distillation; and applying structure compression to transfer knowledge from multiple layers to a single layer. In online settings, an ensemble of similarly structured student networks can serve as the teacher.