Advancements in Knowledge Distillation: Towards New Horizons of Intelligent Systems
Editors: Witold Pedrycz, Shyi-Ming Chen
published by Springer Cham
[Homepage]
Invited by IEEE Fellow Shyi-Ming Chen, we write the first chapter (pages 1-32), titled as "Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillation". In this chapter, we first survey the conventional offline KD works according to their extracted knowledge types, i.e. response, feature and relation. Benefiting the comprehensive supervision from teacher, the student could generalize better over the target task. The teacher-student-based KD has some limitations, for example, high costs for pre-training a large teacher network. Therefore, two Online KD and Self-KD schemes are proposed to improve the student without a pre-trained teacher. In practice, KD applications often face various scenarios, for example, cross-modal KD and data-free KD. Moreover, we also show some popular mechanisms to help distillation perform better, such as multi-teacher KD, attention-based KD and adversarial KD. We survey the representative works for each KD setup and summarize their main ideas and contributions. Finally, we prospect the future challenges of existing KD applications. We hope our survey can inspire future research to develop more advanced KD algorithms for improving the student performance.

《学有所承：研究生毕业传承会撷英》
本书作者来自中国科学院计算所、软件所、数学院、中国科学院大学以及北京大学、清华大学、中国人民大学、大湾区大学等院校。
出版社： 上海科学技术出版社
[Homepage] [JD]
本书是中国科学院计算技术研究所“学有所承”毕业生传承会的文萃。全书共分四篇

​​第一篇“早已智珠在握”​​：辑录毕业生在科研选题、提出思路、读文献、建系统、做实验、组会交流、写论文等方面的具体建议，迹近于“术”；

​​第二篇“更悟学术道通”​​：辑录毕业生们更深层次的思考，包括方法论、心理调适、如何激发“心流”，以及“什么是格局”等思索，迹近于“道”；

​​第三篇“验谈说与后学听”​​：辑录圆桌对话环节上毕业生与在读同学们的对话，是答疑，是解惑；

​​第四篇“雏凤清于老凤声”​​：辑录研究生的导师们的所思所想，是欣慰，是期盼。

中国工程院院士李国杰研究员、计算所徐志伟研究员和大湾区大学副校长李晓明教授倾情作序，其意拳拳，其盼殷殷。