CVPR 2021, Tutorial on

Normalization Techniques in Deep Learning: Methods, Analyses, and Applications

Saturday morning (10:00 AM - 13:30 PM EDT), June 19, 2021
Slides and videos are available on this website.

Organizer and Lecturer


Overview

Normalization methods can improve the training stability, optimization efficiency and generalization ability of deep neural networks (DNNs), and have become basic components in most state-of-the-art DNN architectures. They also have successfully proliferated throughout various areas of deep learning, including but not limited to computer vision, natural language processing, and speech recognition. However, despite the abundance and ever more important roles of normalization techniques, we note that there is an absence of a unifying lens with which to describe, compare and analyze them. In addition, our understanding of theoretical foundations of these methods for their success remain elusive.

This tutorial cover normalization methods, analyses and applications, and will address the following questions:
(1) What are the main motivations behind different normalization methods in DNNs, and how can we present a taxonomy for understanding the similarities and differences between a wide variety of approaches?
(2) How can we reduce the gap between the empirical success of normalization techniques and our theoretical understanding of them?
(3) What recent advances have been made in designing/tailoring normalization techniques for different tasks, and what are the main insights behind them?

This tutorial is based on our survey paper: Normalization Techniques in Training DNNs: Methodology, Analysis and Application with recent update. I also maintain a set of cited papers/codes on this Github


Schedule

10:00 - 10:30 AM . Motivations of Normalization Techniques. [Slides] [Video]      Lecturer: Lei Huang.

10:30 - 11:30 AM . Introduction of Normalization Methods. [Slides] [Video]      Lecturer: Lei Huang.

11:30 - 11:45 AM . Break.

11:45 - 12:15 PM . Analyses of Normalization. [Slides] [Video]      Lecturer: Lei Huang.

12:15 - 13:15 PM . Applications of Normalization. [Slides] [Video]      Lecturer: Lei Huang.

13:15 - 13:30 PM . Q&A .



Previous tutorial on normalization technqiues in deep learning: ECCV 2018 .

Please contact Lei Huang if you have question. The webpage template is by the courtesy of awesome Georgia.