Normalization methods can improve the training stability, optimization efficiency and generalization ability of deep neural networks (DNNs), and have become basic components in most state-of-the-art DNN architectures. They also have successfully proliferated throughout various areas of deep learning, including but not limited to computer vision, natural language processing, and speech recognition. However, despite the abundance and ever more important roles of normalization techniques, we note that there is an absence of a unifying lens with which to describe, compare and analyze them. In addition, our understanding of theoretical foundations of these methods for their success remain elusive.

This tutorial cover normalization methods, analyses and applications, and will address the following questions:
(1) What are the main motivations behind different normalization methods in DNNs, and how can we present a taxonomy for understanding the similarities and differences between a wide variety of approaches?
(2) How can we reduce the gap between the empirical success of normalization techniques and our theoretical understanding of them?
(3) What recent advances have been made in designing/tailoring normalization techniques for different tasks, and what are the main insights behind them?

This tutorial is based on our survey paper: Normalization Techniques in Training DNNs: Methodology, Analysis and Application with recent update. I also maintain a set of cited papers/codes on this Github