Zoneout is proposed in paper: Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations. It is also used in Tacotron 2. In this tutorial, we will introduce what it is and how to implement it using tensorflow.
Smoothing normalization is proposed in paper: Attention-Based Models for Speech Recognition. In this tutorial, we will introduce how to implement it in tensorflow.
In deep learning, we usually place a dropout layer after a dense layer. However, here is a problem? Dropout layer is placed before or after activation function.
Convolution subsampling module is an important part for processing inputs in conformer. In this tutorial, we will introduce how to implement it using tensorflow.
Sinusoidal position encoding or postion embedding is used in transformer. In this tutorial, we will introduce you how to implement it using tensorflow.