Orthogonal Regularization is a regularization technique which is often used in convolutional neural networks. In this tutorial, we will introduce it for deep learning beginners.
What is Orthogonal Regularization
There are two types of Orthogonal Regularization, they are:
L1 Norm Orthogonal Regularization
It is defined as:
L2 Norm Orthogonal Regularization
where \(I\) is an identity matrix, \(W\) should be initialized as an orthogonal matrix.
In tensorflow, in order to create a random orthogonal matrix, you can read:
If you do not use tensorflow, you can read:
In order to compute matrix L1,L2 norm in tensorflow, you can refer: