Understand Orthogonal Regularization in Deep Learning: A Beginner Introduction – Deep Learning Tutorial

By | October 13, 2020

Orthogonal Regularization is a regularization technique which is often used in convolutional neural networks. In this tutorial, we will introduce it for deep learning beginners.

What is Orthogonal Regularization

There are two types of Orthogonal Regularization, they are:

L1 Norm Orthogonal Regularization

It is defined as:

L1 Norm Orthogonal Regularization

L2 Norm Orthogonal Regularization

L2 Norm Orthogonal Regularization

where \(I\) is an identity matrix, \(W\) should be initialized as an orthogonal matrix.

In tensorflow, in order to create a random orthogonal matrix, you can read:

TensorFlow Create a Random Orthogonal Matrix: A Beginner Guide

If you do not use tensorflow, you can read:

Python Create a Random Orthogonal Matrix: A Beginner Guide

In order to compute matrix L1,L2 norm in tensorflow, you can refer:

TensorFlow Calculate Matrix L1, L2 and L Infinity Norm: A Beginner Guide – TensorFlow Tutorial

Leave a Reply

Your email address will not be published. Required fields are marked *