Implement Orthogonal Regularization in TensorFlow: A Step Guide – TensorFlow Tutorial

By | January 7, 2021

Orthogonal Regularization is a regularization technique used in deep learning model. In this tutorial, we will implement it using tensorflow.

What is Orthogonal Regularization?

Orthogonal Regularization is introduced in this tutorial:

Understand Orthogonal Regularization in Deep Learning: A Beginner Introduction

How to implement Orthogonal Regularization using tensorflow?

In this tutorial, we will implement L2 Norm Orthogonal Regularization. Here is an example code:

m2 = tf.matmul(w, w,  transpose_b = True)
ex = tf.eye(self.class_num)
lreg = tf.multiply(m2, (1-ex))
beta = 0.001
loss = beta * tf.nn.l2_loss(lreg)

Here self.class_num is the number of class. Matrix w is self.class_num * m.

loss is the loss value which can be used as a loss function.

Leave a Reply

Your email address will not be published. Required fields are marked *