Understand TensorFlow tf.norm(): Compute the Norm of Vector, not Matrix – TensorFlow Tutorial

By | July 6, 2020

TensorFlow tf.norm() function is often used to calculate the norm of vector and matrix. However, this function is not a good way to compute the norm of matrix. In this tutorial, we will use some examples to show you this truth.

Syntax

tf.norm() is defined as:

tf.norm(
    tensor,
    ord='euclidean',
    axis=None,
    keepdims=None,
    name=None,
    keep_dims=None
)

We can use this function to compute l0, l1, l2 and l infinity norm of a vector and matrix by ord and axis.

Parameters

ord: Order of the norm. It can be 1, 2, np.inf

axis: axis can be None, integer. It determines how to compute vector norm on which axis.

You should notice: if axis = None, tf.norm() will flatten tensor then comput norm.

We will use some examples to introduce how to use tf.norm().

Preliminary

In order to compute the norm of vecters, you should know what is vector norm and how to compute. Here is a tutorial.

Understand Vector Norm: A Beginner Introduction

Compute l1 norm when axis = None

Look at example code below.

import tensorflow as tf
import numpy as np

xs = tf.Variable(np.array([[1, 2],[3, 4]]), dtype = tf.float32)
l1 = tf.norm(xs, axis = None)

xs is a 2*2 tensor, it contains two vectors. We will compute its l1 norm.

init = tf.global_variables_initializer() 
init_local = tf.local_variables_initializer()
with tf.Session() as sess:
    sess.run([init, init_local])
    print(sess.run(l1))

Run this code, you will get the l1 norm of xs is: 10.0

Why the value of l1 norm is 10.0?

Becuase axis = None, tf.norm() will convert 2*2 xs to:[1, 2, 3, 4], then compute the l1 norm of it. The value of l1 norm is 10.0

However, tf.norm() is a good choice to compute the norm of vecters, it is not a good way to compute the norm of matrix. We will use an example to illustrate it.

Compute the l2 norm of vecter xs

We will compute the l2 norm of vecter xs on axis = None, axis = 0 and axis = 1. Here is an example:

l2_none = tf.norm(xs, ord = 2, axis = None)
l2_0 = tf.norm(xs, ord = 2, axis = 0)
l2_1 = tf.norm(xs, ord = 2, axis = 1)

init = tf.global_variables_initializer() 
init_local = tf.local_variables_initializer()
with tf.Session() as sess:
    sess.run([init, init_local])
    print(sess.run(l2_none))
    print(sess.run(l2_0))
    print(sess.run(l2_1))

Run this code, we will get the result.

axis Value
None 5.477226
0 [3.1622777 4.472136 ]
1 [2.236068 5. ]

If tensor xs is a matrix, the value of its l2 norm is:5.4649854. All value above is not 5.4649854. It means tf.norm() can not calculate the l2 norm of matrix correctly.

In order to know how to compute matrix norm in tensorflow, you can read:

TensorFlow Calculate Matrix L1, L2 and L Infinity Norm: A Beginner Guide – TensorFlow Tutorial

As to tf.norm(), it will compute the l1, l2, l infinity norm based on vector norm algorithm, not matrix norm algorithm. If you plan to use it to compute the matrix norm, you must notice this point.

Leave a Reply

Your email address will not be published. Required fields are marked *