Understand tf.layers.Dense(): How to Use and Regularization – TensorFlow Tutorial

By | March 26, 2021

tf.layers.Dense() is widely used in models built by tensorflow. In this tutorial, we will use some examples to show how to use tf.layers.Dense().

tf.layers.Dense()

tf.layers.Dense() is defined as:

__init__(
    units,
    activation=None,
    use_bias=True,
    kernel_initializer=None,
    bias_initializer=tf.zeros_initializer(),
    kernel_regularizer=None,
    bias_regularizer=None,
    activity_regularizer=None,
    kernel_constraint=None,
    bias_constraint=None,
    trainable=True,
    name=None,
    **kwargs
)

It will implement the operation: outputs = activation(inputs * kernel + bias)

Here is an explain:

understand tf.layers.dense() in tensorflow

To understand the structure of tf.layers.Dense(), you can read this tutorial:

Understand Dense Layer (Fully Connected Layer) in Neural Networks – Deep Learning Tutorial

Important Parameters explained

units: dimensionality of the output, for example: 64

activation: can be relu, sigmoid, tanh etc al

use_bias: use bias in activation(inputs * kernel + bias) or not

kernel_initializer: Initializer function for the weight matrix. If None (default), weights are initialized using the default initializer used by tf.get_variable()

Understand How tf.get_variable() Initialize a Tensor When Initializer is None: A Beginner Guide – TensorFlow Tutorial

bias_initializer: Initializer function for the bias.

trainable: Boolean, if True also add variables to the graph collection GraphKeys

name: String, the name of the layer.

reuse: Boolean, whether to reuse the weights of a previous layer by the same name.

How to use tf.layers.Dense()?

We will use an example to show you how to use it.

import tensorflow as tf

x = tf.random_normal([5,3])
y = tf.layers.dense(inputs=x, units=10, activation=tf.nn.relu)
print(y)
                    
with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    print(sess.run(y))

In this example code, the inputs = x, the shape of x is 5*3

units = 10, which means the dimensionality of the output is 5*10.

tf.layers.Dense() will create two tensorflow variables:

w, weight, the shape of it is 3*10

b, bias, the weight of it is 10

Run this code, you will get this result:

y is:

Tensor("dense/Relu:0", shape=(5, 10), dtype=float32)

The value of y is:

[[0.19549479 0.         0.04906832 0.         0.         0.45005807
  0.         0.         0.10907209 0.        ]
 [0.16909868 0.         0.         0.5597176  0.06139323 0.8804685
  0.63529086 0.         0.12375151 0.        ]
 [0.51807237 0.         0.9474018  0.8525198  0.         0.9306468
  1.0625012  0.         0.49360478 1.0925933 ]
 [0.         0.68350804 0.         0.9666059  0.8174535  0.
  0.77449316 0.4195258  0.         1.035249  ]
 [0.         0.57167256 0.         0.         0.6504928  0.
  0.         0.07965806 0.         0.        ]]

How to regularize weights in tf.layers.Dense()?

The simplest way is to get all trainable weights in tf.layers.Dense(). Here is an example:

for n in tf.trainable_variables():
    
    print(n.name)
    print(n)

Run this code, you may get this result:

dense/kernel:0
<tf.Variable 'dense/kernel:0' shape=(3, 10) dtype=float32_ref>
dense/bias:0
<tf.Variable 'dense/bias:0' shape=(10,) dtype=float32_ref>

From the result, we can find:

The name of weight is dense/kernel:0 in tf.layers.Dense().

In order to regularize weights in tf.layers.Dense(),  we can read this tutorial:

Multi-layer Neural Network Implements L2 Regularization in TensorFlow – TensorFLow Tutorial

Leave a Reply

Your email address will not be published. Required fields are marked *