Focal loss is a good method to improve the model performance for imbalance multi label classification. In this tutorial, we will implement it using tensorflow.

## Focal Loss

Focal loss was introduced by Tsung-Yi Lin et al in paper << Focal Loss for Dense Object Detection>>, it is used to handle imbalance prediction in object detection.

Focal loss can be defined as:

\(FL(p_t)=-a_t(1-p_t)^rlog(p_t)\)

Here \(p_t\) is the predicted score, \(r\) is a hyper parameter.

There is a comparison between focal loss and cross entropy loss.

Understand the Gradient of Cross Entropy Loss Function

## How to implement focal loss in tensorflow?

Focal loss can be used in multi label classification, we can use tensorflow to create it. Here is an example code:

def focal_loss(logits, labels, gamma=2.0, alpha=4.0): epsilon = 1.e-9 probs = tf.nn.softmax(logits) y_pred = tf.clip_by_value(probs, epsilon, 1. - epsilon) ce = tf.multiply(labels, -tf.log(y_pred)) weight = tf.multiply(labels, tf.pow(tf.subtract(1., y_pred), gamma)) fl = tf.multiply(alpha, tf.multiply(weight, ce)) reduced_fl = tf.reduce_max(fl, axis=1) # reduced_fl = tf.reduce_sum(fl, axis=1) # same as reduce_max return tf.reduce_mean(reduced_fl)

As to this code, you should notice:

labels is the gold truth, which is batch_size * class_num.

You can use this loss as follows:

self.scores_intent = tf.matmul(self.doc_output, self.weights['softmax_intent']) + self.biases['softmax_intent'] loss_mean = self.focal_loss(logits=self.scores_intent, labels=self.input_y) # self.input_y is 64*10, batch_size = 64