Understand Advanced LSTM: Compare with Conventional LSTM – LSTM Tutorial

By | March 16, 2021

Advanced LSTM is a variation of LSTM, which is proposed in paper ADVANCED LSTM: A STUDY ABOUT BETTER TIME DEPENDENCY MODELING IN EMOTION RECOGNITION. In this tutorial, we will compare it with Conventional LSTM, which will help us to understand it.

Advanced LSTM

The structure of advanced lstm is:

The structure of advanced lstm

We can find the output \(C(t+1)\) and \(h(t+1)\) of \(t+1\) step are computed based the ouputs of previous 3 steps, which is the main difference between advanced lstm and conventional lstm.

Difference between advanced lstm and conventional lstm

The output of conventional lstm is computed based on previous step. However, the advanced lstm is based on previous T steps. For example, T = 3.

The equation of advanced lstm as follows:

The equations of advanced lstm

Because we will use previous T steps to compute current output in advaced lstm, we should determine each weight of per previous step, which means we will use two attention layers to compute¬† \(C’\) and \(h’\).

Warning

If you plan to use advanced lstm to build your model, you must notice:

  • Weight \(W\) is the same in \(W_{h_T}\) and \(W_{C_T}\)
  • You can not compute \(C’\) and \(h’\) in each step.¬† You should compute them every 3 or 4 steps. You may get a worse result if you compute \(C’\) and \(h’\) in each step. Because advaced lstm will capture more duplicated contex in that situation.

Leave a Reply

Your email address will not be published. Required fields are marked *