Understand LSTMP (LSTM with Recurrent Projection Layer): Comparing with LSTM

By | December 14, 2020

LSTMP (LSTM with Recurrent Projection Layer) is an improvement of LSTM with peephole conncections. In this tutorial, we will introduce this model for LSTM Beginners.

Compare LSTMP and LSTM with with peephole conncections

LSTMP is proposed in paper LONG SHORT-TERM MEMORY BASED RECURRENT NEURAL NETWORK ARCHITECTURES FOR LARGE VOCABULARY SPEECH RECOGNITION.

LSTM with peephole conncections LSTMP
Understand LSTM - A Beginner Guide Understand LSTMP (LSTM with Recurrent Projection Layer) - A Beginner Guide

\(m_t\) is the output of the current lstm cell. As to equations (1)-(5) and (7)-(11), they are the same.

We usually set \(y_t = m_t\) in LSTM. However, we add a recurrent projection layer in LSTMP.

This recurrent projection layer is:

\[r_t = W_{rm}m_t\]

\(r_t\) is the output of the current lstm cell. We use a weight \(W_{rm}\) to compress or enlarge \(m_t\).

For example:

If the shape of \(m_t\) is 1*200, \(W_{rm}\) is 200* 100. The shape of \(r_t\) is 1* 100. However, if the shape of \(W_{rm}\) is 200* 300. The output of \(r_t\) will be 1 * 300.

Leave a Reply

Your email address will not be published. Required fields are marked *