Understand Adaptive Gating Mechanism in Deep Learning – Deep Learning Tutorial

By | June 18, 2019

Adaptive Gating Mechanism can dynamically control the information flow based on the current input, which often be a sigmoid function.

In LSTM

lstm three gates

In gated end-to-end memory networks

Gated End-to-End Memory Networks Equation

There are some important questions you should know to understand Adaptive Gating Mechanism.

Q l: What is gate?

Gate is often a sigmoid function, value of it in (0,1)

Q 2: How to understand adaptive?

Adaptive means the value of gate is trained by model, the value of each gate (sigmoid) is determined by inputs and trainable variables. So different inputs will generate different value, which is adaptive.

Q 3: How to control information flow by gate?

The value of each gate is in (0,1), if the value is equal to 0, the information will be quite.

Q4: How to use gate mechanism?

Gate mechanism often be used in two inputs, such as xt and ht-1 or ok and uk, we often use gate to evaluate the different weight of each input.

But if the number of input is better than two, gate mechanism will not wok, we will need use attention mechanim to evaluate the different weight of each input.

Leave a Reply

Your email address will not be published. Required fields are marked *