Analysis of LSTM variants

The paper ‘LSTM : A Search Space Odyssey’ presents a comparative analysis of eight LSTM variants.

This large-scale analysis is based their performance on three different tasks, handwriting recognition, speech recognition, and polyphonic music modeling.

The results shows there is no significant improvement provided by the different variants against the standard LSTM architecture.

Two most critical component in the LSTM blocks are forget gate and output activation function.

It was also observed that the hyperparameters studied during analysis were virtually independent.