On the Memory Mechanism of Tensor-Power Recurrent Model
Published in International Conference on Artificial Intelligence and Statistics, 2021
Recommended citation: Qiu, H., Li, C., Weng, Y., Sun, Z., He, X. and Zhao, Q., 2021, March. On the Memory Mechanism of Tensor-Power Recurrent Models. In International Conference on Artificial Intelligence and Statistics (pp. 3682-3690). PMLR. http://proceedings.mlr.press/v130/qiu21a.html
Tensor-power (TP) recurrent model is a family of non-linear dynamical systems, of which the recurrence relation consists of a p-fold (a.k.a., degree-p) tensor product. Despite such the model frequently appears in the advanced recurrent neural networks (RNNs), to this date there is limited study on its memory property, a critical characteristic in sequence tasks. In this work, we conduct a thorough investigation of the memory mechanism of TP recurrent models. Theoretically, we prove that a large degree p is an essential condition to achieve the long memory effect, yet it would lead to unstable dynamical behaviors. Empirically, we tackle this issue by extending the degree p from discrete to a differentiable domain, such that it is efficiently learnable from a variety of datasets. Taken together, the new model is expected to benefit from the long memory effect in a stable manner. We experimentally show that the proposed model achieves competitive performance compared to various advanced RNNs in both the single-cell and seq2seq architectures.