Journal Article

Seformer: a long sequence time-series forecasting model based on binary position encoding and information transfer regularization

Liu, Pengjie, Zhou, Xiaofeng, Li, Shuai, Zeng, Pengyu, Hu, Guoliang

2023

Overview

Long sequence time-series forecasting (LSTF) problems, such as weather forecasting, stock market forecasting, and power resource management, are widespread in the real world. The LSTF problem requires a model with high prediction accuracy. Recent studies have shown that the transformer model architecture is the most promising model structure for LSTF problems compared with other model architectures. The transformer model has the property of permutation equivalence, which leads to the importance of sequence position encoding, an essential process in model training. Currently, the continuous dynamics models constructed for position encoding using the neural differential equations (neural ODEs) method can model sequence position information well. However, we have found that there are some limitations when neural ODEs are applied to the LSTF problem, including the time cost problem, the baseline drift problem, and the information loss problem; thus, neural ODEs cannot be directly applied to the LSTF problem. To address this problem, we design a binary position encoding-based regularization model for long sequence time-series prediction, named Seformer, which has the following structure: 1) The binary position encoding mechanism, including intrablock and interblock position encoding. For intrablock position encoding, we design a simple ODE method by discretizing the continuum dynamics model, which reduces the time cost required to compute neural ODEs while maintaining their dynamics properties to the maximum extent. In interblock position encoding, a chunked recursive form is adopted to alleviate the baseline drift problem caused by eigenvalue explosion. 2) Information transfer regularization mechanism: By regularizing the model intermediate hidden variables as well as the encoder-decoder connection variables, we can reduce information loss during the model training process while ensuring the smoothness of the position information. Extensive experimental results obtained on six large-scale datasets show a consistent improvement in our approach over the baselines.

• Publisher: Springer US,Springer Nature B.V
• Subject: Architecture / Artificial Intelligence / Coders / Coding / Computer Science / Differential equations / Drift / Efficiency / Eigenvalues / Encoders-Decoders / Explosions / Forecasting / Information transfer / Machine learning / Machines / Manufacturing / Mathematical models / Mechanical Engineering / Natural language processing / Neural networks / Ordinary differential equations / Permutations / Power / Processes / Regularization / Regularization methods / Resource management / Securities markets / Smoothness / Time series / Training / Transformers / Visualization / Weather forecasting