Segformer: Segment-Based Transformer with Decomposition for Long-Term Series Forecasting

Long sequence time-series forecasting has important applications in long-term planning management scenarios. Researches based on Transformer effectively improve the capability for long sequence forecasting, but the quadratic computing complexity causes high resource consumption, limiting its application in long sequence scenarios. Meanwhile, many studies use the dot-product attention mechanism to model time dependency, but don't model the sequential information of time-series. It prevents the capture of more efficient long-term time dependencies. In addition, in the latest researches combining decomposition methods, there are the problem of trend information loss and the limitation of not being able to peel off more subdivided time patterns, which restrict the improvement of prediction ability. Therefore, we propose a Transformer-based model, Segformer. Firstly, Segformer extracts multiple components with obvious dependencies and coordinates the modeling process with the help of multi-component decomposition blocks and collaboration blocks. Secondly, SegAttention, a new variant of the attention mechanism with
$O((\frac{L}{l})^{2})$
computation complexity (whole sequence length L, segment length
$l$
), is proposed to model the dependencies between segments according to the values and orders in the segment sequences, and aggregate segment-level information. Experiments on five real datasets show that Segformer respectively reduces the forecasting error by about 13% and 22% compared with the two advanced benchmarks, and Segformer offers an efficient solution for long-term dependency modeling problem of time-series.