An Enhanced Transformer Encoder Using Inverted Stationarization Decomposition for Long-Term Multivariate Time Series Forecasting
Received: 27 November 2025 | Revised: 23 December 2025, 12 January 2026, 21 January 2026, and 26 January 2026 | Accepted: 28 January 2026 | Online: 4 April 2026
Corresponding author: Suprapto
Abstract
Autoformer is the first Transformer-based model employing auto-correlation and moving average decomposition to extract trend and seasonal patterns from time series data. Although effective in modeling complex temporal dependencies, these mechanisms show limited accuracy when applied to real-world multivariate time series that are highly non-stationary and lack clear periodicity. This limitation arises from the sensitivity of moving average-based decomposition to fluctuations and distribution shifts, as well as the inability of auto-correlation to capture inter-variable dependencies. To overcome these challenges, this study proposes an Inverted Stationarization Decomposition approach integrated with a Transformer encoder for long-term multivariate time series forecasting. The stationarization process reduces non-stationarity in the input data and restores appropriate statistical properties at the output stage, whereas the inverted embedding mechanism enables effective modeling of correlations among variables. The proposed approach was evaluated using the ETT dataset (four subsets), and the Exchange, Weather, ECL, and Traffic datasets. Experimental results demonstrate that the proposed model consistently outperforms Autoformer, achieving an average Mean Squared Error (MSE) reduction of 29.15% and an average Mean Absolute Error (MAE) reduction of 23.82% across all evaluated datasets.
Keywords:
Autoformer, moving average, non-stationary, embedding, forecastingDownloads
References
S. Martelo, D. León, and G. Hernandez, "Multivariate Financial Time Series Forecasting with Deep Learning," in 9th Workshop on Engineering Applications on Applied Computer Sciences in Engineering, Bogotá, Colombia, 2022, pp. 160–169. DOI: https://doi.org/10.1007/978-3-031-20611-5_14
R. Pérez-Chacón, G. Asencio-Cortés, A. Troncoso, and F. Martínez-Álvarez, "Pattern sequence-based algorithm for multivariate big data time series forecasting: Application to electricity consumption," Future Generation Computer Systems, vol. 154, pp. 397–412, May 2024. DOI: https://doi.org/10.1016/j.future.2023.12.021
H. Zhang, J. Wang, Y. Qian, and Q. Li, "Point and interval wind speed forecasting of multivariate time series based on dual-layer LSTM," Energy, vol. 294, May 2024, Art. no. 130875. DOI: https://doi.org/10.1016/j.energy.2024.130875
D. B. N. Assad, J. Cara, and M. Ortega-Mier, "Comparing Short-Term Univariate and Multivariate Time-Series Forecasting Models in Infectious Disease Outbreak," Bulletin of Mathematical Biology, vol. 85, no. 1, Dec. 2022, Art. no. 9. DOI: https://doi.org/10.1007/s11538-022-01112-5
Z. Chen, M. Ma, T. Li, H. Wang, and C. Li, "Long sequence time-series forecasting with deep learning: A survey," Information Fusion, vol. 97, Sept. 2023, Art. no. 101819. DOI: https://doi.org/10.1016/j.inffus.2023.101819
E. A. Nketiah, L. Chenlong, J. Yingchuan, and S. A. Aram, "Recurrent neural network modeling of multivariate time series and its application in temperature forecasting," Plos One, vol. 18, no. 5, May 2023, Art. no. e0285713. DOI: https://doi.org/10.1371/journal.pone.0285713
J. Ju and F.-A. Liu, "Multivariate Time Series Data Prediction Based on ATT-LSTM Network," Applied Sciences, vol. 11, no. 20, Oct. 2021, Art. no. 9373. DOI: https://doi.org/10.3390/app11209373
A. Vaswani et al., "Attention is All you Need," in 31st Conference on Neural Information Processing Systems, Long Beach, CA, USA, 2017, pp. 5998–6008.
Q. Wen et al., "Transformers in Time Series: A Survey," in Thirty-Second International Joint Conference on Artificial Intelligence, Macao, China, 2023, pp. 6778–6786. DOI: https://doi.org/10.24963/ijcai.2023/759
S. Li et al., "Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting," in Proceedings of the 33rd International Conference on Neural Information Processing Systems, Vancouver, Canada, 2019, pp. 5243–5253.
N. Kitaev, Ł. Kaiser, and A. Levskaya, "Reformer: The Efficient Transformer." arXiv, Feb. 18, 2020.
H. Zhou et al., "Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting," Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, no. 12, pp. 11106–11115, May 2021. DOI: https://doi.org/10.1609/aaai.v35i12.17325
H. Han et al., "Time series forecasting model for non-stationary series pattern extraction using deep learning and GARCH modeling," Journal of Cloud Computing, vol. 13, no. 1, Jan. 2024, Art. no. 2. DOI: https://doi.org/10.1186/s13677-023-00576-7
N.-H. Duong, M.-T. Nguyen, T.-H. Nguyen, and T.-P. Tran, "Application of Seasonal Trend Decomposition using Loess and Long Short-Term Memory in Peak Load Forecasting Model in Tien Giang," Engineering, Technology & Applied Science Research, vol. 13, no. 5, pp. 11628–11634, Oct. 2023. DOI: https://doi.org/10.48084/etasr.6181
H. Wu, J. Xu, J. Wang, and M. Long, "Autoformer: decomposition transformers with auto-correlation for long-term series forecasting," in Proceedings of the 35th International Conference on Neural Information Processing Systems, Virtual Event, 2021, pp. 22419–22430.
T. Zhou, Z. Ma, Q. Wen, X. Wang, L. Sun, and R. Jin, "FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting," in Proceedings of the 39th International Conference on Machine Learning, Baltimore, MD, USA, 2022, pp. 27268–27286.
H. Cao, Z. Huang, T. Yao, J. Wang, H. He, and Y. Wang, "InParformer: Evolutionary Decomposition Transformers with Interactive Parallel Attention for Long-Term Time Series Forecasting," Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 6, pp. 6906–6915, June 2023. DOI: https://doi.org/10.1609/aaai.v37i6.25845
M. Höll, K. Kiyono, and H. Kantz, "Theoretical foundation of detrending methods for fluctuation analysis such as detrended fluctuation analysis and detrending moving average," Physical Review E, vol. 99, no. 3, Mar. 2019, Art. no. 033305. DOI: https://doi.org/10.1103/PhysRevE.99.033305
Y. Wang, J. Zhu, and R. Kang, "DESTformer: A Transformer Based on Explicit Seasonal–Trend Decomposition for Long-Term Series Forecasting," Applied Sciences, vol. 13, no. 18, Sept. 2023, Art. no. 10505. DOI: https://doi.org/10.3390/app131810505
Y. Yu, R. Ma, and Z. Ma, "Robformer: A robust decomposition transformer for long-term time series forecasting," Pattern Recognition, vol. 153, Sept. 2024, Art. no. 110552. DOI: https://doi.org/10.1016/j.patcog.2024.110552
Y. Nie, N. H. Nguyen, P. Sinthong, and J. Kalagnanam, "A Time Series is Worth 64 Words: Long-term Forecasting with Transformers." arXiv, Mar. 05, 2023.
Y. Liu, H. Wu, J. Wang, and M. Long, "Non-stationary transformers: exploring the stationarity in time series forecasting," in Proceedings of the 36th International Conference on Neural Information Processing Systems, New Orleans, LA, USA, 2022, pp. 9881–9893. DOI: https://doi.org/10.52202/068431-0718
Y. Zhang and J. Yan, "Crossformer: Transformer Utilizing Cross-Dimension Dependency for Multivariate Time Series Forecasting," in The Eleventh International Conference on Learning Representations, Kigali, Rwanda, 2022.
A. Zeng, M. Chen, L. Zhang, and Q. Xu, "Are Transformers Effective for Time Series Forecasting?," Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 9, pp. 11121–11128, June 2023. DOI: https://doi.org/10.1609/aaai.v37i9.26317
T. Kim, J. Kim, Y. Tae, C. Park, J.-H. Choi, and J. Choo, "Reversible Instance Normalization for Accurate Time-Series Forecasting against Distribution Shift," in The Tenth International Conference on Learning Representations, Virtual Event, 2021.
G. Kavakci, B. Cicekdag, and S. Ertekin, "Time Series Prediction of Solar Power Generation Using Trend Decomposition," Energy Technology, vol. 12, no. 2, Feb. 2024, Art. no. 2300914. DOI: https://doi.org/10.1002/ente.202300914
"Autoformer." Google Drive. https://drive.google.com/drive/folders/1ZOYpTUa82_jCcxIdTmyr0LXQfvaM9vIy.
Downloads
How to Cite
License
Copyright (c) 2026 Nasra Pratama Putra, Suprapto, Agus Sihabuddin
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain the copyright and grant the journal the right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) after its publication in ETASR with an acknowledgement of its initial publication in this journal.
