A Cohesive Transformer-Capsule Network Model for Sentiment and Aspect Analysis Using Contextual Embeddings
Received: 2 December 2025 | Revised: 28 January 2026 and 19 February 2026 | Accepted: 20 February 2026 | Online: 2 March 2026
Corresponding author: C. H. Sumalakshmi
Abstract
In aspect-based sentiment analysis, identifying and associating sentiments with specific aspects is a significant challenge due to the complex interplay between context and language. Traditional methods often struggle to accurately identify sentiments associated with particular aspects, especially in texts characterized by dynamically evolving linguistic contexts. The present study introduces a novel Transformer-Capsule Network for Sentiment and Aspect Analysis (TCSA) model that integrates transformer architectures and Capsule Networks (CapsNets) designed to improve the precision of aspect analysis. This model combines the deep contextual understanding capabilities of RoBERTa with the dynamic routing efficiency of CapsNets. The key strategy employed involves dynamic contextual word embeddings generated by BERT, which are crucial for capturing semantic contexts contributing to accurate aspect analysis. Furthermore, the model uses sophisticated data augmentation algorithms, including synonym substitution, back-translation, and contextual augmentation to expand the training data. The TCSA model utilizes a multi-head self-attention mechanism in the transformer-capsule framework, enabling focused attention on the various data fragments, and thus interpreting complex interactions between textual aspects and their contexts. The model demonstrates excellent performance, achieving an accuracy of 97.15%, a precision of 97.20%, a recall of 98.12%, and an F1-score of 97.30%. The results not only demonstrate the effectiveness of the model for aspect analysis but also set a new standard in the use of dynamic contextual embeddings and complex network architectures for sentiment analysis.
Keywords:
sentiment analysis, transformer networks, capsule networks, dynamic contextual embeddings, multi-head self-attention, RoBERTa, textual semanticsDownloads
References
Based Text Sentiment Classification," IFAC-PapersOnLine, vol. 53, no. 5, pp. 698–703, 2021. DOI: https://doi.org/10.1016/j.ifacol.2021.04.160
B. Liu, Sentiment Analysis and Opinion Mining. Cham, Switzerland: Springer International Publishing, 2012.
Z. M. Zohreh Madhoushi, A. R. Hamdan, and S. Zainudin, "Aspect-Based Sentiment Analysis Methods in Recent Years," Asia-Pacific Journal of Information Technology & Multimedia, vol. 08, no. 01, pp. 79–96, Jun. 2019. DOI: https://doi.org/10.17576/apjitm-2019-0801-07
P. Demotte, K. Wijegunarathna, D. Meedeniya, and I. Perera, "Enhanced Sentiment Extraction Architecture for Social Media Content Analysis Using Capsule Networks," Multimedia Tools and Applications, vol. 82, no. 6, pp. 8665–8690, Mar. 2023. DOI: https://doi.org/10.1007/s11042-021-11471-1
P. Pookduang, R. Klangbunrueang, W. Chansanam, and T. Lunrasri, "Advancing Sentiment Analysis: Evaluating RoBERTa against Traditional and Deep Learning Models," Engineering, Technology & Applied Science Research, vol. 15, no. 1, pp. 20167–20174, Feb. 2025. DOI: https://doi.org/10.48084/etasr.9703
D. Tang, B. Qin, X. Feng, and T. Liu, "Effective LSTMs for Target-Dependent Sentiment Classification," in Proceedings of the 26th International Conference on Computational Linguistics, Osaka, Japan, Dec. 2016, pp. 3298–3307.
Z. Wang, H. Wu, H. Liu, and Q.-H. Cai, "Bert-Pair-Networks for Sentiment Classification," in 2020 International Conference on Machine Learning and Cybernetics, Adelaide, Australia, Dec. 2020, pp. 273–278. DOI: https://doi.org/10.1109/ICMLC51923.2020.9469534
A. Yadav and D. K. Vishwakarma, "Sentiment Analysis Using Deep Learning Architectures: A Review," Artificial Intelligence Review, vol. 53, no. 6, pp. 4335–4385, Aug. 2020. DOI: https://doi.org/10.1007/s10462-019-09794-5
K. M. Karaoğlan and O. Fındık, "Extended Rule-Based Opinion Target Extraction with a Novel Text Pre-Processing Method and Ensemble Learning," Applied Soft Computing, vol. 118, Mar. 2022, Art. no. 108524. DOI: https://doi.org/10.1016/j.asoc.2022.108524
M. Jiang, J. Wu, X. Shi, and M. Zhang, "Transformer Based Memory Network for Sentiment Analysis of Web Comments," IEEE Access, vol. 7, pp. 179942–179953, 2019. DOI: https://doi.org/10.1109/ACCESS.2019.2957192
Q.-H. Vo, H.-T. Nguyen, B. Le, and M.-L. Nguyen, "Multi-Channel LSTM-CNN Model for Vietnamese Sentiment Analysis," in 2017 9th International Conference on Knowledge and Systems Engineering, Hue, Vietnam, Oct. 2017, pp. 24–29. DOI: https://doi.org/10.1109/KSE.2017.8119429
P. Liu, X. Qiu, and X. Huang, "Recurrent Neural Network for Text Classification with Multi-Task Learning." arXiv, 2016.
J. Han, J. Chen, P. Chen, J. Liu, and D. Peng, "Chinese Text Sentiment Classification Based on Bidirectional Temporal Deep Convolutional Network," Computer Applications and Software, vol. 36, no. 12, pp. 225–231, 2019.
S. Liao, J. Wang, R. Yu, K. Sato, and Z. Cheng, "CNN for Situations Understanding Based on Sentiment Analysis of Twitter Data," Procedia Computer Science, vol. 111, pp. 376–381, 2017. DOI: https://doi.org/10.1016/j.procs.2017.06.037
Z. Jianqiang, G. Xiaolin, and Z. Xuejun, "Deep Convolution Neural Networks for Twitter Sentiment Analysis," IEEE Access, vol. 6, pp. 23253–23260, 2018. DOI: https://doi.org/10.1109/ACCESS.2017.2776930
R. Johnson and T. Zhang, "Effective Use of Word Order for Text Categorization with Convolutional Neural Networks," in Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Denver, CO, USA, 2015, pp. 103–112. DOI: https://doi.org/10.3115/v1/N15-1011
M. Cai, "Sentiment Analysis of Tweets Using Deep Neural Architectures," in 32nd Conference on Neural Information Processing Systems, Montréal, Canada, 2018, pp. 1–8.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, "Pre-Training of Deep Bidirectional Transformers for Language Understanding," in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, MN, USA, 2019, pp. 4171–4186. DOI: https://doi.org/10.18653/v1/N19-1423
S. Sabour, N. Frosst, and G. E. Hinton, "Dynamic Routing Between Capsules," in 31st Conference on Neural Information Processing Systems, Long Beach, CA, USA, 2017.
P. Demotte and S. Ranathunga, "Dual-State Capsule Networks for Text Classification." arXiv, 2021.
X. Yu, S.-N. Luo, Y. Wu, Z. Cai, T.-W. Kuan, and S.-P. Tseng, "Research on a Capsule Network Text Classification Method with a Self-Attention Mechanism," Symmetry, vol. 16, no. 5, Apr. 2024, Art. no. 517. DOI: https://doi.org/10.3390/sym16050517
J. Su, S. Yu, and D. Luo, "Enhancing Aspect-Based Sentiment Analysis with Capsule Network," IEEE Access, vol. 8, pp. 100551–100561, 2020. DOI: https://doi.org/10.1109/ACCESS.2020.2997675
Z. Yang, Z. Dai, Y. Yang, J. Carbonell, R. Salakhutdinov, and Q. V. Le, "XLNet: Generalized Autoregressive Pretraining for Language Understanding." arXiv, Jan. 02, 2020.
J. Kim and J.-H. Lee, "Modeling Inter-Speaker Relationship in XLNet for Contextual Spoken Language Understanding." arXiv, 2019.
B. AlBadani, R. Shi, J. Dong, R. Al-Sabri, and O. B. Moctard, "Transformer-Based Graph Convolutional Network for Sentiment Analysis," Applied Sciences, vol. 12, no. 3, Jan. 2022, Art. no. 1316. DOI: https://doi.org/10.3390/app12031316
S. Longpre, Y. Lu, Z. Tu, and C. DuBois, "An Exploration of Data Augmentation and Sampling Techniques for Domain-Agnostic Question Answering," in Proceedings of the 2nd Workshop on Machine Reading for Question Answering, Hong Kong, China, 2019, pp. 220–227. DOI: https://doi.org/10.18653/v1/D19-5829
J. Liu, "515K Hotel Reviews Data in Europe." Kaggle, 2014, [Online]. Available: https://www.kaggle.com/datasets/jiashenliu/515k-hotel-reviews-data-in-europe.
A. Gulli and S. Pal, Deep Learning with Keras: Implement Neural networks with Keras on Theano and TensorFlow. Birmingham, UK: Packt Publishing, 2017.
D. M. W. Powers, "Evaluation: From Precision, Recall and F-Measure to ROC, Informedness, Markedness and Correlation," Journal of Machine Learning Technologies, vol. 2, no. 1, pp. 37–63, 2011.
H. Yang, C. Zhang, and K. Li, "PyABSA: A Modularized Framework for Reproducible Aspect-Based Sentiment Analysis," in Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, Birmingham, UK, Oct. 2023, pp. 5117–5122. DOI: https://doi.org/10.1145/3583780.3614752
X. Bai, P. Liu, and Y. Zhang, "Investigating Typed Syntactic Dependencies for Targeted Sentiment Classification Using Graph Attention Neural Network," IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 503–514, 2021. DOI: https://doi.org/10.1109/TASLP.2020.3042009
M. Pontiki, D. Galanis, J. Pavlopoulos, H. Papageorgiou, I. Androutsopoulos, and S. Manandhar, "SemEval-2014 Task 4: Aspect Based Sentiment Analysis," in Proceedings of the 8th International Workshop on Semantic Evaluation, Dublin, Ireland, 2014, pp. 27–35. DOI: https://doi.org/10.3115/v1/S14-2004
Downloads
How to Cite
License
Copyright (c) 2026 Laxmi Pamulaparthy, C. H. Sumalakshmi
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain the copyright and grant the journal the right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) after its publication in ETASR with an acknowledgement of its initial publication in this journal.
