Large Language Models for Arabic Sentiment Analysis and Machine Translation
Received: 11 November 2024 | Revised: 15 December 2024 | Accepted: 29 December 2024 | Online: 23 January 2025
Corresponding author: Mohamed Zouidine
Abstract
Large Language Models (LLMs) have recently demonstrated outstanding performance in a variety of Natural Language Processing (NLP) tasks. Although many LLMs have been developed, only a few models have been evaluated in the context of the Arabic language, with a significant focus on the ChatGPT model. This study assessed three LLMs on two Arabic NLP tasks: sentiment analysis and machine translation. The capabilities of LLaMA, Mixtral, and Gemma under zero- and few-shot learning were investigated, and their performance was compared against State-Of-The-Art (SOTA) models. The experimental results showed that, among the three models, LLaMA tends to have better comprehension abilities for the Arabic language, outperforming Mixtral and Gemma on both tasks. However, except for the Arabic-to-English translation, where LLaMA outperforms the transformer model by 4 BLEU points, in all cases, the performance of the three LLMs fell behind that of the SOTA model.
Keywords:
Arabic Natural Language Processing (NLP), Gemma, Large Language Models (LLM), LLaMA, machine translation, mixtral, sentiment analysisDownloads
References
S. Minaee et al., "Large Language Models: A Survey." arXiv, 2024.
W. X. Zhao et al., "A Survey of Large Language Models." arXiv, 2023.
H. Touvron et al., "LLaMA: Open and Efficient Foundation Language Models." arXiv, 2023.
OpenAI et al., "GPT-4 Technical Report." arXiv, 2023.
A. Abdelali et al., "LAraBench: Benchmarking Arabic AI with Large Language Models," in Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), St. Julian’s, Malta, Nov. 2024, pp. 487–520.
Z. Alyafeai, M. S. Alshaibani, B. AlKhamissi, H. Luqman, E. Alareqi, and A. Fadel, "Taqyim: Evaluating Arabic NLP Tasks Using ChatGPT Models." arXiv, 2023.
A. Al-Thubaity et al., "Evaluating ChatGPT and Bard AI on Arabic Sentiment Analysis," in Proceedings of ArabicNLP 2023, 2023, pp. 335–349.
M. D. Alahmadi, M. Alharbi, A. Tayeb, and M. Alshangiti, "Evaluating Large Language Models’ Proficiency in Answering Arabic GAT Exam Questions," Engineering, Technology & Applied Science Research, vol. 14, no. 6, pp. 17774–17780, Dec. 2024.
K. Yu, Y. Liu, A. G. Schwing, and J. Peng, "Fast and Accurate Text Classification: Skimming, Rereading and Early Stopping," presented at the ICLR 2018 Workshop Program Chairs, Feb. 2018.
M. T. I. Khondaker, A. Waheed, E. M. B. Nagoudi, and M. Abdul-Mageed, "GPTAraEval: A Comprehensive Evaluation of ChatGPT on Arabic NLP," in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023, pp. 220–247.
K. Kadaoui et al., "TARJAMAT: Evaluation of Bard and ChatGPT on Machine Translation of Ten Arabic Varieties," in Proceedings of ArabicNLP 2023, 2023, pp. 52–75.
A. Q. Jiang et al., "Mixtral of Experts." arXiv, 2024.
Gemma Team et al., "Gemma: Open Models Based on Gemini Research and Technology." arXiv, 2024.
"zouidine/AMT_LLMs." Jul. 17, 2024, [Online]. Available: https://github.com/zouidine/AMT_LLMs.
"zouidine/ASA_LLMs." Jun. 20, 2024, [Online]. Available: https://github.com/zouidine/ASA_LLMs.
M. Aly and A. Atiya, "LABR: A Large Scale Arabic Book Reviews Dataset," in Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Sofia, Bulgaria, Dec. 2013, pp. 494–498.
A. Elnagar, Y. S. Khalifa, and A. Einea, "Hotel Arabic-Reviews Dataset Construction for Sentiment Analysis Applications," in Intelligent Natural Language Processing: Trends and Applications, vol. 740, K. Shaalan, A. E. Hassanien, and F. Tolba, Eds. Springer International Publishing, 2018, pp. 35–52.
M. Cettolo, C. Girardi, and M. Federico, "WIT3: Web Inventory of Transcribed and Translated Talks," in Proceedings of the 16th Annual Conference of the European Association for Machine Translation, Trento, Italy, Dec. 2012, pp. 261–268.
"Groq Products," Groq, Oct. 18, 2021. https://groq.com/products/.
J. White et al., "A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT." arXiv, 2023.
W. Antoun, F. Baly, and H. Hajj, "AraBERT: Transformer-based Model for Arabic Language Understanding." arXiv, Mar. 07, 2021.
A. Vaswani et al., "Attention is All you Need," in Advances in Neural Information Processing Systems, 2017, vol. 30.
M. Zouidine, M. Khalil, and A. I. El Farouk, "Pre-processing and Pre-trained Word Embedding Techniques for Arabic Machine Translation," in Intelligent Systems Design and Applications, 2023, pp. 115–125.
K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, "Bleu: a Method for Automatic Evaluation of Machine Translation," in Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Philadelphia, PA, USA, Apr. 2002, pp. 311–318.
T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi, "BERTScore: Evaluating Text Generation with BERT," presented at the International Conference on Learning Representations, Sep. 2019.
Downloads
How to Cite
License
Copyright (c) 2025 Mohamed Zouidine, Mohammed Khalil
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain the copyright and grant the journal the right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) after its publication in ETASR with an acknowledgement of its initial publication in this journal.
