Benchmarking Transformer Models for Low-Resource Language Translation: A Case Study on the Tegalan-Indonesian Language Pair
Received: 18 November 2025 | Revised: 16 December 2025 | Accepted: 26 December 2025 | Online: 4 April 2026
Corresponding author: Dwi Intan Af'idah
Abstract
This study investigates how well Transformer models can translate a low-resource local language, Tegalan, to Indonesian. Three Transformer-based models, mBART-50, mT5, and NLLB-200, were tested using a new parallel Tegalan-Indonesian dataset, collected from everyday conversations and online news texts. The translations were manually reviewed by native speakers and cultural experts. The preprocessing steps included case folding, spelling normalization, and subword tokenization to ensure consistency and handle dialect differences. Each model was fine-tuned under controlled conditions, with manual adjustment of hyperparameters. BLEU, METEOR, and TER were used to evaluate the models, offering insight into word-level accuracy, semantic alignment, and the required number of edits. The results show that NLLB-200 achieved the highest performance, with BLEU scores of 85.51, METEOR scores of 76.91, and TER scores of 17.73, clearly exceeding both mBART-50 and mT5. A qualitative review of the output indicated that NLLB-200 generated more natural and contextually appropriate translations. The results suggest that Transformer models can be a practical option for translation in low-resource environments and may also contribute to ongoing efforts to document and maintain regional languages. Further work is planned to enlarge the dataset and examine the extent to which semi-supervised techniques might strengthen both accuracy and overall model robustness.
Keywords:
Low-Resource Language, Tegalan–Indonesian, Transformer Models, Neural Machine TranslationDownloads
References
V. Peltokorpi and E. Vaara, "Language Policies and Practices in Wholly Owned Foreign Subsidiaries: A Recontextualization Perspective," in Language in International Business, M. Y. Brannen and T. Mughan, Eds. Springer International Publishing, 2017, pp. 93–138. DOI: https://doi.org/10.1007/978-3-319-42745-4_5
I. Rivera-Trigueros, "Machine translation systems and quality assessment: a systematic review," Language Resources and Evaluation, vol. 56, no. 2, pp. 593–619, June 2022. DOI: https://doi.org/10.1007/s10579-021-09537-5
D. Khurana, A. Koli, K. Khatter, and S. Singh, "Natural language processing: state of the art, current trends and challenges," Multimedia Tools and Applications, vol. 82, no. 3, pp. 3713–3744, Jan. 2023. DOI: https://doi.org/10.1007/s11042-022-13428-4
S. Bird, "Decolonising Speech and Language Technology," in Proceedings of the 28th International Conference on Computational Linguistics, Barcelona, Spain, 2020, pp. 3504–3519. DOI: https://doi.org/10.18653/v1/2020.coling-main.313
A. F. Aji et al., "One Country, 700+ Languages: NLP Challenges for Underrepresented Languages and Dialects in Indonesia," in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Dublin, Ireland, 2022, pp. 7226–7249. DOI: https://doi.org/10.18653/v1/2022.acl-long.500
F. Li, B. Liu, H. Yan, P. Xie, J. Li, and Z. Zhang, "Incorporating bilingual translation templates into neural machine translation," Scientific Reports, vol. 15, no. 1, Feb. 2025, Art. no. 5547. DOI: https://doi.org/10.1038/s41598-025-86754-w
F. Reza, Z. Rohmah, and R. Ismail, "Dialect shift and cultural dynamism among Betawi community in urban Jakarta in Palang Pintu and Rebut Dandang traditional ceremonies," Cogent Arts & Humanities, vol. 11, no. 1, Dec. 2024, Art. no. 2410542. DOI: https://doi.org/10.1080/23311983.2024.2410542
U. Kulsum, A. D. Darnis, and A. Asdari, "Language Shift and Endangerment of Sundanese Banten Dialect in South Tangerang," Insaniyat : Journal of Islam and Humanities, vol. 7, no. 2, pp. 99–111, May 2023. DOI: https://doi.org/10.15408/insaniyat.v7i2.28905
D. A. Sulistyo, "LSTM-Based Machine Translation for Madurese-Indonesian," Journal of Applied Data Sciences, vol. 4, no. 3, pp. 189–199, Sept. 2023. DOI: https://doi.org/10.47738/jads.v4i3.113
D. E. Messaoudi and D. Nessah, "Enhancing Neural Arabic Machine Translation using Character-Level CNN-BILSTM and Hybrid Attention," Engineering, Technology & Applied Science Research, vol. 14, no. 5, pp. 17029–17034, Oct. 2024. DOI: https://doi.org/10.48084/etasr.8383
A. Vaswani et al., "Attention Is All You Need." arXiv, 2017.
Z. Tan et al., "Neural machine translation: A review of methods, resources, and tools," AI Open, vol. 1, pp. 5–21, 2020. DOI: https://doi.org/10.1016/j.aiopen.2020.11.001
S. Ranathunga, E. S. A. Lee, M. P. Skenduli, R. Shekhar, M. Alam, and R. Kaur, "Neural Machine Translation for Low-Resource Languages: A Survey." arXiv, 2021.
L. Susanto, R. Diandaru, A. Krisnadhi, A. Purwarianti, and D. T. Wijaya, "Replicable Benchmarking of Neural Machine Translation (NMT) on Low-Resource Local Languages in Indonesia," in Proceedings of the First Workshop in South East Asian Language Processing, Nusa Dua, Bali, Indonesia, 2023, pp. 100–115. DOI: https://doi.org/10.18653/v1/2023.sealp-1.8
Y. Tang et al., "Multilingual Translation with Extensible Multilingual Pretraining and Finetuning." arXiv, Aug. 02, 2020.
L. Xue et al., "mT5: A massively multilingual pre-trained text-to-text transformer." arXiv, 2020. DOI: https://doi.org/10.18653/v1/2021.naacl-main.41
NLLB Team et al., "Scaling neural machine translation to 200 languages," Nature, vol. 630, no. 8018, pp. 841–846, June 2024. DOI: https://doi.org/10.1038/s41586-024-07335-x
S. M. Singh and T. D. Singh, "Low resource machine translation of english–manipuri: A semi-supervised approach," Expert Systems with Applications, vol. 209, Dec. 2022, Art. no. 118187. DOI: https://doi.org/10.1016/j.eswa.2022.118187
B. Namdarzadeh, S. Mohseni, L. Zhu, G. Wisniewski, and N. Ballier, "Fine-tuning MBART-50 with French and Farsi data to improve the translation of Farsi dislocations into English and French," in Proceedings of Machine Translation Summit XIX, Vol. 2: Users Track, Macau SAR, China, June 2023, pp. 152–161.
T. N. Son, N. A. Tu, and N. M. Tri, "An Efficient Approach for Machine Translation on Low-resource Languages: A Case Study in Vietnamese-Chinese." arXiv, Jan. 31, 2025.
R. Oida-Onesa and M. A. Ballera, "Fine Tuning Language Models: A Tale of Two Low-Resource Languages," Data Intelligence, vol. 6, no. 4, pp. 946–967, Dec. 2024. DOI: https://doi.org/10.3724/2096-7004.di.2024.0016
V. N. M. Abadi and F. Ghasemian, "Enhancing Persian text summarization through a three-phase fine-tuning and reinforcement learning approach with the mT5 transformer model," Scientific Reports, vol. 15, no. 1, Jan. 2025, Art. no. 80. DOI: https://doi.org/10.1038/s41598-024-78235-3
V. Akerman et al., "The eBible Corpus: Data and Model Benchmarks for Bible Translation for Low-Resource Languages." arXiv, Apr. 19, 2023.
D. Degenaro and T. Lupicki, "Experiments in Mamba Sequence Modeling and NLLB-200 Fine-Tuning for Low Resource Multilingual Machine Translation," in Proceedings of the 4th Workshop on Natural Language Processing for Indigenous Languages of the Americas (AmericasNLP 2024), Mexico City, Mexico, 2024, pp. 188–194. DOI: https://doi.org/10.18653/v1/2024.americasnlp-1.22
S. Lee et al., "A Survey on Evaluation Metrics for Machine Translation," Mathematics, vol. 11, no. 4, Feb. 2023, Art. no. 1006. DOI: https://doi.org/10.3390/math11041006
"Machine Translation | Seq2Seq | LSTMs." Kaggle, [Online]. Available: https://kaggle.com/code/harshjain123/machine-translation-seq2seq-lstms.
D. N. De Oliveira and L. H. D. C. Merschmann, "Joint evaluation of preprocessing tasks with classifiers for sentiment analysis in Brazilian Portuguese language," Multimedia Tools and Applications, vol. 80, no. 10, pp. 15391–15412, Apr. 2021. DOI: https://doi.org/10.1007/s11042-020-10323-8
T. Bergmanis, A. Stafanovičs, and M. Pinnis, "Robust Neural Machine Translation: Modeling Orthographic and Interpunctual Variation," in Frontiers in Artificial Intelligence and Applications, A. Utka, J. Vaičenonienė, J. Kovalevskaitė, and D. Kalinauskaitė, Eds. IOS Press, 2020. DOI: https://doi.org/10.3233/FAIA200606
K. Imamura and M. Utiyama, "An Empirical Study of Multilingual Vocabulary for Neural Machine Translation Models," in Proceedings of the Eleventh Workshop on Asian Translation (WAT 2024), Miami, FL, USA, Aug. 2024, pp. 22–35. DOI: https://doi.org/10.18653/v1/2024.wat-1.2
Y. Liu et al., "Multilingual Denoising Pre-training for Neural Machine Translation," Transactions of the Association for Computational Linguistics, vol. 8, pp. 726–742, Dec. 2020. DOI: https://doi.org/10.1162/tacl_a_00343
C. Raffel et al., "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer," Journal of Machine Learning Research, vol. 21, no. 140, pp. 1–67, 2020.
NLLB Team et al., "No Language Left Behind: Scaling Human-Centered Machine Translation." arXiv, 2022.
D. A. Sulistyo, A. P. Wibawa, D. D. Prasetya, F. A. Ahda, I. N. G. A. Astawa, and F. A. Dwiyanto, "Multilingual Parallel Corpus for Indonesian Low-Resource Languages," JOIV : International Journal on Informatics Visualization, vol. 9, no. 5, pp. 2176–2182, Sept. 2025. DOI: https://doi.org/10.62527/joiv.9.5.3412
Downloads
How to Cite
License
Copyright (c) 2026 Dwi Intan Af'idah, Sharfina Febbi Handayani, Ratri Wikaningtyas
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain the copyright and grant the journal the right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) after its publication in ETASR with an acknowledgement of its initial publication in this journal.
