A Fine-Tuned BART Pre-trained Language Model for the Indonesian Question-Answering Task
Received: 4 December 2024 | Revised: 30 December 2024 and 16 January 2025 | Accepted: 19 January 2025 | Online: 3 April 2025
Corresponding author: Seng Hansun
Abstract
The information extraction process from a given context can be time consuming and a Pre-trained Language Model (PLM) based on the transformer architecture could reduce the time needed to obtain the information. Moreover, PLM is easily fine-tuned to accomplish certain tasks, one of which is the Question-Answering (QA) task. In literature, QA tasks are generally fine-tuned using encoder-based PLMs, such as the Bidirectional Encoder Representations from Transformers (BERT), where the generated answers come from the extraction process of the context. In order to be able to return more abstract answers, a PLM with Natural Language Generation (NLG) capability, such as the Bidirectional and Auto-Regressive Transformer (BART), is needed. In this study, we aim to fine-tune the NLG PLM using BART to build a more abstractive generative QA task. Based on the experimental results, the fine-tuned BART model performs well with an 85.84 F1 score and a 59.42 Exact Match (EM) score.
Keywords:
BART, BERT, NLG, PLM, QA taskDownloads
References
A. Bouziane, D. Bouchiha, N. Doumi, and M. Malki, "Question Answering Systems: Survey and Trends," Procedia Computer Science, vol. 73, pp. 366–375, Jan. 2015.
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding," in Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Minneapolis, MN, USA, Mar. 2019, pp. 4171–4186.
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, "Language Models are Unsupervised Multitask Learners," 2019. Available: https://cdn.openai.com/better-language-models/language_
models_are_unsupervised_multitask_learners.pdf
S. Giddaluru, S. M. Maturi, O. Ooruchintala, and M. Munirathnam, "Stance Detection in Hinglish Data using the BART-large-MNLI Integration Model," Engineering, Technology & Applied Science Research, vol. 14, no. 4, pp. 15477–15481, Aug. 2024.
M. Lewis et al., "BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension," in 58th Annual Meeting of the Association for Computational Linguistics, Jul. 2020, pp. 7871–7880.
A. Vaswani et al., "Attention is all you need," in 31st International Conference on Neural Information Processing Systems, Long Beach, CA, USA, Dec. 2017, pp. 6000–6010.
T. B. Brown et al., "Language models are few-shot learners," in 34th International Conference on Neural Information Processing Systems, Vancouver, BC, Canada, Dec. 2020, pp. 1877–1901.
Y. Liu et al., "Multilingual Denoising Pre-training for Neural Machine Translation," Transactions of the Association for Computational Linguistics, vol. 8, pp. 726–742, Dec. 2020.
Y. Tang et al., "Multilingual Translation from Denoising Pre-Training," in Findings of the Association for Computational Linguistics: ACL-IJCNLP, Aug. 2021, pp. 3450–3466.
J. H. Clark et al., "TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages," Transactions of the Association for Computational Linguistics, vol. 8, pp. 454–470, Jul. 2020.
T. Wolf et al., "Transformers: State-of-the-Art Natural Language Processing," in Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Jul. 2020, pp. 38–45.
K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, "BLEU: a method for automatic evaluation of machine translation," in 40th Annual Meeting on Association for Computational Linguistics, Philadelphia, PA, USA, Jul. 2012, pp. 311–318.
C.-Y. Lin, "ROUGE: A Package for Automatic Evaluation of Summaries," in Text Summarization Branches Out, Barcelona, Spain, Apr. 2004, pp. 74–81.
R. A. Putri and A. Oh, "IDK-MRC: Unanswerable Questions for Indonesian Machine Reading Comprehension," in Conference on Empirical Methods in Natural Language Processing, Abu Dhabi, United Arab Emirates, Dec. 2022, pp. 6918–6933.
M. Fuadi and A. D. Wibawa, "Automatic Question Generation from Indonesian Texts Using Text-to-Text Transformers," in International Conference on Electrical and Information Technology, Malang, Indonesia, Sep. 2022, pp. 84–89.
B. Richardson and A. Wicaksana, "Comparison of indobert-lite and roberta in text mining for Indonesian language question answering application," International Journal of Innovative Computing, Information and Control, vol. 18, no. 6, pp. 1719–1734, Dec. 2022.
M. M. Henry, G. N. Elwirehardja, and B. Pardamean, "Automatic question generation for bahasa indonesia examination using copynet," Procedia Computer Science, vol. 245, pp. 953–962, Jan. 2024.
P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang, "SQuAD: 100,000+ Questions for Machine Comprehension of Text," in Conference on Empirical Methods in Natural Language Processing, Austin, TX, USA, Nov. 2016, pp. 2383–2392.
P. Rajpurkar, R. Jia, and P. Liang, "Know What You Don`t Know: Unanswerable Questions for SQuAD," in 56th Annual Meeting of the Association for Computational Linguistics, Melbourne, Australia, Jul. 2018, pp. 784–789.
M. Artetxe, S. Ruder, and D. Yogatama, "On the Cross-lingual Transferability of Monolingual Representations," in 58th Annual Meeting of the Association for Computational Linguistics, Jul. 2020, pp. 4623–4637.
L. Xue et al., "mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer," in Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Jun. 2021, pp. 483–498.
Y. Liu et al., "RoBERTa: A Robustly Optimized BERT Pretraining Approach." arXiv, Jul. 26, 2019.
V. Sanh, L. Debut, J. Chaumond, and T. Wolf, "DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter," in 33rd Conference on Neural Information Processing Systems, Vancouver, BC, Canada, Dec. 2019, pp. 1–5.
C. Raffel et al., "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer," Journal of Machine Learning Research, vol. 21, no. 140, pp. 1–67, 2020.
S. Hansun, A. Suryadibrata, R. Nurhasanah, and J. Fitra, "Tweets sentiment on ppkm policy as a covid-19 response in indonesia," Indian Journal of Computer Science and Engineering, vol. 13, no. 1, pp. 51–58, Feb. 2022.
S. Hansun, A. Wicaksana, and A. Q. M. Khaliq, "Multivariate cryptocurrency prediction: comparative analysis of three recurrent neural networks approaches," Journal of Big Data, vol. 9, no. 1, Apr. 2022, Art. no. 50.
T. A. Wotaifi and B. N. Dhannoon, "An Effective Hybrid Deep Neural Network for Arabic Fake News Detection," Baghdad Science Journal, vol. 20, no. 4, pp. 1392–1392, Aug. 2023.
A. K. Jassim, M. J. Hamzah, and A. H. Aliwy, "Using Graph Mining Method in Analyzing Turkish Loanwords Derived from Arabic Language," Baghdad Science Journal, vol. 19, no. 6, pp. 1369–1369, Dec. 2022.
B. Wilie et al., "IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding," in 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, Suzhou, China, Dec. 2020, pp. 843–857.
S. Cahyawijaya et al., "IndoNLG: Benchmark and Resources for Evaluating Indonesian Natural Language Generation," in Conference on Empirical Methods in Natural Language Processing, Nov. 2021, pp. 8875–8898.
A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. Bowman, "GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding," in EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, Brussels, Belgium, Nov. 2018, pp. 353–355.
S. Gehrmann et al., "The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics," in 1st Workshop on Natural Language Generation, Evaluation, and Metrics, Bangkok, Thailand, Aug. 2021, pp. 96–120.
Downloads
How to Cite
License
Copyright (c) 2025 Alfonso Darren Vincentio, Seng Hansun
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain the copyright and grant the journal the right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) after its publication in ETASR with an acknowledgement of its initial publication in this journal.
