Current Practices in Quality Assessment of Systematic Reviews in Computing: Exploring Automation Potential
Received: 19 November 2024 | Revised: 17 January 2025 | Accepted: 5 February 2025 | Online: 3 April 2025
Corresponding author: Ghader Reda Kurdi
Abstract
Systematic Literature Reviews (SLRs) play a crucial role in evidence synthesis within computing research. However, the quality of SLRs can vary significantly, affecting their reproducibility and trustworthiness. This study addresses the problem of poorly understood practices in SLR quality assessment. It investigates the current landscape of quality instruments used to assess SLRs in computing by analyzing 97 tertiary studies across various computing domains. The analysis focuses on identifying the dominant quality instruments, and examining reported modifications or adaptations made to them. A qualitative analysis is conducted on the interpretations and scoring of widely utilized quality criteria. The analysis reveals diverse interpretations and potential inconsistencies in the application of quality instruments, owing to the absence of concrete examples. The findings provide valuable insights for both SLR authors and consumers in computing research, pointing out the most widely deployed quality instruments, common customization and interpretive practices, and potential areas for improvement. This study contributes to the ongoing discussions on enhancing SLR quality in computing, forming the basis for automating the quality assessment process.
Keywords:
mapping study, quality assessment, quality criteria, reporting quality, reproducibility, systematic literature review, scoping review, tertiary studyDownloads
References
B. A. Kitchenham, "Systematic reviews," in 10th International Symposium on Software Metrics, 2004. Proceedings., Sep. 2004, pp. xii–xii.
M. Usman, N. bin Ali, and C. Wohlin, "A Quality Assessment Instrument for Systematic Literature Reviews in Software Engineering," e-Informatica Software Engineering Journal, vol. 17, no. 1, 2023, Art. no. 230105.
B. Kitchenham, L. Madeyski, and D. Budgen, "SEGRESS: Software Engineering Guidelines for REporting Secondary Studies," IEEE Transactions on Software Engineering, vol. 49, no. 3, pp. 1273–1298, Mar. 2023.
Z. Hamad and N. Salim, "Systematic literature review (SLR) automation: A systematic literature review," Journal of Theoretical and Applied Information Technology, vol. 59, no. 3, pp. 661–672, 2014.
N. bin Ali and M. Usman, "A critical appraisal tool for systematic literature reviews in software engineering," Information and Software Technology, vol. 112, pp. 48–50, Aug. 2019.
D. Budgen, P. Brereton, S. Drummond, and N. Williams, "Reporting systematic reviews: Some lessons from a tertiary study," Information and Software Technology, vol. 95, pp. 62–74, Mar. 2018.
N. B. Ali and M. Usman, "Reliability of search in systematic reviews: Towards a quality assessment framework for the automated-search strategy," Information and Software Technology, vol. 99, pp. 133–147, Jul. 2018.
X. Zhou, Y. Jin, H. Zhang, S. Li, and X. Huang, "A Map of Threats to Validity of Systematic Literature Reviews in Software Engineering," in 23rd Asia-Pacific Software Engineering Conference, Hamilton, New Zealand, Dec. 2016, pp. 153–160.
J. Kruger, C. Lausberger, I. von Nostitz-Wallwitz, G. Saake, and T. Leich, "Search. Review. Repeat? An empirical study of threats to replicating SLR searches," Empirical Software Engineering, vol. 25, no. 1, pp. 627–677, Jan. 2020.
B. Kitchenham et al., "Systematic literature reviews in software engineering – A tertiary study," Information and Software Technology, vol. 52, no. 8, pp. 792–805, Aug. 2010.
F. Q. B. da Silva, A. L. M. Santos, S. Soares, A. C. C. Franca, and C. V. F. Monteiro, "Six Years of Systematic Literature Reviews in Software Engineering: an Extended Tertiary Study," in International Conference on Software Engineering, Cape Town, South Africa, Dec. 2010, pp. 1–10.
D. Costal, C. Farre, X. Franch, and C. Quer, "How Tertiary Studies perform Quality Assessment of Secondary Studies in Software Engineering." arXiv, Oct. 07, 2021.
G. Kurdi, "Building an Electronic Resource of Systematic Reviews in Computing," in IEEE International Conference on Computing, Langkawi, Malaysia, Oct. 2023, pp. 95–100.
B. Kitchenham, O. Pearl Brereton, D. Budgen, M. Turner, J. Bailey, and S. Linkman, "Systematic literature reviews in software engineering – A systematic literature review," Information and Software Technology, vol. 51, no. 1, pp. 7–15, Jan. 2009.
F. K. Kamei, "The Use of Grey Literature Review as Evidence for Practitioners," ACM SIGSOFT Software Engineering Notes, vol. 44, no. 3, Jul. 2020, Art. no. 23.
J. M. Verner, O. P. Brereton, B. A. Kitchenham, M. Turner, and M. Niazi, "Risks and risk mitigation in global software development: A tertiary study," Information and Software Technology, vol. 56, no. 1, pp. 54–78, Jan. 2014.
M. Cubric, "Drivers, barriers and social considerations for AI adoption in business and management: A tertiary study," Technology in Society, vol. 62, Aug. 2020, Art. no. 101257.
R. Kachouie, S. Sedighadeli, and A. B. Abkenar, "The Role of Socially Assistive Robots in Elderly Wellbeing: A Systematic Review," in International Conference on Human-Computer Interaction, Vancouver, BC, Canada, Jul. 2017, pp. 669–682.
S. Singhal, N. Jatana, B. Suri, S. Misra, and L. Fernandez-Sanz, "Systematic Literature Review on Test Case Selection and Prioritization: A Tertiary Study," Applied Sciences, vol. 11, no. 24, Jan. 2021, Art. no. 12121.
E. Engstrom, P. Runeson, and M. Skoglund, "A systematic review on regression test selection techniques," Information and Software Technology, vol. 52, no. 1, pp. 14–30, Jan. 2010.
B. Suri and S. Singhal, "Implementing Ant Colony Optimization for Test Case Selection and Prioritization," International Journal on Computer Science and Engineering, vol. 3, no. 5, pp. 1924–1932, 2011.
S. Qureshi, S. U. R. Khan, J. Iqbal, and Inayat-Ur-Rehman, "A Study on Mitigating the Communication and Coordination Challenges During Requirements Change Management in Global Software Development," IEEE Access, vol. 9, pp. 88217–88242, Jan. 2021.
S. Imtiaz, M. Bano, N. Ikram, and M. Niazi, "A tertiary study: experiences of conducting systematic literature reviews in software engineering," in 17th International Conference on Evaluation and Assessment in Software Engineering, Porto de Galinhas, Brazil, Apr. 2013, pp. 177–182.
R. Yaqoob, Sanaa, S. U. R. Khan, and M. A. Shah, "Tertiary study on landscaping the review in code smells," IET Conference Proceedings, vol. 2021, no. 4, pp. 131–136, Oct. 2021.
L. Yang et al., "Quality Assessment in Systematic Literature Reviews: A Software Engineering Perspective," Information and Software Technology, vol. 130, Feb. 2021, Art. no. 106397.
P. Malcher, O. Barbosa, D. Viana, and R. Santos, "Software Ecosystems: A Tertiary Study and a Thematic Model." arXiv, Dec. 20, 2022.
P. A. S. Duarte, F. M. Barreto, P. A. C. Aguilar, J. Boudy, R. M. C. Andrade, and W. Viana, "AAL Platforms Challenges in IoT Era: A Tertiary Study," in 13th Annual Conference on System of Systems Engineering, Paris, France, Jun. 2018, pp. 106–113.
J. L. Barros-Justo, F. B. V. Benitti, and S. Matalonga, "Trends in software reuse research: A tertiary study," Computer Standards & Interfaces, vol. 66, Oct. 2019, Art. no. 103352.
Q. Xu, X. Chen, S. Li, H. Zhang, M. A. Babar, and N. K. Tran, "Blockchain-based Solutions for IoT: A Tertiary Study," in 20th International Conference on Software Quality, Reliability and Security Companion, Macau, China, Dec. 2020, pp. 124–131.
M. Raatikainen, J. Tiihonen, and T. Mannisto, "Software product lines and variability modeling: A tertiary study," Journal of Systems and Software, vol. 149, pp. 485–510, Mar. 2019.
S. P. Pillai, S. D. Madhukumar, and T. Radharamanan, "Consolidating evidence based studies in software cost/effort estimation — A tertiary study," in TENCON 2017 - 2017 IEEE Region 10 Conference, Penang, Malaysia, Nov. 2017, pp. 833–838.
A. Idri and L. Cheikhi, "A survey of secondary studies in software process improvement," in 13th International Conference of Computer Systems and Applications, Agadir, Morocco, Dec. 2016, pp. 1–8.
F. Q. B. da Silva, A. L. M. Santos, S. Soares, A. C. C. França, C. V. F. Monteiro, and F. F. Maciel, "Six years of systematic literature reviews in software engineering: An updated tertiary study," Information and Software Technology, vol. 53, no. 9, pp. 899–913, Sep. 2011.
A. B. Marques, R. Rodrigues, and T. Conte, "Systematic Literature Reviews in Distributed Software Development: A Tertiary Study," in Seventh International Conference on Global Software Engineering, Porto Alegre, Brazil, Aug. 2012, pp. 134–143.
J. M. Verner, O. P. Brereton, B. A. . Kitchenham, M. Turner, and M. Niazi, "Systematic literature reviews in global software development: A tertiary study," in 16th International Conference on Evaluation & Assessment in Software Engineering, Ciudad Real, Spain, Dec. 2012, pp. 2–11.
C. Marimuthu and K. Chandrasekaran, "Systematic Studies in Software Product Lines: A Tertiary Study," in 21st International Systems and Software Product Line Conference, Sevilla, Spain, Sep. 2017, pp. 143–152.
A. Ligthart, C. Catal, and B. Tekinerdogan, "Systematic reviews in sentiment analysis: a tertiary study," Artificial Intelligence Review, vol. 54, no. 7, pp. 4997–5053, Oct. 2021.
S. Saleem, N. F. Khan, S. Zafar, and N. Raza, "Systematic literature reviews in cyberbullying/cyber harassment: A tertiary study," Technology in Society, vol. 70, Aug. 2022, Art. no. 102055.
A. A. Khan, J. Keung, M. Niazi, S. Hussain, and H. Zhang, "Systematic Literature Reviews of Software Process Improvement: A Tertiary Study," in European Conference on Software Process Improvement, Ostrava, Czech Republic, Sep. 2017, pp. 177–190.
K. Curcio, R. Santana, S. Reinehr, and A. Malucelli, "Usability in agile software development: A tertiary study," Computer Standards & Interfaces, vol. 64, pp. 61–77, May 2019.
N. F. Khan, A. Yaqoob, M. S. Khan, and N. Ikram, "The cybersecurity behavioral research: A tertiary study," Computers & Security, vol. 120, Sep. 2022, Art. no. 102826.
R. Hoda, N. Salleh, J. Grundy, and H. M. Tee, "Systematic literature reviews in agile software development: A tertiary study," Information and Software Technology, vol. 85, pp. 60–70, May 2017.
L. Villalobos Arias, C. U. Quesada López, A. Martínez Porras, and M. Jenkins Coronas, "A tertiary study on model-based testing areas, tools and challenges: Preliminary results," 2018.
L. Villalobos-Arias, C. Quesada-Lopez, A. Martinez, and M. Jenkins, "Model-based testing areas, tools and challenges: A tertiary study," CLEI Electronic Journal, vol. 22, no. 1, Apr. 2019, Art. no. 3.
N. Rios, M. G. de Mendonça Neto, and R. O. Spinola, "A tertiary study on technical debt: Types, management strategies, research trends, and base information for practitioners," Information and Software Technology, vol. 102, pp. 117–145, Oct. 2018.
J. W. M. Lai and M. Bower, "Evaluation of technology use in education: Findings from a critical analysis of systematic literature reviews," Journal of Computer Assisted Learning, vol. 36, no. 3, pp. 241–259, 2020.
M. Bano, D. Zowghi, and N. Ikram, "Systematic reviews in requirements engineering: A tertiary study," in 4th International Workshop on Empirical Requirements Engineering, Karlskrona, Sweden, Aug. 2014, pp. 9–16.
H. J. Junior and G. H. Travassos, "Consolidating a common perspective on Technical Debt and its Management through a Tertiary Study," Information and Software Technology, vol. 149, Sep. 2022, Art. no. 106964.
Z. Kotti, R. Galanopoulou, and D. Spinellis, "Machine Learning for Software Engineering: A Tertiary Study," ACM Comput. Surv., vol. 55, no. 12, Nov. 2023, Art. no. 256.
D. Budgen, P. Brereton, N. Williams, and S. Drummond, "What Support do Systematic Reviews Provide for Evidence-informed Teaching about Software Engineering Practice?," e-Informatica Software Engineering Journal, vol. 14, no. 1, pp. 7–60, 2020.
H. Cadavid, V. Andrikopoulos, and P. Avgeriou, "Architecting systems of systems: A tertiary study," Information and Software Technology, vol. 118, Feb. 2020, Art. no. 106202.
G. A. Garcia-Mireles and M. E. Morales-Trujillo, "Gamification in Software Engineering: A Tertiary Study," in International Conference on Software Process Improvement, Guanajuato, Mexico, Oct. 2019, pp. 116–128.
M. U. Khan, S. Sherin, M. Z. Iqbal, and R. Zahid, "Landscaping systematic mapping studies in software engineering: A tertiary study," Journal of Systems and Software, vol. 149, pp. 396–436, Mar. 2019.
M. Goulao, V. Amaral, and M. Mernik, "Quality in model-driven engineering: a tertiary study," Software Quality Journal, vol. 24, no. 3, pp. 601–633, Sep. 2016.
E. Dutra, B. Diirr, and G. Santos, "Human Factors and their Influence on Software Development Teams - A Tertiary Study," in XXXV Brazilian Symposium on Software Engineering, Joinville, Brazil, Oct. 2021, pp. 442–451.
T. N. Kudo, R. F. Bulcao-Neto, and A. M. R. Vincenzi, "Requirement patterns: a tertiary study and a research agenda," IET Software, vol. 14, no. 1, pp. 18–26, 2020.
H. K. V. Tran, M. Unterkalmsteiner, J. Borstler, and N. bin Ali, "Assessing test artifact quality—A tertiary study," Information and Software Technology, vol. 139, Nov. 2021, Art. no. 106620.
S. Das and K. Gary, "Agile Transformation at Scale: A Tertiary Study," in International Conference on Agile Software Development, Jun. 2021, pp. 3–11.
D. Budgen, S. Drummond, P. Brereton, and N. Holland, "What scope is there for adopting evidence-informed teaching in SE?," in 34th International Conference on Software Engineering, Zurich, Switzerland, Jun. 2012, pp. 1205–1214.
V. Garousi and M. V. Mantyla, "A systematic literature review of literature reviews in software testing," Information and Software Technology, vol. 80, pp. 195–216, Dec. 2016.
B. J. Shea et al., "AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews," Journal of Clinical Epidemiology, vol. 62, no. 10, pp. 1013–1020, Oct. 2009.
B. Kitchenham and S. Charters, "Guidelines for performing Systematic Literature Reviews in Software Engineering," University of Durham, Durham, UK, Technical Report EBSE-2007-01, Jan. 2007.
G. Duck, G. Nenadic, A. Brass, D. L. Robertson, and R. Stevens, "Extracting patterns of database and software usage from the bioinformatics literature," Bioinformatics, vol. 30, no. 17, pp. i601–i608, Sep. 2014.
A. Kovacevic, Z. Konjovic, B. Milosavljevic, and G. Nenadic, "Mining methodologies from NLP publications: A case study in automatic terminology recognition," Computer Speech & Language, vol. 26, no. 2, pp. 105–126, Apr. 2012.
D. S. Cruzes and T. Dyba, "Synthesizing evidence in software engineering research," in ACM-IEEE International Symposium on Empirical Software Engineering and Measurement, Bolzano-Bozen, Italy, Sep. 2010, pp. 1–10.
F. Lau, C. Kuziemsky, M. Price, and J. Gardner, "A review on systematic reviews of health information system studies," Journal of the American Medical Informatics Association, vol. 17, no. 6, pp. 637–645, Nov. 2010.
D. S. Cruzes and T. Dyba, "Research synthesis in software engineering: A tertiary study," Information and Software Technology, vol. 53, no. 5, pp. 440–455, May 2011.
A. C. C. dos Santos, I. H. de F. Junior, H. P. de Moura, and S. Marczak, "A Systematic Tertiary Study of Communication in Distributed Software Development Projects," in Seventh International Conference on Global Software Engineering, Porto Alegre, Brazil, Aug. 2012, pp. 182–182.
I. Nurdiani, J. Börstler, and S. A. Fricker, "The impacts of agile and lean practices on project constraints: A tertiary study," Journal of Systems and Software, vol. 119, pp. 162–183, Sep. 2016.
Z. I. Batouta, R. Dehbi, M. Talea, and O. Hajoui, "Automation in code generation: Tertiary and systematic mapping review," in 4th IEEE International Colloquium on Information Science and Technology, Tangier, Morocco, Oct. 2016, pp. 200–205.
E. Oliveira, T. Conte, M. Cristo, and N. Valentim, "Influence Factors in Software Productivity — A Tertiary Literature Review," International Journal of Software Engineering and Knowledge Engineering, vol. 28, no. 11n12, pp. 1795–1810, Nov. 2018.
A. Ampatzoglou, S. Bibi, P. Avgeriou, M. Verbeek, and A. Chatzigeorgiou, "Identifying, categorizing and mitigating threats to validity in software engineering secondary studies," Information and Software Technology, vol. 106, pp. 201–230, Feb. 2019.
L. Bedu, O. Tinh, and F. Petrillo, "A Tertiary Systematic Literature Review on Software Visualization," in Working Conference on Software Visualization, Cleveland, OH, USA, Oct. 2019, pp. 33–44.
V. Delavari, E. Shaban, M. Janssen, and A. Hassanzadeh, "Thematic mapping of cloud computing based on a systematic review: a tertiary study," Journal of Enterprise Information Management, vol. 33, no. 1, pp. 161–190, Sep. 2019.
G. T. G. Neto, W. B. Santos, P. T. Endo, and R. A. A. Fagundes, "Multivocal literature reviews in software engineering: Preliminary findings from a tertiary study," in ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, Porto de Galinhas, Brazil, Sep. 2019, pp. 1–6.
D. Akdur and O. Demirors, "Systematic Reviews in Model-Driven Engineering: a Tertiary Study," Journal of Aeronautics and Space Technologies, vol. 13, no. 1, pp. 57–68, 2020.
J. L. Barros-Justo, F. B. V. Benitti, and J. S. Molleri, "Risks and risk mitigation in global software development: An update," Journal of Software: Evolution and Process, vol. 33, no. 11, 2021, Art. no. e2370.
X. Huang, H. Zhang, X. Zhou, D. Shao, and L. Jaccheri, "A Research Landscape of Software Engineering Education," in 28th Asia-Pacific Software Engineering Conference, Taipei, Taiwan, Dec. 2021, pp. 181–191.
E. M. Arvanitou, A. Ampatzoglou, S. Bibi, A. Chatzigeorgiou, and I. Deligiannis, "Applying and Researching DevOps: A Tertiary Study," IEEE Access, vol. 10, pp. 61585–61600, Jan. 2022.
H. Y. Fuchigami, M. R. Severino, L. Yamanaka, and M. R. de Oliveira, "A Literature Review of Mathematical Programming Applications in the Fresh Agri-Food Supply Chain," in International Joint conference on Industrial Engineering and Operations Management, Novi Sad, Serbia, Jul. 2019, pp. 37–50.
F. Kamei, G. Pinto, I. Wiese, M. Ribeiro, and S. Soares, "What Evidence We Would Miss If We Do Not Use Grey Literature?," in 15th ACM / IEEE International Symposium on Empirical Software Engineering and Measurement, Bari, Italy, Oct. 2021, pp. 1–11.
N. Salleh and A. Nordin, "Trends and perceptions of evidence-based software engineering research in Malaysia," in The 5th International Conference on Information and Communication Technology for The Muslim World, Kuching, Malaysia, Nov. 2014, pp. 1–6.
M. Neumann, "The Integrated List of Agile Practices - A Tertiary Study," in International Conference on Lean and Agile Software Development, Jan. 2022, pp. 19–37.
J. Barata, "The fourth industrial revolution of supply chains: A tertiary study," Journal of Engineering and Technology Management, vol. 60, Apr. 2021, Art. no. 101624.
D. Budgen, P. Brereton, N. Williams, and S. Drummond, "The contribution that empirical studies performed in industry make to the findings of systematic reviews: A tertiary study," Information and Software Technology, vol. 94, pp. 234–244, Feb. 2018.
C. Fu, H. Zhang, X. Huang, X. Zhou, and Z. Li, "A Review of Meta-ethnographies in Software Engineering," in 23rd International Conference on Evaluation and Assessment in Software Engineering, Copenhagen, Denmark, Apr. 2019, pp. 68–77.
D. Srivastava and V. Kumar Soni, "A Systematic Review On Sentiment Analysis Approaches," in 2nd International Conference on Advance Computing and Innovative Technologies in Engineering, Greater Noida, India, Apr. 2022, pp. 1–6.
H. Tang, Y. Zhou, X. Huang, and G. Rong, "Does pareto’s law apply to evidence distribution in software engineering? an initial report," in 3rd International Workshop on Evidential Assessment of Software Technologies, Nanjing, China, Dec. 2014, pp. 9–16.
F. Kamei et al., "Grey Literature in Software Engineering: A critical review," Information and Software Technology, vol. 138, Oct. 2021, Art. no. 106609.
P. Singh, M. Galster, and K. Singh, "How do Secondary Studies in Software Engineering report Automated Searches? A Preliminary Analysis," in 22nd International Conference on Evaluation and Assessment in Software Engineering, Christchurch, New Zealand, Jun. 2018, pp. 145–150.
D. L. Cortes-Murcia, W. J. Guerrero, and J. R. Montoya-Torres, "Supply Chain Management, Game-Changing Technologies, and Physical Internet: A Systematic Meta-Review of Literature," IEEE Access, vol. 10, pp. 61721–61743, Jan. 2022.
G. K. Hanssen, D. Smite, and N. B. Moe, "Signs of Agile Trends in Global Software Engineering Research: A Tertiary Study," in Sixth International Conference on Global Software Engineering Workshop, Helsinki, Finland, Aug. 2011, pp. 17–23.
J. C. Epifanio and L. F. Da Silva, "Scrutinizing Reviews on Computer Science Technologies for Autism: Issues and Challenges," IEEE Access, vol. 8, pp. 32802–32815, Jan. 2020.
E. Bayram, B. Dogan, and V. Tunali, "Bibliometric Analysis of the Tertiary Study on Agile Software Development using Social Network Analysis," in Innovations in Intelligent Systems and Applications Conference, Istanbul, Turkey, Oct. 2020, pp. 1–4.
S. Jiang, K. Jakobsen, L. Jaccheri, and J. Li, "Blockchain and Sustainability: A Tertiary Study," in IEEE/ACM International Workshop on Body of Knowledge for Software Sustainability, Madrid, Spain, Jun. 2021, pp. 7–8.
H. Abedinnia, C. H. Glock, E. H. Grosse, and M. Schneider, "Machine scheduling problems in production: A tertiary study," Computers & Industrial Engineering, vol. 111, pp. 403–416, Sep. 2017.
M. A. M. S. Lemstra and M. A. de Mesquita, "Industry 4.0: a tertiary literature review," Technological Forecasting and Social Change, vol. 186, Jan. 2023, Art. no. 122204.
B. Cartaxo, "Integrating evidence from systematic reviews with software engineering practice through evidence briefings," in 20th International Conference on Evaluation and Assessment in Software Engineering, Limerick, Ireland, Jun. 2016, pp. 1–4.
M. E. Esandi, I. Gutierrez-Ibarluzea, N. Ibargoyen-Roteta, and B. Godman, "An evidence-based framework for identifying technologies of no or low-added value (NLVT)," International Journal of Technology Assessment in Health Care, vol. 36, no. 1, pp. 50–57, Jan. 2020.
T. Taipalus, V. Isomottonen, H. Erkkila, and S. Ayramo, "Data Analytics in Healthcare: A Tertiary Study," SN Computer Science, vol. 4, no. 1, Dec. 2022, Art. no. 87.
K. Archer, R. Savage, S. Sanghera-Sidhu, E. Wood, A. Gottardo, and V. Chen, "Examining the effectiveness of technology use in classrooms: A tertiary meta-analysis," Computers & Education, vol. 78, pp. 140–149, Sep. 2014.
K. S. S. Santos, L. B. L. Pinheiro, and R. S. P. Maciel, "Interoperability Types Classifications: A Tertiary Study," in XVII Brazilian Symposium on Information Systems, Uberlandia, Brazil, Jun. 2021, pp. 1–8.
T. Cerny, A. Al Maruf, A. Janes, and D. Taibi, "Microservice Anti-Patterns and Bad Smells. How to Classify, and How to Detect Them. A Tertiary Study." Social Science Research Network, Rochester, NY, USA, Jan. 18, 2023.
X. Liu et al., "Research on Microservice Architecture: A Tertiary Study." Social Science Research Network, Rochester, NY, Aug. 30, 2022.
C. Alves, J. Cunha, and J. Araujo, "On the Pragmatics of Requirements Engineering Practices in a Startup Ecosystem," in 28th International Requirements Engineering Conference, Zurich, Switzerland, Sep. 2020, pp. 311–321.
A. Yasin, R. Fatima, L. Wen, W. Afzal, M. Azhar, and R. Torkar, "On Using Grey Literature and Google Scholar in Systematic Literature Reviews in Software Engineering," IEEE Access, vol. 8, pp. 36226–36243, Jan. 2020.
B. Napoleao, K. Felizardo, E. Souza, and N. Vijaykumar, "Practical similarities and differences between Systematic Literature Reviews and Systematic Mappings: a tertiary study," in 29th International Conference on Software Engineering and Knowledge Engineering, Pittsburgh, PA, USA, Jul. 2017, pp. 85–90.
K. Champion, S. Khatri, and B. M. Hill, "Qualities of Quality: A Tertiary Review of Software Quality Measurement Research." arXiv, Jul. 29, 2021.
Y. Zhou, H. Zhang, X. Huang, S. Yang, M. A. Babar, and H. Tang, "Quality assessment of systematic reviews in software engineering: a tertiary study," in 19th International Conference on Evaluation and Assessment in Software Engineering, Nanjing, China, Apr. 2015, pp. 1–14.
S. Jiang, K. Jakobsen, J. Bueie, J. Li, and P. H. Haro, "A Tertiary Review on Blockchain and Sustainability With Focus on Sustainable Development Goals," IEEE Access, vol. 10, pp. 114975–115006, Jan. 2022.
G. Lacerda, F. Petrillo, M. Pimenta, and Y. G. Gueheneuc, "Code smells and refactoring: A tertiary systematic review of challenges and observations," Journal of Systems and Software, vol. 167, Sep. 2020, Art. no. 110610.
K. Feichtinger, K. Meixner, R. Rabiser, and S. Biffl, "A Systematic Study as Foundation for a Variability Modeling Body of Knowledge," in 47th Euromicro Conference on Software Engineering and Advanced Applications, Palermo, Italy, Sep. 2021, pp. 25–28.
M. Alghobiri, "A Comparative Analysis of Classification Algorithms on Diverse Datasets," Engineering, Technology & Applied Science Research, vol. 8, no. 2, pp. 2790–2795, Apr. 2018.
Downloads
How to Cite
License
Copyright (c) 2025 Ghader Reda Kurdi, Budoor Ahmad Allehyani
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain the copyright and grant the journal the right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) after its publication in ETASR with an acknowledgement of its initial publication in this journal.
