Leveraging a Modified Contrastive Language-Image Pre-training Model to Align Images and Text for Generating Remedy Text for Malus Pumila Lamina Images
Received: 17 December 2024 | Revised: 5 February 2025 and 22 February 2025 | Accepted: 24 February 2025 | Online: 11 March 2025
Corresponding author: M. Sudha
Abstract
The increasing threat of leaf diseases to the productivity of precision farming necessitates systematic, logical, and scalable leaf identification methodologies. Conventional plant disease detection approaches are often slow, inefficient, and limited in their applicability, restricting the effective management of leaf diseases. This research work recommends a hybrid multimodal model that uses different modes of activities for leaf disease detection and can integrate image and text data in a single frame to improve the accuracy and proficiency of disease classification. The text data include custom-generated remedy descriptors specifically designed for the proposed model. The latter combines Machine Learning (ML) techniques, such as OTSU thresholding, Gaussian filtering, and modified Contrastive Language-Image Pre-training (mCLIP), to classify diseased leaves and propose suitable remedial actions. The proposed mCLIP model combines image and label data to enhance the effectiveness of multi-class image classification and suitable remedy description generation. Unlike existing multimodal approaches that primarily output text describing image features, the proposed model generates remedy text as the output for specific diseases. This novel approach offers a comprehensive solution for leaf disease detection and renders optimistic results for real-time and automated disease identification in agricultural practices, facilitating early intervention and better crop management. The proposed model obtained an accuracy of 98.1%.
Keywords:
OTSU, Gausian filter image segmentation, multimodal, mCLIPDownloads
References
J. Dong, Y. Yao, A. Fuentes, Y. Jeong, S. Yoon, and D. S. Park, "Visual information guided multi-modal model for plant disease anomaly detection," Smart Agricultural Technology, vol. 9, Dec. 2024, Art. no. 100568.
S. Pandey, A. Somaddar, S. Jain, S. Perti, and U. Pathak, "A Multimodal Deep Learning Approach for Advanced Plant Disease Prediction," in 1st DMIHER International Conference on Artificial Intelligence in Education and Industry 4.0, Wardha, India, Nov. 2023, vol. 1, pp. 1–6.
H. Lee, Y.-S. Park, S. Yang, H. Lee, T.-J. Park, and D. Yeo, "A Deep Learning-Based Crop Disease Diagnosis Method Using Multimodal Mixup Augmentation," Applied Sciences, vol. 14, no. 10, Jan. 2024, Art. no. 4322.
A. Musa, M. Hassan, M. Hamada, and F. Aliyu, "Low-Power Deep Learning Model for Plant Disease Detection for Smart-Hydroponics Using Knowledge Distillation Techniques," Journal of Low Power Electronics and Applications, vol. 12, no. 2, Jun. 2022, Art. no. 24.
Md. S. H. Shovon, S. J. Mozumder, O. K. Pal, M. F. Mridha, N. Asai, and J. Shin, "PlantDet: A Robust Multi-Model Ensemble Method Based on Deep Learning For Plant Disease Detection," IEEE Access, vol. 11, pp. 34846–34859, Jan. 2023.
J. Li et al., "A Multi-Modal Open Object Detection Model for Tomato Leaf Diseases with Strong Generalization Performance Using PDC-VLD," Plant Phenomics, vol. 6, Aug. 2024, Art. no. 0220.
Y. Lu et al., "Application of Multimodal Transformer Model in Intelligent Agricultural Disease Detection and Question-Answering Systems," Plants, vol. 13, no. 7, Jan. 2024, Art. no. 972.
M. Jung et al., "Construction of deep learning-based disease detection model in plants," Scientific Reports, vol. 13, no. 1, May 2023, Art. no. 7331.
B. Yang et al., "A novel plant type, leaf disease and severity identification framework using CNN and transformer with multi-label method," Scientific Reports, vol. 14, no. 1, May 2024, Art. no. 11664.
B. Khan et al., "Bayesian optimized multimodal deep hybrid learning approach for tomato leaf disease classification," Scientific Reports, vol. 14, no. 1, Sep. 2024, Art. no. 21525.
J. Liu and X. Wang, "A multimodal framework for pepper diseases and pests detection," Scientific Reports, vol. 14, no. 1, Nov. 2024, Art. no. 28973.
S. S. Harakannanavar, J. M. Rudagi, V. I. Puranikmath, A. Siddiqua, and R. Pramodhini, "Plant leaf disease detection using computer vision and machine learning algorithms," Global Transitions Proceedings, vol. 3, no. 1, pp. 305–310, Jun. 2022.
Y. M. Abd Algani, O. J. Marquez Caro, L. M. Robladillo Bravo, C. Kaur, M. S. Al Ansari, and B. Kiran Bala, "Leaf disease identification and classification using optimized deep learning," Measurement: Sensors, vol. 25, Feb. 2023, Art. no. 100643.
A. Abisha and B. Bharathi, "A Hybrid Feature Extraction and Classification using Xception-RF for Multiclass Disease Classification in Plant Leaves," Applied Artificial Intelligence, vol. 37, no. 1, Dec. 2023, Art. no. 2176614.
A. Dash, P. K. Sethy, S. G. K. Patro, and A. O. Salau, "Deep feature extraction based cascading model for the classification of Fusarium stalk rot and charcoal rot disease in maize plant," Informatics in Medicine Unlocked, vol. 42, Jan. 2023, Art. no. 101363.
G. Heller et al., "Multisource neural network feature map fusion: An efficient strategy to detect plant diseases," Intelligent Systems with Applications, vol. 19, Sep. 2023, Art. no. 200264.
B. Mazumder, M. S. I. Khan, and K. M. Mohi Uddin, "Biorthogonal wavelet based entropy feature extraction for identification of maize leaf diseases," Journal of Agriculture and Food Research, vol. 14, Dec. 2023, Art. no. 100756.
M. V. Shewale and R. D. Daruwala, "High performance deep learning architecture for early detection and classification of plant leaf disease," Journal of Agriculture and Food Research, vol. 14, Dec. 2023, Art. no. 100675.
G. Dai, J. Fan, Z. Tian, and C. Wang, "PPLC-Net:Neural network-based plant disease identification model supported by weather data augmentation and multi-level attention mechanism," Journal of King Saud University - Computer and Information Sciences, vol. 35, no. 5, May 2023, Art. no. 101555.
S. Pudumalar and S. Muthuramalingam, "Hydra: An ensemble deep learning recognition model for plant diseases," Journal of Engineering Research, vol. 12, no. 4, pp. 781–792, Dec. 2024.
A. O. Anim-Ayeko, C. Schillaci, and A. Lipani, "Automatic blight disease detection in potato (Solanum tuberosum L.) and tomato (Solanum lycopersicum, L. 1753) plants using deep learning," Smart Agricultural Technology, vol. 4, Aug. 2023, Art. no. 100178.
S. P. Singh, K. Pritamdas, K. J. Devi, and S. D. Devi, "Custom Convolutional Neural Network for Detection and Classification of Rice Plant Diseases," Procedia Computer Science, vol. 218, pp. 2026–2040, Jan. 2023.
B. M. Abuhayi and A. A. Mossa, "Coffee disease classification using Convolutional Neural Network based on feature concatenation," Informatics in Medicine Unlocked, vol. 39, Jan. 2023, Art. no. 101245.
T. Poblete et al., "Detection of symptoms induced by vascular plant pathogens in tree crops using high-resolution satellite data: Modelling and assessment with airborne hyperspectral imagery," Remote Sensing of Environment, vol. 295, Sep. 2023, Art. no. 113698.
R. Mumtaz, M. H. Maqsood, I. ul Haq, U. Shafi, Z. Mahmood, and M. Mumtaz, "Integrated digital image processing techniques and deep learning approaches for wheat stripe rust disease detection and grading," Decision Analytics Journal, vol. 8, Sep. 2023, Art. no. 100305.
H. H. Alshammari and H. Alkhiri, "Optimized recurrent neural network mechanism for olive leaf disease diagnosis based on wavelet transform," Alexandria Engineering Journal, vol. 78, pp. 149–161, Sep. 2023.
L. L. Janneh, Y. Zhang, M. Hydara, and Z. Cui, "Deep learning-based hybrid feature selection for the semantic segmentation of crops and weeds," ICT Express, vol. 10, no. 1, pp. 118–124, Feb. 2024.
D. R. F. R. Anandhi and S. Sathiamoorthy, "Enhanced Sea Horse Optimization with Deep Learning-based Multimodal Fusion Technique for Rice Plant Disease Segmentation and Classification," Engineering, Technology & Applied Science Research, vol. 13, no. 5, pp. 11959–11964, Oct. 2023.
R. Rajamohanan and B. C. Latha, "An Optimized YOLO v5 Model for Tomato Leaf Disease Classification with Field Dataset," Engineering, Technology & Applied Science Research, vol. 13, no. 6, pp. 12033–12038, Dec. 2023.
A. Abbas, U. Maqsood, S. U. Rehman, K. Mahmood, T. AlSaedi, and M. Kundi, "An Artificial Intelligence Framework for Disease Detection in Potato Plants," Engineering, Technology & Applied Science Research, vol. 14, no. 1, pp. 12628–12635, Feb. 2024.
A. Gatla et al., "Optimizing Edge AI for Tomato Leaf Disease Identification," Engineering, Technology & Applied Science Research, vol. 14, no. 4, pp. 16061–16068, Aug. 2024.
S. Vimalkumar and R. Latha, "Maize Leaf Disease Detection using Manta-Ray Foraging Optimization with Deep Learning Model," Engineering, Technology & Applied Science Research, vol. 14, no. 5, pp. 17068–17074, Oct. 2024.
D. Rahadiyan, S. Hartati, Wahyono, and A. P. Nugroho, "Feature aggregation for nutrient deficiency identification in chili based on machine learning," Artificial Intelligence in Agriculture, vol. 8, pp. 77–90, Jun. 2023.
"PlantVillage Dataset." https://www.kaggle.com/datasets/abdallahalidev/plantvillage-dataset.
Downloads
How to Cite
License
Copyright (c) 2025 Dhanasekaran Menaga, M. Sudha
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain the copyright and grant the journal the right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) after its publication in ETASR with an acknowledgement of its initial publication in this journal.
