Application of deep learning and machine learning workflows for field-scale phenotyping

Mirnezami, Seyed Vahid
Major Professor
BASKAR Ganapathysubramanian
Committee Member
Journal Title
Journal ISSN
Volume Title
Research Projects
Organizational Units
Mechanical Engineering
Organizational Unit
Journal Issue
Mechanical Engineering

Tassel is the male inflorescences organ of the maize plant that develops atop the plant. Coarse features of tassels, including shape and size, can influence shedding pollen, fertilization, and subsequently grain yield. Therefore, understanding tassel dynamics and characteristics as well as how it evolves during the plant growth can help the plant scientist community to increase the grain yield as a final goal. To do so, first, tassels were investigated in one time points. The tassels were cut in the field and their images were captured in a lightbox. Coarse features were measured using novel image processing approaches. 351 tassels with different genotypes were used for the experiment. Tassel length, first lowest branch length, and angle as well as central spike length were measured by applying image processing and machine learning techniques. Tassels were also classified to open and close structures to obtain accurate predictions for the traits. The results show that R2 values for the tassel length and central spike length were 0.92 and 0.80, respectively. In addition, the R2 values for the first lowest branch length and angle were 0.63 and 0.91, respectively. The R2 values for the first lowest branch length was low compared to others because locating the first lowest branch point and its corresponding branch tip was hard due to branches occlusion. This study was done to create a robust algorithm for tassel phenotyping. Challenges were figured out for better tassel phenotyping in the field. Then, we looked at a diverse panel in the field, using stationary cameras to capture 6 tassels every 10 minutes for 8 hours per day during a month. Traditional approaches for phenotyping anthesis progression are time-consuming, subjective, and labor-intensive and are thus impractical for phenotyping large populations in multiple environments. In this work, we utilize a high throughput phenotyping approach that is based on extracting time-lapse information of anthesis progress from digital cameras. The major challenge is identifying the region of the interest (i.e. location of tassels in the imaging window) in the acquired images. Camera drift, different types of weather, including fog, rain, clouds, and sun and additionally, occlusion of tassels by other tassels or leaves complicated this problem. We discussed the associated challenges for object detection and localization under noisy conditions. In addition, a framework was developed to utilize Amazon Mechanical Turk to allow turkers to annotate the images and evaluate them to create an object detection dataset. Finally, we illustrated a promising deep-learning approach to tassel recognition and localization that is based on Faster-RCNN which has shown the strong capability for detection and localization. This method was improved using a boosting method to improve the dataset. This approach is able to reliably identify a diverse set of tassel morphologies with the mAP of 0.81. Tassel flowering pattern is the most important and complex trait. Tassel maize as a male structure is responsible to produce pollen for the silk as a female organ on the same plant. The amount of pollen and shedding time is important for the breeders as well as the biologists. This study introduced an automated end-to-end pipeline by coupling various deep learning, machine learning and image processing approaches. Inbred lines from both SAM and NAM panels were grown at Curtiss farm at Iowa State University, Ames, IA. A stationary camera was installed for every two plants. Tassels architecture, weather type, tassels and camera movements are the most important challenges of the research. To address these issues, deep learning algorithms were utilized. Tassel detection, classification, and segmentation. In addition, advanced image processing approaches were used to crop the tassel main spike and track the during tassel evolution. The results showed that deep learning is a powerful tool to detect, classify and segment the tassels. The mAP for the tassel detection was 0.91. The F1-score obtained for the tassel classification was 0.93. In addition, the accuracy of semantic segmentation for creating a binary image from the RGB tassel images was 0.95. The width of the flowering was obtained using graph theory in image processing and the time and location of the flowering can be obtained from the width data over the main spike branch. In addition to tassel structures, crop growth simulation models can help farmers and breeders predict crop performance, and in maize, Leaf Appearance Rate (LAR) is an important parameter used in crop performance simulation models such as APSIM. Since breeders and biologists would like to minimize human involvement in monitoring LAR, this trait can be monitored by applying a high-throughput phenotyping system. Engineers have entered the picture in collaboration with plant scientists to establish different and robust phenotyping methods, and in this study, maize leaf appearance rate was investigated using high-throughput phenotyping approaches. We developed an imaging system for automatically capturing a time-series of images of maize plants under field conditions, with 380 RGB cameras were used to capture images from 380 rows. There were 6 plants with the same genotype in each row that had different genotypes differed row-by-row, and the images were taken for 9 hours daily at 20- minute intervals for more than one month during a growing season. An end-to-end deep learning method was then used to count the numbers of leaves in the images. The dataset for the deep learning algorithm, obtained using the Amazon Mechanical Turk platform, was created by one expert turker along with a well-trained turker. Results demonstrated that an end-to-end model with training based on the expert turker dataset performed very well, handling variation in images that included leaf occlusions and weather type. The R2 between the ground truth obtained by the expert turker and predicted values was approximately 0.73, 0.74, and 0.95 for three testing cameras. The model's prediction performance demonstrated that the number of leaves increase with different slopes for different genotypes. The data can be used for further genotypic analysis.