Automated crop plant detection based on the fusion of color and depth images for robotic weed control

Tang, Lie
Gai, Jingyao
Steward, Brian
Tang, Lie
Steward, Brian
Major Professor
Committee Member
Journal Title
Journal ISSN
Volume Title
Research Projects
Organizational Units
Journal Issue
Agricultural and Biosystems EngineeringHuman Computer InteractionPlant Sciences Institute

Robotic weeding enables weed control near or within crop rows automatically, precisely and effectively. A computer‐vision system was developed for detecting crop plants at different growth stages for robotic weed control. Fusion of color images and depth images was investigated as a means of enhancing the detection accuracy of crop plants under conditions of high weed population. In‐field images of broccoli and lettuce were acquired 3–27 days after transplanting with a Kinect v2 sensor. The image processing pipeline included data preprocessing, vegetation pixel segmentation, plant extraction, feature extraction, feature‐based localization refinement, and crop plant classification. For the detection of broccoli and lettuce, the color‐depth fusion algorithm produced high true‐positive detection rates (91.7% and 90.8%, respectively) and low average false discovery rates (1.1% and 4.0%, respectively). Mean absolute localization errors of the crop plant stems were 26.8 and 7.4 mm for broccoli and lettuce, respectively. The fusion of color and depth was proved beneficial to the segmentation of crop plants from background, which improved the average segmentation success rates from 87.2% (depth‐based) and 76.4% (color‐based) to 96.6% for broccoli, and from 74.2% (depth‐based) and 81.2% (color‐based) to 92.4% for lettuce, respectively. The fusion‐based algorithm had reduced performance in detecting crop plants at early growth stages.


This is the peer-reviewed version of the following article: Gai, Jingyao, Lie Tang, and Brian L. Steward. "Automated crop plant detection based on the fusion of color and depth images for robotic weed control." Journal of Field Robotics (2019), which has been published in final form at DOI: 10.1002/rob.21897. This article may be used for non-commercial purposes in accordance with Wiley Terms and Conditions for Self-Archiving.