【作者】 项荣；

【导师】 应义斌； 蒋焕煜；

【作者基本信息】 浙江大学 ， 农业机械化工程， 2013， 博士

【摘要】 果蔬识别和定位的准确性与实时性将直接影响果蔬采摘机器人的采摘成功率和采摘效率。开放环境中果蔬的识别和定位受非结构环境因素的影响很大,因此一直以来是采摘机器人研究领域的研究重点之一本研究以番茄为研究对象,利用双目立体视觉技术,详细探讨了开放环境中番茄图像分割算法,被枝叶遮挡番茄识别算法,成簇番茄识别算法及番茄3D定位方法,部分解决了当前番茄采摘机器人视觉系统研究中存在的问题,所开发的番茄识别和定位算法将为番茄采摘机器人视觉系统的改善奠定一定理论和方法基础。主要研究内容及研究结论如下：(1)比较了基于色差、归一化色差、颜色分量比这三类颜色特征的番茄图像分割算法的分割性能。基于归一化色差、颜色分量比的番茄图像分割算法可实现不同光照强度下的番茄图像分割；光照较弱时.归一化色差法的分割效果较好；顺光条件下,光照较强时,双颜色分量比法的分割效果较好。(2)提出了融合番茄光斑识别及分段阈值的番茄图像分割算法。该算法基于R分量与光照度的相关性,融合了基于归一化色差及颜色分量比这两类番茄图像分割算法的优点,进一步结合番茄光斑区域识别方法实现了不同光照强度下的番茄图像分割。对184幅图像共计750个含光斑的番茄区域的试验表明：基于该方法的图像分割正确率为93.6%,明显优于基于归一化色差法的36.3%；平均执行时间为71ms；该算法对对顺光条件下的番茄图像具有较好的整体分割性能。(3)对比研究了基于圆回归及圆Hough变换的被遮挡番茄识别方法。首先两类方法均进行边缘曲率分析,去除曲率异常边缘,然后对曲率正常边缘分别通过圆回归及圆Hough变换实现被遮挡番茄的识别。对220幅存在遮挡的番茄图像的试验结果表明：轻微遮挡时,两类方法的识别正确率分别为90.8%,89.1%；中度遮挡时,两类方法的识别正确率分别为50.4%,74.8%；严重遮挡时,两类方法的识别性能均较差；可见存在遮挡时,圆Hough变换法的整体识别性能优于圆回归法；两类方法的平均执行时间均约为100ms。(4)对比研究了基于数学形态学、圆回归、圆Hough变换及双目立体视觉的四类成簇番茄识别方法。数学形态学法通过条件腐蚀和循环膨胀实现成簇番茄的识别；圆回归与圆Hough变换法与被遮挡番茄识别所用方法相似；双目立体视觉法根据成簇区域内前后番茄区域深度差,基于迭代Otsu法将成簇番茄分为两类：重叠番茄和粘连番茄,然后使用圆回归法实现粘连番茄的识别,对经深度图像边缘分割后的彩色图像边缘应用圆回归法实现重叠番茄识别。经138对存在轻微遮挡的成簇番茄立体图像的试验结果显示：存在轻微遮挡时,四类方法的识别正确率分别为60.9%,69.0%,71.1%,82.5%；存在较严重遮挡时,四类方法的识别性能均不理想；四类方法的平均执行时间分别为3、85、138、500ms。试验结果表明：双目立体视觉法对粘连及重叠番茄的识别性能均较好,其余三类方法对粘连番茄的识别性能较好,而对重叠番茄的识别性能均较差；双目立体视觉法的整体识别性能要优于其余三类方法,特别是对重叠番茄。(5)提出了番茄3D测量误差校正方法。研究了基于形心特征匹配,区域匹配及组合匹配的三类立体匹配方法,分析了基于三类立体匹配的3D定位误差,分析了造成深度测量误差的要因,提出了相应的校正方法,讨论了遮挡对3D定位精度的影响,测试分析了三类立体匹配方法的实时性。组合匹配将粗匹配阶段通过形心匹配获得的番茄粗略视差作为精匹配阶段区域匹配的视差范围中心,从而获得了一个中心随图像采集距离变化而变化的动态视差范围,在确保获得与区域匹配相近匹配精度的同时,减少了立体匹配计算量,图像采集距离在300～１000mm范围内时,所需匹配时间为区域匹配的三分之一。1349对立体图像的试验表明：1)基于三类立体匹配得到的x坐标测量误差相对较小,分别为：[0,12.8],[1.3,13.8],[1.3,13.8]mm。y坐标及深度测量误差相对较大,且与图像采集距离间分别呈现近似线性递减和递增的关系。2)经二元分段线性回归y坐标预测模型校正后,基于三类立体匹配的y坐标测量误差范围分别为：[-7,-0.8],[-6.1,1.4],[-6.1,1.8]mm。3)番茄大小是影响深度测量误差的要因。4)经基于深度测量值及番茄大小的二元线性回归深度预测模型校正后,基于三类立体匹配的深度测量误差范围分别为：[-10.4,28.7],[-6,12.8],[-5.6,5.3]mm。5)x、y坐标测量结果比深度测量值更易受遮挡影响。6)三类匹配方法的执行时间分别为：[0.4,4],[24,288.8],[9.6,98.8]ms。(6)编制了基于C语言的番茄识别和定位软件。完成了番茄识别和定位算法应用流程,动态链接库及测试软件的设计,应用于番茄采摘机器人的视觉系统后,进行了实验室及温室的采摘试验。23次实验室采摘试验,识别成功18次,采摘成功11次；10次温室采摘试验,识别成功9次,采摘成功8次；采摘效率约为30s/个。上述研究成果将为进一步提高番茄采摘机器人视觉系统对开放环境的适应能力奠定一定的方法和技术基础。更多还原

【Abstract】 The accuracy and real time performance of recognition and localization methods for fruits and vegetables is the key to the success rate and efficiency of fruits and vegetables harvesting robots. Recognition and localization of fruits and vegetables is much influenced by the environment factors under open environments, so it is always a difficult task in the research field of harvesting robots.Tomatoes are used as samples in this study. Image segmentation methods for tomatoes, recognition methods for occluded tomatoes, recognition methods for clustered tomatoes and localization methods for tomatoes were detailed studied by using binocular stereo vision technology. Some problems in the recognition and localization of tomatoes were solved in this study. The developed recognition and localization algorithms offered help for improving the vision system of tomato harvesting robots. Main contents and results were listed as follows:(1) Segmentation performance was compared among three image segmentation methods for tomatoes based on color difference, normalized color difference and color component ratio. The method based on normalized color difference and the method based on color component ratio could realize the image segmentation for tomato images captured under different lighting conditions; the method based on normalized color difference was better under darker lighting conditions; the method based on double ratios of color components was better under front lighting conditions.(2) A piecewise thresholding segmentation method for tomatoes fusing a recognition method for light spots was presented. Based on the dependency between R component values of pixels in the region of a tomato and the illuminance on the surface of the tomato, this method used both advantages of two methods:the method based on normalized color difference and the method based on color component ratio. Furthermore, the recognition method for light spots was also used. Employing this method, tomato image segmentation could be realized under different lighting conditions. Test results from184images in which there were750tomatoes with light spots on their surface showed that the success rate of image segmentation was93.6%. which was much better than the one of the method based on normalized color difference, which was onlv36.3%. The average running time was71ms. This method had good segmentation performance to tomato images which were captured under front lighting conditions.(3) Two recognition methods for occluded tomatoes were studied:the method based on circle regression and the method based on circle Hough transformation. Firstly, the two methods were both based on curvature analysis, and then edge points with abnormal curvature values were discarded, after that, occluded tomatoes were recognized through the circle regression method and the circle Hough transformation method for the remaining edges, respectively. Test results from220images in which there were tomaoes occluded by leaves or branches showed that for tomatoes which were occluded by branches or leaves slightly, the recognition success rates of these two methods were90.8%and89.1%, respectively. For tomatoes occluded moderately, the recognition success rates of these two methods were90.8%and89.1%, respectively. For tomatoes occluded seriously, two methods both had bad recognition performance. Overall, for occluded tomatoes, the performance of the method based on circle Hough transformation was much better than that of the other. The running times for two methods were both about100ms.(4) Four recognition methods for clustered tomatoes were presented:the method based on mathematic morphology, the method based on circle regression, the method based on Hough transformation and the method based on binocular stereo vision. The method based on mathematic morphology was realized through the operation of conditional corrosion and circular expansion. The method based on circle regression and the method based on Hough transformation was similar to those used in the recognition of occluded tomatoes. In the method based on binocular stereo vision, clustered tomatoes were classified into two types:overlapping tomatoes and adhering tomatoes, based on the depth difference between the front tomato and the back one in a same clustered region employing an iterative Otsu method. Then, adhering tomatoes were recognized using the circle regression method. On the other hand, overlapping tomatoes were recognized using the circle regression method for the edges of the clustered region in color image which was segmented into several segments by the edges in depth map. Furthermore, the number of edge segments was same to the number of tomatoes in this clustered region. Test results from138pairs of stereo images of tomatoes occluded slightly showed that the recognition success rates of these four methods were60.9%,69.0%,71.1%,82.5%, respectively. However, for tomatoes occluded seriously, the recognition performance of these four methods were all not good enough. The running times of these four methods were3,85.138, and500ms. respectively. Test results showed that the recognition performance of the method based on binocular stereo vision was good to both two types of clustered tomatoes. Otherwise, the recognition performance of other three methods was good to adhering tomatoes, but was not good to overlapping tomatoes. Overall, the performance of the method based on binocular stereo vision was better than other three methods, especially for overlapping tomatoes.(5) Correction models for3D measurement error were presented. Three stereo matching methods were studied:the centroid-based stereo matching method, the area-based stereo matching method and the combination stereo matching method.3D localization errors were analyzed for the results produced based on these three stereo matching methods, the main factor to depth measurement errors was analyzed, and correction models were also presented. Furthermore, the influence to the accuracy of3D localization for tomatoes caused by occlusion was also discussed. Finally, the real time performance of these three stereo matching methods was tested, too. Combination stereo matching included two stages:rough stereo matching and precise stereo matching. Disparity acquired through centroid-based stereo matching at the rough matching stage was used as the center of the disparity range which was used in the area-based stereo matching at the precise matching stage. Then a dynamic disparity range was acquired of which the center moved with the shoot distance. By this way. not only the accuracy of the stereo matching was promised to be similar to that of the area-based stereo matching method, but also the time costs of stereo matching were also reduced. The time costs reduced to one third of that of area-based stereo matching method when the shoot distances were ranged from300to1000mm. Test results from1349pairs of stereo images showed that:1) for three stereo matching methods,x coordinate measurement errors were all smaller:the range of x coordinate measurement errors based on three stereo matching methods were [0,12.8],[1.3,13.8].[1.3.13.8] mm, respectively; but y coordinate and depth measurement errors were larger; y coordinate errors increased and depth errors decreased as the distances got larger.2) After the correction using binary piece wise linear regression models for y coordinate prediction, the ranges of y coordinate measurement errors based on three stereo matching methods were [-7,-0.8].[-6.1.1.4],[-6.1,1.8] mm. respectively3) Tomato size was the main factor to the depth measurement errors.4) After correction using binary linear regression models for depth prediction, the ranges of depth measurement errors based on three stereo matching methods were [-10.4,28.7],[-6,12.8],[-5.6,5.3] mm, respectively.5) x and y coordinate measurement results were more easily influenced by occlusion than depth measurement results.6) The ranges of running times of three stereo matching methods were [0.4,4],[24,288.8],[9.6,98.8] ms, respectively.(6) Finally, softwares of the recognition and localization methods for tomatoes under open environments were designed based on the C program language. The application flow chart of the recognition and localization methods for tomatoes and its dynamic link library produced from the C program of the recognition and localization methods was designed. Moreover, a VB program interface which was used to test the DLL was also designed. Harvesting experiments were executed both in laboratory and greenhouse after the DLL was employed by the vision system of a tomato harvesting robot. Among23times harvesting attempts in laboratory, the number of successful recognition was18, and the number of successful harvesting was11. Moreover, the numbers of successful recognition and successful harvesting were9and8in10times of harvesting attempts in greenhouse. The harvesting efficiency was about30s per tomato.The above work provided method and technology foundation for improving the adaption performance of the vision system of tomato harvesting robots under open environments.更多还原