【作者】 王德鑫；

【导师】 张茂军；

【作者基本信息】 国防科学技术大学 ， 控制科学与工程， 2010， 博士

【摘要】 理想的人机交互模式是利用人类自然形成的与自然界沟通的认知习惯和形式进行的自由交互。多重触控技术就是一种以用户为中心的自然人机交互方式,通过构建一种能同时检测到多个触点的多重触控平台,允许用户双手自由交互甚至多人协同交互,并以用户本能的交互手势进行交互,降低了用户的认知负担,充分释放了用户交互潜力,成为人机交互技术的研究热点之一。它在军事\民用指挥决策、商业零售、公共信息查询、信息家电、大众传媒以及教育等许多领域有着广泛的应用前景。多重触控技术的研究包括多重触控平台的构建及其多触点检测、定位和跟踪方法,以及多重触控手势描述与识别方法。目前多重触控平台构建方法是其中研究最深入的部分,也涌现出了大量的成果并得到成功应用,促使了多重触控技术的兴起,但是现有的多重触控平台大多具有成本高昂,尤其是构建大范围交互平台成本高昂,另外对交互环境和用户的交互行为有特殊要求,而且安装不方便,需要对现有显示平台进行改装和占用较大的空间,不具有移植性,此外大部分多重触控平台的多触点检测、定位和跟踪方法无法解决多触点之间的遮挡问题,定位精度较低,这些因素一定程度上限制了多重触控技术的发展。多重触控手势是多重触控平台与用户之间的桥梁,目前的多重触控应用局限于设定的几种手势,不能根据应用约束以及用户的交互习惯定制用户手势,需要用户去学习,记忆和适应指定的手势,限制了用户的自由和潜力,脱离了以人为中心的自然交互理念,且手势识别方法大都需要复杂的训练,手势定义的变动容易颠覆现有的识别结果,而且对用户操作的差异不够灵活,成为限制多重触控技术广泛应用的主要瓶颈。本文分别对其展开研究,具体内容如下:多重触控平台构建方法。提出一种四路摄像头协同的多重触控平台构建方法,该方法只需要简单的红外光源和四路广角红外摄像头,适用于构建各种尺寸的多重触控平台,且不需要对现有显示器进行改装,能直接安装于显示器前方,具有构建简单,安装方便,便于移植,成本低廉等特点。多触点检测定位跟踪方法。首先基于灰度梯度的单幅图像目标检测实现多触点的亚像素级检测,然后利用查找表法/消失点法/立体标定法构建触点在每个摄像头中的方向线,并利用四路摄像头空间约束关系去除触点间遮挡的影响,实现多个触点的精确定位,最后采用改进最近邻数据关联法,利用触点目标的位置、速度和方向特征对落在卡尔曼滤波预测范围内的目标构建加权二部图的权,并用KM算法求解加权二部图的最优匹配来实现多触点目标的稳定跟踪。多重触控手势描述。提出一种应用导向的多重触控语义交互手势描述方法,且基于OWL的多重触控手势本体描述结果能利用推理工具进行检验,发现并指导解决冲突,从而使得用户可以根据应用和自己的喜好自定义手势,能够方便地增加,删除,修改手势定义,或从现有手势继承派生出新的手势,且该方法适用于任意多重触控平台,具有较好的扩展性、重用性和语义性。多重触控手势识别。提出一种基于轨迹形状特征及约束关系的多重触控手势识别方法,使得手势模板建立简单,且用户修改手势定义或从现有手势继承派生出的新手势能直接存储为模板,不需要进行复杂的重新训练,而且能适应不同用户操作速度,幅度上的差异,具有较好的鲁棒性。此外,该方法适用于任意多重触控平台的任意条轨迹组成的复杂手势。多重触控系统实现。设计开发了四路摄像头协同多重触控系统,构建了2.4 m*1.35 m尺寸的四路摄像头协同多重触控平台,开发了应用导向的多重触控手势描述工具,并针对3维对象操控任务设计调查问卷采集面向三维物体操控的多重触控手势,并组织大量用户进行测试,对本文方法进行了验证,达到了较好的效果。更多还原

【Abstract】 The ideal human-machine interaction pattern is to use the cognitive habits and styles, which are naturally formed during people’s communications with the nature. Multi-touch technology is a user-centered natural computer interaction method, and refers to a set of interaction techniques, including a touch screen or touchpad, as well as software that recognizes multiple simultaneous contact points. It allows users to interact with a system through the concurrent use of several fingers and permits multiple users to work together through collaborative hand gestures, which reduces the cognitive burdens on users and fully releases potentials of users’interactions, thereby it has become one of the hot spots of human-computer interaction research. It has widespread application prospects in many areas, such as the military or civilian command and decision-making, commercial retail, public information inquiry, information appliances, the mass media as well as the education and so on.Multi-touch system research includes the multi-touch platform construction and its detection, localization and tracking methods of multi contacts, as well as multi touch gesture description method and recognition method. Currently multi-touch platform construction method is one of the most extensive sections, and has emerged in a large number of platforms with successful applications, which contributes to the emergence of multi touch technology. However, most of the existing multi touch platforms are costly, particularly in the construction of large-size platforms, in addition to special requirements on the interactive environment or user interactions, and inconvenient installations, which require modification of the existing display platforms and occupies large spaces, therefore they owns little transplantation. Furthermore most of the detecting, locating and tracking methods are unable to resolve occlusions between contacts, and with low locating accuracies. These issues limit the popularization of multi-touch to some extend.Multi-touch gesture is the bridge between the platform and the user. Current multi-touch applications are limited to a set of predefined gestures, which could not be customized according to application constraints and users’interaction habits, consequently require users to learn, memorize and adapt to specified gestures and limit user’s freedoms and potentials, which are separated from user-center natural interaction concept. Furthermore, the multi-touch gesture recognition methods mostly require sophisticated trainings, and any definition changes may easily subvert recognition results of existing gestures, moreover, it is lack of flexibility for users’differences. These problems make it the main bottleneck to promote the widespread of multi-touch application.This dissertation addresses these issues as follows:1) Four collaborative cameras based multi-touch platform construction method.It requires only simple infrared light sources and four wide-angle infrared cameras, and is suitable for construction of various sizes of multi touch platforms, and requires no alterations to existing monitors, on the contrary it can be installed directly in front of existing monitor. It has such characteristics as easily setup, convenient installation, and transplantation, low cost, etc.2) Multiple contacts detection, location and tracking methods.Detecting multiple contacts in sub-pixel based on gray gradient of a projected image for tracking points, and then use the lookup table method or Vanishing Point method or stereo calibration method to build direction lines in each camera, and solve the occlusion by position and space constraints, which can achieve precise location. Finally use an improved nearest neighbor data association method to achieve the stable tracking of contacts. The improvement includes building the weighted bipartite graphs for contacts falling in the estimation of Kalman filter based on location, speed and direction features, and use the KM algorithm to find the maximum matching of the weighted bipartite graphs to build the data association of multiple contacts.3) A semantic and application-oriented interactive multi-touch gestures description method.The description results of multi-touch ontology based on OWL can utilize reasoning tools to identify and guide the resolution of conflicts, which allows the user to customize gestures according to application constraints and their own preferences, and it can easily add, delete, or modify definitions, or derive new gestures from existing results. It is applicable to arbitrary multi-touch platform; therefore it has good extensibility, reusability and semantics.4) Multi-touch gesture recognition method based on shape features and constraints of trajectories.It simplifies the templates building templates, and users’modifications of gesture definitions or new gestures inherited from existing gestures can be directly stored as templates, which requires no complex re-trainings. Moreover, it is robust to different users’operating speed, amplitude differences. In addition, the method is suitable for any complex composition of trajectories for any multi-touch platform.5) Multi-touch system implementation.A four collaborative cameras based multi-touch system is set forth, including a multi-touch platform of 2.4m*1.35m, an application-oriented multi-touch gestures description tool, and a 3-dimensional object manipulation application. A questionnaire for the application is designed and analyzed to build the mostly acceptable gesture sets, and a set of users are invited to testing methods advanced here. Results prove the applicability of these methods.更多还原