【作者】 张旭；

【导师】 杨基海； 陈香；

【作者基本信息】 中国科学技术大学 ， 生物医学工程， 2010， 博士

【摘要】 随着计算设备融入人类生活环境的方方面面,真实世界、数字世界和作为主体的人成为一个有机整体。人与环境的无缝沟通和自由互动的需求促使了人体动作识别成为未来多通道多模态人机交互研究中的热点。人体动作识别是指计算机自动检测、分析和理解人体各类运动和行为,如手指、手腕、手臂、头、面部或身体等姿态或运动模式,以判断人的意图并提供相应服务的过程。人体的任何一个动作都是由多组肌群在神经系统的支配下相互协调、共同完成的。肌电信号(EMG)是伴随肌肉活动产生的一种重要的生物电信号。由表面肌电(SEMG)传感器在相应肌群皮肤表面捕获的肌肉活动信息不但能反映关节的伸屈状态和伸屈强度,还能反映动作完成过程中肢体的形状和位置等信息,是感知人体动作的重要方式。本文以手势动作、肩颈部动作和腿部运动为研究对象,对基于SEMG的多种类人体动作的检测与识别技术进行了深入探索,设计并实现了基于人体动作的实时交互系统,开展了一定规模的用户测试实验,为实现自然和谐的人机交互提供了几种具体有效的解决方案。研究成果将促进多模态智能人机接口技术的进一步发展与推广应用,同时在人类行为理解、康复医学、情境感知、普适计算和导航定位等应用研究中也具有重要价值。本文具体的研究工作和创新点包括：1.基于多通道SEMG的手势动作识别研究。此研究的目标是为基于手势的人机交互系统提供有效的识别算法,并为手势命令集的选择和SEMG传感器的安置提供理论依据。一方面,以8类常用手势动作为研究对象,对包含信号采集、活动段检测、特征提取以及分类识别在内的多种动作SEMG识别方法进行了研究,提出了一种适用于实时交互的优化算法,并在此基础上,构建了基于SEMG的实时手势识别系统。另一方面,结合解剖学知识,对涉及多种精细手势动作在内的20种手势开展了手势命令集及SEMG传感器安放位置的优化研究,并在此基础上提出了一种实用的交互控制应用方案。在实时系统上进行的特定用户、多用户以及与用户无关的实验结果表明,该交互方案具有很好的鲁棒性,此研究成果可为交互应用中手势集的选取及传感器的安置提供参考依据。2.基于加速度与SEMG信息融合的手语手势识别研究。此研究是对基于多传感器信息检测和融合的手势识别技术的初步探索。放置在手前臂的加速计适合区分具有不同运动轨迹的手臂挥划,而SEMG信息更适于表达手指手腕等精细动作引起的具有不同模式的肌肉活动。针对这两种传感器捕获手语手势信息的互补性,本文提出了一种有效融合加速度和SEMG信息的手语手势动作识别方法框架,利用多流HMM和决策树融合两种异质传感器信息,对30种中国手语单手词和16种情景对话例句展开了分类识别研究,并实现了采用18类手势动作对虚拟魔方进行控制的实时手势交互系统。3.基于SEMG的肩颈部动作识别研究。头肩颈部动作在自然和谐的人机交互中可作为辅助的交互手段。利用从背部、肩部和颈部等相关肌群检测到的多通道SEMG,对7类肩颈部动作进行了分类识别研究,探索其用于交互的可行性与有效性。同时,在手势实时识别与交互系统基础上进行了相应改进,实现了基于SEMG的肩颈部动作实时识别系统。4.融入肌电信息的个人导航研究。步行者正常行走时,左右两腿交替迈步,相应腓肠肌轮流收缩施力完成身体前向运动。从腓肠肌表面采集的SEMG信号强度随步行时腿部用力大小表现出显著的节律性。针对这一运动生理学特点,本文提出了一种基于SEMG的步行者运动分析技术,并与数字罗盘相结合,实现了一种新型的步行者航位推算(PDR)方法。该方法采用叠加窗分帧技术、样本熵特征提取对双腿腓肠肌SEMG信号进行处理,用HMM分类器对步行者正常步行还是站立动作进行分类,由正常步行的SEMG信号检测迈步周期和估计步长,结合航向信息即得到步行者的位置和航迹。在此基础上进一步结合GPS接收机,验证了该方法实现个人室内外无缝导航的有效性。本论文的研究得到国家863高科技研究发展计划“基于肌电传感器和加速计的手势交互设备研究”(2009AA01Z322)、国家自然科学基金项目“基于表面肌电的中国手语手势识别研究”(60703069)、NOKIA赫尔辛基(Helsinki)研究中心及北京研究院合作项目和中国科学技术大学研究生创新基金的资助。更多还原

【Abstract】 As computing devices have become involved into all aspects of human living environment, the real world, digital world and the people who constitute the main body of the world are organized as an integrated whole. The demand of seamless communication and liberated interaction between people and the environment promotes the human body gesture recognition to become the hot spot of the research on future multi-channel and multi-modal human-computer interaction (HCI). Human body gesture recognition is known as the process that the computer automatically capture, analyze and understand the various types of gestures and human behaviors, such as fingers, wrists, arms, head, face and body posture or gesture patterns, to determine people’s intentions and provide the corresponding services.Arbitrary body movements are completed by groups of muscles which are coordinated and work closely together under the control of the nervous system. The electromyographic (EMG) signal caused by muscular activities is regarded as one kind of important bioelectric signals. Attached on the surface of skin above relevant muscles, Surface EMG (SEMG) sensor can capture the information of human muscular activities, which not only reflect the state and strength of flexion and extension of the joints, but also reflect the information of limb postures and positions. The EMG processing technologies provide us with important opportunities to capture human body gestures.Aiming at the recognition of hand gestures, neck and shoulder gestures and leg motions, the dissertation investigates the detection and recognition of various kinds of human body gestures based on SEMG signals, designs real-time gesture-based interactive systems, conducts a certain amount of user testing experiments, and provides some practical solutions for the natural and harmonious HCI. The research will promote further the development, application and extension of multi-modal intelligent HCI techniques. Moreover, the research achievements are of sufficient importance in the fields of human behavior understanding, rehabilitation medicine, context awareness, pervasive computing, and navigation. The main work and achievements of the dissertation focuses on the following aspects:1. Hand gesture recognition based on the multi-channel SEMG. The purpose of this study is to realize effective algorithms for hand gesture-based HCI systems, and to provide theoretical foundations for the selection of input hand gesture commands and SEMG sensor placements. In one aspect, the SEMG processing and recognition methods including signal measurement, active segmentation, feature extraction and classification are studied to classify 8 kinds of commonly used hand gestures. Subsequently, an optimized algorithm is proposed for real-time interaction. And on this basis, a SEMG-based real-time hand gesture recognition system is established. In the other aspect, according to the anatomical knowledge, optimization study on the definition of hand gesture commands and SEMG sensor placement is conducted on the classification of 20 kinds of hand gestures including some subtle finger movements. Thereby, a practical interactive scheme for HCI applications is proposed. The user testing experiments are conducted on the real-time hand gesture recognition system in user-specific, multi-user, and user-independent classification. The experimental results demonstrate the robust performance of proposed interactive scheme. The achievements of this study can provide important references on the selection of input hand gestures and the placement of SEMG sensors in applications of SEMG-based HCI.2. Sign language recognition based on the information fusion of acceleration (ACC) and SEMG The aim of this study is to investigate the hand gesture recognition technique based on the information fusion of multiple sensors. ACC-based methods are capable of distinguishing larger scale gestures with different hand trajectories of forearm movement, whereas SEMG-based recognition systems are capable of distinguishing subtle gestures with different muscular activities, such as subtle finger or wrist movements. Considering the complementary characteristics of ACC- and SEMG-based measurements, a framework for hand gesture recognition based on the information fusion of a 3-axis ACC and multi-channel SEMG is presented. The framework utilizes multi-stream HMM and decision tree for information fusion of the two heterogeneous sensors. Based on the framework, the classification of 30 kinds of Chinese sign language (CSL) words and 16 CSL dialog sentences is implemented. Furthermore, a promising real-time interactive system is built for the control of virtual Rubik’s cube game using 18 kinds of hand gestures.3. Neck and shoulder gesture recognition based on SEMG sensors. Neck and shoulder gestures can be regarded as a supplementary mean of natural and harmonious HCI. The feasibility and practicability of building muscle-computer interfaces starting from SEMG-based neck and shoulder gesture recognition is investigated. The multi-channel SEMG signals are measured from the relevant back, shoulder and neck muscles to classify 7 kinds of neck and shoulder gestures. Then, a real-time SEMG-based neck and shoulder gesture recognition and interaction system is established by the improvement of real-time hand gesture recognition system.4. Personal navigation fused with SEMG information. Considering the characteristics of the left and right leg alternately making each pace and contractions of muscles are cyclic when a pedestrian is walking, the EMG signal from the surface skin of calf (Gastrocnemius) showed a significant rhythm according with strength of every pace exerted by leg muscles. Taking advantage of human physiological characteristics during walking, a novel pedestrian dead reckoning (PDR) method is proposed with the fusion of SEMG-based technique for analyzing pedestrian’s activities and digital compass based technique for measuring azimuth. In the PDR method, overlapped windowing schemes and sample entropy feature extraction are firstly utilized to process gastrocnemius SEMG signals, and HMM classifiers are used to classify pedestrian activities such as walking or standing still. Then the SEMG-based step detection and step length estimation are implemented during walking, and are combined with the heading of each step measured by digital compass to determine the trace and position of the pedestrian. The field tests demonstrate that a GPS receiver integrated with our proposed PDR method has great potential to provide feasible and effective solutions to seamless outdoor/indoor pedestrian navigation.The research is supported by the National High Technology Research and Development Program of China (The 863 Program) "Research on the Gesture Input Devices Based on the Accelerometers and Surface EMG sensors" (2009AA01Z322), National Natural Science Foundation of China "Chinese Sign Language Recognition based on Surface Electromyogram" (60703069), cooperation projects with NOKIA Research Center (Helsinki & Beijing) and Graduate Innovation Foundation of USTC.更多还原