返回
类型 应用研究 预答辩日期 2018-03-03
开始(开题)日期 2012-01-12 论文结束日期 2017-03-23
地点 显示中心一楼会议室 论文选题来源 学校自选项目    论文字数 7 (万字)
题目 基于计算机视觉的手势识别及人机交互技术的应用研究
主题词 人机交互,手势分割,手势识别,复杂环境
摘要 手势交互具有自然、直观等优点,已成为人机交互领域中重要部分与研究热点。手势交互技术可按输入设备分为多种类型,例如数据手套、加速传感器、触摸屏、单目摄像机、深度摄像机等。本论文研究的是基于单目视觉的手势交互技术,它通过分析裸手的图像信息来识别手势的类型或语义。该技术不需要额外的设备,更符合人类的交互习惯,但是有限的输入信息导致了一些具有挑战性的问题。人手的高自由度、环境的复杂性等均会影响手势识别的正确率与运算速度。为此,本文提出了能用于复杂环境的手势分割与识别算法,尤其解决了人脸的干扰问题。然后,在这些算法的基础上设计开发了一个实时的手指鼠标系统,实现了一些鼠标的功能。本论文主要内容与贡献如下: (1)针对复杂环境下的手势分割问题,本文提出一种结合肤色提取与背景差分的手势分割算法。为了减少光照影响,我们根据已有研究选择了亮度与色度分离的的YCbCr颜色空间,再利用色度分量Cb与Cr建立了一个基于直方图的肤色模型。考虑到手势位置的改变会造成其颜色的变化,以及背景中类肤色的物体也会对手势分割造成干扰,于是我们利用前景目标的色度信息实时地更新该肤色模型。然后,我们提出了一种结合帧间差分的背景分割法。为了适应动态背景,该方法通过统计非前景点的信息来实时地更新背景图。实验表明,我们的算法能在复杂背景下快速且准确地提取目标。 (2)针对背景中存在人脸干扰的问题,我们提出一种新颖的手势检测与分割算法,称之为基于边缘修复的手势部件分割法。由于我们最终的目的是将手势识别算法应用于一个可替代鼠标的交互系统,所以用户的人脸经常出现于摄像头视野内。人脸与手的颜色与纹理特征非常类似,会对手势识别的稳定性造成很大的影响。因此,如何精确地分离手与人脸区域是本课题最重要的内容之一。首先,当手与人脸重合时,利用分层Chamfer距离匹配算法定位手势区域。然后,考虑到手指的灵活性,我们将人手分为手掌与各手指的子部分,再利用基于图结构与梯度方向直方图的支持向量机分类器来检测这些子部分。接着,由于在手与人脸的交界处的边缘信息很可能模糊不清,我们提出一种边缘修复法来获得完整的子部分轮廓线。最后,将这些修复后的轮廓组合起来即可得到精确的手势区域。实验表明,该算法成功地实现了手与人脸重合时的手势分割,并且对头部运动、手势几何变形以及不同用户的差异性具有鲁棒性。 (3)针对静态手势识别问题,我们讨论了Hu矩特征、支持向量机分类器以及特征手算法,并在此基础上提出一种基于Hu矩、轮廓凸性与紧性的支持向量机法。考虑到本文的应用环境“手指鼠标系统”,我们定义了6种静态手形,并利用MU HandImages ASL 手势数据库与自行采集的手势视频序列进行测试。实验表明该识别算法对手势方向、尺寸变化以及手指结构变化均体现了令人满意的识别正确率与实时性。此外,我们提出一种结合手指形状与指尖位置特征的指尖检测法,它对手的方向与尺度变化具有鲁棒性,并用实验证明了其对各手形的手指数与指尖位置的检测精确度。 (4)结合提出的各种手势分割与识别算法,我们设计了一个基于单目视觉的手指鼠标系统。该系统可配置于一个自带或外接摄像机的个人计算机,用户面对电脑执行预定义的手势即可操作电脑,例如移动光标、移动文件、缩放、单击、双击、复制粘贴。我们为该系统开发了两套平台,即Windows平台以及DSP平台,并设计了系统的交互模式、框架与算法流程。实验表明该系统成功实现了上述各种类似鼠标的功能,使用户在与计算机的交互中有自然与舒适的体验。
英文题目 ALGORITHM OF COMPUTER VISION-BASED HAND GESTURE RECOGNITION AND APPLICATION OF HUMAN-COMPUTER INTERACTION
英文主题词 Human-computer interaction, hand segmentation, hand recognition, complex background
英文摘要 Hand gesture interaction is an important and hot research topic in human-computer interaction because of its naturalness and intuition. Technologies of hand gesture interaction can be classified according to the input devices, such as data glove, acceleration sensor, touch screen, monocular camera, depth camera and so on. This dissertation is dedicated to the hand gesture interaction technology based on monocular vision which identifies hand gestures by analyzing the images of bare hands. This technology conforms to human habits because no extra device is required. However, the limited input information leads to some challenging problems. For example, the accuracy and efficiency of hand gesture recognition is sensitive to the high degree of freedom of human hand and the complexity of the scenario. In this dissertation, several algorithms of hand gesture segmentation and recognition are proposed to solve the problems mentioned above, especially the influence of human faces. Furthermore, a real-time finger mouse system which achieves some functions of the mouse is developed by using the proposed algorithms. The main content and contributions of this dissertation is summarized as follows: (1) For hand segmentation under the complex scenario, a hand segmentation method based on skin color extraction and background subtraction is proposed. First, in order to reduce the influence of the illumination, YCbCr color space is employed in which the luminance and chrominance information are separated. A histogram-based model of skin color is built by using the chrominance information. Since the colors of the hands are various at different locations and the presence of other skin-like objects makes the discrimination of the bare hand difficult, the skin color model is updated in real time by the chrominance information of the foreground target. Second, a background subtraction method combining temporal differencing is proposed. In order to adapt the dynamic background, the background image is updated by the statistical analysis of the background pixels. The experimental results show that the target regions can be extracted fast and accurately in the complex background. (2) For the influence of human face, an edge repair-based hand subpart segmentation algorithm is proposed to detect and segment the hands. Since all the proposed algorithms will be applied in an interaction system as an alternative of the mouse, the user’s face appears in the camera view frequently. The efficiency of hand recognition is largely influenced by the face because of the uncertain movement and the similarity of color and texture to the hand. Therefore, how to distinguish the hand from the face is one of the most important parts in this dissertation. First, a hierarchical chamfer matching algorithm is employed to locate the hand region roughly. Second, in consideration of the flexibility of the fingers, the hand is divided into palm and fingers, which are separately detected by combining pictorial structures and HOG characteristics. Since edge information is probably blurry at the border of the hand and face, a completely connected curve representing the hand silhouette is difficult to figure out. Consequently, an edge repair method based on pre-stored templates is presented for each subpart. Finally, the repaired contours are combined to extract the precise hand region. The experimental results show that our algorithm is robust to the movement of the head, different users and geometric transformation of hands. (3) For the static hand gesture recognition, support vector machine based on Hu moments and eigenhand are first reviewed in this dissertation. Then a new support vector machine algorithm based on Hu moments, convexity and compactness of hand contours is proposed. Six hand postures are defined in consideration of the requirement of the finger mouse system and evaluated by using both the MU HandImages ASL dataset and a dataset built by ourselves. Experimental results show that the proposed method achieves a high recognition rate and operation speed for the discrimination of hand postures performed by different users and the geometric transformation of hands. Besides, a fingertip detection method is proposed which combines finger shape and fingertip position characteristics. The experimental results prove that the method is robust to hand scaling and rotation and it can detect the number of fingers and the positions of fingertips accurately. (4) A finger mouse system based on monocular vision is developed by using the proposed algorithms of hand segmentation and recognition. The system can be installed in a personal computer and allows users to communicate with the computer without a mouse. The functions implemented by the system include moving the cursor, moving a document, zoom in/out, click, copy and paste. Two platforms of Windows and DSP are built for the system and then the interaction mode, the system framework and the algorithm flow are designed. Experimental results show that the system works successfully and offers a natural and comfortable interaction experience.
学术讨论
主办单位时间地点报告人报告主题
东南大学显示中心张雄教授组 2013年6月20日 显示中心三楼会议室 徐军 画圆法指尖检测法与背景模型更新法
东南大学显示中心张雄教授组 2015年3月12日 显示中心三楼会议室 徐军 手指鼠标交互系统的搭建
东南大学显示中心张雄教授组 2013年4月18日 显示中心三楼会议室 徐军 基于视觉的人机交互系统的研究现状与进展
东南大学显示中心张雄教授组 2013年5月23日 显示中心三楼会议室 徐军 门限阈值肤色分割与Camshift手势跟踪
东南大学显示中心张雄教授组 2013年9月9日 显示中心三楼会议室 徐军 最小二乘曲线拟合法
东南大学显示中心张雄教授组 2013年12月2日 显示中心三楼会议室 徐军 Introduction of the researches of the 6th International Conference on Machine Vision
东南大学显示中心张雄教授组 2014年3月31日 显示中心三楼会议室 徐军 手势特征点提取法与Canny边缘检测法
东南大学显示中心张雄教授组 2014年5月5日 显示中心三楼会议室 徐军 手与人脸发生重合时的边缘修复法
     
学术会议
会议名称时间地点本人报告本人报告题目
Asia Display 2011年11月 江苏,昆山 Study of Interactive Display System Based on Hand-gesture Recognition
International Conference on Machine Vision 2013年11月 英国,伦敦 Finger Mouse System Based on Computer Vision in Complex Backgrounds
     
代表作
论文名称
Finger Mouse System Based on Computer Vision in Complex Backgrounds
A real-time hand detection system during hand over face occlusion
A High-Security and Smart interaction System Based on hand Gesture Recognition for Internet of Thing
 
答辩委员会组成信息
姓名职称导师类别工作单位是否主席备注
王鸣 正高 教授 博导 南师大
蒯曙光 正高 研究员 博导 华东师范大学
李青 正高 教授 博导 东南大学
李晓华 正高 教授 博导 东南大学
屠彦 正高 教授 博导 东南大学
      
答辩秘书信息
姓名职称工作单位备注
王莉莉 副高 副教授 东南大学