Hand gesture interaction is an important and hot research topic in human-computer interaction because of its naturalness and intuition. Technologies of hand gesture interaction can be classified according to the input devices, such as data glove, acceleration sensor, touch screen, monocular camera, depth camera and so on. This dissertation is dedicated to the hand gesture interaction technology based on monocular vision which identifies hand gestures by analyzing the images of bare hands. This technology conforms to human habits because no extra device is required. However, the limited input information leads to some challenging problems. For example, the accuracy and efficiency of hand gesture recognition is sensitive to the high degree of freedom of human hand and the complexity of the scenario. In this dissertation, several algorithms of hand gesture segmentation and recognition are proposed to solve the problems mentioned above, especially the influence of human faces. Furthermore, a real-time finger mouse system which achieves some functions of the mouse is developed by using the proposed algorithms. The main content and contributions of this dissertation is summarized as follows:
(1) For hand segmentation under the complex scenario, a hand segmentation method based on skin color extraction and background subtraction is proposed. First, in order to reduce the influence of the illumination, YCbCr color space is employed in which the luminance and chrominance information are separated. A histogram-based model of skin color is built by using the chrominance information. Since the colors of the hands are various at different locations and the presence of other skin-like objects makes the discrimination of the bare hand difficult, the skin color model is updated in real time by the chrominance information of the foreground target. Second, a background subtraction method combining temporal differencing is proposed. In order to adapt the dynamic background, the background image is updated by the statistical analysis of the background pixels. The experimental results show that the target regions can be extracted fast and accurately in the complex background.
(2) For the influence of human face, an edge repair-based hand subpart segmentation algorithm is proposed to detect and segment the hands. Since all the proposed algorithms will be applied in an interaction system as an alternative of the mouse, the user’s face appears in the camera view frequently. The efficiency of hand recognition is largely influenced by the face because of the uncertain movement and the similarity of color and texture to the hand. Therefore, how to distinguish the hand from the face is one of the most important parts in this dissertation. First, a hierarchical chamfer matching algorithm is employed to locate the hand region roughly. Second, in consideration of the flexibility of the fingers, the hand is divided into palm and fingers, which are separately detected by combining pictorial structures and HOG characteristics. Since edge information is probably blurry at the border of the hand and face, a completely connected curve representing the hand silhouette is difficult to figure out. Consequently, an edge repair method based on pre-stored templates is presented for each subpart. Finally, the repaired contours are combined to extract the precise hand region. The experimental results show that our algorithm is robust to the movement of the head, different users and geometric transformation of hands.
(3) For the static hand gesture recognition, support vector machine based on Hu moments and eigenhand are first reviewed in this dissertation. Then a new support vector machine algorithm based on Hu moments, convexity and compactness of hand contours is proposed. Six hand postures are defined in consideration of the requirement of the finger mouse system and evaluated by using both the MU HandImages ASL dataset and a dataset built by ourselves. Experimental results show that the proposed method achieves a high recognition rate and operation speed for the discrimination of hand postures performed by different users and the geometric transformation of hands. Besides, a fingertip detection method is proposed which combines finger shape and fingertip position characteristics. The experimental results prove that the method is robust to hand scaling and rotation and it can detect the number of fingers and the positions of fingertips accurately.
(4) A finger mouse system based on monocular vision is developed by using the proposed algorithms of hand segmentation and recognition. The system can be installed in a personal computer and allows users to communicate with the computer without a mouse. The functions implemented by the system include moving the cursor, moving a document, zoom in/out, click, copy and paste. Two platforms of Windows and DSP are built for the system and then the interaction mode, the system framework and the algorithm flow are designed. Experimental results show that the system works successfully and offers a natural and comfortable interaction experience.