Multimodal Hand Gesture Dataset from Wrist-worn sensors (MuWiGes)

About the Project

In the framework of a project entitle "Edge Intelligence based Hand Gestures Recognition using Wearable Multimodal Sensors for Human Machine Interaction" supported by the Air Force Office of Scientific Research under award number FA2386-20-1-4053.

Description

Hand gesture is one of the most natural and intuitive means of communication between human and machine. In recent years, many datasets and deep learning models have been proposed for hand gesture recognition. However, it still lacks publicly available datasets collected by multimodal wrist-worn sensors. As a result, methods for hand gesture recognition captured from such sensors are very limited. In this paper, we design a prototype of wrist-worn device that is able to capture both RGB video streams and motion information of hand gestures. Then we build a hand gesture dataset using this device. This new benchmark dataset, named MuWiGes with sufficient size, variety, and real-world elements, is able to train and evaluate deep neural networks. It contains more than 5408 samples focused on interaction with home appliances. This dataset could be the first multimodal dataset released for the research community using wrist-worn sensors. The MuWiGes dataset contains 12 actions performed by 50 persons (30 males and 20 females) with age and gender distribution as follows:

Figure 1: Age Distribution of 50 Subjects

Figure 2: Gender Distribution of 50 Subjects

Each subject performs continously all 12 dynamic hand gestures. All frames, accelerometer data (gx, gy, gz) and gyroscope data (ax, ay, az) are synchronized. Furthermore, to facilitate the labeling process, the starting and ending points of each gesture are also marked during data acquisition via a keypad or a remote control device

Figure 3: An example of gesture G1: 8 frames extracted uniformaly from the original sequence from the third person view are shown in the top row while the accelerometer and gyroscope signal are in the middle row and 8 frames captured by the prototype are in the bottom row

Hardware

In this section we describe the layout of the system the collection data MuWiGes.

We build a the wearable device consists of three main components: 1) the first component includes a wide-angle camera, an accelerometer and a gyroscope sensor; 2) the second component is an embedded device (i.e. jetson nano) that is in charge of data acquisition, processing and storage; 3) the final component is the power supply in Figure 4.

Figure 4: Main components of the device

The sensors are mounted on a watch-like hand band so that the user can wear it easily at his wrist as Figure 5.

Figure 5: Design wearable device

We use a lowcost wide-angle RGB camera for the purpose of ordinary usage. The camera model is IMX219-160 which gives the highest resolution of 3280 × 2464 at 15 fps, the camera side field of view of 160o. At 1280 × 720 resolution, the acquisition rate may reach 90 fps. The MPU6050 module is used to receive accelerometer and gyroscope signals. The device is worn on the backside of the user’s right wrist. As a result, the camera will capture images of the hand back. The sensors are connected to an embedded computer (i.e. Jetson Nano) to transfer data through CSI port. Figure 6 illustrates the final design of our prototype.