SensCapsNet: Deep Neural Network for Non-Obtrusive Sensing Based Human Activity Recognition:
SensCapsNet
Recently, the recent advancement of deep learning with the capacity to perform automatic high-level feature extraction has achieved promising performance for sensor-based human activity recognition (HAR). Among different deep learning methods, Convolutional Neural Network (CNN) and Long Short Term Memory (LSTM) have been widely adopted. However, scalar outputs and pooling in CNN only allow to get the invariance but not the equivariance. The capsule networks (CapsNet) with the vector output and routing by agreement is able to capture the equivariance. In this paper, we propose a method for recognizing human activity from wearable sensors based on a capsule network named SensCapsNet. The architecture of SensCapsNet is designed to be suitable for spatial-temporal data coming from wearable sensors. Experimental results show that the proposed network outperforms CNN and LSTM methods. The performance of the proposed CapsNet architecture is assessed by altering dynamic routing between capsule layers. The proposed SensCapsNet yields improved accuracy values of 77.7% and 70.5% for 1 routing on two testing datasets in comparison with the baseline methods based on CNN and LSTM that yields the F1-score of 67.7% and 69.2% for the first dataset and 65.3% and 67.6% for the second dataset respectively. Moreover, even several human activity datasets are available, privacy invasion and obtrusive concerns have not been carefully taken in to consideration in dataset building. Toward to build a non-obstructive sensing based human activity recognition method, in this paper, a dataset named 19NonSens is designed and collected from twelve subjects wearing e-Shoes and a smart watch to perform 19 activities under multiple contexts. This dataset will be made publicity available. Finally, thanks to the promising results obtained by the proposed method, we develop a life logging application which achieves a real-time computation and the accuracy rate greater than 80% for 5 common upper body activities.
Publications:
Cuong Pham, Son Nguyen-Thai, Huy Tran-Quang, Son Tran, Hai Vu, Thanh-Hai Tran, Thi-Lan Le*, SensCapsNet: Deep Neural Network for Non-obtrusive Sensing based Human Activity Recognition, IEEE Access, 2020.
3D skeleton-based action recognition with convolutional neural networks:
Overall framework for action recognition using skeleton data.
Activity recognition based on skeletons has drawn a lot of attention due to its wide applications in human computer interaction, surveillance system. Compare with image
data, a skeleton has a benefit of the robustness with background changing and computing efficiently dues to its low dimensional representation. With the rise of deep neural networks, a lot of
works has been applied using both CNN and LSTM networks to solve this problem. In this paper, we proposed a framework for action recognition using skeleton data and evaluate it with
different network architectures. We first modify the feature representation by adding motion information to a skeleton image, which gives useful information to the networks. After
that, different networks architectures have been employed and evaluated to give insight into how well it will perform on this kind of data. Finally, we evaluated the system on two public
datasets NTU-RGB+D and CMDFall to show the efficiency and feasibility of the system. The proposed method achieves 76.8% and 45.23% on NTU-RGB+D and CMDFall, respectively, which
is competitive results.
Publications:
Van Nam Hoang, Van-Toi Nguyen, Thi-Lan Le, Hai Vu and Thanh-Hai Tran, 3D skeleton-based action recognition with convolutional neural networks, 2nd Int. Conference on Multimedia Analysis and Pattern Recognition (MAPR), May, 2019.
Novel Skeleton-based Action Recognition Using
Covariance Descriptors on Most Informative Joints:
Determination of most informative joints for Action 6 (throw). The most informative joints are marked in red color.
Human action recognition has attracted much attention of research community in recent years due to its large application domain such as human-robot interaction, gaming, surveillance, etc. Action recognition can be implemented in either
single-modal (color, depth, skeletal) or multi-modal schemes. Our research focuses on human action recognition based on skeletal information. In literature, a Temporal Hierarchy of Covariance
Descriptors on 3D Joints (Cov3DJ) was proposed to exploit information from time dimension of skeletal data. Since each joint has a certain level of engagement in an action, another
approach aims at selecting joints which are most informative for action recognition. In this paper, a novel framework, named as Covariance Descriptor on Most Informative Joints (CovMIJ),
is proposed to benefit from the simplicity of representation via covariance descriptor and noise immunity by using only Most Informative Joints (MIJ) for action recognition. Extensive
experiments between CovMIJ, Cov3DJ and state-of-the-art Res-TCN (Temporal Convolutional Neural Networks with Residual Units) on two public datasets (MSR-Action3D and CMDFALL) show the efficiency of our proposal. On MSR-Action3D dataset,
accuracy of CovMIJ achieves 93.6% while that of Cov3DJ is only 90.53%. Improvement is found on CMDFALL dataset with accuracy of 64.72% for CovMIJ compared to 61.34% for Cov3DJ. On CMDFALL dataset, CovMIJ with F1 score of 62.5%
outperforms the deep learning network Res-TCN with F1 score
of 39.38%.
Publications:
Tien-Nam Nguyen, Dinh-Tan Pham, Thi-Lan Le, Hai Vu, Thai-Hai Tran, Novel Skeleton-based Action Recognition Using Covariance Descriptors on Most Informative Joints, 10th International Conference on Knowledge and Systems Engineering (KSE 2018) - November 2018, HCM, Vietnam.
Human posture recognition using human skeleton provided by Kinect:
Important joints defined for 4 main postures representation
Human posture recognition is an attractive and challenging topic in computer vision because of its wide range of application. The coming of low cost device Kinect with its SDK gives us a possibility to resolve with ease some difficult problems encountered when working with conventional cameras. In this paper, we explore the capacity of using skeleton information provided by Kinect for human posture recognition in a context of a heath monitoring framework. We conduct 7 different experiments with 4 types of features extracted from human skeleton. The obtained results show that this device can detect with high accuracy four interested postures (lying, sitting, standing, bending)
Publications:
Thi-Lan Le, Minh-Quoc Nguyen, Thanh-Mai Nguyen, Human posture recognition using human skeleton provided by Kinect, The International Conference on Computing, Management and Telecommunications (ComManTel 2013), 21-24 January 2013, Ho Chi Minh city, VietNam.
