TWO-STAGE FRAMEWORK FOR CONTINUOUS HAND GESTURE SPOTTING AND RECOGNITION USING 3D SKELETON JOINT KINEMATICS

D.V Srihari Babu, Y Narasimha Reddy, M Nagaraju, Bakkala Santha Kumar

Abstract: Continuous hand gesture recognition needs not only the spotting (temporal localization) but also the gesture identification (classification) under the ambiguity of transitions and motion variation. This paper introduces a two-stage, cloud-deployable framework that somehow performs gesture spotting based on a compact 3D skeletal kinematic descriptor, and on the other hand, classifies isolated gestures based on key joint trajectories. We perform a comparative ablation study for three spotting variants, namely BiLSTM+BCE, BiLSTM+weighted BCE and attention+focal loss, and report the frame accuracy and mean Jaccard overlap, followed by a skeletal LSTM classifier accuracy. In controlled continuous stream experiments, the best spotting variant has 85.07% frame accuracy, 67.35% of mean Jaccard, and 94.43% of classifier accuracy. Finally, we address the scalability and deployment by the proposed model export for the cloud inference and the evaluation on ChaLearn LAP ConGD and the representation pretraining using Human3.6M as future works.

Keywords: Continuous Hand Gesture Recognition, Gesture Spotting, Temporal Localization, Skeleton Kinematic Descriptor, Skeleton-Based Action Recognition

DOI: 10.24874/PES08.02B.027

Recieved:   Revised:   Accepted:   
UDC:

Reads: 0   

---