Self-Occluded Human Pose Recovery in Monocular Video Motion Capture

Monocular video motion capture is a popular alternative to more expensive technologies such as marker-based optical motion capture. However, motions that are occluded from the single camera view, for example, due to self-occlusion, are difficult to recover. In this paper, we propose a machine learning-based method that is used in post-processing to reconstruct the incorrect motions that are caused by self-occlusion. The post-processing network is trained on a dataset acquired from three subjects doing 30 different basic exercise motions that include self-occlusion. The collected data comprise single video camera footage and optical motion capture data as the ground truth. To correctly reconstruct the occluded motion, action recognition information is used to select a machine learning model that is trained on the specific motion. The performance of predictive and non-predictive networks are compared to each other and also with the state of the art in human motion estimation. The results show a significant reduction of the overall pose error and the pose error for selected body parts with a large degree of self-occlusion.