Neural Non Rigid Tracking
Aljaž Božič1     Pablo Palafox1     Michael Zollhöfer2     Angela Dai1     Justus Thies1     Matthias Nießner1    
    1Technical University of Munich     2Meta
Proc. Neural Information Processing Systems (NeurIPS) 2020

We introduce a novel, end-to-end learnable, differentiable non-rigid tracker that enables state-of-the-art non-rigid reconstruction. Given two input RGB-D frames of a non-rigidly moving object, we employ a convolutional neural network to predict dense correspondences. These correspondences are used as constraints in an as-rigid-as-possible (ARAP) optimization problem. By enabling gradient back-propagation through the non-rigid optimization solver, we are able to learn correspondences in an end-to-end manner such that they are optimal for the task of non-rigid tracking. Furthermore, this formulation allows for learning correspondence weights in a self-supervised manner. Thus, outliers and wrong correspondences are down-weighted to enable robust tracking. Compared to state-of-the-art approaches, our algorithm shows improved reconstruction performance, while simultaneously achieving 85x faster correspondence prediction than comparable deep-learning based methods.