1Technical University of Munich 2Facebook Reality Labs
Proc. Neural Information Processing Systems (NeurIPS) 2020
We introduce a novel, end-to-end learnable, differentiable non-rigid tracker that enables state-of-the-art non-rigid reconstruction.
Given two input RGB-D frames of a non-rigidly moving object, we employ a convolutional neural network to predict dense correspondences.
These correspondences are used as constraints in an as-rigid-as-possible (ARAP) optimization problem.
By enabling gradient back-propagation through the non-rigid optimization solver, we are able to learn correspondences in an end-to-end manner such that they are optimal for the task of non-rigid tracking.
Furthermore, this formulation allows for learning correspondence weights in a self-supervised manner.
Thus, outliers and wrong correspondences are down-weighted to enable robust tracking.
Compared to state-of-the-art approaches, our algorithm shows improved reconstruction performance, while simultaneously achieving 85x faster correspondence prediction than comparable deep-learning based methods.