Learning to Optimize Non-Rigid Tracking
Yang Li1     Aljaž Božič2     Tianwei Zhang1     Yanli Ji3     Tatsuya Harada1     Matthias Nießner2    
    1The University of Tokyo     2Technical University of Munich     3University of Electronic Science and Technology of China
Proc. Computer Vision and Pattern Recognition (CVPR), IEEE, June 2020

One of the widespread solutions for non-rigid tracking has a nested-loop structure: with Gauss-Newton to minimize a tracking objective in the outer loop, and Preconditioned Conjugate Gradient (PCG) to solve a sparse linear system in the inner loop. In this paper, we employ learnable optimizations to improve tracking robustness and speed up solver convergence. First, we upgrade the tracking objective by integrating an alignment data term on deep features which are learned end-to-end through CNN. The new tracking objective can capture the global deformation which helps Gauss-Newton to jump over local minimum, leading to robust tracking on large non-rigid motions. Second, we bridge the gap between the preconditioning technique and learning method by introducing a ConditionNet which is trained to generate a preconditioner such that PCG can converge within a small number of steps. Experimental results indicate that the proposed learning method converges faster than the original PCG by a large margin.