Visual Servoing on a deformable object with pre-planned trajectory-based geometric primitives

Abstract This paper presents a novel approach to rigidly align a prespective camera to the deformation of a non-rigid object. We assume having a reference pre-planned camera trajectory viewing a non-rigid object. Our goal is to align this trajec-tory at the execution time given a prior only on the most relevant landmarks. Our method does not assume any parametric or non-parametric prior model on the physics of deformation. The proposed method is formalized as a tracking problem embedded in an optimal visual control framework. The tracking problem encodes visual servoing of geometric features from the deformable object between pre-planned and execution time. The optimal visual control framework is formalized with a weighted least-square criterion that minimizes the distance between the images of the reference features and the features at runtime. The weights are time-dependent smooth functions that encode the relevance of the visible object’s features. We experimentally prove that our method can adapt to multiple pre-planned trajectories and multiple types of deformations (both linear and nonlinear) without any prior knowledge on them. Robustness to noise in the detection of image features is also experimentally demonstrated.