Towards real-time video inpainting

Image inpainting refers to the task of restoring missing information in an image in a way such that a human observer is unable to tell if the image has been manipulated or not. It can be used, for example, to restore old images and photographs by removing scratches or other signs of wear and tear. However, it can also be used to get rid of superimposed text or to remove certain objects or subjects from a video. Video inpainting is an extension of image inpainting into a third, temporal dimension. On the one hand this extension provides additional information for the inpainting of a single frame but on the other hand introduces new challenges such as the need to restore the missing information in a temporally coherent way. Video inpainting can be used in various non-performance critical scenarios such as film editing or film restoration. However, there are also scenarios in which a video must be processed in real-time, which imposes certain constraints on the video inpainting method. Examples of such scenarios are live TV broadcasting or the usage in surveillance systems. This work proposes a modular real-time video inpainting method which is based on a regular video inpainting method and evaluates it with respect to different inpainting scenarios and compares it to the original method. Because of the additionally imposed constraints the results produced by the proposed method have a lower visual quality compared to the results of the original method. However, if the actual use cases of the method are known beforehand, the proposed method can be adapted and further optimized, for example, by relaxing some of the constraints.