Internal Seminar: M. E. Brix Doest, Technical University of Denmark
Mads Emil Brix Doest from DTU presents today his research on Single-Shot Analysis of Refractive Shape Using Convolutional Neural Networks. The appearance of a transparent object is determined by a combination of refraction and reflection, as governed by a complex function of its shape as well as the surrounding environment. Prior works on 3D reconstruction have largely ignored transparent objects due to this challenge, yet they occur frequently in real-world scenes. Brix Doest presents an approach to estimate depths and normals for transparent objects using a single image acquired under a distant but otherwise arbitrary environment map. In particular, they use a deep convolutional neural network (CNN) for this task. Unlike opaque objects, it is challenging to acquire ground truth training data for refractive objects, thus, they propose to use a large-scale synthetic dataset. To accurately capture the image formation process, they use a physically-based renderer and demonstrate that a CNN trained on our dataset learns to reconstruct shape and estimate segmentation boundaries for transparent objects using a single image, while also achieving generalization to real images at test time. In the presented experiments, they extensively study the properties of our dataset and compare to baselines demonstrating its utility.