On the Road to Large-Scale 3D Monocular Scene Reconstruction using Deep Implicit Functions

Autonomous driving relies on building detailed models of a vehicles surroundings, including all hazards, obstacles and other road users. At present, much of the autonomous driving literature reduces the world to a collection of parametric 3D boxes. While this framework is sufficient for many driving scenarios, other important scene details (e.g. overhanging structures, open car doors, debris, potholes etc.) are not modelled. Recently deep implicit functions have been shown to be suitable for representing fine grained details at arbitrarily high resolutions using images alone. However, they have predominantly been employed in constrained situations, such as reconstructing individual objects or small-scale scenes. In this work we explore the application of deep implicit functions to larger scenes in the context of real-world autonomous driving scenarios. In particular we focus on the challenging case where only monocular images are available at test time. While most implicit function networks rely on watertight meshes for training, these are not in general available for real world scenes. We therefore propose an alternative training scheme using LiDAR to provide approximate ground truth occupancy supervision. We also show that incorporating priors such as pre-detected object bounding boxes can improve the quality of reconstruction. Our method is evaluated on a real-world autonomous driving dataset.