Portrait relighting for 3D light-field display based on radiance fields

Relighting a portrait scene from a photographs or video requires accounting for high-order light transport effects and accurate estimation of portrait scene geometry, specular reflections, shadows, and subsurface scattering. A delicate manipulation of the lighting can be performed while keeping the scene albedo and geometry unaltered. A radiance field-based approach to light-field rendering of portrait scenes is proposed in this work. The method requires only a mobile phone with a flash and an inexpensive polarization foil for data acquisition. Two groups of portrait images are captured using cross-polarized light and parallel polarized light. Subsequently, two multilayer perceptrons (MLPs) are employed to extract the material information (Albedo, Normals, Specular) of the portrait individually. The extracted material information, along with an ambient light map, is utilized in physically-based rendering to generate portrait images. The final result is optimized using a coarse-to-fine optimization scheme. Finally, a light-field image is generated through light-field encoding. Our approach achieves higher quality light-field portrait scenes with controllable illumination, realistic shadows, and subsurface scattering (as shown in Fig. 1). The method we propose can be applied to small-sized light-field displays to enhance the lighting quality of portrait reconstructions, and lay the foundation for future applications in lightfield communication and related fields.