Real-time spatially adaptive deconvolution for CFA images

In numerous applications, such as surveillance, industrial inspection, medical imaging and security, high resolution is of crucial importance for the performance of the computer vision systems. Besides spatial resolution, high frame rate is also of high importance in these applications. While the resolution of CMOS imaging sensors is following Moores law, for optics it is becoming increasingly challenging to follow such a development. In order to follow the pixel size reduction, lenses have to be constructed with much more precision, while the physical size increases dramatically. Moreover, expertise needed to construct a lens of sufficient quality is available at a very few locations in the world. The use of lower quality lenses with high resolution imaging sensors, lead to numerous artifacts. Due to the different light refraction indexes for different wavelengths, primary color components do not reach their targeted pixels in the sensor plane, which causes lateral chromatic aberration artifacts. These artifacts manifest as false colors in high contrast regions around the edges. Moreover, due to the variable refraction indexes, light rays do not focus on the imaging sensor plain, but in front or behind it, which leads to blur due to the axial aberration. Due to the increased resolution, the size of the pixel is significantly reduced, which reduces the amount of light it receives. As a consequence, the amount of noise increases dramatically. The amount of noise further increases due to the high frame rate and therefore shorter exposure times. In order to reduce the complexity and the price, most cameras today are built using one imaging sensor with spatial color multiplexing filter arrays. This way, camera manufacturers avoid using three imaging sensors and beam splitters, which significantly reduces the price of the system. Since not all color components are present at each pixel location it is necessary to interpolate them, i.e. to perform demosaicking. In the presence of lateral chromatic aberration, this task becomes more complex, since many pixels in the CFA do not receive a proper color, which creates occlusions which in turn create additional artifacts after demosaicing. To prevent this type of artifacts, occlusion inpainting has to be performed. In this paper we propose a new method for simultaneous correction of all artifacts mentioned above. We define operators representing spatially variable blur, subsampling and noise applied to the unknown artifacts free image, and perform reconstruction of the artifacts free image. First we perform lens calibration step in order to acquire the lens point spread (PSF) function at each pixel in the image using point source. Once we obtained PSFs we perform joint deconvolution using variable kernels obtained from the previous step.