On joint optimization of motion compensation, quantization and baseline entropy coding in H.264 with complete decoder compatibility

This paper presents a framework for jointly designing motion compensation, quantization and entropy coding in a hybrid video coding structure to minimize a rate distortion cost. Given motion compensation, a soft decision-based quantization algorithm is first designed to reduce the rate distortion cost by adapting quantization outputs to the baseline entropy coding method in the newest standard H.264. Motion compensation is then optimized by searching for a prediction to further reduce the rate distortion cost based on given quantization outputs. By alternating these two steps, an iterative method is then proposed. The proposed algorithms have been implemented based on the reference encoder of H.264 with complete baseline decoder compatibility. Comparative studies show that the baseline-based iterative optimization method achieves coding performance comparable or sometimes superior to that afforded by the main profile encoder.