MacSR: Macroblock-aware Lightweight Video Super-Resolution

The mobile video quality can be improved by video super-resolution (SR) especially when bandwidth is limited. To achieve real-time SR, the latest work, ClassSR (CVPR 19), divides frames into equal-size image blocks (IBs), and different-complexity SR models are used respectively to reduce the computational burden. We propose a lightweight IB-based SR scheme, MacSR, for on-demand videos. In Figure 1, the server first runs the time modeling algorithm based on decoding information to estimate video quality (i.e., PSNR) improvement of each 16x16 IB SR (i.e., SR impact). Then, for different types of IBs, we use forward propagating and intra-IB sorting to select IBs and generate corresponding sets (i.e., intra and inter lists). Finally, we use the block searching algorithm to search the appropriate inter-IB and intra-IB proportions according to video quality requirements. The algorithm finally obtains the SR IB indexes saved in patch lists. On the client side, we redesign the decoder which super-resolves the selected IBs in patch lists, while other IBs rely on the high-resolution reference frame queue for decoding. In this way, the terminal can output high-quality videos with low computation. [GRAPHICS] We measure the latency of decoding and SR on the Xeon E5-2650 CPU to quantify the acceleration, compared to per-frame SR in six 240p videos (three types). In the IB-based SR mode, all schemes use the same SR model, and the input frame is divided into 16x16 blocks. The adopted SR model is a variation of MDSR (CVPRW 17). In Figure 2, MacSR achieves 32%, 23.2%, and 24% acceleration compared with Nemo (MobiCom 20). The acceleration further increases to 48.75%,39.2%, and 105.5%, compared with Fast (CVPRW 17). MacSR maintains robustness with video content changes. The average video quality of MacSR is 0.25dB higher than Fast. The average extra bandwidth brought by patch lists is less than 3% of video chunks.