DAVE: Dynamic Adaptive Video Encoding for Real-time Video Streaming Applications

Real-time video streaming applications have become tremendously popular in recent years, such as remote control and video conferencing applications. A key characteristic that differentiates these applications from traditional live streaming applications is that these applications have a very low-latency requirement for interactivity. The stricter low-latency requirement brings many challenges: the video has to be encoded in a real-time manner; the substantial resources on the server or cloud cannot be utilized for encoding; and the adaptation strategies in live streaming applications are not adequate for real-time video streaming, such as adaptive bitrate selection (ABR). In addition, the video perceptual quality of current real-time video streaming systems is usually sacrificed to meet the very low-latency requirement. To address these challenges, in this paper, a new real-time video streaming protocol, DAVE (Dynamic Adaptive Video Encoding for real-time video streaming applications), is proposed. In the proposed real-time video streaming system, captured video frames are encoded with different configurations. Since the video encoding configuration determines the video data size, quality, and encoding time, we first conduct an experimental study on the impact of each configuration parameter. Based on our experimental findings, we then propose a super resolution based video encoding configuration selection algorithm which does not use a fixed strategy to determine the encoding configurations as in existing real-time video streaming systems but uses a reinforcement learning based model to learn the optimal video encoding configuration that includes the configuration of both regular video encoding parameters and the up-scale of super resolution models. As a result, DAVE can optimize the performance of real-time video streaming systems based on user Quality of Experience (QoE) metrics. To the best of our knowledge, this is the first work that incorporates super resolution and reinforcement learning in the protocol design for real-time video streaming systems. Extensive evaluations show that DAVE can substantially improve the video perceptual quality by 15% and can also reduce the endto-end latency by 20%, as compared with existing systems(1).