RL for dynamic ABR for better user experience when streaming videos.
Notes
Main Idea: Pensieve, a reinforcement learning system for learned adaptive bitrate (ABR) algorithms
Learns ABR decisions via the performance of past decisions.
Improves average quality of experience (QoE) by 12-25%
Background about ABR Algorithms:
Adaptive bitrate (ABR) algorithms are used by providers to optimize video quality (maximize QoE)
Selecting the right bitrate is challenging because of variable network throughput, conflicting video QoE requirements, cascading effects of bitrate decisions, and the coarse-grained nature of ABR decisions.
Existing ABR algorithms have fixed control rules and are tuned manually for one QoE objective. They don't adapt well to changes in QoE goals.
Pensieve is a system that learns ABR algorithms automatically via RL and optimizes the control policy for the characteristics of the network. The high-level architecture can be seen below:
This is the image that can be found in the article, displaying the architecture of RL in Bitrate Adaptation. The ABR agent is trained on a neural network policy using RL to pass back QoE-based rewards, providing feedback on refined bitrate decisions.
Design of Pensieve:
Simulated video streaming environment (because real-world training would have been too slow). The simulator maintained the client playback buffer with the assigned download time based on the bitrate and input network traces for fast, repeatable training.
Utilized an A3C (Asynchronous Advantage Actor-Critic) RL algorithm to train the neural network.
This is a powerful deep RL algorithm that uses multiple agents training in parallel and asynchronously updating a shared model.
State: past throughput, download time, buffer size, video chunk sizes, number of chunks remaining, previous bitrate decision
Action: select the bitrate for the next video chunk from a set of available bitrates
Reward: QoE metric that linearly balances video quality, rebuffering time, and smoothness of video quality
Generalizes to unseen networks and works on synthetic training data. This transfers across videos with different bitrates and lengths.
Evaluated against MPC, BOLA, and buffer-based approaches on real-world network traces.
Outcome of Pensieve:
Pensieve learns different policies for different QoE objectives and will automatically balance bitrate utility, rebuffering penalty, and smoothness penalty based on the QoE function. This aids long-term planning.
