This paper is proposing a new method for humans to provide feedback for complex tasks, reducing the amount of feedback by orders of magnitude (since full human feedback would require a ton of time).
In summary, we desire a solution to sequential decision problems without a well-specified reward
function that:
- enables us to solve tasks for which we can only recognize the desired behavior, but not
necessarily demonstrate it,
- allows agents to be taught by non-expert users,
- scales to large problems, and
- is economical with user feedback.
Essentially, humans would compare short video clips of what the agent does, making evaluation easier. This also may be specific to the experiments that this paper runs since both can be visually interpreted by humans: Atari games and robotics tasks.
The paper has a policy π and reward function r^, each containing deep neural networks. Those are each updated by:
- Policy π interacts with the environment -> produces trajectories {τ1,...,τi}. π is updated as traditional RL to maximize the predicted rewards seen as rt=r^(ot,at).
- Atari games: advantage actor-critic (A2C)
- Robotics tasks: Trust Region Policy Optimization (TRPO). Adjusted hyperparameter for entropy bonus in TRPO.
- Rewards normalized to a mean of zero + constant SD
- Select segments (σ1,σ2) from trajectories {τ1,...,τi} from step 1, send them to the human to compare.
- Humans compare two visualizations of trajectory segments, each clip being 1-2 seconds long.
- Human indicates which segment they prefer. This is recorded in the database as (σ1,σ2,μ) where μ is over the distribution of {1,2}.
- Human can choose to mark the comparison as incomparable, which will not include it in the database.
- The parameters of the mapping r^ are optimized via supervised learning to fit comparisons collected by the human so far.
- Follows the Bradley-Terry model for estimating score functions from pairwise preferences.
More formal math presented in the paper.
Results showed that human feedback performed the same or slightly worse with the same number of labels, possibly attributed to human error in labeling and consistency.
