Learning-to-Rank with Nested Feedback

Many platforms on the web present ranked lists of content to users, typically optimized for engagement-, satisfaction- or retention-driven metrics. Advances in the Learning-to-Rank (LTR) research literature have enabled rapid growth in this application area. Several popular interfaces now include nested lists, where users can enter a 2nd-level feed via any given 1st-level item. Naturally, this has implications for evaluation metrics, objective functions, and the ranking policies we wish to learn. We propose a theoretically grounded method to incorporate 2nd level feedback into any 1st-level ranking model. Online experiments on a large-scale recommendation system confirm our theoretical findings.