Evaluating Decision-Aware Recommender Systems

The main goal of a Recommender System is to suggest relevant items to users, although other utility dimensions - such as diversity, novelty, confidence, possibility of providing explanations - are often considered. In this work, in order to increase the amount of relevant items presented to the user, we analyse how the system could measure the confidence on its own recommendations, so it has the capability of taking decisions about whether an item should be recommended or not. A direct consequence of this design is that the number of suggested items decreases, impacting in some of the beyond-accuracy dimensions (especially, coverage). We present an evaluation of different decision-aware techniques that can be applied to some families of recommender systems, and explore evaluation metrics that allow to combine more than one evaluation dimension. Empiric results show that large precision improvements are obtained when using these approaches at the expense of user and item coverage.