Calibration in recommender systems

Recommender systems are ubiquitously employed to help users of various digital platforms select items (such as music tracks, videos, movies, news articles, products, job advertisements, or profile pages of other users) from catalogs which may contain millions of entries. They relieve users' cognitive load by shrinking their choice scope to short lists of personalized recommendations. However, generating sufficiently personalized recommendation lists remains a challenge, not least due to fluidity and multi-faceted nature of human tastes and preferences. Previous work introduced the calibration view on personalization, which combines methods of quantifying and improving recommendation alignment with user preferences. In particular, calibration is concerned with maintaining a personalized level of diversity in their recommendations. Generally, recommendations provided to a user are deemed calibrated with respect to a certain item attribute (such as genre or popularity) when the distribution of said item attribute in the recommendations matches its distribution in their respective interaction history. This thesis consists of three parts that critically examine and expand the notion of item-attribute calibration in recommender systems. In the first part, we propose novel approaches to measuring and improving item popularity calibration in recommender systems. Proposed metrics complement established evaluation methods, primarily revolving around mean and divergence measures, with higher-order moment and user demographic analyses. We investigate discrepancies in the effectiveness of an established post-processing calibration framework between different recommendation algorithms and user groups. We then extend the framework with a hyper-parameter selection method allowing to exploit the user-group level discrepancies for achieving a more favorable overall trade-off between recommendation accuracy and calibration. Additionally, we introduce CaliTune, an in-processing alternative to the prevalent post-processing calibration approaches, that often incur notable computational overhead at inference time. CaliTune improves popularity calibration of established recommenders through a fine-tuning procedure combining large-scale dynamic sampling of item-candidates, heuristic list-wise calibration loss and a universal list-wise utility objective, allowing it to maintain competitive accuracy. In the second part, we investigate the connection between a common definition of popularity miscalibration based on Jensen-Shannon Divergence (JSD) and experiences of real users. We demonstrate that, while users perceive differences between lists calibrated to various extents, calibration does not necessarily carry an obvious value to them. Furthermore, the degree to which JSD aligns with user perception strongly depends on such factors as recommendation list popularity composition and user familiarity with the recommended items. In the third part, we propose a calibration-informed view on evaluation of potential cultural impact of recommender systems. We introduce item country of origin as calibration attribute, propose user-dependent attribute category assignment (domestically produced items) and conduct a feedback loop simulation study. Our results show that recommender systems tend to steer recommendations provided to users from countries whose music is less represented in the item catalog away from their domestically produced items, potentially contributing to globalization of user preferences in the long term.