Associated with the Linz Institute of Technology and the Institute of Computational Perception @ JKU Linz, Austria

Computational Versus Perceived Popularity Miscalibration in Recommender Systems

CompVsUserPercep Teaser

Abstract

Popularity bias in recommendation lists refers to over-representation of popular content and is a challenge for many recommendation algorithms. Previous research has suggested several offline metrics to quantify popularity bias, which commonly relate the popularity of items in users' recommendation lists to the popularity of items in their interaction history. Discrepancies between these two factors are referred to as popularity miscalibration. While popularity metrics provide a straightforward and well-defined means to measure popularity bias, it is unknown whether they actually reflect users' perception of popularity bias.To address this research gap, we conduct a crowd-sourced user study on Prolific, involving 56 participants, to (1) investigate whether the level of perceived popularity miscalibration differs between common recommendation algorithms, (2) assess the correlation between perceived popularity miscalibration and its corresponding quantification according to a common offline metric. We conduct our study in a well-defined and important domain, namely music recommendation using the standardized LFM-2b dataset, and quantify popularity miscalibration of five recommendation algorithms by utilizing Jensen-Shannon distance (JSD). Challenging the findings of previous studies, we observe that users generally do perceive significant differences in terms of popularity bias between algorithms if this bias is framed as popularity miscalibration. In addition, JSD correlates moderately with users' perception of popularity, but not with their perception of unpopularity.


Citation

Oleg Lesota, Gustavo Escobedo, Yashar Deldjoo, Bruce Ferwerda, Simone Kopeinik, Elisabeth Lex, Navid Rekab-saz, Markus Schedl
Computational Versus Perceived Popularity Miscalibration in Recommender Systems
Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, 1889--1893, doi:10.1145/3539618.3591964, 2023.

BibTeX

@inproceedings{Lesota2023CompVsUserPercep,
    title = {Computational Versus Perceived Popularity Miscalibration in Recommender Systems},
    author = {Lesota, Oleg and Escobedo, Gustavo and Deldjoo, Yashar and Ferwerda, Bruce and Kopeinik, Simone and Lex, Elisabeth and Rekab-saz, Navid and Schedl, Markus},
    booktitle = {Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval},
    publisher = {Association for Computing Machinery},
    address = {New York, NY, USA},
    location = {Taipei, Taiwan},
    doi = {10.1145/3539618.3591964},
    url = {https://doi.org/10.1145/3539618.3591964},
    pages = {1889--1893},
    year = {2023}
}