Abstract
Recent research has suggested different metrics to measure the inconsistency of recommendation performance, including the accuracy difference between user groups, miscalibration, and popularity lift. However, a study that relates miscalibration and popularity lift to recommendation accuracy across different user groups is still missing. Additionally, it is unclear if particular genres contribute to the emergence of inconsistency in recommendation performance across user groups. In this paper, we present an analysis of these three aspects of five well-known recommendation algorithms for user groups that differ in their preference for popular content. Additionally, we study how different genres affect the inconsistency of recommendation performance, and how this is aligned with the popularity of the genres. Using data from Last.fm, MovieLens, and MyAnimeList, we present two key findings. First, we find that users with little interest in popular content receive the worst recommendation accuracy, and that this is aligned with miscalibration and popularity lift. Second, our experiments show that particular genres contribute to a different extent to the inconsistency of recommendation performance, especially in terms of miscalibration in the case of the MyAnimeList dataset.
Citation
Dominik Kowald,
Gregor Mayr,
Markus
Schedl,
Elisabeth Lex
A Study on Accuracy, Miscalibration, and Popularity Bias in Recommendations
Advances in Bias and Fairness in Information Retrieval,
1--16, doi:10.1007/978-3-031-37249-0_1, 2023.
BibTeX
@inproceedings{Kowald2023AccMiscPop,
title = {A Study on Accuracy, Miscalibration, and Popularity Bias in Recommendations},
author = {Kowald, Dominik and Mayr, Gregor and Schedl, Markus and Lex, Elisabeth},
booktitle = {Advances in Bias and Fairness in Information Retrieval},
editor = {Boratto, Ludovico and Faralli, Stefano and Marras, Mirko and Stilo, Giovanni},
publisher = {Springer Nature Switzerland},
address = {Cham},
location = {Taipei, Taiwan},
doi = {10.1007/978-3-031-37249-0_1},
url = {https://doi.org/10.1007/978-3-031-37249-0_1},
pages = {1--16},
year = {2023}
}