What’s something that is relevant to your interests that you nonetheless dislike? Why is it relevant and what do you dislike about it?

It is truly bizarre to see how many #RecSys paper authors apparently think NDCG can only support binary relevance and therefore they must convert datasets with _explicitly graded relevance_ by binarizing users’ item ratings against a threshold value. That’s not…

oh no, the nothing

chickpea and potato stew with ham, chorizo, and smoked turkey. paired with cava.

After a fair amount of reading about constructing social science concepts with some degree of rigor, I have concluded that the #RecSys definition of popularity doesn’t even have face validity.

“Lots of people interacted with it” is not equivalent to “well-liked by many people”, even on the surface with a colloquial definition.

As they say with unfortunate correctness: “The amount of energy needed to refute bullshit is an order of magnitude bigger than that needed to produce it.”

I had never even heard of at least half of the movies Netflix considers “the most popular of all time”. They have a strange definition of “popular” and “all time” though:

People have been noting issues with combining implicit and explicit feedback for 20+ years. The only papers I can find that make a serious attempt also have a model architecture diagram straight out of the Cthulhu mythos that appears calculated to look sophisticated and confuse reviewers.

Meanwhile, practitioners have to do this all the time and functionally nothing in the recommender systems research literature is any help.

It’s really hard to ground practical recommender systems techniques in the existing research literature

What you find over and over again are papers that either mention the problem and make no attempt to solve it, or papers that consider a weird unrealistic version of the problem and do something convoluted (presumably to make a novel research contribution)

It’s almost impossible to find papers that consider the likely “average case” of real platforms and solve actual problems that come up

Why would this ever matter? Suppose you would like to index into a tensor with an arbitrary non-integer value like a string. Almost anything can be hashed to construct a UUID and a 128 bit UUID can be split into four 32 bit integers. A four dimensional tensor where each dim’s size is the max of uint32 allows you to use a sparse tensor like a Python dictionary. No additional mapping between vocabulary and indices required!

Apparently PyTorch sparse tensors compute their number of elements as the product of sparse dimension sizes, leading to integer overflows for notionally large tensors that have few non-zero elements 🤦

“Cultural capital participates in the process of domination by legitimizing certain practices as naturally superior to others and by making these practices seem superior even to those who do not participate, who are thus led, through a negative process of inculcation, to see their own practices as inferior and to exclude themselves from legitimate practices.”

- Bourdieu on Art, Literature, Culture (Johnson, 1993)

“The process that carries works along is the product of the struggle among agents who, as a function of their position in the field, of their specific capital, have a stake in conservation […] or subversion, i.e. a return to sources, to original purity, to heretical criticism, and so forth.”

- Principles of a Sociology of Cultural Works (Bourdieu, 1991)

Research papers are cultural works.

If you systematically ignore (or denigrate) work on data quality, offline evaluation, algorithmic mechanisms, and the process of actually serving recommendations to real people, I can’t find much sympathy for the lack of impact on industry practice that is sure to result from those choices.

Somehow people are continually surprised by this though. Who did you think was going to do the work to make your ideas practically usable and useful?

“What is badly defined is likely to be badly measured”

- OECD Composite Indicator Handbook

Meanwhile in #RecSys, we have a collection of vaguely defined concepts that we don’t agree with each other on, that users don’t agree with us on, that we’re not sure to operationalize or measure, but that have been repeated ad nauseum for twenty years like self-evident truths or some kind of gospel.

“Does popularity bias exist? A cultural studies approach to off-policy evaluation”

This is either the kind of genius/madness that’ll result in rocks thrown at me from three different directions or such profound stupidity that no one will care

It’s amazing how deep the rabbit hole goes when you reject a simplistic view of popularity. Suddenly sociology, psychology, cultural theory, media studies, and even OECD guides for developing composite economic indicators become relevant. I may get out of this hole alive, but the “short paper” I was working on definitely will not.

happy Earth, Wind & Fire day to those who celebrate

There’s a paper that criticizes the MovieLens recommendation datasets because their interaction generation mechanism is different from an actual streaming platform. That much is true, but I can guarantee that the interaction mechanism isn’t just “recalling movies you have watched” because a single prolific user has enough interactions that it would take 90 years to watch all those movies at a rate of one per day.

At some point we’re going to have to admit to ourselves that evaluations are the product of many design choices, and in no way correspond (rigorously or otherwise) with any objective, underlying reality that’s independent of our priorities, preferences, and decisions.

I’m not holding my breath though.

in related news, I’m getting way more value from a Hulu subscription than I expected