Recommender Systems

Who chose what you watched?

Most of what you watched last week, you did not search for. It was put in front of you: the next video, the row of films, the song after this one, the post at the top of your feed. Someone’s software chose it, from a catalogue far too big for you to browse.

The numbers from the companies themselves are striking. Netflix’s product leaders wrote in 2015 that their recommender system influences choice for about 80% of hours streamed, and estimated that personalization and recommendations save the company more than a billion dollars a year, mostly by keeping subscribers from cancelling. At CES in 2018, YouTube’s chief product officer said that more than 70% of the time people spend watching YouTube comes from its recommendations. TikTok’s For You feed, Spotify’s mixes, Amazon’s “customers also bought” and the ordering of every social feed are the same kind of system.

Recommenders are probably the most widely used AI systems on Earth, and among the least understood by the people they serve. This lesson opens one up, from the simplest idea (“people who liked this also liked that”) to the architecture of a modern feed, how they are measured, how they can go wrong, and how large language models are starting to change them.

What is a recommender system?

A recommender system predicts how much a person will value an item they have not yet seen, and uses those predictions to choose a short, ordered list. The items can be films, products, songs, news articles, jobs, people to follow or ads. The output is almost always a ranking: of all the things we could show you, these ten, in this order.

How is the problem usually framed?

Picture a giant table with one row per person and one column per item. Each cell holds what we know about that person and that item: a star rating, a purchase, a play count. Almost every cell is empty, because nobody has seen more than a sliver of the catalogue. Recommendation is the job of filling in the empty cells well enough to pick the best ones for each row.

A worked example of just how empty: the dataset Netflix released for its 2006 competition had 480,189 customers and 17,770 films, so 8.53 billion possible cells, of which 100,480,507 held a rating. That is 1.18% filled. A shop with ten million products and a hundred million customers is emptier still.

What counts as a signal?

The cells can be filled with two very different kinds of evidence:

Modern systems run mostly on implicit feedback, because there is so much more of it. That changes the maths. With ratings, an empty cell means “unknown”. With clicks, an empty cell might mean “not interested” or “never saw it”, and you cannot tell which. A classic answer from Hu, Koren and Volinsky (2008) treats every cell as a guess of 0 or 1 (“prefers” or not) with a confidence weight that grows with how much the person engaged: watching a show for ten hours is stronger evidence than a five-second preview.

Key takeaways

• A recommender predicts how much each person will value unseen items and turns those predictions into a short ranked list.
• The core data is a person-by-item matrix that is almost entirely empty: about 1.2% filled in the Netflix Prize data, far less in most shops.
• Explicit ratings are clear but rare; implicit signals like clicks and watch time are plentiful but ambiguous, and dominate modern systems.

Recommending by description

The most obvious way to recommend is to describe things. If you liked a tense space film, find other tense space films. This is content-based filtering, and it works without knowing anything about other people.

What is content-based filtering?

Every item is described by features: genre tags, cast, keywords, price range, or today more often an embedding of its text, images or audio. A person is described by a profile in the same feature space, built from the items they liked. Recommendations are the items whose features are closest to the profile.

How does it work?

Say each film has eight tag weights between 0 and 1. The profile is the average of the tag vectors of films you liked. Each candidate film gets a score equal to the cosine similarity between its vector and your profile: 1 if it points the same way, 0 if it shares nothing.

Worked example: you liked Interstellar (space 1, mind-bending 0.8, plus small weights) and The Matrix (action 1, mind-bending 1). Your profile averages them, so “mind-bending” is your strongest tag at 0.9, with action at 0.6 and space at 0.5. Inception, which is mostly action and mind-bending, scores highly. Notting Hill, a romantic comedy, scores close to zero: all it shares is a faint trace of romance.

With only Toy Story liked, the top of the list is Up and WALL-E, followed by Notting Hill purely because both are tagged “funny”: crude features give crude matches. That is content-based filtering’s signature strength and weakness in one screen. It needs no other users, so it works for a brand-new item on the day it arrives, and every recommendation comes with a reason (“shares: animated, family”). But it can only recommend more of the same: nothing in the tags can tell it that fans of Toy Story often love Spirited Away’s strangeness or a particular documentary. The features also have to exist and be good, which is expensive for films and hard for things like songs or jokes.

Why does it still matter?

Content features are the standard cure for the cold-start problem you will meet later, and modern embeddings have made them far richer than hand-made tags. A text or image embedding of a product description captures style and subject without anyone labelling it. Most production systems blend content signals with the collaborative methods of the next two sections.

Key takeaways

• Content-based filtering describes items with features and people with a profile in the same space, then ranks by similarity.
• It needs no data about other users, so it handles new items well and explains itself naturally.
• It cannot surprise you: it only finds items that resemble what you already liked, and it is only as good as its features.

Collaborative filtering

Here is a stranger idea that turned out to be more powerful: ignore what items are, and only look at who liked them. If you and I agreed about twenty films, my opinion of the twenty-first is a good guess at yours.

What is collaborative filtering?

Collaborative filtering predicts a person’s taste from the behaviour of many people together, using only the ratings matrix. It knows nothing about genres or plots. GroupLens did this for Usenet news articles in 1994, and the idea has powered shops and streaming services ever since.

There are two classic flavours:

• User-user: find people whose past ratings correlate with yours (your “neighbours”), then average their opinions of the item, weighted by how similar each one is to you.
• Item-item: find items whose ratings correlate with the item in question across all users, then look at how you rated those neighbours.

How is similarity measured?

People use the rating scale differently: one person’s 3 is another’s 5. So before comparing, each rating is centred on its rater’s average. A 4 from someone who averages 2 is enthusiastic; a 4 from someone who averages 4.8 is lukewarm. Similarity is then the cosine between the centred vectors, computed only over items both have rated. For two users this is essentially the Pearson correlation.

User-user prediction: your average plus a weighted average of how far each neighbour rated the item above or below their own average.

In user-user mode Ana, who loves Alien and The Matrix like you, gets a similarity near 1, while Chloe and Dev, who prefer Titanic and Notting Hill, come out negative. The prediction for Die Hard is pulled up by Ana and Ben, who both rated it above their own averages. Tap your Titanic rating until it reads 5 and watch the neighbours reshuffle: the same table now says you are more like Chloe.

Item-item mode reaches similar answers by a different route: Die Hard’s column of ratings moves with Alien’s and The Matrix’s, so your high ratings for those carry over.

Why Amazon chose items over users

With millions of customers, comparing you against everyone at request time is slow, and your neighbours change every time anyone buys anything. Items are more stable. In a 2003 paper, Linden, Smith and York described Amazon’s item-to-item collaborative filtering: compute the table of similar items offline, then at request time just look up the neighbours of the handful of things you bought or viewed. It is fast, scales to huge catalogues, and explains itself: “because you bought X”.

Why does it matter, and where does it break?

Collaborative filtering discovers connections that no tag would capture, such as the fact that people who like a certain comedian also like a certain cookbook. But it has three weaknesses you can already see in the small table. Sparsity: two people with few films in common produce a similarity computed from almost nothing (the lab shrinks such similarities towards zero). Cold start: a new film with no ratings has no neighbours at all. Popularity: blockbusters co-occur with everything, so they tend to crowd into every neighbour list.

Key takeaways

• Collaborative filtering predicts from the ratings matrix alone: people who agreed before will probably agree again.
• Centre ratings on each person’s average before comparing, then weight neighbours by similarity.
• Item-item scales better than user-user and explains itself, but both struggle with sparse data, new items and popularity bias.

Matrix factorization

In October 2006 Netflix offered a million dollars to anyone who could predict its customers’ ratings 10% more accurately than its own system. It took almost three years, thousands of teams, and one idea above all others: matrix factorization.

What is matrix factorization?

Instead of comparing rows or columns directly, give every person and every item a short list of numbers, say k = 20, called latent factors. The predicted rating is the dot product of the two lists. Nobody chooses what the factors mean: training discovers whatever dimensions best explain the ratings. On films, some turn out to resemble serious versus silly, or action versus romance; many are not interpretable at all.

The name comes from linear algebra: the huge ratings matrix R (people × items) is approximated by the product of two thin matrices, P (people × k) and Q (items × k), so R ≈ P Qᵀ. A matrix with 8.5 billion cells is summarised by about 10 million numbers.

How is it trained?

The standard recipe, set out by Koren, Bell and Volinsky in 2009 from their prize-winning work, adds bias terms for generous raters and universally loved items:

μ is the global average, b_u and b_i are the person and item biases, p_u and q_i the latent factor vectors.

Training minimises squared error on the known ratings only, plus a penalty λ on the size of all the parameters so that a film with three ratings cannot get extreme factors. Stochastic gradient descent visits each known rating, computes the error e = r − r̂, and nudges: p_u moves towards q_i, q_i moves towards p_u, both scaled by e and shrunk by λ. An alternative, alternating least squares, fixes P and solves exactly for Q, then fixes Q and solves for P, and parallelises well.

A worked step: with μ = 3.6, b_u = 0.2, b_i = 0.3, p_u = (0.5, 0.2) and q_i = (0.6, −0.5), the prediction is 3.6 + 0.2 + 0.3 + (0.30 − 0.10) = 4.3. If the true rating was 5, the error is 0.7, so the model will raise b_i a little and rotate p_u and q_i slightly towards each other.

At the start the factors are small random numbers and every blank is predicted near the average. Within a few dozen epochs the films split into two groups on the right-hand plot, action and sci-fi on one side, animation and romance on the other, and each person’s diamond swings towards the films they rated highly. The model was never told any genre. With λ = 0 the training error keeps falling because the model is free to memorise; with λ = 0.5 the factors stay short and the predictions stay closer to the averages. Choosing λ well is the difference between learning taste and memorising noise.

The Netflix Prize, 2006 to 2009

Netflix released 100,480,507 anonymised ratings and challenged teams to cut the error of its Cinematch system (an RMSE of 0.9525 on the hidden test set) by 10%. Simon Funk’s widely read blog post in late 2006 showed that SGD matrix factorization alone jumped up the leaderboard, and factor models became the backbone of nearly every leading team. On 21 September 2009 the prize went to BellKor’s Pragmatic Chaos, a merger of three teams, with a test RMSE of 0.8567. A rival team, The Ensemble, matched the score but had submitted 20 minutes later.

Two postscripts matter as much as the result. The winning entry blended hundreds of models, and Netflix later explained that it put two of the earlier methods, matrix factorization and restricted Boltzmann machines, into production, but that the final ensemble’s extra accuracy did not justify the engineering effort; by then streaming had changed what mattered. And the data was not as anonymous as hoped: Narayanan and Shmatikov showed that a handful of ratings and approximate dates could re-identify subscribers by matching them against public IMDb reviews. After a lawsuit and privacy concerns from the US Federal Trade Commission, Netflix cancelled a planned second competition in 2010.

Key takeaways

• Matrix factorization gives every person and item a short learned vector; a predicted rating is their dot product plus bias terms.
• It is trained on known ratings only, by SGD or alternating least squares, with regularization λ to stop rare items overfitting.
• It dominated the 2006 to 2009 Netflix Prize, and its core idea, learned embeddings compared by dot product, runs through every modern recommender.

Retrieval, then ranking

A video platform has hundreds of millions of videos and must answer in a fraction of a second. You cannot run a large model on every video for every request. So every large recommender is built as a funnel.

What does a modern recommender look like?

YouTube’s 2016 description of its deep learning recommender made the pattern famous, and nearly every large system now follows it. A cheap candidate retrieval stage narrows millions of items to a few hundred plausible ones. An expensive ranking stage scores each of those carefully. A final re-ranking step applies rules the model does not know about.

How does retrieval find a few hundred items among millions?

The workhorse is the two-tower model, a neural descendant of matrix factorization. One network (the user tower) turns everything known about you and your current context into a vector u. A second network (the item tower) turns each item’s ID and content into a vector v. They are trained so that u · v is high for items you engaged with and low for others, typically by treating the other items in the same training batch as negatives, with a correction so popular items are not over-penalised for appearing as negatives so often.

The key property is that the towers only meet at the dot product. Item vectors can be computed in advance and loaded into an approximate nearest-neighbour index (the same HNSW and IVF structures used for document search). At request time the system computes one user vector and asks the index for the items with the largest dot products, which takes milliseconds even over hundreds of millions of items. Because item content feeds the item tower, a new item gets a sensible vector as soon as it is uploaded.

What does the ranking model add?

The ranker only sees a few hundred candidates, so it can afford a large model and hundreds of features that would be too slow for retrieval: how you interacted with this creator before, how fresh the item is, what you watched in the last five minutes, the device you are on. It predicts not one number but several: probability of a click, expected watch time, probability of a like, a share, or a “not interested”.

Those predictions must be combined into a single score to sort by, and the weights are a product decision, not a technical one. Covington and colleagues explained that YouTube ranked by expected watch time rather than click probability, because ranking by clicks promotes deceptive videos that people click and then abandon.

Ranked by clicks alone, the “you won’t believe” clip and the 30-second outrage clip take the top two slots, even though people abandon them almost immediately and often press “not interested”. Ranked by expected watch time, the long documentary and the lecture rise instead. The balanced preset penalises dismissals and gives the football highlights and the tutorial a chance. None of these orderings is “correct”: each encodes a view of what a good session is.

Key takeaways

• Large recommenders are funnels: cheap retrieval narrows millions of items to hundreds, then an expensive ranker orders them.
• Two-tower models learn user and item vectors that meet only at a dot product, so item vectors can be pre-indexed for fast nearest-neighbour search.
• Rankers predict several outcomes at once, and the weights that combine them are a value judgement about what the product is for.

Cold start and exploration

A recommender only learns about what it shows. If a new song never gets played, it never earns the plays that would get it recommended. Breaking that circle means deliberately showing things the model is unsure about.

What is the cold-start problem?

Collaborative methods need interaction history. A new item has none, so it has no collaborative position; a new user has none either, so there is nothing to personalise from. Every platform faces both constantly: YouTube receives new videos every minute, and every shop has first-time visitors.

Standard remedies, usually combined:

• Content features (text, image and audio embeddings, metadata) so a new item has a position before anyone interacts with it.
• Onboarding questions (“pick three artists you like”) and context such as country, device and the page someone arrived from.
• Popular or trending items as a fallback for people with no history.
• A deliberate exploration budget: some impressions reserved for items whose value is uncertain.

How does exploration work?

This is the explore-exploit trade-off from reinforcement learning, in its simplest form the multi-armed bandit. Greedy always shows the items with the best observed like-rate. ε-greedy does the same, except that a small fraction ε of slots go to random items. Thompson sampling keeps a probability distribution over each item’s true like-rate (wide for items with little data) and ranks by a random draw from each: uncertain items sometimes draw high and get shown, and as evidence accumulates their distributions narrow.

Contextual bandits extend this to personalised choices. In a well-known 2010 experiment on the Yahoo! front page, the LinUCB algorithm raised clicks on news articles by 12.5% over a bandit that ignored user context.

Greedy never shows any new item: with no data its estimated like-rate is zero, so it can never collect the data that would raise it. Its likes-per-100 stays below the best possible forever, and N1’s bar stays flat. ε-greedy stumbles on N1 through random slots and, once the estimate is good, starts showing it all the time, but it keeps wasting a fixed share of slots on items it already knows are poor. Thompson sampling tries every new item early, drops the weak ones quickly and settles on N1, typically earning the most likes over the run. That is why variants of it are common in production for new content and ads.

Key takeaways

• New items and new users have no interaction history, so collaborative methods alone cannot place them.
• Content features, onboarding, popularity fallbacks and deliberate exploration are the standard remedies.
• Greedy recommenders can lock out good new items forever; ε-greedy and Thompson sampling spend some impressions to learn.

Measuring recommendations

How do you know a recommender is good? The honest answer is that offline numbers tell you which models are worth testing, and only experiments with real people tell you which is better.

What are offline ranking metrics?

Take logged data, hide the most recent interactions for each person, train on the rest, and check whether the hidden items appear near the top of each person’s ranked list. Common metrics at a cut-off k (the number of slots actually shown):

• Precision@k: what fraction of the top k were relevant.
• Recall@k: what fraction of all the relevant items made the top k.
• NDCG@k (normalised discounted cumulative gain): each hit earns 1/log₂(position + 1), so a hit at position 1 is worth 1, at position 2 about 0.63, at position 5 about 0.39. The total is divided by the best achievable score, giving a number between 0 and 1.
• Reciprocal rank: one over the position of the first hit, averaged over people as MRR.

Precision@5 and recall@5 only care whether the relevant films are in the top five. Swap Arrival into first place and neither moves, but NDCG rises, because people look at the top of a list first and a hit there is worth more. Bring Get Out into the top five without pushing Alien out and recall jumps from 0.67 to 1.0: a cut-off metric is blind to everything below the line.

Why are offline metrics not enough?

Logged data only contains reactions to what the old system chose to show. A new model that recommends something never shown before gets no credit offline, even if people would have loved it. Offline metrics also reward popularity: popular items are over-represented among held-out interactions, so a “most popular” list scores surprisingly well. And a better score on a proxy (predicting clicks) may not mean a better product (people satisfied and returning next month).

So production teams run online A/B tests: a random slice of users gets the new model, and the teams compare engagement, retention and surveys over weeks. Netflix’s 2015 paper describes this pipeline, using offline results to decide which ideas deserve an A/B test and retention as the deciding metric.

Key takeaways

• Offline metrics such as recall@k and NDCG@k check whether held-out interactions land near the top of each ranked list.
• Logged data is biased towards what the old system showed and towards popular items, so offline wins must be confirmed with online A/B tests.
• Simple baselines are strong: a surprising number of published models failed to beat them when checked carefully.

Feedback loops and society

Recommenders do not just predict what people want; they shape it, because people can only choose among what they are shown. That makes them one of the most debated technologies of the decade, and the evidence is more mixed than either side usually admits.

What is a feedback loop?

A recommender trains on logs of what people did with its own past recommendations. Items it showed get data; items it did not show get none. Over time the system can become more certain about a narrowing set of items, and the logs look like confirmation.

In simulations, Chaney, Stewart and Engelhardt showed that recommenders retrained on their own feedback make users’ behaviour more alike without making it more useful to them. Popularity bias is the most visible symptom: items that are already popular get recommended more, which makes them more popular, while niche items and new creators struggle to be seen. The exploration demo in the previous section showed a small version of the same lock-in.

Do recommenders create filter bubbles?

The “filter bubble” worry is that personalisation traps people in ever narrower, more extreme content. Some audits of YouTube around 2019 reported pathways from mainstream to fringe channels. But the most careful causal studies so far find smaller effects than the worry suggests:

• Hosseinmardi and colleagues (2024) ran bots that copied real users’ YouTube histories, then had some bots follow only the recommendations. The recommendation-following bots ended up consuming less partisan content than the real users, suggesting that what people chose mattered more than what the algorithm pushed. The sidebar “forgot” a partisan history within about 30 videos.
• In a large experiment during the 2020 US election, Guess and colleagues switched consenting Facebook and Instagram users to a chronological feed for three months. People spent substantially less time on the platforms and saw more political and more untrustworthy content, yet measured polarization and political attitudes did not change significantly.

These studies have limits: they measure particular platforms, periods and outcomes, and a few months may be too short to see slow effects. What they rule out is the simple story that turning off the algorithm would fix polarization.

Engagement versus wellbeing

The deeper criticism is about objectives. A system trained to maximise watch time will find whatever keeps people watching, and that is not the same as what people would endorse on reflection. Platforms have responded by adding survey-based “satisfaction” predictions, penalties for regretted content and time-spent reminders, and researchers study how to rank by stated preference rather than by behaviour alone. The Guess study’s finding that chronological feeds cut time on the platform is a reminder that ranking does change how much people use a product, even where it does not change their beliefs.

Regulation: the EU Digital Services Act

Europe now regulates recommender systems directly. Under the Digital Services Act, Article 27 requires online platforms to explain in their terms and conditions, in plain language, the main parameters their recommender systems use and any options users have to change them. Article 38 goes further for very large platforms and search engines (those with more than 45 million monthly users in the EU): they must offer at least one feed option that is not based on profiling. That is why many apps now have a “following” or chronological feed in Europe. Very large platforms must also assess and mitigate systemic risks, including those their recommenders create, and submit to independent audits.

Key takeaways

• Recommenders learn from logs of their own choices, which feeds popularity bias and can narrow what people see.
• Careful causal studies of YouTube and Facebook found smaller effects on partisanship and polarization than the filter bubble story predicts, though their scope is limited.
• The EU Digital Services Act requires platforms to explain their recommender parameters, and very large ones to offer a feed not based on profiling.

Generative recommenders

Language models predict the next word. A person’s history of watches or purchases is also a sequence. Since 2023 a fast-growing line of work asks: what if a recommender simply generated the next item?

What is generative recommendation?

Sequence models have been used for recommendation for years: a transformer reads your recent interactions and predicts what comes next, much as a language model reads a sentence. The obstacle is the vocabulary. A language model chooses among perhaps 100,000 tokens; a video platform has hundreds of millions of items, and new ones every second.

Rajput and colleagues at Google (2023) proposed semantic IDs. Each item’s content embedding is compressed into a short sequence of codes by residual quantization, so similar items share prefixes. A transformer is then trained to generate the next item’s code, token by token. Retrieval becomes generation, and a new item gets a meaningful ID from its content alone.

How far has it gone?

Several large platforms have reported replacing parts of the retrieval-then-ranking funnel with a single generative model. The short-video company Kuaishou’s OneRec (2025) generates whole sessions of videos with an encoder-decoder model, then fine-tunes it with a reward model and direct preference optimisation, the same alignment technique used for chatbots. The authors report a 1.6% increase in watch time in their main feed, which at that scale is a large gain. Meta, Pinterest, Snapchat and others have published related work, and the field is moving quickly.

Large language models also enter recommenders in gentler ways: writing item descriptions and features for cold-start content, explaining recommendations in natural language, and conversational recommendation, where you describe what you want (“something like Arrival but less slow”) and an assistant searches a catalogue with tools. What they have not done is make the core problems go away: the feedback loop, the choice of objective, evaluation beyond logged data, and the need to serve billions of requests cheaply.

Key takeaways

• Generative recommenders treat a history of interactions as a sequence and generate the next item, as a language model generates the next word.
• Semantic IDs, short codes derived from content embeddings, give huge and changing catalogues a vocabulary a transformer can generate.
• Production systems such as Kuaishou’s OneRec report gains from unifying retrieval and ranking in one model, but feedback loops, objectives and evaluation remain open problems.

Check your understanding

Seven situations from building and running recommenders. Each asks you to apply an idea from the lesson.

A bookshop launches a recommender. Thousands of new titles arrive each month and most customers buy only two or three books a year. Which approach should carry most of the weight for new titles?

References

Papers, company write-ups and legislation cited in this lesson, from the collaborative filtering systems of the 1990s to generative recommenders of 2025. To train a recommender yourself, open the Recommender Lab.

Sources

1. [1]

   The Netflix Recommender System: Algorithms, Business Value, and Innovation(opens in a new tab)

   Gomez-Uribe and Hunt, 2015

   ACM Transactions on Management Information Systems 6(4). Netflix executives describe their algorithms and report that recommendations influence about 80% of hours streamed.

2. [2]

   CES: YouTube's AI is the puppetmaster over what you watch(opens in a new tab)

   CBS News, 2018

   Reports YouTube chief product officer Neal Mohan's statement at CES 2018 that more than 70% of watch time comes from recommendations.

3. [3]

   Collaborative Filtering for Implicit Feedback Datasets(opens in a new tab)

   Hu, Koren and Volinsky, 2008

   ICDM 2008. Treats implicit signals such as viewing time as a binary preference with a confidence weight, trained with alternating least squares.

4. [4]

   GroupLens: An Open Architecture for Collaborative Filtering of Netnews(opens in a new tab)

   Resnick, Iacovou, Suchak, Bergstrom and Riedl, 1994

   CSCW 1994. An early automated collaborative filtering system that predicted ratings from the ratings of correlated users.

5. [5]

   Amazon.com Recommendations: Item-to-Item Collaborative Filtering(opens in a new tab)

   Linden, Smith and York, 2003

   IEEE Internet Computing 7(1). Explains why Amazon compared items rather than users: item similarities can be computed offline and scale to huge catalogues.

6. [6]

   Matrix Factorization Techniques for Recommender Systems(opens in a new tab)

   Koren, Bell and Volinsky, 2009

   IEEE Computer 42(8). The standard introduction to latent factor models with biases, regularization, SGD and ALS, drawn from the Netflix Prize.

7. [7]

   Netflix Recommendations: Beyond the 5 stars (Part 1)(opens in a new tab)

   Amatriain and Basilico, 2012

   Netflix Technology Blog. Explains that Netflix adopted two Progress Prize algorithms (matrix factorization and RBMs) but judged the final Grand Prize ensemble not worth the engineering effort.

8. [8]

   Robust De-anonymization of Large Sparse Datasets(opens in a new tab)

   Narayanan and Shmatikov, 2008

   IEEE Symposium on Security and Privacy 2008. Showed that Netflix Prize subscribers could be re-identified by matching a few ratings against public IMDb reviews.

9. [9]

   Deep Neural Networks for YouTube Recommendations(opens in a new tab)

   Covington, Adams and Sargin, 2016

   RecSys 2016. Describes YouTube’s two-stage design: a candidate generation network narrows millions of videos to hundreds, then a ranking network orders them.

10. [10]

    Sampling-Bias-Corrected Neural Modeling for Large Corpus Item Recommendations(opens in a new tab)

    Yi et al., 2019

    RecSys 2019. A two-tower retrieval model trained with in-batch negatives and a correction for popularity bias, deployed for YouTube.

11. [11]

    A Contextual-Bandit Approach to Personalized News Article Recommendation(opens in a new tab)

    Li, Chu, Langford and Schapire, 2010

    WWW 2010. LinUCB on the Yahoo! Front Page Today Module, reporting a 12.5% click lift over a context-free bandit.

12. [12]

    Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches(opens in a new tab)

    Ferrari Dacrema, Cremonesi and Jannach, 2019

    RecSys 2019. Of 18 neural methods, 7 could be reproduced, and 6 of those were beaten by simple, well-tuned baselines.

13. [13]

    How Algorithmic Confounding in Recommendation Systems Increases Homogeneity and Decreases Utility(opens in a new tab)

    Chaney, Stewart and Engelhardt, 2018

    RecSys 2018. Simulations in which recommenders trained on their own feedback make users’ behaviour more alike without making it more useful to them.

14. [14]

    Causally Estimating the Effect of YouTube's Recommender System Using Counterfactual Bots(opens in a new tab)

    Hosseinmardi et al., 2024

    PNAS 121(8). Bots that followed only recommendations consumed less partisan content than the real users they copied; the sidebar “forgot” partisan history within about 30 videos.

15. [15]

    How Do Social Media Feed Algorithms Affect Attitudes and Behavior in an Election Campaign?(opens in a new tab)

    Guess et al., 2023

    Science 381. Switching consenting Facebook and Instagram users to chronological feeds for three months cut time on the platforms, but did not significantly change polarization.

16. [16]

    Regulation (EU) 2022/2065 on a Single Market for Digital Services (Digital Services Act)(opens in a new tab)

    European Parliament and Council of the European Union, 2022

    Article 27 requires platforms to explain the main parameters of their recommender systems; Article 38 requires very large platforms to offer at least one option not based on profiling.

17. [17]

    Recommender Systems with Generative Retrieval(opens in a new tab)

    Rajput et al., 2023

    NeurIPS 2023. TIGER: items get “semantic IDs” (short code sequences from a quantized content embedding) and a transformer generates the next item’s ID.

18. [18]

    OneRec: Unifying Retrieve and Rank with Generative Recommender and Iterative Preference Alignment(opens in a new tab)

    Deng et al., 2025

    Kuaishou replaces its retrieval-then-ranking cascade with one generative model, aligned with DPO; reports a 1.6% watch-time increase in deployment.