How YouTube Decides What You Watch

Every time your YouTube homepage loads, a system evaluates a corpus of billions of videos and hands you about thirty of them in a few hundred milliseconds. It cannot score billions of videos per request; no hardware on Earth does that at YouTube's traffic volume. So it cheats, in a very principled way, and the shape of that cheat, a two-stage funnel of retrieval followed by ranking, has become the standard architecture for nearly every large-scale recommender system in production today. Having built and debugged systems in this family, I can tell you the interesting part is not the neural networks. It is the sequence of failures that forced each piece of the design into existence.

Everything below is grounded in what YouTube and Google have published, primarily Covington, Adams, and Sargin's 2016 RecSys paper and Zhao et al.'s 2019 RecSys paper on multi-task ranking, plus public statements from the team. Where I am inferring from that public research rather than reporting confirmed internals, I will say so.

You Cannot Rank Billions, So You Retrieve Hundreds

The architecture is a funnel with two stages that solve two different problems.

Candidate generation takes the full corpus, billions of videos, and cuts it down to a few hundred candidates that are plausibly relevant to you. This stage optimizes for recall: don't miss anything you might love. It has to be brutally fast, so it uses cheap models and clever indexing, and it barely looks at the video-user pair in detail.

Ranking takes those few hundred candidates and scores each one with a much heavier model that can afford to look at hundreds of features per video: your watch history with this channel, how long ago you saw a similar video, the video's fresh performance statistics, device, time of day. This stage optimizes for precision: get the ordering at the top of the page exactly right.

The economics are what make this non-negotiable. If your ranking model costs 10 milliseconds of compute per candidate, you can afford a few hundred candidates per request. You cannot afford a few billion. Every serious recommender, YouTube, TikTok, Instagram Reels, converges on this funnel because the arithmetic gives no other option.

The 2016 Paper That Set the Template

Covington et al.'s "Deep Neural Networks for YouTube Recommendations" (RecSys 2016) is the canonical public description, and a decade later it is still the best single document for understanding the system's bones.

The candidate generation model in that paper reframes recommendation as extreme multiclass classification: predict which video, out of millions of classes, the user will watch next.

P(watch = v_i | user U, context C) = exp(v_i · u) / Σ_j exp(v_j · u)

where u is a dense user embedding produced by a feedforward network from the user's watch history (embedded video IDs, averaged), search history, and demographics, and each v_j is a learned video embedding. Training a softmax over millions of classes is done with sampled negatives; nobody computes the full denominator.

The trick that makes this deployable is that at serving time you do not need probabilities at all. You need the top few hundred videos by v_j · u, which is a maximum inner product search problem. Compute the user embedding once per request, then find its nearest neighbors in the video embedding space using an approximate nearest neighbor (ANN) index. Modern ANN methods, hierarchical graphs like HNSW, or the quantization-based approach in Google's own ScaNN, retrieve top candidates from billion-item corpora in single-digit milliseconds with recall in the high 90s. You trade a sliver of exactness for four to five orders of magnitude of speed.

This structure, a user tower producing an embedding, an item tower producing embeddings, dot product as the score, is what the field now calls a two-tower model. Its defining constraint is that the user and the item never interact inside the network; they only meet at the final dot product. That is exactly what makes ANN indexing possible (item embeddings can be precomputed and indexed) and exactly what makes the model coarse (it cannot represent "this user likes this creator's long videos but not their shorts" as richly as a model that crosses features). Coarse and fast is precisely what stage one calls for. YouTube has since published follow-up work on two-tower retrieval with in-batch negative sampling and corrections for the fact that popular videos appear too often as negatives (Yi et al., RecSys 2019), so the design clearly persisted well beyond the 2016 paper.

One more detail from Covington et al. that deserves more fame than it has: the example age feature. ML models trained on historical data learn the past's popularity distribution, which is stale for a platform where fresh uploads matter enormously. They fed the age of each training example into the model, then set it to zero at serving time, letting the model explicitly learn and then discount popularity's time decay. Without it, the model implicitly recommended two-week-old videos because that is what training data reflected. It is a one-feature fix for a subtle distribution mismatch, and it is the kind of fix that separates papers from production systems.

A Score Is a Weighted Bundle of Predictions

Here is the question people almost never ask precisely: when the ranking stage assigns a video a "score," what is that number?

It is not relevance. It is not quality. It is not one number a single model produces. Based on the published work, it is a weighted combination of separately predicted user behaviors. The ranking model is a multi-task network that, for each candidate video, predicts several things at once:

• Engagement tasks: probability of a click; expected watch time if clicked.
• Satisfaction tasks: probability of a like, of a share, of a dismissal ("not interested"), and, notably, the predicted response to YouTube's post-watch survey questions ("was this video worth your time?"). YouTube has publicly confirmed it trains models to predict survey responses for the vast majority of views that never get an actual survey.

The final score is, conceptually:

score(v, u) = w_1 · E[watch_time] + w_2 · P(like) + w_3 · P(survey_satisfied)
            - w_4 · P(dismiss) + ...

The weights w_i are not learned. They are set by humans and tuned through live A/B experiments against long-term metrics. This is the single most consequential fact about the whole system: the machine learning predicts what you will do; the weight vector encodes what the company wants. Every debate about "the algorithm's" values is, mechanically, a debate about a handful of hand-tuned scalars sitting on top of some very good conditional probability estimates. The Zhao et al. 2019 paper says almost exactly this, that the combination function is manually tuned, and I would bet the tuning process consumes more senior-engineer hours than the model architecture does.

Why Watch Time Dethroned Clicks

The choice of what to predict has its own history, and it is a case study in Goodhart's law.

Until 2012, YouTube's ranking optimized heavily for clicks. The failure mode was mechanical and inevitable: a click is decided by the thumbnail and title, not the video. Optimize for clicks and you are running a global optimization over thumbnails. Creators responded rationally, thumbnails got more misleading, viewers clicked, felt cheated, and left after twenty seconds. The metric went up while the thing the metric was supposed to measure went down.

In March 2012, YouTube publicly switched its ranking objective to watch time, on the theory that you can be tricked into clicking but not into watching thirteen minutes. Some creators saw views drop 30 to 40 percent overnight; the platform judged this correctly as the metric getting more honest, not the content getting worse. Predicting expected watch time is also why Covington et al. trained their models with weighted logistic regression, positive examples weighted by observed watch time, so that the model's odds output approximates expected watch time directly rather than click probability.

Watch time has its own Goodhart failure, of course: it rewards whatever is maximally retentive, which is not the same as worthwhile, hence the later addition of survey-predicted satisfaction to the objective bundle around 2015 and after. Each objective in that weighted sum is a scar from the previous objective's exploit.

Shared Experts, Separate Heads: MMoE

Predicting engagement and satisfaction in one network creates a classic multi-task tension: the tasks are correlated but not aligned. A video can be highly clickable and highly unsatisfying, that is the entire clickbait phenomenon, so a shared-bottom network where both tasks consume the same representation ends up compromising both.

Zhao et al.'s 2019 RecSys paper ("Recommending What Video to Watch Next") describes YouTube's answer: Multi-gate Mixture-of-Experts (MMoE). Instead of one shared trunk, the network has several expert sub-networks, and each task gets its own learned softmax gate that mixes the experts:

task_k output = h_k( Σ_i  g_k,i(x) · expert_i(x) )

The engagement head can lean on experts that capture "will this hold attention," the satisfaction head can lean on experts that capture "will this be liked and rated well," and the gates learn the division of labor from data instead of an engineer hand-partitioning features. The paper reports that MMoE improved both engagement and satisfaction metrics in live experiments over the shared-bottom baseline, with the gating adding negligible serving cost. There is an amusing production detail in there too: naive MMoE training suffered gate polarization (one expert per task, defeating the point), which they mitigated by adding dropout on the gating softmax. Real systems papers always contain one paragraph like this, and it is always the most useful paragraph.

The Model Grades Its Own Homework

Now the deepest problem, and the reason recommender systems are epistemically weirder than most ML.

The training data is clicks and watch time on videos the previous version of the system chose to show, in positions the previous system chose. Two corruptions follow.

The first is position bias. Item one on the page gets clicked more than item five partly because it is better and partly because it is item one. Train naively and the model learns "things shown at position one get clicked," a self-fulfilling prophecy. The Zhao et al. paper describes YouTube's correction: a shallow tower that takes position and device as input and learns a bias logit that is added to the main model's output during training, then removed (position treated as missing) at serving. The main tower is thereby pushed to learn actual relevance while the shallow tower absorbs the artifact. They report position bias this technique measured and removed; the CTR difference across positions is substantial even conditioned on relevance.

The second corruption is the feedback loop. The system only observes outcomes for what it recommends, so its picture of your preferences is a picture of its own past behavior reflected back. If it never shows you woodworking videos, it never learns whether you like woodworking; confidence and coverage collapse toward the already-recommended. Left unchecked, this narrows both individual users (rabbit holes) and the ecosystem (rich-get-richer dynamics for established channels, since new uploads have no interaction data at all).

The standard counterweight is deliberate exploration: spend a small fraction of impressions on candidates the model is uncertain about, and treat the resulting data as disproportionately valuable because it came from outside the loop. Google's research groups have published extensively on exploration for recommenders, including REINFORCE-based policy methods with off-policy correction evaluated on YouTube (Chen et al., WSDM 2019) and follow-up work arguing exploration measurably grows the corpus of viable content. Exactly how much traffic YouTube devotes to exploration and how it is targeted is not public; that they do it is, and any system of this scale that didn't would slowly strangle its own training signal. This is inference from the public research, but it is not a daring inference.

The Objective Function Is the Product

Strip away the embeddings and the experts and the ANN indexes, and YouTube's recommender is a machine for answering one question a few hundred thousand times per second: given everything we know, what is the expected value, under our current definition of value, of showing this person this video? The neural networks have gotten steadily better at the "expected" part. The hard, contested, endlessly revised part is the definition of value, that little hand-tuned weight vector where clicks lost to watch time in 2012 and watch time has been ceding ground to predicted satisfaction ever since.

Which means the most important line of code in the system is not a model. It is a config. And every platform now racing to build the next feed is going to relearn, exploit by exploit, why YouTube's config looks the way it does.