Shawn NgRecommendation Is Vector Search With a Query You Never Typed
Type something into YouTube's search box and it hands you videos. Open the homepage, type nothing at all, and it hands you videos anyway. Those look like two different products, and underneath they are one operation: turn a query into a vector, turn every item in the catalog into a vector of the same shape, return whatever sits closest. The field calls this dense retrieval, and the only thing separating the two cases is who wrote the query. In search you wrote it. In recommendation the system wrote one on your behalf, out of your watch history, your device, your country, and the hour of the day. The first time I traced a homepage request and a search request through the same service and watched them converge on the same index call, the recommender stopped being a mysterious object to me and became something far more tractable: a search engine that has to guess the query first.
Check out the full video on my YouTube channel Divide and Quantum.
Everything below is grounded in published work: Covington, Adams and Sargin in 2016, Yi and colleagues in 2019, Guo and colleagues in 2020, and the approximate-nearest-neighbor literature sitting underneath all three. This is the retrieval half of the story only. The ranking stage that scores the survivors got its own post, and where I am inferring from public research rather than reporting confirmed internals, I will say so.
Two Ways to Ask the Same Question
Write both problems down formally and they collapse into each other. A search engine takes a string, runs it through an encoder, and gets a vector. A recommender takes a user state (recent watches, longer-term history, device, country, hour) runs it through an encoder, and gets a vector. From there down, nothing distinguishes them:
search: query text -> encoder -> q # you wrote it
recommendation: user state -> encoder -> q # the system wrote it
both: return the top k items by q · v # identical from here down
The index does not know which of you authored the query, and would not behave any differently if it did. That is why the same machinery keeps surfacing in places with nothing else in common: the product rail under an Amazon listing, Spotify's daily mixes, the document lookup sitting in front of a language model. Karpukhin and colleagues' Dense Passage Retrieval for Open-Domain Question Answering (EMNLP 2020) is nominally a question-answering paper, and its retrieval stage is the same object a recommender uses to fill a homepage.
One honest qualifier before the geometry swallows everything. Lexical retrieval did not die: BM25 still beats dense methods out of domain, which is the headline result of the BEIR benchmark (Thakur et al., NeurIPS 2021). Search systems keep it as a fallback and blend the two. A recommender gets no such luxury, because there is no text to match against. It is dense all the way down, and every weakness in the geometry is one it has to live with.
Meaning as Coordinates, and the Negatives That Carve Them
Sparse retrieval represents an item by which words it contains: a vocabulary-length vector of mostly zeros, where every dimension carries a name you can read. Dense retrieval represents that same item as a few hundred continuous numbers, none of which corresponds to anything nameable. Computer vision ran this arc first, when networks trained end to end retired decades of hand-designed feature descriptors, and retrieval ran it a few years behind. The trade buys an enormous amount: similarity becomes arithmetic, one dot product per candidate, cheap enough to do a hundred million times. You pay for it permanently in explanation. You cannot grep an embedding.
Nothing in the architecture produces that geometry. Training produces it, contrastively: pull the vectors of pairs that really happened toward each other, push apart pairs that never did, repeat across billions of examples. The negatives do the real work here, which is easy to miss. A model rewarded only for pulling true pairs together learns to map everything to a single point and scores perfectly. The pushing is what creates structure.
Which makes negative sampling closer to cartography than to a training detail. Draw negatives from the batch, the cheap and obvious approach, and an item appears as a negative in proportion to how often it appears at all. Popular items get shoved away from everything far harder than rare ones, and the space quietly acquires a bias against exactly the content most people want. Yi and colleagues (RecSys 2019) estimate item frequency in a streaming fashion and subtract its log from the sampled logits to correct for this. It usually gets filed as a sampling fix. It is more honest to call it a repair to the map, because the uncorrected space is not a noisier version of the right answer, it is the wrong shape.
Nobody Scans a Hundred Million Vectors
Start with what the index exists to avoid, which is reading the entire corpus on every request. Published YouTube corpus figures have been climbing: "millions" in Covington et al. (2016), "tens of millions" in Yi et al. (2019), whose actually-served index was around 10 million videos, and "O(100) millions" in Singh et al. (2024). That is short of the billions usually quoted, including by me in the ranking post before I went back and checked the sources. Take the most recent figure and the retrieval towers' published output width of 128 dimensions:
100M items x 128 dims x 4 bytes = 51.2 GB read per query
at ~50 GB/s of usable bandwidth = ~1 second, for one query, one user
The bandwidth figure is my assumption rather than anyone's spec, and a fat multi-channel server will beat it several times over. It does not matter. Covington et al. put the serving budget in the tens of milliseconds, for every simultaneous viewer on Earth, and no amount of memory bandwidth closes a gap that size. A hash table would give you exact lookup in constant time by turning a key into an address, but similarity is not equality: you cannot hash a vector into the bucket holding its neighbors, because finding the neighbors is the entire question.
HNSW (Malkov and Yashunin, posted to arXiv in 2016 and published in IEEE TPAMI vol. 42 no. 4 in April 2020) is the answer most of the industry reaches for. Picture the corpus wired into a layered graph. The bottom layer connects every point to its near neighbors, each layer above holds a thinner sample, and the top layer is a sparse highway where a single edge crosses most of the space. Searching is graph traversal: enter at the top, step to whichever neighbor sits closer to the query, drop a layer when no neighbor improves, and repeat until you bottom out. You touch a few hundred to a few thousand nodes instead of a hundred million, and the cost goes polylogarithmic. It is the bargain A* makes, using structure to avoid exploring the whole graph, minus the guarantee. Greedy descent stops in local minima, and the recall you lose is the price of the ticket.
ScaNN (Guo et al., Accelerating Large-Scale Inference with Anisotropic Vector Quantization, ICML 2020) takes the other road entirely: partition the space into clusters, probe a few of them, and compress everything with quantization. The clever part is the loss function. Ordinary quantization minimizes reconstruction error equally in all directions, which is the wrong objective for an inner product, because error parallel to the datapoint wrecks the dot product while error perpendicular to it barely registers. Anisotropic vector quantization penalizes the parallel component harder and buys real fidelity at the same bit budget.
So which road does YouTube take? A Google Cloud codelab calls ScaNN "the same technology behind Google Search, YouTube, and Google Play," which is developer marketing and names no specific system. The better evidence is in the papers: Yi et al. describe their retrieval index as tree-and-quantization approximate maximum-inner-product search with learned coarse and product quantizers, citing the Google quantization work ScaNN descends from. That is the quantization road rather than the graph road, stated by the team that built it. The memory arithmetic points the same way, since HNSW's graph costs roughly 60 to 450 bytes per object on top of the vectors, a tax that overtakes the payload the moment those vectors get compressed. Covington et al. then add a deflating footnote to the whole argument: their A/B results were not particularly sensitive to which nearest-neighbor algorithm they used. The index is not an optimization layered onto the geometry. It is the permission slip that lets the geometry exist at all.
It also runs on a different clock from everything around it, and the split is purely economic. Item vectors are expensive, so they are stale by construction: Yi et al. rebuild their index every few hours, explicitly so that freshly uploaded videos enter the corpus at all. The query vector is a single forward pass, computed now, on this request, and the context wrapped around it (the hour in your timezone, what is trending this minute, what you clicked ninety seconds ago) costs almost nothing and is always current. That split explains the most misread behavior in these systems. You finish a video, the homepage reshuffles, and it feels like the machine just learned something about you. It did not. The item vectors are hours old and your long-term representation is not retrained between page loads. What moved was the query.
Cold Start, and Buying the Coordinates You Lack
Cold start usually gets described as missing data, which undersells how awkward it is. A video uploaded five minutes ago does get a vector. The question is whether that vector means anything. Where the item tower is content-derived, it faithfully encodes what the video is about and nothing whatsoever about who wants to watch it, which is the only question retrieval actually asks. Where it is ID-derived, and YouTube's published retrieval tower is, unseen video IDs land in random hash buckets, so the vector encodes neither. That second failure is precisely the motivation Singh and colleagues give for replacing random hashes with learned semantic identifiers. A brand new user is the same problem inverted: no history means no input to encode, so the system falls back on locale, device and raw popularity.
The uncomfortable part is that nothing in the geometry marks the difference. A vector inferred from a title and a vector distilled from ten million watch events are the same shape, sit in the same index, and get scored by the same dot product. The uncertainty is real, and the representation has nowhere to put it. A cold-start item is not a missing row in a table, it is a confident coordinate with no evidence underneath it.
If uncertainty cannot live in the vector, it has to live in the policy that decides what to show. That is the contextual bandit framing, whose canonical form is LinUCB (Li, Chu, Langford and Schapire, A Contextual-Bandit Approach to Personalized News Article Recommendation, WWW 2010), evaluated on Yahoo's front page for a 12.5% click lift over a context-free bandit baseline:
choose the item maximizing: x · θ # what the model predicts
+ α · sqrt(x^T A^-1 x) # how little the model knows
That second term is the whole idea. An item earns a slot either because it scores well or because the system genuinely does not know how it scores, and since A accumulates the features already served, the bonus shrinks along directions you have been probed on before. Exploration is not generosity toward new creators, and it is not diversity as a virtue. It is the system buying coordinates it does not have, with your attention as the currency. LinUCB itself is a Yahoo news algorithm with no published connection to YouTube, and the folk wisdom that exploration is a tax paid now for accuracy later is contested by Google's own recommender group, who argue the near-term value is real and measurable (Chen et al., RecSys 2021, on an unnamed industrial platform). Either way, every feed you scroll contains a few cells of an experiment nobody asked you to join.
Filter Bubbles Are a Property of the Metric
Your query vector is a function of your history, so it moves when your history moves. Watch three cooking videos and the encoder input shifts, the query lands further into the cooking region, items near it score higher, more get shown, more get watched, and the query moves again. Nobody wrote that rule down anywhere. It is what a fixed point looks like from the inside.
The trap tightens because neighborhoods are not uniformly dense. Popular regions hold more items, so as your query drifts into one, the top few hundred neighbors come from an ever narrower slice of the space, and anything outside that cluster has to outscore several thousand things inside it just to be seen. Escape requires being shown something far away, and the probability of that falls off with distance. Positive feedback on distance is more or less the definition of a trap.
That is an argument about mechanism, not a measurement of outcome, and the difference matters. Whether real feeds narrow real people is an empirical question the geometry does not settle. What the geometry does settle is intent. Nobody built the bubble. The metric has one, and you are standing in it.
When the Index Becomes a Language Model
The most interesting recent work does not improve the geometry. It walks away from it. In Recommender Systems with Generative Retrieval (Rajput, Mehta and colleagues, NeurIPS 2023), each item gets a semantic ID: a short sequence of discrete codewords from a residual-quantized variational autoencoder, three of them plus a fourth to break collisions. Retrieval then stops being a nearest-neighbor problem and becomes sequence prediction. Feed a transformer the semantic IDs of what someone recently watched and have it decode the next one, the way a language model decodes the next token, except the language it speaks is items. The framework is called TIGER, for Transformer Index for Generative Recommenders, and the word index in there is the radical part. There is no data structure. The index is the weights.
Now the deflating numbers, because this is exactly the sort of result that gets oversold. TIGER was evaluated on three Amazon Product Reviews categories holding roughly 12,000 to 18,000 items each, with no live experiment and no YouTube data anywhere in it. Saying it is "not yet running at YouTube scale" flatters it considerably. What has reached a production model is the representation rather than the retrieval: Singh et al. (RecSys 2024) swapped random-hash video IDs for semantic IDs inside YouTube's production ranking model and improved generalization on new and long-tail videos, in offline experiments on held-out logged data. The paper reports no launch. Semantic IDs have been tested inside a production model; generative retrieval remains a research result on Amazon reviews.
It is still worth watching, because it is where retrieval and language modeling stop being separate disciplines, and that should make you cautious for a fairly ordinary engineering reason. Nearest-neighbor search carries one guarantee you never think about until it is gone: it cannot return an item that does not exist. A decoder can.
The Bet Underneath All of It
Every piece above descends from a single decision, made decades ago and rarely revisited: meaning is coordinates, and relevance is proximity. Commit to that and most of the architecture stops being a design space and becomes a consequence. There are two towers because the item side has to be precomputable and indexable. There is an approximate index because coordinates you cannot scan are coordinates you cannot use. There are two clocks because geometry is expensive and context is free. Cold start is a point with no honest coordinates, exploration is how the system buys the ones it lacks, and a filter bubble is what a point does when every step drags it toward the mass of its own neighborhood.
The ranking post ended on the objective function being the product, and for the ranker that is exactly right. One layer down it is not. The objective decides what the system wants; the coordinate system decides what it can reach, and no weighting of any task will retrieve a video the index never offered. The geometry quietly caps your ceiling months before anyone thinks to look at the loss curve.
Which is why generative retrieval deserves attention regardless of what becomes of TIGER specifically. It is the first serious proposal to stop using coordinates at all. The next retrieval architecture may not have a space you can point at. It will have a vocabulary.