The part most engineers miss is that this shift changes what's worth learning. Hand-crafted feature pipelines were a career for years; now the leverage sits in whoever can frame behaviour as text the model can interpret.
That reframing skill is quietly repricing engineering salaries far outside Netflix.
Worth watching as a comp signal, not just an architecture one.
Prefill-only scoring is the quiet hero here. Once you drop autoregressive decode, the cost model looks a lot more like a dense ranker than a chat completion — and prefix caching + aggressive context engineering suddenly matter more than model size. The cold-start path still feels underspecified to me: when a title is hours old and has almost no co-occurrence trail, do you lean harder on verbalized metadata in the prompt, refresh only the catalog embeddings, or keep a separate retrieval lane that bypasses GenRec until the title has enough engagement signal?
The offline/online split is the part worth sitting with. Offline ranking metrics tell you the model learned something; only the live experiment tells you users actually keep watching. I've seen the same gap in my own eval work: a model can look great on a labeled set and still miss in production because the labels don't capture what users do. The inference-cost discipline here is the other lesson. Prefill-only inference, prefix caching, tight token budgets: that's the unglamorous work that decides whether a good model is a shippable one. Measure cost per request like a first-class metric, not an afterthought.
A ranking head over catalog embeddings can't recommend a title Netflix doesn't carry, and prefill-only means there's no decode bill at all. With a 0.006% long-term lift, I'd read the real win as retiring thousands of hand-built features, not a sharper ranker.
The part most engineers miss is that this shift changes what's worth learning. Hand-crafted feature pipelines were a career for years; now the leverage sits in whoever can frame behaviour as text the model can interpret.
That reframing skill is quietly repricing engineering salaries far outside Netflix.
Worth watching as a comp signal, not just an architecture one.
Prefill-only scoring is the quiet hero here. Once you drop autoregressive decode, the cost model looks a lot more like a dense ranker than a chat completion — and prefix caching + aggressive context engineering suddenly matter more than model size. The cold-start path still feels underspecified to me: when a title is hours old and has almost no co-occurrence trail, do you lean harder on verbalized metadata in the prompt, refresh only the catalog embeddings, or keep a separate retrieval lane that bypasses GenRec until the title has enough engagement signal?
The offline/online split is the part worth sitting with. Offline ranking metrics tell you the model learned something; only the live experiment tells you users actually keep watching. I've seen the same gap in my own eval work: a model can look great on a labeled set and still miss in production because the labels don't capture what users do. The inference-cost discipline here is the other lesson. Prefill-only inference, prefix caching, tight token budgets: that's the unglamorous work that decides whether a good model is a shippable one. Measure cost per request like a first-class metric, not an afterthought.
A ranking head over catalog embeddings can't recommend a title Netflix doesn't carry, and prefill-only means there's no decode bill at all. With a 0.006% long-term lift, I'd read the real win as retiring thousands of hand-built features, not a sharper ranker.