The 37.5% lines up exactly with the 5:3 split, since 3 of every 8 layers stop storing their own keys and values. I'm curious whether the quality cost shows up mostly on long context tasks, because all of those later layers attend through one fixed layer's keys. Does it also save prefill time, since those layers can skip their K and V projections entirely?
This is a brilliant explanation and I loved reading through it, highly appreciate the detail explanation on how well balanced the model and computer are engineered.
The number that matters here is not 3 billion. It is 1.39 gigabytes. A 3-billion-parameter model at 16-bit precision needs almost 6GB; palettization gets it to 1.39GB, and the later build pushes the decoder to 2 bits a weight. That is not just an engineering trick. It is a pricing decision. Every token generated on the phone is a token that is never billed by a cloud provider. Apple has vertically integrated inference the same way it vertically integrated silicon: the metered service becomes a sunk cost inside a device it already sold.
That is the fork this piece is really about, and it runs through the whole AI trade. The cloud model depends on inference being a metered service — tokens billed per use, which is the revenue the hyperscalers need to justify the buildout. On-device inference breaks the meter. Once the model ships in the phone, the marginal cost of a query is battery, not a bill. And AFM 3 Core Advanced points the same way: a 20-billion-parameter sparse model that activates only 1 to 4 billion at a time, with expert weights loaded from flash on demand. The direction is 'activate only what you need,' which is the opposite of the cloud model, where you pay for the whole model on every token.
The Private Cloud Compute section is the hedge, and it is worth reading as one. Apple still needs the cloud for the hard queries — and it now runs PCC workloads on NVIDIA GPUs in Google Cloud, which means it is renting the very infrastructure the AI trade is built on while keeping the cheap work on the device. The question is where the boundary moves. Quantization-aware training, cross-layer KV sharing, sparse experts: each of these moves more work onto the phone. The cloud keeps only the queries the phone cannot do, and that set shrinks every generation.
So the investor question is the one the piece leaves hanging. The hyperscalers' buildout assumes inference demand grows with usage. On-device inference is demand that never reaches their data centers. If the most common queries — summarization, extraction, classification, the everyday work — never leave the phone, what exactly is the cloud selling to the rest of the market? The piece presents the on-device/cloud boundary as a technical decision. It is also a revenue decision, and Apple just voted with its silicon.
The 5:3 layer split makes sense for kv cache, but the handoff between on device and Private Cloud Compute is where the real tradeoff lives. What signal does Apple use to decide a query leaves the phone, latency budget or just prompt length, and does the routing decision itself run in a tiny on device model?
The on-device check and Private Cloud Compute are two different models, not one model that sometimes leaves the phone. The piece says the on-device model handles summarization, extraction, and classification, and a more complex request is encrypted for validated PCC nodes, where a separate server model runs it. AFM 3 Core Advanced stores the full sparse model on flash and loads only the expert weights the prompt selected. I would record which boundary answered, and which experts were loaded, next to the completion. Otherwise a later mismatch looks like the phone model changed when the request actually crossed into PCC, or when a different expert set was pulled into RAM.
Nice breakdown. The part about the model deciding when to use the cloud is the one I'd like to see more of, since that's where privacy and speed trade off for everyday users.
With decoder weights at 2 bits, the eval that counts is the one run on the palettized build with each adapter loaded, because the fp16 checkpoint can pass a feature the shipped model fails. Long documents deserve their own slice of that eval, since that's where the cross-layer KV sharing does the most work.
The weights-versus-KV-cache split is the part I'd turn into a device test matrix. I'd measure peak memory for a short request, a long input, and repeated requests while switching adapters, alongside time to first token. A model that loads successfully can still exceed its working budget later. That would make the runtime-resources conclusion testable without treating the compressed weight size as the app's full memory cost.
well well.. no I will not be joining that webinar. .very interesting information.. I will send that to someone who can figure that out.. and writes App I do not like PHONE APPs. at all.. I use them only when necessary but it is too late.. my phone communicates w/ the laptop
RIGHT THAT IS REAL APP CODING TALK over my head for sure won't even go there.
The 37.5% lines up exactly with the 5:3 split, since 3 of every 8 layers stop storing their own keys and values. I'm curious whether the quality cost shows up mostly on long context tasks, because all of those later layers attend through one fixed layer's keys. Does it also save prefill time, since those layers can skip their K and V projections entirely?
This is a brilliant explanation and I loved reading through it, highly appreciate the detail explanation on how well balanced the model and computer are engineered.
The number that matters here is not 3 billion. It is 1.39 gigabytes. A 3-billion-parameter model at 16-bit precision needs almost 6GB; palettization gets it to 1.39GB, and the later build pushes the decoder to 2 bits a weight. That is not just an engineering trick. It is a pricing decision. Every token generated on the phone is a token that is never billed by a cloud provider. Apple has vertically integrated inference the same way it vertically integrated silicon: the metered service becomes a sunk cost inside a device it already sold.
That is the fork this piece is really about, and it runs through the whole AI trade. The cloud model depends on inference being a metered service — tokens billed per use, which is the revenue the hyperscalers need to justify the buildout. On-device inference breaks the meter. Once the model ships in the phone, the marginal cost of a query is battery, not a bill. And AFM 3 Core Advanced points the same way: a 20-billion-parameter sparse model that activates only 1 to 4 billion at a time, with expert weights loaded from flash on demand. The direction is 'activate only what you need,' which is the opposite of the cloud model, where you pay for the whole model on every token.
The Private Cloud Compute section is the hedge, and it is worth reading as one. Apple still needs the cloud for the hard queries — and it now runs PCC workloads on NVIDIA GPUs in Google Cloud, which means it is renting the very infrastructure the AI trade is built on while keeping the cheap work on the device. The question is where the boundary moves. Quantization-aware training, cross-layer KV sharing, sparse experts: each of these moves more work onto the phone. The cloud keeps only the queries the phone cannot do, and that set shrinks every generation.
So the investor question is the one the piece leaves hanging. The hyperscalers' buildout assumes inference demand grows with usage. On-device inference is demand that never reaches their data centers. If the most common queries — summarization, extraction, classification, the everyday work — never leave the phone, what exactly is the cloud selling to the rest of the market? The piece presents the on-device/cloud boundary as a technical decision. It is also a revenue decision, and Apple just voted with its silicon.
The 5:3 layer split makes sense for kv cache, but the handoff between on device and Private Cloud Compute is where the real tradeoff lives. What signal does Apple use to decide a query leaves the phone, latency budget or just prompt length, and does the routing decision itself run in a tiny on device model?
The on-device check and Private Cloud Compute are two different models, not one model that sometimes leaves the phone. The piece says the on-device model handles summarization, extraction, and classification, and a more complex request is encrypted for validated PCC nodes, where a separate server model runs it. AFM 3 Core Advanced stores the full sparse model on flash and loads only the expert weights the prompt selected. I would record which boundary answered, and which experts were loaded, next to the completion. Otherwise a later mismatch looks like the phone model changed when the request actually crossed into PCC, or when a different expert set was pulled into RAM.
Nice breakdown. The part about the model deciding when to use the cloud is the one I'd like to see more of, since that's where privacy and speed trade off for everyday users.
With decoder weights at 2 bits, the eval that counts is the one run on the palettized build with each adapter loaded, because the fp16 checkpoint can pass a feature the shipped model fails. Long documents deserve their own slice of that eval, since that's where the cross-layer KV sharing does the most work.
The weights-versus-KV-cache split is the part I'd turn into a device test matrix. I'd measure peak memory for a short request, a long input, and repeated requests while switching adapters, alongside time to first token. A model that loads successfully can still exceed its working budget later. That would make the runtime-resources conclusion testable without treating the compressed weight size as the app's full memory cost.
well well.. no I will not be joining that webinar. .very interesting information.. I will send that to someone who can figure that out.. and writes App I do not like PHONE APPs. at all.. I use them only when necessary but it is too late.. my phone communicates w/ the laptop
RIGHT THAT IS REAL APP CODING TALK over my head for sure won't even go there.