This video discusses a new optimization technique developed by Google called "frozen multi-token prediction" (MTP) that significantly speeds up the Gemini Nano model on Pixel devices without requiring additional training or a larger model. The core idea is to freeze the main Gemini Nano model and attach a small prediction head to its final layers. This head leverages the model's existing internal activations and key-value cache, eliminating the need for extra copies or wasted prefill steps. By doing this, the model can generate multiple tokens in a single pass, resulting in a bit-for-bit identical output to the original, slower method because the main model still validates each token. The benefits observed on Pixel devices include over 50% faster generation, nearly two additional tokens accepted per pass, up to 55% improved acceptance rate for smart replies, and a saving of 230 megabytes per instance due to the elimination of a separate draft model. This approach is presented as superior to previous methods like speculative decoding and Eagle, which rely on a secondary model for prediction. Frozen MTP, in contrast, efficiently reuses the existing model's capabilities. The technology is already powering features like notification summaries and proofreading on Pixel 9 and Pixel 10, offering performance improvements and longer battery life without user intervention. Google plans to further enhance performance with future techniques like parallel decoding and branching verification. The video concludes by posing a question to the audience about whether this is a practical, real-world improvement or merely a benchmark enhancement.

DIY Smart Code

Google just made Gemini Nano on Pixel over 50% faster. No new training, no bigger model. They built it on one tiny trick. And the output stays exactly the same.

It is called frozen multi-token prediction. Normally, a model writes one token at a time. That is the slow part. So, Google freezes the whole Gemini Nano backbone. Then they attach a tiny prediction head to the final layers. That head taps the model's own internal activations. It even shares the frozen key value cache. So, there is no extra copy and no wasted prefill.

Now, the model drafts several tokens in a single pass. And because the main model still checks every token, the result is bit-for-bit identical. The numbers on a Pixel 9 are big. Over 50% faster generation. Nearly two extra tokens accepted per pass. Up to 55% better acceptance on smart replies. And 230 megabytes saved per instance because there is no separate draft model.

This beats older tricks like speculative decoding and Eagle. Those bolt on a second model to guess ahead. Frozen MTP just reuses the brain that is already there. It already powers notification summaries and proofread on Pixel 9 and Pixel 10. And you do not need a research lab. Your Pixel just gets faster on-device models and longer battery for free.

Next, Google wants parallel decoding and branching verification to push it even further. So, is this a real-world win or just a benchmark flex? Drop your verdict below.

This frozen MTP trick just changed on-device AI forever #GeminiNano #AI #GooglePixel

Google just made Gemini Nano on Pixel
over 50% faster. No new training, no
bigger model.
They built it on one tiny trick. And the
output stays exactly the same.
It is called frozen multi-token
prediction.
Normally, a model writes one token at a
time. That is the slow part.
So, Google freezes the whole Gemini Nano
backbone. Then they attach a tiny
prediction head to the final layers.
That head taps the model's own internal
activations. It even shares the frozen
key value cache. So, there is no extra
copy and no wasted prefill.
Now, the model drafts several tokens in
a single pass. And because the main
model still checks every token, the
result is bit-for-bit identical. The
numbers on a Pixel 9 are big. Over 50%
faster generation. Nearly two extra
tokens accepted per pass. Up to 55%
better acceptance on smart replies.
And 230 megabytes saved per instance
because there is no separate draft
model.
This beats older tricks like speculative
decoding and Eagle. Those bolt on a
second model to guess ahead. Frozen MTP
just reuses the brain that is already
there. It already powers notification
summaries and proofread on Pixel 9 and
Pixel 10.
And you do not need a research lab. Your
Pixel just gets faster on-device models
and longer battery for free.
Next, Google wants parallel decoding and
branching verification to push it even
further.
So, is this a real-world win or just a
benchmark flex? Drop your verdict below.

Get Our Mobile App

This frozen MTP trick just changed on-device AI forever #GeminiNano #AI #GooglePixel

Transcription