Transcription
Breaking news. Google potentially solved one of AI's most expensive problems.
Every time you're deep in a conversation with the language model, say 20 or 30 messages in, the model is holding that whole conversation in its active memory. That memory is called KV cache, and it grows with every message. Meaning, the longer you have a conversation, the more memory it eats up, and things become slower and more expensive. This is one of the biggest bottlenecks in AI right now.
And just a few days ago, researchers at Google published a paper called Turboquan. And it works like this. Say you're 30 messages deep in with the language model. Every word in that conversation gets converted into vectors, which is just text converted into numbers. And normally each one of those numbers takes up 16 bits of memory. Now, if you multiply that across thousands of numbers across every single message, you're holding massive amounts of data in memory just to keep the conversation alive.
Turbo Quant introduces a simple compression that takes each one of those numbers down to just 3.5 bits. It's about five times smaller. And when the model reads them back, the output is near identical, meaning the model doesn't lose any context.
Now, I've seen a bunch of people making videos about this, saying that this will bring language models to your phone. I want to set this straight. To run a language model on your device, you need the model weights to fit in memory. You need a chip that's fast enough to run it and an effective KV cache. This paper potentially solves that last part, but running on your phone with just this is a crazy stretch.
This compression does a few things. Longer conversations, faster responses, and potentially cheaper to run on serverside AI. Follow and I'll keep you posted.