Transcription
Google just published one of the most important breakthroughs of the year. It's called TurboQuant. And it has everything to do with how we use memory in LLMs. Not just in agents, not just in applications, but like the core LLM architecture becoming more memory efficient. And that is a huge deal because right now one of the biggest crises in the industry is the fact that intelligence and demand for intelligence are scaling way way faster than memory.
And if you're hearing this and you're thinking TurboQuant, this sounds like Silicon Valley the television show. You're absolutely right. In fact, that's what most of the newspapers said. They said Turboquant is the new Pied Piper. They were kidding around because in the television show Silicon Valley, the little startup Pied Piper actually makes a compression algorithm that saves a ton of hard disk space and memory and that's how they're valued and that's like their whole startup thesis.
In this case, I would argue it's even more valuable because TurboQuant compresses the way LLMs handle processing of text in a way that is lossless. And that's a big big deal. So, Pi Piper compressed video files and Turbo Quant compresses the memory that LLM use to think called the key value cache or the KV cache. It's the thing that determines how much an AI can hold in its head at a time. And what TurboQuant showed is that they can do a six times memory reduction in the KV cache and up to an 8x speed up on chip without losing even one bit of data.
That's the the biggest news in the world, right? Like that is a huge deal from an AI perspective because it means that the structural economics of the memory crisis that we've been walking into for the last couple of years may actually be addressable. Now, just to give you the 30 secondond view of the memory crisis, supply is structurally constrained. HBM, high bandwidth memory, is getting harder and harder and harder to make. Partly that's because there's so much demand for it. And partly because it's literally harder to make because the conflict in Iran means that there's less helium and more expensive power prices, both of which impact the ability to make memory. And so, for multiple reasons, memory is very hard to make right now.
On top of that, demand is exploding because agents happened. And that means that the average conversation length or the average token usage for a particular interaction went from how long it takes you to have a chat to a thousandx that because agents can burn so many tokens. It's not unusual for agents to burn 100 million tokens, even a billion tokens. Now that means there's more and more demand for exactly the working memory that this Google paper addresses. And just to underline how big a deal this is, token consumption is already reaching 25 billion tokens a year for enterprises with AI native workers. That's that's a year per engineer just to be clear, not for the enterprise as a whole. And also memory prices are soaring. Memory prices are multiple hundreds of percent up which is increasing the bill of materials and costs for everything that we use for computing including our personal computers. And the squeeze is relentless. Like we are looking at a situation where for the next half decade this is going to be difficult because bringing more fabrication units online is not easy. So that's the problem space. That's why everyone's stressed about memory. In the middle of this turbo quant looks like a possible way out.
Now, I grant you it's a working paper. It's not yet in production systems. I don't want to overpromise, but it's worth understanding how it works because it starts to paint a picture for how we can use memory more efficiently. And that's a big deal. Look, traditional methods for compressing AI memory are really, really problematic. I want to go through a couple and then explain without math why Turbo Quant is so much better. So if you wanted to compress AI memory before Turbo Quant came out, something that you could do would be called vector quantization, which is a fancy way of saying that you can compress data, but then you need to add data to make sure the data is easily retrievable. So you add something called quantization constants, which sort of somewhat defeats the purpose of the compression because you're adding more data back in after you press it. And that overhead actually literally adds one to two extra bits per number that you compress. And it sort of defeats some of the purpose of the compression. It's like packing a suitcase by folding everything tightly, but you have to carry a separate bag with the folding instructions, right? That would be sort of silly, but that's a little bit like what vector quantization does.
Turboqu Quant makes things easier. So, TurboQuant eliminates that overhead, the packing instruction, so to speak, in a couple of stages. First, Polar Quant rotates the data into a standard coordinate system. So what I mean by that is that the data structure becomes predictable enough that you don't need special normalization to read it per block going through the LLM transformer head. So think of it as converting go three blocks east and four blocks north into go five blocks at a 37 degree angle. Both of those are technically the same thing, but one is a shorter way to say it. And so the radius captures the signal strength and the angles capture the meaning. And it's a a more efficient way to pack up that data. in this analogy because the angles capture all of that data. You don't need to carry the extra bag of folding instructions and it's just a clean, lossless, more efficient way to carry data.
But we're not done there because the second technique is what makes this really brilliant. Even if you compress it and you make sure that you like carefully represent all of the original data in a slightly smaller form, it's still possible for tiny errors to creep in. And if you're an LLM and you want to be not tolerant of tiny errors, then that's unacceptable because you actually have to do long running steps over many many layers of context and it's really really important to get it exactly right. Let's say in our example that three blocks east and four blocks north translated to 37°. It's actually 36 and a half. It's not 37, but it's close enough for most purposes. Well, Google went farther. Google didn't just say this is close enough for most purposes. It's almost perfect. They also added a second technique called QJL or if you want to have a tongue twister, quantized Johnson Linden Strauss. Say that five times fast. That's basically a fancy name for a process that takes the tiny residual error that 36 12 versus 35 degrees or whatever it is. And it corrects it and it corrects it efficiently using just a single bit, a mathematical error checker that eliminates the bias and attention scores. And the combination leads to net net zero overhead and a perfect compression.
And so the result is eye opening. So where you have a KV cache that might have 16 or 30 bits in it for a key value, you can compress that up to 10x from 32 down to three bits of value without any loss. And that was tested across a variety of fields that we care about for LLM. So if you're wondering, is this just theoretical? I mean, yes, it's a paper, but it was tested. So it was tested across question answering, it was tested across code generation, it was tested across summarization, and it was tested critically across needle in a haststack retrieval. So if you have a big piece of context and you do this compression on it, can the LLM still find a specific tiny word or phrase in that gigantic context? And so they ran and threw a 100,000 traditional tokens at this compression system and TurboQuant compressed it. And then they said, "Okay, now can you find this tiny little phrase that we've put into this 100,000 tokens?" And it could. And the beautiful thing about this is that it's what we call a data oblivious algorithm. So it's not specific to a specific data set. It's not specific to a specific large language model. It's actually a mathematical property we're working with here, which makes it more easy to translate.
Now, if you're wondering, okay, this sounds great. Why don't we all have it? The answer is that rolling something to production takes time, and it's important to understand how it actually works. I'm going to give you an example of that. Here's why this matters and why we need to think beyond the spec sheet to make sure we get this right. When you compress the KV cache by 6x or 8x or 10x, however big the number ends up being in production, you don't just save memory, you change the way concurrency math works on a chip. In other words, you change the number of simultaneous users that a single GPU can serve. And this is the number that determines whether the inference workloads that you're running end up being profitable for you with your GPU investment or not. But one of the things that's interesting is if the concurrency number gets high, you may have to change the way your enterprise deployments work, the way your firmware works on top of the chips to enable more concurrency on a single chip because chips typically have concurrency limits that they may have put in place before. In fact, certainly put in place. chips have concurrency limits typically that have been put in place long before Turbo Quant came out and that you may have to think about how you address when you want to take advantage of this. And so one of the things I want to call out is that whenever you try and production scale something, you have to think about the whole stack. And especially if you're thinking about something as near to the metal as memory use in a KV cache, you have to think about all of the implications for the stack before you can roll it out. And that's why as much fun as this is, there's still work to be done before this is fully available for production.
And so you might say, Nate, well, why are we talking about this? It's just theory. Well, the answer is even if it's theory today, this is still the fastest possible path for us to solve this problem. And the reason why is that it can move at the speed of software. It doesn't necessarily have to move at the speed of hardware. Because if you're trying to fix these fab timelines, I talked about the issues with making HBM. It's a half a decade timeline, right? If you're trying to address how demand works in the system, I'm sorry, but demand is just exploding for AI and that's not going anywhere. So that's an immovable force that's getting bigger and bigger all the time. In that world, software is sort of our only way through the memory problem.
Now, if you're wondering what is a KV cache, I'm going to explain it really simply. KV cache is the memory for the language model. So it's the model's working memory while it does computation across a prompt. So every token the model has ever seen gets stored as a key value pair or in the KV cache. And the model computes over all of those pairs for every token generated. And so the KV cache is what lets a model connect token number 89,031 to token number 2354 in your giant prompt. It's what allows the model to hold a conversation, to follow an argument, to track a codebase. It's super super important. And so if model weights are effectively the processor that allows you to do the computing, the KV cache is like a hard drive. It's like RAM. It allows you to remember things. And so if you were to invent a piece of software that effectively an algorithm that effectively compresses and makes the hard drive you already have potentially six, seven, eight times more efficient. That's a really really big deal.
And I think that one of the things I want to call out is that this TurboQuant paper is happening in the context of a larger set of innovations that are around the core architecture of LLMs that we should be paying attention to. I'll give you one more example. This is from a company called Percepa, which is figuring out how to embed a computer inside a large language model. Now, one of the things I want to call out for people who are like, "But an LLM is a computer." The answer is actually no. It's not a computer. The LLM is a neural network architecture and it's inherently probabilistic. And so it does not compute in the classical sense normally. And so when you see great results where like the LLM does math these days, what you're really seeing is the LM calling a tool and using a tool like Python to do the math. And that's how that works. But that may not be how that works in the future.
So what they figured out is they could get the model to deterministically solve a Sudoku puzzle by actually computing the answer logically step by step with 100% accuracy over a lengthy number of steps. We're talking a million plus steps at 33,000 tokens a second which is very very fast. If you want to get into the details, what they did is they compiled a web assembly interpreter directly into the weight matrix of a standard PyTorch transform. Not as an external tool call, not as a code interpreter sandbox that was running alongside the model that it could use, but actually they compiled the computer inside the weights of the model. And so the model can execute C programs through a forward pass step by step and emits a stack trace as tokens. For the nerds out there, that's how they did it.
And the implications of having a computer in your LLM are really interesting. Not because again all of our LLMs will immediately have this, the production piece comes out here too. But because if you look at the combination of the memory piece where Google's pushing, the computer piece that Percepa is pushing, you start to see a changing capability envelope. What happens in 6 months or 8 months if our LLMs can now run native compute inside the LLM weights as something they can invoke when they need to run particular programs and they don't need a tool call and what happens if that is also something that is super efficient because the KV cache has been compressed six or eight times and now you can do more. What you're looking at if you start to chain together some of these insights we're having at the cutting edge is a world of a stepchanging capability, right? A world where it's not the LLM itself getting smarter that makes the LLM better. It's the fact that the LLM architecture is changing. And so the LM is much better at memory, holds way more memory without effort, seemingly holds six, seven, 8x more memory without working too hard, and also at the same time, oh by the way, doesn't have to call tools to do computing anymore. That is looking a lot like a revolutionary change in architecture that we may start to see in the second half of 2026 as it starts to roll toward production systems.
This is how innovation works, right? You have a transformer architecture. There are a trillion dollars being poured into making this whole system better. And people are like constantly banging their heads on the wall figuring out how do we improve this? These two breakthroughs that I'm talking about are not the only ones, but I've picked them because they highlight where the industry is going as a whole. The industry is thinking about how you can make true computing possible and a more flexible architecture that doesn't require all of this tool calling. The industry is thinking about how you can efficiently solve memory. And when you put those two together, you get a capability breakthrough. That is a big big deal.
Now let's imagine a world where we start to see this breakthrough architecture. Just walk back with me and look at the strategic implication for a second. Google wins twice in this world. They wrote TurboQuant and they also run Gemini. And Google has explicitly stated that the KV cache is a bottleneck for Gemini. They've had trouble securing HPM high bandwidth memory. And so if they can actually get Turbo Quan implemented in Gemini, they effectively have a compounding cost advantage on top of their TPU stack and they are able to start to roll this out faster than anybody else because they had the breakthrough themselves originally. This also frees them from some of the competitive dynamic around acquiring memory which is going to be a structural advantage for them long term.
Now for Nvidia, the narrative gets complicated. Jensen spent GTC arguing that Vera Rubin's 500x memory increase solves the inference bottleneck. And Turbo Quant effectively says, "Why not just compress the cash and you get 6x more out of the GPUs you already have?" Well, Nvidia makes money selling chips. They kind of want you to buy more chips to solve that problem. If Nvidia ends up selling fewer chips because compression works really well, they've got a problem. Now, so far that hasn't been the case because the world needs so much AI and is demanding so much AI that no matter what we do, Jensen keeps selling more chips. But it is something that complexifies Nvidia's narrative a little bit as we start to move into a world where memory becomes more of a software fungeible constraint.
The other thing I want to call out is that middleware continues to not win here. So middleware is sitting on top of these foundation models. And if you're looking at where you acrew value, the foundation models are where you can acrue value. That's where the KV cache is being optimized. That's where the tool sets and tool calling capabilities are being optimized. And if you're sitting on top of these models, they may or may not pass their margins on to you. Certainly, they won't pass full margins on to you and you may still continue to get squeezed even as the foundation models reap the gains of these kinds of efficiencies. Enterprises on the other hand are in a really good spot because what enterprises can do is they can start to say, "Hey, we want more for our existing chips. How can we get more out of our existing chips? Can we use something like TurboQuant and actually implement the memory in a way that works?"
Now, one thing I want to call out here is you might walk away thinking TurboQuant is the only answer. TurboQuant is the only breakthrough on memory. And that is not true. There are other forces out there, other breakthroughs, other research papers out there on me. And so really what makes this whole memory conversation powerful, it's it's that it's not a single paper. It's actually the breadth of the attack on the memory problem as a whole. So smart people are hitting this from at least five distinct angles and I want to outline them briefly so you understand the broader landscape.
Quantization is how TurboQuant does it, right? They're representing the same vectors in the memory space in fewer bits. And that's the core insight, right? And before TurboQuant did this, KV demonstrated two-bit asymmetric quantization. If you want to get nerdy, zip cache has an approach here as well. So quantization is a stable vector of attack across at least three different research areas.
Eviction and sparity is actually a completely different strategy from Turboquan. Instead of compressing everything and throwing away the tokens that don't matter, H2O, which is uh Oracle's heavy hitter, keeps only the tokens that attention scores the highest and evicts the rest. Uh, Snap KV has an approach here where it does something similar during decoding. Streaming LLM keeps this sliding window of recent tokens plus a few attention sync tokens. So, essentially, eviction and sparity means we can pick the tokens that matter and keep those. It's not going to be lossless like Turboquant, but it is in production and it does help some. And there's there's three or four angles there as well.
Architectural redesign. Uh this attacks the problem at a model level. Deepseek v2 did this the first time. They introduced multi head latent attention which for the nerds projects keys and values into a lower dimensional latent space during training and then shrinks the KV cache footprint by design rather than after the fact. And there's been other work done here too, right? Hybrid architectures like IBM's Granite 4.0, uh Nvidia's Neotronh. These replace most of the quadratic attention mechanism, which is how LLMs traditionally work with uh what would we would call like a linear time state space model solution. In other words, it makes the memory problem smaller by definition because it's not quadratic attention. And these require training from scratch, right? You have to just train them from from the the get-go. And that limits immediate adoption, but it represents another direction the architecture is heading to make memory more efficient.
A fourth approach is offloading and taring, right? It keeps the full cache, but it shifts it around strategically. So, shadow KV will store compressed keys on a GPU and offload those values to a CPU, which allows you to achieve much larger batch sizes and uh very high throughput on the right chip. And it treats GPU memory and CPU memory as effectively a hierarchy, right? You can have RAM and solid state memory on your machine in the same way you have like GPUs and CPUs. Flex Gen is also going after this uh and they take it even further with really aggressive offloading to a disk for throughput optimized not latency optimized workload. So this is where you want like high throughput. You don't care how long it takes. And so that's a way to get memory off of the model's immediate KV cache but still make it accessible to the model.
Attention optimization uh is the fifth and final approach we'll talk about. It makes the computation itself cheaper without shrinking down the data. So flash attention is an example of how companies are attacking this problem, right? Uh flash attention restructures how attention which is what LLMs do reads and writes GPU memory to minimize input output reads and writes which in theory improves performance dramatically on NVIDIA chips. Percept has been in this business for a while. Uh Perceptor's whole KVach also goes after this. It uses two-dimensional attention heads to reduce attention complexity, which by the way, the two-dimensional attention heads are what made that millionstep computation that we talked about earlier with percepta possible. So, they've been working on this for a while.
The point here is not that any of these is quote unquote the answer. I don't believe in a silver bullet. The point is there are now dozens of research groups and companies attacking the memory problem as a software problem or attacking it algorithmically from multiple directions. And the results are going to compound as we continue to innovate. And this represents one of our most efficient ways to solve one of the biggest problems in AI and really in computing in society today. Memory is something that is blocking us on harvesting a lot of the value of AI. And it's something that I think we don't even realize because we can't even imagine a world where LLMs actually have excellent memory over a long term. Like we're excited that we have fragments of memory in chat GPT right now. We're excited that that that maybe Claude has a little bit of memory that it can remember us with and and maybe we can adjust that or edit that or maybe we can tell it to use an MCP server. Great. Memory is such a big deal that we can't imagine a world where the LM just is ambiently aware and has persistent memory over a long period of time. But that's where we're investing and that's where we're going. That is the long-term vision.
And the reason I made this video today is because the breakthroughs that we have, including Turboon, which I think is a huge deal, are going to enable us to start to build toward that world and ultimately build really interesting customer experiences, really interesting business experiences, unlock a world that feels more Star Trekian than the world we have today, more like there's just ambient compute and memory everywhere. But we have to build our way there. We have to innovate our way there. And it is extremely difficult to do that in a world where it's getting harder and harder to make memory in, which is the world we live in today. I hope this little tour, slightly nerdy down memory lane has been helpful for you. Uh, and I hope you have a sense of where the industry is going on solving some of these really complex problems.
The biggest takeaway for you if you're looking like what do I do and how can I apply this is pretty simple. Make sure that you have a plan for how you want to handle your own memory and context layer. There is going to be personal memory and context layer stuff out there. I've recommended like you should have an open source one because then no company owns it. Uh that's why I launched uh open brain as an open source protocol. But regardless, you should think of memory as a long-term constraint in your life and in the life of your company if you have one uh or if you work at a company. And you should be treating it that way. Treat it as something that is a yearslong problem. And you want to make sure that what you are storing is something that you control, that you're okay with, and that you can retrieve aically. That's really important because the alternative is some company deciding for you. And then in that world, if the LMS get better at memory, it's just easier for you because you can throw more at them and it's fine and you don't have to worry about it. If you take anything away, it's what I would call sovereign memory. You should own your memory. You should decide what your memory does. Somebody else should own it for you.
All right, best of luck and uh thank you Google for making a really cool breakthrough and uh allowing me to reference Silicon Valley, which is like the best Silicon Valley TV show. Chips.