Transcription
Yesterday, Google published a blog, uh, that was actually the culmination of three ongoing research papers, but this is going to change how local models work and how we use them. Frankly, like, this is going to revolutionize running a model on your device.
For those of you who don't know me, my name is Timothy Carbat, and I'm the founder of Anything LLM, which is basically this all-in-one app that does everything that you would expect from something like a ChatGPT. So, chat with documents, run AI agents, automations, do all that stuff, but we do it with local models. That's all we do. That's what we prioritize. That's like my whole focus. My day job is building this tool with local models. So, I'm particularly excited about this research, which, by the way, I'm not going to get into the math or any crazy nuance like that. There are other channels. I'm going to link their videos in the description where you can get a super heavy, maybe math or programming take. But what I'm going to talk about is the impact. Any definitions I think that need to be defined, I'll just explain.
So, what we're talking about today, and you've probably heard about it if you have any interest in local models at all, is something called TurboQuant. The TL;DR of TurboQuant is that this is going to enable people to not only run reasonably sized models on consumer hardware, which you can already do today, but with a much more practical context window. Your context window is basically your chat. This is fundamentally the thing that gives your LLM any amount of short-term memory. So, things it can recall from previous conversations. This includes the system prompt, any examples you gave it, your current chat history, uh, any tool calls or available tools, uh, any context from documents that got injected, just everything that the model is going to use to answer your question goes into this context window.
But that context window is exactly the limitation that is holding local models back from honestly giving a competitive experience to a cloud model like ChatGPT or Claude because there is something that is called the KV cache. Now, that might sound complicated. It really isn't. Just think of it as essentially the history of what you've been talking about. So, every time you send a chat, it will then check the KV cache, and then whenever it answers the prompt, it'll basically send that back into the cache. So that basically, this should improve the model's ability to answer and recall information that you've already talked about in that session.
The issue with this entire concept on local models is that this cache takes up a lot of memory because as you chat more, more gets into the cache, which grows the cache, and it takes up more of your GPU's RAM or your NPU's RAM or just your regular RAM. And also, keep in mind, you have to load the model itself. So, if you're talking about the 3B range at the top of this graph, these are basically idiot models. So, these are models that, you know, sure, you get the ability to get a larger context window, but it's basically just like having more conversations with a dumb person. The really where you want to be is anywhere around like 7 to 9. Um, 13 is a great middle ground for most consumers. 30B if you can do it. 70B is incredible. Anything greater than that is awesome. There's this whole new concept, or really not new, but it's becoming more popular, is a thing called a mixture of experts where you can basically get the smartness of a big model but only run the 3B part of the model that's relevant to your prompt. That's called or mixture of experts.
But in general, this can be a very demanding process, right? This is just to load the model. And for a basic context window of something like 8K or 16K, that's additional memory that needs to be allocated. So, think of the device that you're currently on, even a phone, right? You maybe have 12 gigs of RAM. Maybe you have a GPU, and that's anywhere from, let's say, 8 to 12. Maybe you got a high-end one, 32. Great. Most people are on modest devices. You know, they're sitting with like 32 gigs of RAM. Maybe they have an 8-gigabyte uh GPU. So, these are modest devices. The model is going to take up space.
But there is a part of this, the context window, that KV cache, that is what TurboQuant has optimized. And essentially, what this boils down to is being able to fit six times more tokens in the same amount of space, essentially. Now, of course, this depends on the model. It's still early days on this implementation, but all of the benchmarks are showing that this is working.
So, what I want to talk about is this graph. Now, you can ignore the graph on the left and the right. In the middle graph, we only want to focus on the far left value and the far right value for now. The other two aren't really important for this discussion. The F16 column that you can see right there, that is basically what you would get today if you pulled the model and ran it locally, right? It would take up that much RAM for basically a given context window. And the only thing that's really important is that the red bar on the right side of this middle graph is basically four times smaller than the gray bar. So, what this means is that on the exact same hardware, with the exact same model, this person is able to have a four times less memory footprint than if they were just running the regular version of the model at that same context.
And there are people right now trying to merge this into Llama.cpp, which is by far and away the most popular way to run models locally. It's what basically everybody uses or everybody builds on top of. What this boils down to is if you were previously running, let's just say a 7B model or something like that on your hardware, but you were limited to, you know, 8,000 tokens because you just don't have enough RAM to really kind of go crazy, right? The model supports up to 200,000, but you can only get eight because of your hardware. You can now easily jump up to 32. And the difference is staggering.
One of the easiest ways to drive home this idea of like, well, okay, I went from 8 to 32 context. What does that even mean? Let's just put it this way. These are just random podcasts on YouTube. If you have an 8K context window, you won't be able to summarize the transcript of any of these podcasts. It's just too many words. You can't do it. You might be able to do it with 16, but you're going to be really pushing it, and nothing else on your device is going to be can be running because it'll be competing for RAM. At 32K context, it is trivial to do this. Uh, 32K tokens is very easily doable.
To give an idea, just because I ran it as a benchmark, Anything LLM has a meeting assistant in it where we can arbitrarily, uh, like listen to Zoom calls to help summarize them for you or Teams or whatever. It works with anything, but you can also just like drag and drop audio. And so one of the test cases I had was I was like, "Okay, who talks more than anyone on the planet?" And it was like a 3-hour Lex Freedman podcast, that was still 48,000 tokens. So, unlocking the ability for consumers to jump from 8K context to 32K context is a legitimate step function in what it means to what can you actually do with a local model. It used to be you can't really do much because you're limited by your hardware, and if your hardware is bad, you're just going to get a bad experience. TurboQuant makes it to where that's not the case anymore. There are going to be tons of tasks, even complex workflows, that you can run entirely on your device.
Now, will there still be a place for cloud models? Of course, there will be because there's always going to be the performance, the ability to just send it somewhere else and get inference back. And of course, you're still going to have your million-token kind of workloads that are going to just consume tokens. And sure, you can't do that on your computer. TurboQuant does not make your computer have an infinite context window now. It just basically multiplies what you can now do on it.
TurboQuant also couldn't have come at a better time. So, you can see this graph right here, which is the price of 32 gigs worth of DDR5 memory. You can see it's been relatively consistent as time has gone on, but obviously, recently, it's just gone through the roof. Now, there's a bunch of reasons for why that is. But the reason that this is important to you if you're watching this video is that that means the current hardware you're on is now more capable for running AI workloads, basically, and you didn't do anything. And at a time where PC prices are going through the roof and just in general, equipment is harder to get. You're going to see not only like your corporate machines kind of cost-constrained to give you, you know, newer hardware but still relatively low spec. The consumers are going to be squeezed by the same exact market. However, being able to separate people's workloads from cloud models is still, if anything, even more important because that cost that cloud is charging today is not the real cost, and that is going to be very expensive in the future.
And so, that's basically it. That is what TurboQuant means to you. That's what it means to me. I'm not getting into any of the technical details. Honestly, that's for someone that's like far more experienced than me. Like I said, I'll be linking some videos in the description if you want those deep dives, but if you wanted the practical take on what this research means, this is where you can get it.
If you like this video, hit subscribe. Um, I'm always building Anything LLM every day. If you use the tool, please leave me some feedback on it. Otherwise, I'll see you in the comments.