The video discusses the practical limitations of running large language models (LLMs) locally, specifically focusing on the discrepancy between model file size and the actual memory required for context and cache.

Initially, the presenter highlights that while a 9B model might fit on a laptop (5.6 GB file size), the context window and cache can significantly exceed available memory. A short chat session can add 0.25 GB of cache, but 'agents' accumulate context over time. One session can load thousands of tokens, with real-world sessions easily surpassing 50,000 tokens and long ones reaching 128,000 tokens.

The presenter explains the 'cache math': each token costs approximately 32 KB of KV cache. At 128,000 tokens, this amounts to 4 GB of cache alone. Combined with model weights and overhead, the total can reach around 10 GB, with the context and cache alone consuming 8 GB even in an efficient scenario. Further complicating matters, not all layers are equally efficient, and a dense model might require 16 GB just for these components, especially when benchmarks run at 256,000 tokens.

The video then addresses the misconception that models 'forget' due to limitations. The presenter clarifies that the model itself might support a large context window (e.g., 262,000 tokens), but the software running it (Ollama in this case) often has a much smaller default context limit (4,096 tokens), which is buried in its FAQ. This default setting, not the model's inherent memory, is the reason for perceived forgetting.

Finally, the presenter offers solutions: enabling 'flash attention' and setting an environment variable to store the cache at Q8 precision. This halves the memory requirement for the cache (e.g., 4 GB becomes 2 GB), making a 16 GB machine viable again. The advice is to size the context according to available RAM, either by quantizing the cache or capping the context length.

DIY Smart Code

You pulled the 9B because it fits your laptop. But a comment under my last video nailed the catch. The model fits, the context for your session likely won't. So, let's check it.

The model file is 5.6 GB. A short chat adds a quarter GB of cache. It fits. That's the rent. Because agents don't chat, they accumulate. Watch what one session loads. A file, 12,000 tokens, a diff, three more. Test output, seven more. Real sessions blow past 50,000 and long ones push 128,000.

Here's the cache math. The model's config prices every token at about 32 KB of KV cache. At 128,000 tokens, that's 4 GB. If your sessions crawl, this is why. Weights plus cache plus overhead, be kind. About 10 GB max the context and the cache alone hits eight. And that's the efficient case. Just eight of 32 layers grows the cache. A dense model needs 16 GB. The benchmark ran at 256,000 tokens.

But the cache isn't the sneakiest part. Stop blaming the model for forgetting. It supports 262,000 tokens, but Ollama's default is 4,096, buried in the FAQ. It was never remembering your files.

So, here's what actually works. Turn on flash attention and one environment variable will store the cache at Q8. Half the memory, very small precision loss per Ollama's own FAQ. 4 GB of cache becomes two. A 16 GB machine is back in play. Size the context to your RAM. Quantize the cache or cap the context. Which one are you running? One word below. And subscribe for more local model deep dives.

5.6GB model, 10GB problem—here's the fix #localllm #ollama #devtips

You pulled the 9B because it fits your
laptop.
But a comment under my last video nailed
the catch. The model fits, the context
for your session likely won't. So, let's
check it.
The model file is 5.6 GB. A short chat
adds a quarter GB of cache. It fits.
That's the rent.
Because agents don't chat, they
accumulate.
Watch what one session loads. A file,
12,000 tokens, a diff, three more. Test
output, seven more.
Real sessions blow past 50,000 and long
ones push 128,000.
Here's the cache math.
The model's config prices every token at
about 32 KB of KV cache.
At 128,000
tokens, that's 4 GB. If your sessions
crawl, this is why.
Weights plus cache plus overhead, be
kind.
About 10 GB max the context and the
cache alone hits eight. And that's the
efficient case.
Just eight of 32 layers grows the cache.
A dense model needs 16 GB.
The benchmark ran at 256,000
tokens.
But the cache isn't the sneakiest part.
Stop blaming the model for forgetting.
It supports 262,000 tokens, but Ollama's
default is 4,096,
buried in the FAQ. It was never
remembering your files. So, here's what
actually works. Turn on flash attention
and one environment variable will store
the cache at Q8. Half the memory, very
small precision loss per Ollama's own
FAQ. 4 GB of cache becomes two. A 16 GB
machine is back in play. Size the
context to your RAM. Quantize the cache
or cap the context. Which one are you
running? One word below. And subscribe
for more local model deep dives.

Get Our Mobile App

5.6GB model, 10GB problem—here's the fix #localllm #ollama #devtips

Transcription