Transcription
This project uses Apple Silicon's architecture to run a 26 billion parameter model on just 2 GB of RAM, and it's actually usable. I tried it out locally and was able to get 23 tokens per second.
The project was created a couple of weeks ago. Turbo Fieldfare avoids the large memory footprint of Gemma 4, which is around 14 gig, by keeping only a tiny fraction of the model in memory, then streaming the experts direct from SSD. The entire thing is written in Swift and Metal, which is Apple's low-level GPU API. So, you can launch it as a Mac app, which is incredibly user-friendly, and it takes advantage of the specific architecture that you get from Apple Silicon chips.
So, today I want to dive into how Turbo Fieldfare uses Apple Silicon specifically to squeeze incredible performance out of a 26 billion parameter model. So, if we head over to the GitHub repo, we can run a series of commands to clone the repo and get the app up and running. And once up and running, we need to download the model itself, load the model, then we can send our first message. You can see for me on an M3 Max, I'm getting 23.4 tokens per second with a memory footprint of just 2.15 GB. This feels completely usable to me. So, let's take a deeper look at the architecture.
And if you enjoy content like this, then don't forget to subscribe to Better Stack. First, we need to understand how Gemma 4 works because that leads into the system design of the application itself. Gemma 4 is a mixture of experts model. Most models use one big feed-forward neural network, but Gemma is different. As a layer, it instead has 128 small feed-forward blocks. These are the experts plus a tiny router. For each token, the router picks the top eight experts and only those eight run. So, 26 billion parameters total, but only roughly 3.9 billion actually do any work on any given token. 85% of the file is idle at any instance. You don't need 14.3 GB in RAM. You need whatever the current token happens to touch.
Turbo Fieldfare takes advantage of this design, keeping the always-needed parts in RAM, leaving the experts on SSD and fetching them just in time. So, let's take a look at what Turbo Fieldfare is doing with Apple Silicon. When you install the bundle, it gets split into two separate piles. The first is everything the token needs, no matter what. The attention, the router, the embeddings, and one shared expert that always runs, that comes to about 1.35 gig. It gets memory-mapped straight off disk and it's always resident on memory the whole time the model is loaded. The second pile is the experts themselves. 30 layers with 128 experts each, about 3.36 megabytes a piece. So, roughly 12.9 GB, and that never gets loaded at all. It just sits on the SSD and gets pulled in a few megabytes at a time as and when the model asks for it.
So, what actually happens when you produce a token? The model has 30 layers and a token passes through every one of them in order, and each layer does the same two things. First up is attention, and that's the step where the model looks back over everything that's been written so far and works out what matters right now. If it's about to write the word "after the cat sat on the", the attention is what makes "cat" count more than "the". And the handy thing is that attention runs entirely on that 1.35 gig that's already in memory, though we haven't touched the disk yet at all.
Then the router takes what attention produced and it names the eight experts it wants out of the 128. And that's where a problem lies because you can't know which experts you need until you've already done half the work on that layer. There's no reading ahead and no prefetching since the choice depends on this token and every token before it. And it only gets made a fraction of a millisecond before those weights are needed. So, 30 times per token, the CPU has to stop and go to disk.
Which brings us to why this specifically is a Mac project. On a PC with a discrete GPU, getting a weight in front of the GPU means reading it off the SSD into system RAM. Then pushing it across the PCI bus into the card's own VRAM. That's two copies and a bus hop. And doing that thousands of times a second would tank performance. Apple Silicon, however, has unified memory. So, the CPU and the GPU are looking at the same physical RAM, and there's no VRAM to copy into. A Metal buffer is just memory that both the CPU and the GPU can see. So, the CPU reads by straight off the SSD into a buffer the GPU is about to run against, meaning we can skip a bunch of work.
And they lean on these optimizations harder with the file format. Normally, weights on disk sit in a storage format that needs to be unpacked and converted before a GPU can use them. But Turbo Fieldfare stores them in exactly the layout the Metal kernel consumes, right down to the 4-bit quantized values. The installer rearranges the data without ever re-encoding it. So, reading the file is loading the weight with no conversion step in between.
So, what's the GPU doing while the CPU is off fetching from disk? Well, remember the shared expert sitting on the resident pile that runs for every token no matter what the router picked. That's what it works on. The disk read hides inside work the model had to do anyway. So, you get most the fetch for free. So, we've got multiple work streams happening async, which squeezes the time down even further.
And even more so, it's not even going to disk every time either. Each layer keeps 16 of its 128 experts parked in memory. So, when the router picks one that's already there, you get it instantly. And when it doesn't, the expert comes off the SSD and pushes out whichever one has been used least. That's called an LFU cache, which is least frequently used. This is different to an LRU cache where you throw out whatever was touched longest ago because routing here isn't random. Some experts get picked constantly across all sorts of tokens and others hardly come up at all. So, counting how often an expert gets used keeps the popular ones around better than counting how recently. It does mean the whole design is betting on routing being predictable because if every token wanted a different random eight, you'd miss the cache nearly every time and lose performance.
So, the system design here is very specific to take advantage of both the architecture of Gemma 4 and Apple Silicon. You can try this out yourself and head over to the repo we've included in the description. And I'm excited to see how far we can squeeze these models. You know, maybe one day we'll have a fable-level model running on our watch. We've also filmed a video showing how you can run this type of architecture on even smaller machines. You can watch that here. Otherwise, thank you so much for watching, guys. I hope you enjoyed that one, and I'll see you in the next one.