Transcription
Last week Justine said they've released llama file, which should support Android. I was like, "That's so hype!" And then I tried it this afternoon, like 2 hours ago, but I still got this error, which was the original error that, um, we got. Mmap error. Um, but then Justine was like, "I think I know the issue," and sent me some test scripts to run, and they ran fine. They will make a new release probably, um, very soon with the working Android version of llama file. But yeah, I'm going to quickly show you guys how to get that running so you can run any LLM model on your phone and potentially even run an inference server off of your Android phone.
So the first thing you want to do is download llama file. Um, so it'll probably be version 0.8.9 by the time you're watching this. Um, so yeah, click releases and you can just download the base llama file here. Now, same file is portable across any OS, so Windows, Linux, and now Android. Um, and yeah, it works really well.
So you download that. You also want a GGUF model to run with it. So GGUF models are like quantized models that work really well with llama file and, um, like these kind of llama.cpp frameworks. So you can run it with just your CPU, and you can also use GPU offloading if you want. And to find them, you can just type whatever model. We're going to do TinyLlama, um, because it's the smallest one, so it'll run really, really easily. And then just put GGUF. And typically, you'll find them on Hugging Face. So I guess you could type the same thing in Hugging Face search. Um, and then, yeah, find it.
I've downloaded this one. You go into files here. Um, when, when you find the, when you find the project on Hugging Face, you'll be on the model card tab, um, by default. You should go into the files tab and download one of these. So you can see these are all different quantized versions. Um, so the, the smallest one, Q2, um, will be the smallest, um, but it might not be the smartest. And then Q8 is like has the most kind of precision, but it's also the biggest file. I found, um, that the sweet spot is actually Q4_K_M. Supposedly, shout out to this person on the open interpreter Discord, cyanide. They linked this, um, Reddit post where they kind of test the different quant versions. And supposedly, uh, Q4_K_M is a sweet spot where like anything lower than Q4 kind of loses a lot of quality, but Q4 seems to be a sweet spot. So I don't know, I took that knowledge and I downloaded Q4_K_M, which is only like about 700 megs.
When you get those two files, um, you want to put them into your phone. So I don't know how, whatever way you normally put files on your phone. I use Android File Transfer on Mac. Um, on Windows, you don't need a third-party app, but on Mac, you do, which pretty much, it's just a file explorer. Um, I put it in the downloads folder. I made a new folder in downloads called llama. I put it, put it in here. Um, um, but, um, I'll show you, you can't actually run it from your internal storage on Android, um, on Termux for some reason. So there's a couple of other steps. So let's hop over to my phone screen and I'll show you.
So let's open up Termux. You need Termux. This is an app. You can get it from the App Store, but I recommend getting it from F-Droid, um, you'll have access to more packages from. And, um, if you put it in your downloads folder like I did, we can just CD into storage downloads and I put it into a llama folder here. Now, um, you might see your files here. So I'm just going to make an example llama file, um, as an example here. So you can see we have example.llopa. Now, if this was an executable, we can't actually run, um, executables on local storage, internal storage, or something on Termux for some reason. So I actually had to move it to the home folder, which you can get with, um, dollar sign home, um, which is in this directory here. So what I did was I made a directory in home, um, called llama. Now we already have that, and then I moved all the files in the downloads into that directory with this command here. MV star for everything into home llama. Now I've already done that, so I'm just going to CD into home llama and you can see here I have a bunch of those scripts that, um, Justine sent me to test. And the two main files here, llama file and TinyLlama, they're the ones that we're most interested in.
So we can just run that now. You, uh, you first have to enable the executable permissions for llama files. So you can do that with CH mod plus X and then the llama file that you downloaded. Yours will probably be something like llama file_v0.8.8 or 9ine something like that. U run this command, and then that'll change the permissions for you to be able to actually execute it. Once you've done that, you just execute it with ./llama file and then we can load in a model with -m and then I'm going to load in the TinyLlama model. Um, I'm pressing tab for completion here, which you can press this button here for the tab completion. And yeah, just like that, I think we can press enter now and it will load that model.
Now, this is in, yeah, and you can see it actually opened up the inference server here. So it's running locally on my Android phone, and we can prompt it just through this web app here. Like, "How is your day?" And, um, we can see we have locally running TinyLlama on my Android through this web app. Now, because it's running locally on an inference server, we could theoretically interface with it with other agents and stuff on the Android phone as well, completely offline. So that's super, super cool. Uh, yeah, and that's the tutorial.