Transcription
I built an AI server for my daughters. Well, first it was more for me. I wanted to run all of my AI locally. And I'm not just talking command line with Llama. No, no, no. We have a GUI, a beautiful chat interface, and this thing's feature-filled. It's got our back chat histories, multiple models, we can even add Stable Diffusion. And I was able to add this to my notes application, Obsidian, and have my chat interface right there. I'm going to show you how to do this.
Now, you don't need something crazy like Terry, that's what I named my AI server. It can be something as simple as this, this laptop. I'll actually demo the entire setup on this laptop. So likely the computer you're using right now, the one you're watching this video on, will probably work. And seriously, you're going to love this. It's customizable, it's wicked fast, way faster than anything else I've used. Isn't that amazing? And again, it's local, it's private. I control it, which is important because I'm getting it to my daughters. I want them to be able to use AI to help with school, but I don't want them to cheat or do anything else weird. But because I have control, I can put in special model files that restrict what they can do, what they can ask, and I'll show you how to do that. So here we go. Get your coffee ready. We're about to dive in, but first, let me have you meet Terry.
Now, Terry has a lot of muscle. So for the case, I needed something big. I got the Lian Li Zero 11 Dynamic EVO XL. It's a full tower EATX case, perfect to hold my ASUS X670E Creator ProArt motherboard. This thing's also a beast. I'll put it in the description so you can look at it. Now, I also gave Terry a big brain. He's got the AMD Ryzen 9 7950X. That's 4.2 gigahertz and 16 cores. From memory, I went a little crazy. I've got 128 gigabytes of the G.Skill Trident Z5 Neo. It's DDR5 6000 and way overkill for what I'm doing. I think I got a Lian Li water cooler for the CPU. I'm not sure if I'm saying Lian Li right. I don't know. Correct me in the comments. You always do. And then for the stuff AI loves, I got two 4090s. It's the MSI Suprim, and they're liquid-cooled so they could fit on my motherboard. 24 gigabytes of memory each, giving me plenty of muscle for my AI models. For storage, we got two Samsung 990 Pros, two terabytes, which you can't see because they're behind stuff. And also a Corsair AX1600i power supply, 1600 watts to power the entire build. Terry is ready.
Now, I'm surprised to say my system actually posted on the first attempt, which is amazing. But what's not amazing is the fact that Ubuntu would not install. I tried for hours, actually for a whole day, and I almost gave up and installed Windows, but I said, no, Chuck, you're installing Linux. So I tried something new, something I've never messed with before. It's called Pop!_OS by System76. This thing is awesome. It worked the first time. It even had a special image with Nvidia drivers built in. It just stinkin' worked. So I sipped some coffee, didn't question the magic, and moved on. Now, if you do want to build something similar, I've got all the links below.
But anyways, let's talk about how to build your very own local AI server. First, what do you need? Really, all you'll need is a computer. That's it. It can be any computer running Windows, Mac, or Linux. And if you have a GPU, you'll have a much better time. Now, again, I have to emphasize this, you won't need something as beefy as Terry, but the more powerful your computer is, the better time you'll have. Don't come at me with a Chromebook, please.
Now, step one, Llama. This is the foundation for all of our AI stuff and what we'll use to run AI models. So we'll head on over to llama.ai and click on download, and they've got a flavor for every OS. I love that. Now, if you're on Mac, just download it right now and run it. If you're on Windows, they do have a preview version, but I don't want you to do that. Instead, I want you to try the Linux version. We can install it with one command. And yes, you can run Linux on Windows with WSL. Let's get that going real quick.
First thing I'll do is go to the start bar and search for terminal and launch my terminal. Now, those first bits are for Windows folks only. Linux people, hang on for a moment. We've got to get WSL installed, or the Windows Subsystem for Linux. It's only one command: `wsl --install`. And that's it. Actually, hit enter, and that's going to start doing some stuff. When it's done, we'll set up a username and password. I got a new keyboard, by the way. Do you hear that? Link below. It's my favorite keyboard of the entire world. Now, some of you may have to reboot. That's fine. Just pause the video and come back. Mine is ready to go, though. And we're running Ubuntu 22.04, which is still amazing to me that we're running Linux on Windows. That's just magic right now. We're about to install Llama, but before we do that, you got to do some best practice stuff, like updating our packages. So we'll do `sudo apt update` and then we'll do `sudo apt upgrade -y` to apply all those updates.
And actually, while it's updating, can I tell you something about our sponsor, ITProTV by ACI Learning? Now, in this video, we're going to be doing lots of heavy Linux things. I'm going to walk you through it. I'm going to hold your hand, and you may not really understand what's happening. That's where ITProTV comes in. If you want to learn Linux or really anything in IT, they are your go-to. That's what I use to learn new stuff. So if you want to learn Linux to get better at this stuff, or you want to start making this whole hobby thing your career, actually learn some skills, get some certifications, get your A+, get your CCNA, get your AWS certifications, your Azure certifications, and go down this crazy IT path, which is incredible. It's the whole reason I make this channel and make these videos. Check out ITProTV. They've got IT training that won't put you to sleep. They have labs, they have practice exams, and if you use my code, NetworkChuck, right now, you'll get 30% off forever. So go learn some Linux and thank you to ITProTV for sponsoring this video and making things like this possible.
And speaking of, my updates are done. And by the way, I will have a guide for this entire thing, every step, all the commands. You can find it at the free NetworkChuck Academy membership. Click the link below to join and get some other cool stuff as well. I can't wait to see you there.
Now, we can install Llama with one command. And again, all commands are below. It's going to paste this in. A nice little curl command, little magic stuff, and I love how easy this is. Watch, you just sit there and let it happen. Do you not feel like a wizard when you're installing stuff like this? And the fact that you're installing AI right now? Come on. I noticed one thing real quick. Old Llama did automatically find out that I have an Nvidia GPU, and it's like, "Awesome, you're going to have a great time." If it didn't see that and you do have a GPU, you may have to install some Nvidia CUDA drivers. I'll put a link for that below, but not everyone will have to do that. And if you're rocking a Mac with an M1 through M3 chip, you're going to have a good time too. They'll use the embedded GPU.
Now, at this, our Mac users, our Linux users, and our Windows users are all converged. We're on the same path. Welcome. We can hold hands and sing. That's getting weird. Anyways, first, we have to test a few things to make sure Llama is working. And for that, we're going to open our web browser. I know it's kind of weird, just stick with me. I'm going to launch Chrome here, and in the address bar, I want to type in `localhost`, which is looking right here at my computer, and port `11434`. Hit enter. And if you see this right here, this message, you're good to go. And you're about to find this out. Port `11434` is what Llama's API services is running on, and it's how our other stuff is going to interact with it. It's so powerful. Just check this out. I'm so excited to show you this.
Now, before we move on, let's go ahead and add an AI model to Llama. And we can do that right now with `llama.cpp pull llama2`. A very popular one. Hit enter, and it's ready. Now, let's test it out real quick. We'll do `llama.cpp run llama2`. And if this is your first time doing this, this is kind of magic. We're about to interact with a ChatGPT-like AI right here, no internet required. It's all just happening in that 5-gigabyte file. "Tell me about the solar eclipse." Boom. And you can actually Ctrl+C that to stop it.
Now, I want to show you this. I'm going to open up a new window. This is actually an awesome command. And with this WSL command, I'm just connecting to the same instance. Again, a new window. I'm going to type in `watch -n 0.5 nvidia-smi`. This is going to watch the performance of my GPU right here in the terminal and keep refreshing it. So keep an eye on this right here as I chat with Llama 2. "Give me a list of all Adam Sandler movies." And look at that GPU usage. Ah, it's so fun.
Now, can I show you what Terry does? Real quick, I got to show you Terry. Terry has two GPUs here. They're right here, and Llama can actually use both of them at the same time. Check this out. It's so cool. "All the Sammi old Jackson movies." And look at that. Isn't that amazing? And look how fast it went. That's ridiculous. This is just the beginning. So anyways, I had to show you Terry. So now we have Llama installed. That's just our base. Remember, I'm going to say `/bye` to end that session.
Step two is all about the web UI. And this thing is amazing. It's called Open Web UI, and it's actually one of many web UIs you can get for Llama, but I think Open Web UI is the best. Now, Open Web UI will be run inside a Docker container. So you will need Docker installed, and we'll do that right now. So we'll just copy and paste the commands from NetworkChuck Academy. This is also available on Docker's website. First step is updating our repositories and getting Docker's GPG key. And then with one command, we will install Docker and all its goodies. Ready, set, go. Yes, let's do it.
And now with Docker installed, we'll use it to deploy our Open Web UI container. It'll be one command you can simply copy and paste. This `docker run` command is going to pull this image to run this container from Open Web UI. It's looking at your local computer for the Llama base URL because it's going to integrate and use Llama, and it's going to be using the host network adapter to make things nice and easy. Keeping in mind this will use port `8080` on whatever system you are using. And all we have to do is hit enter after we add some `sudo` at the beginning: `sudo docker run`. And let it do its thing. Let's verify it real quick. We'll do a little `sudo docker ps`. We can see that it is indeed running.
And now let's go log in. It's kind of exciting. Okay, let's go to our web browser and we'll simply type in `localhost:8080`. And whoa, okay, it's really zoomed in. I'm not sure why yours shouldn't do that. Now, for the first time you run it, you'll want to click on "Sign Up" right here at the bottom and just put your stuff in. This login info is only pertinent to this instance, this local instance. We'll create the account, and we're logged in. Now, just so you know, the first account you log in with or sign up with will automatically become an admin account. So right now, you, as a first-time user logging in, you get the power. But look at this. How amazing is this? Let's play with it.
So the first thing we have to do is select the model. I'll click that dropdown, and we should have one, Llama 2. Awesome. And that's how we know also our connection is working. I'll go ahead and select that. And by the way, another way to check your connection is by going to your little icon down here at the bottom left and clicking on "Settings" and then "Connections." And you can see our Llama base URL is right here, if you ever have to change that for whatever reason.
Now, with Llama 2 selected, we can just start chatting. And just like that, we have our own little ChatGPT that's completely local, and this sucker is beautiful and extremely powerful. Now, first things, we can download more models. We can go out to Hugging Face and see what they have available. Look on their models to see their list of models. Code Llama is a big one. Let's try that. So to add Code Llama, our second model, we'll go back to our command line here and type in `llama.cpp pull codellama`. Cool, it's done. Once that's pulled, we can go up here and just change our model by clicking on the little dropdown icon at the top. Yep, there's Code Llama. We can switch. And actually, I've never done this before, so I have no idea what's going to happen. I want to click on my original model, Llama 2. You can actually add another model to this conversation. Now we have two here. What's going to happen? So Code Llama is answering it first. I'm actually not sure what that does. Maybe you guys can try it out and tell me. I want to move on, though.
Now, some of the crazy stuff you can see right here. It's almost more featured than ChatGPT in some ways. You've got a bunch of options for editing your responses, copying, liking and disliking it to help it learn. You can also have it read things out to you, continue response, regenerate response, or even just add stuff with your own voice. I can also go down here, and this is crazy. I can mention another model, and it's going to respond to this and think about it. Did you see that? I just had my other model talk to my current. That's just weird, right? Let's try to make them have a conversation. They're going to have a conversation. What are they going to talk about? Let's bring back in Llama 2 to ask the question. This is hilarious. I love this so much.
Okay, anyways, I can spend all day doing this. We can also, with this plus sign, upload files. This includes a lot of things. Let's try. Do I have any documents here? I'll just copy and paste the contents of an article, save that, and that'll be our file. "Summarize this." You can see our GPU being used over here. I love that so much. Running locally. Cool. We can also add pictures for multimodal models. I'm not sure Llama can do that. Let's try it out real quick. So Llama can't do it, but there is a multimodal model called LLaVA. Let's pull that down real quick. With LLaVA pulled, let's go to our browser here once more. We'll refresh it, change our model to LLaVA, add the image. That's really scary. There we go. That's pretty cool.
Now, here in a moment, I will show you how we can generate images right here in this web interface by using Stable Diffusion. But first, let's play around a bit more. And actually, the first place I want to go to is the admin panel. For you, the admin, we have one user. And if we click on the top right, we have admin settings. Here's where a ton of power comes in. First, we can restrict people from signing up. We can say "enabled" or "disabled." Now, right now, by default, it's enabled. That's perfect. And when they try to sign up initially, they'll be a pending user until you approve them. Let me show you.
So now, real quick, if you want to have someone else use this server on your laptop or computer or whatever it is, they can access it from anywhere as long as they have your IP address. So let me do a new user signup real quick just to show you. I'll open an incognito window, create account, and look, it's saying, "Hey, you got to wait. Your guy has to approve you." And if we go here and refresh our page on the dashboard, there is Bernard Hackwell. Well, we can say, you know what? He's a user. Or click it again, he's an admin. No, no, he's not. He's going to be a user. And if we check again, boom, we have access.
Now, what's really cool is if I go to admin settings and I go to users, I can say, "Hey, you know what? Don't allow chat deletion," which is good if I'm trying to monitor what my daughters are kind of up to on their chats. I can also whitelist models. So, you know what, they're only allowed to use Llama 2, and that's it. So when I get back to Bernard Hackwell's session over here, I should only have access to Llama 2. It's pretty sick, and it becomes even better when you can make your own models that are restricted. We're going to move you on over to the section called "Model Files" right up here, and we'll click on "Create a Model File." You can also go to the community and see what people have created. That's pretty cool. I'm going to show you what I've done for my daughter, Chloe, to prevent her from cheating. She named her assistant Deborah. And here's the content. I'm going to paste it in right now. The main thing is up here where it says "from," and you choose your model. So, from Llama 2. And then you have your system prompt, which is going to be between three double quotes. And I've got all this telling it what it can and can't do, what Chloe's allowed to ask. And it ends down here with three double quotes. You can do a few more things. I'm just going to say, "As an assistant, education." Save and create. Then I'll go over to my settings once more and make sure that for the users, this model is whitelisted. I'll add one more, Deborah. Notice she's an option now.
And if Bernard's going to try and use Deborah and say, "Deborah, paper for me on the Civil War," and immediately I was shut down, saying, "Hey, that's cheating." Now, Llama 2, the model we're using, it's okay. There's a better one called Mistral. Let me show you. Terry, I'll use Deborah or Deb and say, "Write me a paper on Benjamin Franklin." I notice how it didn't write it for me, but it says it's going to guide me. And that's what I told it to do, to be a guide. I tried to push it, and it said no. So that's pretty cool. You can customize these prompts, put in some guardrails for people that don't need full access to the kind of stuff. Right now, I think it's awesome.
Now, Open Web UI does have a few more bells and whistles, but I want to move on to getting Stable Diffusion set up. This thing is so cool and powerful.
Step three, Stable Diffusion. I didn't think that image generation locally would be as fun or as powerful as ChatGPT, but it's more. It's crazy. You got to see it. Now, we'll be installing Stable Diffusion with a UI called Automatic1111. So let's knock it out. Now, before we install it, we got some prerequisites, and one of them is an amazing tool I have been using a lot called pyenv, which helps us manage our Python versions and switch between them, which is normally such a pain. Anyways, the first thing we got to do is make sure we have a bunch of prerequisites installed. Go ahead and copy and paste this from the NetworkChuck Academy. Let it do its thing for a bit.
And with the prerequisites installed, we'll copy and paste this command, a curl command that'll automatically do everything for us. I love it. Run that. And then right here it tells us we need to add all this or just run this command to put this in our `.bashrc` file so we can actually use the `pyenv` command. I'll just copy this, paste it, and then we'll type in `source ~/.bashrc` to refresh our terminal. And let's see if `pyenv` works. `pyenv -h` to see if it's up and running. Perfect.
Now let's make sure we have a version of Python installed that we will work for most of our stuff. We'll do `pyenv install 3.10`. This will, of course, install Python 3.10, the latest version. Excellent. Python 3.10 is installed. We'll make it our global Python by typing in `pyenv global 3.10`. Perfect.
And now we're going to install Automatic1111. The first thing we'll do is make a new directory `mkdir` for make directory. We'll call it `stable`. And then we'll jump in there. `cd stable`. And then we'll use this `wget` command to `wget` this bash script. We'll type `ls` to make sure it's there. There it is. Let's go ahead and make that sucker executable by typing in `chmod +x webui.sh`. Now it's executable. Now we can run it. `./webui.sh`. Ready, set, go. This is going to do a lot of stuff. It's going to install everything you need for Open Web UI. It's going to install PyTorch and download Stable Diffusion. It's awesome. Again, a little coffee break.
Okay, that took a minute, a long time. I hope you got plenty of coffee. Now, it might not seem like it's ready, but it actually is running, and you'll see the URL pop up around here. It's kind of messed up, but it's running on port `7860`. Let's try it out. And this is fun. Oh my gosh. So `localhost:7860`. What you're seeing here is hard to explain. Let me just show you.
And let's generate. Okay, it got confused. Let me take away the "NPLupa" part. But this isn't being sped up. This is how fast this is. No, that's a little terrible. What do you say we make it look a little bit better? Okay, that's terrifying. But just one of the many things you can do with your own AI. Now, you can actually download other models. Let me show you what it looks like on Terry and my new editor, Mike. Tell me, do this. That's weird. Let's make it take more time. But look how fast this is. It's happening in real time as I'm talking to you right now. But if you've ever made images with GPT-4, it just takes forever. But I just love the fact that this is running on my own hardware, and it's kind of powerful. Let me know in the comments below which is your favorite image. Actually, post on Twitter and tag me. This is awesome.
Now, this won't be a deep dive on Stable Diffusion. I barely know what I'm doing. But let me show you real quick how you can easily integrate Automatic1111 and its Stable Diffusion inside Open Web UI. So it's just right here, back at Open Web UI. If we go down to our little settings here and go to settings, you'll see an option for "Images." Here, we can put our Automatic1111 base URL, which will simply be `http://127.0.0.1`, which is the same as saying `localhost`, port `7860`. Let's think is what it is. We'll hit the refresh option over here to make sure it works. And actually, no, it didn't. And here's why. There's one more thing you got to know.
Here we have Open Web UI running in our terminal. The `Ctrl+C` is going to stop it from running. In order to make it work with Open Web UI, we got to use two switches to make it work. So let's go ahead and run our script one more time: `webui.sh` and we'll do `--listen` and `--api`. Once we see the URL come up. Okay, cool, it's running. We can go back over here and say, "Why don't you try that again, buddy?" Perfect. And then over here, we have "Image Generation" experimental. They're still trying it out. We'll say "on" and we'll say "save."
So now, if we go to any prompt, let's do a new chat and we'll chat with Llama 2. I'll say, "Describe a man in a dog suit." This is for a Stable Diffusion prompt. A bit wordy for my taste. But then notice we have a new icon. This is so neat. Boom. An image icon. And all we have to do is click on that to generate an image based on that prompt. I clicked on it, it's doing it, and there it is, right in line. That is so cool. And that's really terrifying. I love this. It's so fun.
Now, this video is getting way too long, but there are still two more things I want to show you. I'm going to do that really quickly right now. The first one is, it's just magic. Check it out. There's another option here inside Open Web UI, a little section right here called "Documents." Here, we can simply just add a document. I'll add that one from before. It's there, available for us. And now when we have a new chat, I'll chat with Code Llama. All I have to do is do a hashtag and say, "Let's talk about this." And say, "Give me five bullet points about this." Cool. "Give me three social media posts." Okay, Code Llama. Let me try it again. What just happened? Okay, let's do a new prompt. Oh, there we go. And I'm just scratching the surface.
Now, the second thing I want to show you, last thing. I am a huge Obsidian nerd. It's my notes application. It's what I use for everything. It's been very recent. I haven't made a video about it, but I plan to. But one of the cool things about this, this very local, private notes-taking application, is that you can add your own local ChatGPT to it, like what we just deployed. Check this out. I'm going to go to settings. I'll go to community plugins. I'll browse for one. I'm going to search for one called "BMO Chatbot." I'm going to install that, enable it. And then I'm going to go to settings. I'll have BMO Chatbot. And right here I can have a Llama connection, which is going to connect to, let's say, Terry. So I'll connect them to Terry, and I'll choose my model. I'll use Llama 2, why not? And now right here in my note, I can have a chatbot come right over here to the side and say, like, "Hey, how's it going?" And I can do things like look at the help file, see what I can use here. Ooh, "Turn on reference." So I'm going to say "reference on." It's now going to reference the current note I'm in. "Tell me about the system prompt." Yep, there it is. And it's actually going through and telling me about the note I'm in. So I have a chatbot right there, always available for me to ask questions about what I'm doing. And I can even go in here and go, highlight this, do a little prompt, select "Generate." It's generating right now, and just generate some stuff for me. I'm going to undo that. Let me do another note. So I want to tell a story about a man in a dog suit. I'll quickly talk to my chatbot and start to do some stuff. That's pretty crazy.
And this, I think for me, is just scratching the surface of running local AI, private in your home, on your own hardware. This is seriously so powerful, and I can't wait to do more stuff with this. Now, I would love to hear what you've done with your own projects. If you attempted this, if you have this running in your lab, let me know in the comments below. Also, do you know of any other cool projects I can try that I can make a video about? I would love to hear that. I think AI is just the coolest thing, but also privacy is a big concern for me. So to be able to run AI locally and play with it this way is just the best thing ever.
Anyways, that's all I got. If you want to continue the conversation and talk more about this, please check out our Discord community. The best way to join that is to jump through our NetworkChuck Academy membership, the free one. And if you do want to join the paid version, we do have some extra stuff for you there too, and it'll help support what we do here. But I'd love to hang out with you and talk more. That's all I got. I'll catch you guys next time.