Transcription
[Music]
Hi guys, how are you doing? Welcome back!
Let's talk about AI and machine learning. Artificial intelligence and machine learning have been around for some time. If we look at Wikipedia real quick, we can see that we believe the start of AI was in 1956.
During the years that came after that, it has been worked on and developed. Well, now we can see it in the way we use it with OpenAI and ChatGPT. It's very fast, and it is coming up with answers that make us believe we are talking to an intelligent piece of software. But that's all it is; it is software.
All those mathematical models were available in the 1950s, 1970s, and the '90s. They were already there because those models are basically the same. The thing that we couldn't use them then, like we are using them now, is—well, I am dumbing it down—but basically what's happening is that back then, when it started, we didn't have large digital data sets.
That's what we have nowadays, right? For example, social media, all the apps you're using on your phones, all the websites, the data collecting—it's a major privacy issue, but that's not the subject for this video.
The fact is that we do have those large digital data sets now, and in order to train those models on them, we have computers that are really fast. We all know that AI models, AI ChatGPT, and OpenAI are using a lot of hardware resources from a computer perspective. They are using a lot of dedicated GPUs, and they are using a lot of storage.
The thing is, with the use of all those GPUs, which are needed to do all the computational stuff very fast, so you don't have to wait a long time when you ask it something, they consume a lot of power.
How much power are they consuming? We can see it here. This is just one report from The Verge, where we can see that there has been a paper published which calculates that by 2027, the AI sector could consume between 18 to 134 terawatt-hours each year.
That means that's about the same as the annual energy demand of the PhD candidates in the Netherlands, which is also where I live. So that's very, very huge. I mean, AI is consuming a lot of power. You need a lot of GPUs, and that's one of the reasons it is very difficult to run it locally at home, not connected to the internet, on modest hardware.
This is exactly where Deep Seeker comes in. It has surprised me; it has surprised the whole world in how efficiently it can run. It doesn't need all those GPUs from dedicated vendors like Nvidia and Radeon and AMD. It doesn't need all that computational power to achieve the same, or maybe better, results compared to ChatGPT.
OpenAI is a company that is doing groundbreaking stuff in optimizing the AI models and the machine learning models we have nowadays, making them more efficient and able to run on a CPU.
In my lab environment, which I'm going to show you in a bit, I don't have GPUs running in my VMware lab. It is just an ASC server with virtual machines, and I am able to deploy this model, run it completely offline, and have it perform well—acceptably.
The lab environment I have doesn't have the latest CPUs; those CPUs are a few generations old. But still, this model is running very capably on that platform.
Let's get into it.
All right, so this is a virtual machine running Ubuntu on my ESXi environment. The hardware is not that spectacular. I mean, my lab environment is getting a little old, but it is still running perfectly fine for a lab environment, and that's okay.
So let's look at the hardware configuration of this virtual machine. Let's get into edit settings, and we can see I have four CPUs dedicated to this virtual machine. Just a basic virtual machine with Ubuntu running on it. I gave it 32 GB of RAM, and I have it configured with 500 GB.
In the end, it will nowhere near use that 500 GB because I'm not using the largest model from Deep Seeker to run it locally. I have a network adapter connected, and nothing special—no video card, no GPU. I don't have GPUs in my lab environment, so this is what I chose to test in my lab environment.
From a VMware perspective, it is a basic virtual machine. Everyone who is running ESXi 8 will be able to run this model, and the performance, as you will see in a minute, is acceptable. It is not lightning fast, but it is acceptable for a CPU-only environment from a virtual machine running on VMware.
It is very much acceptable to have this model running in your lab. So let's go to the monitoring and see the performance overview. The machine at the moment is not doing anything. The memory usage is fairly low; basically, the machine is idling.
So what we're going to do is log in using SSH. Let me restart the session here real quick. And there we are; this is the machine I'm logged in to right now.
Because this is a lab environment, I can just run everything as root. Of course, if you do this in a production environment, running applications as root is not the way to go. But as this is a lab environment, running this application as root is perfectly fine.
So let's start with installing AMA because AMA is the model you need to install, and that will give you the ability to talk to that Deep Seeker AI that we will install.
The first step is to run and install AMA. AMA will give you the ability to run large language models on your local server, so you don't need to connect to the internet, and that is exactly what we want.
Let's start by installing AMA. It has to go down to the internet and download it. Once again, if you had GPUs in this environment, the install of AMA would see that GPU and configure it to use it for the models.
It is only using CPU in my VMware lab environment to achieve the results I expect a model to achieve. Let's be honest; we have outfitted every data center in the world with a lot of CPUs, right? So why not use those CPUs to do all the AI stuff and the ML stuff?
AMA is installed. I have a warning—that's exactly what I expected to tell me. I will be running it in CPU-only mode, so the VMware virtual machine does not have a GPU attached to it.
AMA is installed. Let's see what the status is. It is active; that's good. Just to make sure that it starts every time I boot my machine, let's make sure that the service is enabled to start on boot.
Clear up some. All right, so that's also done.
Now that we have AMA installed, we need to install Deep Seek AI. Deep Seek AI has several model sizes. I am going with size 7B, and that's the model I chose to run on my virtual machine.
Let's see how it goes. What I will do as well—I told you guys I will show you how to interact with the