Transcription
Hi guys, welcome back. And today, we're talking about how you can use the power of large language models running within your machine locally, uh, and using it for development and testing purposes.
This is one of the questions that came to me while I was doing a conference here in Auckland, New Zealand. And one of the participants asked me that we don't really have so much of potential and power within our machine to run a large language model because every time while we try to run it, there is going to be a memory issue, or it couldn't able to handle any of the requests. So, what exactly is the way that we can do it without spending any cost on that? Because if you're running the test on your local machine, you know that you don't really have to spend any money for that because it's all going to run within your machine, because the large language model itself is hosted within your machine. If you're using Olama or any other means. So, how exactly is to get around the particular problem?
Well, guess what? There is a way that you can actually do it. We are going to use the same Olama event this time, but just that we are going to be using the Olama cloud model this time. Again, this model is going to be running on the Olama platform itself, but they have claimed that it is going to be super secure or something like that, and you don't really feel like you are going to be running within the cloud because the way that you can do it is going to be very, very straightforward.
So, if I'm going to go to the settings of the Olama over here, you can see that this is the hourly usage that I have got. This is the weekly usage, and you can see that I will never touch any uh weekly usage beyond like 10% for that matter, even though I use a lot uh of the cloud models as well. But I don't really exceed that particular plan over there.
So, how do we actually uh use this particular model? It's very, very simple in Olama. All you have to do is just go to the models over here in the Olama, and you see that there is an option called as "cloud," where these models which are tagged as "cloud" are the cloud models. So, you can use GPT OSS model, which has got even 120 billion parameters. You can also use Qwen3 vision language model, which has got even 235 billion parameters, or even Q3 coder model, which has got 248 billion parameters. See that how many models we have got, and this is going to keep on improving. Even the Gemini 3 Pro preview model is also available for you to try out and see how this [snorts] is actually going to work. And it was released just like yesterday, due to the time of recording, which is quite amazing.
So, if you wanted to do any one of these operations over here, we can actually use the power of the Olama for our testing and development purposes. So, you can see that this is the code that I have got over here, which is building agents with the RAGs tool support with the local large language model. So, if I want to run this particular code, you can see that I am going to be actually using the local large language model, which is the Qwen2.5 model over here, and I'm also doing quite a lot of things here. I'm doing the embedding of the uh data, and also doing storing it in the vector data store, and then I'm also doing the uh calling of an agent, and also a Playwright test agent to do a UI automation, and I'm doing quite a lot of different operations over here, as you can see.
This is part of my Udemy course, which is available over here, as you can see. Uh, we're just going to go to the fourth page. Uh, this is the course that you can see over here: "Build and Test AI Agent Chatbot RAGs and Local Large Language Model." So, this particular course, as you can see over here, has got around 2,000 enrollments, and you can see that these are the details that are available in this course already, and it is fully upgraded to LangChain version 1.0 as well. So, that's why you can see all those bold symbols there, which means it's all fully upgraded.
So, this particular course actually uses everything running in the local large language model, as well as on the cloud model, and that's exactly what I'm going to be showing over here. So, if I'm going to run this particular code over here right now. So, let's say I'm just going to clear all the output. Uh, and if I'm going to run them all, it is currently running the test on my uh local large language model. So, it is going to invoke the tokenization as well as the vectorization and storing it in the database, and also embedding, everything is going to happen over here, as you can see.
And now you can see that it is calling the agent uh to perform the operation. And you'll you can see that the time it's going to take the particular operation is going to be quite heavy because even though I'm running a Mac M1 Max machine within my uh local system over here, it is going to take quite a lot of time for sure. It is going to take a long time to perform this operation. But if you're going to be using a model which is running on the cloud, it is going to be way faster, and you don't really need any compute power for that matter.
So, this is going to answer your question who are asking in the conference saying, "How exactly can we use the power of the large language model running locally and still harnessing the power without our machine having those power?" I know this is still not going to be running on your local machine. The prompt is the language model is still running on the cloud, but it feels like it is running within your machine, and at the same time, you're not going to be incurring any of the cost. Again, if you talk about security and things, it's going to be only applicable if you're going to be developing or building an application which is going to be deployed uh on your enterprise-grade servers and things of that nature.
Well, as that said, I'm also going to quickly show you how you can use the cloud model within your local machine as well, quickly. So, if you just go to any of the terminal over here, you can see that what I do, "list" over here, it's going to show you the Olama, the Qwen3 coder 480 billion parameter cloud prefix as well, like, sorry, cloud suffix as well. So, this cloud suffix means that this particular model is a cloud-enabled model. You can actually download this from the uh Olama over here.
So, if you're just going to go to the Olama and go to the models, you see that we have got these models. Just go and choose any one of the models. For example, if I'm going to use any other model, let's say the Deepseek version 3.1 U model over here. So, I'm going to go copy the 671 billion parameter. This things will never work within your machine forever because you know that it requires a lot of uh GPU powers and things. So, this is not going to work for us.
So, I'm going to just say "Olama run deepseek v3 670 billion parameter cloud." And you see that it is still pulling it as a registry uh to be used within your Olama. And that's it. It is currently connected. And I'm going to say, "How are you doing?" If I'm going to do that, you see that it's just talking to the actual large language model running or hosted on the cloud for you over there. And it's going to be pretty fast if you're going to do it within this particular uh machine over here.
So, if I want to use this particular model or maybe uh any other model, you can just use it. Let's say I'm going to use this particular list recently downloaded model. So, I'm going to go copy that over here, and I'm going to go to the code that I was executing. I think it is executed. Oh, yeah, there we go. You see that it took like 55 minutes to complete this particular operation, which is huge, to be honest.
But if you're going to run the same code via the cloud-based model, which is this one, and I'm going to clear all the output, reset, um, and I'm going to run them all. So, we'll see how that works. So, ideally, because this is going to be running on the cloud model, the the way it is going to work is going to be way faster, if I'm not wrong. So, let's just wait for the execution to happen. There we go. It took only 13 seconds to complete, opposed to 1 minute which was taking before, and you see that it is just working fine as expected.
So, this is how you can use the power of the large language model running on the cloud, and you don't really have to spend a lot of money as well, as I just told you, because right now I just used like one time there, and if I go to the settings, you see that it's just 2.4%. And even if I'm going to use this multiple times, it is never going to exceed ever for the time it has got. But if you still want to use even further, you can go for the billing option and you can upgrade it so that you get even more uh usage and things. But for now, you can just gracefully use uh all the uh the the usage they have given us in the Olama, which can be used for your testing and development purposes.
So, that's it, guys. Once again, thank you so much for watching this video, and I'm sure this is the way that you should go while you do the development and testing within your local machine, at least not for the enterprise-grade development. But you can see that this is quite awesome to see how fast they are [music] uh while doing any operation. See, now this time it took only 9.4 seconds to complete this operation, which is fabulous.