Transcription
We got some pretty major news out of OpenAI today. They just released their openweight model that they've been promising for a few months now, and I think it's a lot better than most people were expecting, especially for an openweight model.
So, this brand new model is called GPTOSS, and it comes in two flavors, two sizes, right? You've got GPTOSS 120 billion parameter and GPTOSS 20B, which is a lot less parameters but should be a lot faster and be able to run on more machines. They also released them under the Apache 2.0 license, meaning that people can download these models, fine-tune them how they want, and pretty much iterate off of this model. And I do believe this is going to be the new openweight king. I don't believe that any of the Llama models or Mistral models or any of the other open models are nearly as powerful as this one that just came out.
Now, if you're wondering why would you care about one of these openweight models, openweight models can be downloaded to your computer and run completely offline. You don't need internet access to run these. You do need a pretty decent GPU to run them, but you don't need internet access. So, you can be having chats with your AI of choice while 30,000 ft in the air on a plane, or if the internet goes out at your house, but you still need to have conversations with a ChatGPT-like bot. You can do that with these openweight models.
Also, none of the data gets sent to the cloud. So, whatever conversations you have with one of these models, it's not getting sent to Google or OpenAI or Microsoft or whoever. It all stays locally on your computer. Nobody can see any of the conversations you have with these models. It's also completely free to use. Once you have it downloaded on your computer, it's yours to use. So, you don't have to pay a ChatGPT or a Claude or a Google fee to use these models at all. They're yours. They're on your computer running locally now.
So let's take a peek at some of the details around this model, and then we'll actually try to run it and see what it can do. So again, two models, 120 billion parameter, 20 billion parameter. It's using a Mixture of Experts model, meaning that it's kind of not pinging all of the parameters at once. It's just pinging a segment of 5.1 billion parameters or 3.6 billion parameters, which theoretically, you know, speeds up the inference when using these models. It's also got a context length of 128,000 tokens, meaning between combined input and output, you're going to be able to use about 96,000 words.
If we take a peek at the benchmarks here in the Codeforces competition code benchmark, we can see that the 120 billion parameter model got a 2,622 score. If we look at GPT-4 Turbo and GPT-4 Omni, the current state-of-the-art pretty much best models ChatGPT has right now, it's pretty on par. It just performed slightly worse than those, but I mean, very slightly. And it actually outperforms GPT-4 Turbo by quite a bit. In this Humanity's Last Exam, these openweight models perform about on par as GPT-4 Omni. On Healthbench, it actually performs close to GPT-4 Turbo and better than GPT-4 Omni and GPT-4 Turbo. In competition math, it actually outscores GPT-4 Turbo and GPT-4 Turbo, and just barely gets beat out by GPT-4 Omni. So, these models are really good at math and code. In the GPQA, which is Google-proof, meaning stuff that you can't really just find the answer on Google. The 120 billion parameter model outperformed GPT-4 Turbo and performs pretty on par with GPT-4 Turbo and GPT-4 Omni. And again, openweight model, free to use, can download on your computer, and it is pretty close to matching the current state-of-the-art closed models from OpenAI. It's pretty mind-blowing, honestly.
These are also chain-of-thought models. So when you give it your prompt, you can actually see it think through the prompt, and then you can sort of give it the amount of thinking you want it to do. So you can have it think for short, medium, or longer term so that it better sort of logics through the response. And they're available right now. You can download them on Hugging Face, and you can run them in tools like Ollama and LM Studio and Together AI and all of these various services here. I tend to use LM Studio, and that's what we'll use in a minute when we test this out. But just a few other things I want to show you.
Sam Altman tweeted as soon as this went live, "GPTOSS is out. We made an open model that performs at the level of GPT-4 Omni and runs on a high-end laptop." That's stretching. I mean, like if you're using the 120 billion parameter model here, it does say it requires an 80 GB GPU, which there aren't many consumer GPUs that will do that. If you have a high-end Mac Studio, you can run it on a high-end Mac Studio. I actually have a Mac Studio with 256 GB of RAM. So, I should be able to run this model. We'll find out in a minute. But the smaller 20 billion parameter model can run with just 16 GB of memory, which you can get on most modern consumer GPUs.
Now, I went to Perplexity real quick and said, "Which GPUs have 16 GB of VRAM or more?" And we can see there's a few AMD Radeon models, RTX 5060, 5070, 5080, of course the 5090, the 4080s, 4090s, 3090s, 3090 Ti. Quite a bit of GPUs that can run the model that requires 16 GB.
And here's some more detailed thoughts from Sam on this new model. "GPTOSS is a big deal. It's a state-of-the-art openweight reasoning model with strong real-world performance comparable to GPT-4 Omni that you can run locally on your own computer or phone with the smaller size. We believe this is the best and most usable open model in the world." He says he wants to get it in the hands of as many people as possible and that "we believe far more good than bad will come from it." The 120 billion parameter model performs about as well as GPT-4 Turbo on challenging health issues, and they've worked to mitigate the most serious safety issues, especially around biosecurity. "Although we believe most people will want to use a convenient service like ChatGPT, people should be able to directly control and modify their own AI when they need to, and the privacy benefits are obvious, 'cause you can do it completely offline. We expect a meaningful uptick in the rate of innovation in our field and for many more people to do important work than were able to do before."
It also looks like this is going to be the model that's going to be used in the future on Windows PCs. As part of today's release, Microsoft is also bringing GPU-optimized versions of the GPTOSS 20B model to Windows devices. If this is good at code, this could honestly replace Claude for a lot of people because it's an open model, meaning you're not paying API fees every time you need to use code. And well, Claude code keeps on bumping up their prices. If you could do this all locally on your own computer, you're not worried about API cost. You're just worried about your own power usage of your own computer.
So, let's go ahead and actually install this on our computer. If you're into branding of any kind, this will change your workflow. Recraft is an AI image generation tool built for designers. You can create consistent, high-quality illustrations, logos, mockups, and icons without giving up creative control. Say you need vector art. Recraft is the only AI that generates real SVGs. I picked a style, typed my prompt, "Create a banner for my MattWolf YouTube channel main page about AI and tech," and got a scalable design instantly.
Some of you may know I have a website called futuretools.io. It's run by a pretty scrappy team, so we don't have a ton of branding besides this logo. Let's see if Recraft can help. I'm going to upload the logo as a reference image, and Recraft instantly generates consistent, on-brand assets I can use across the website and anywhere else I need branding, like slide decks or on YouTube. This feature alone just saved me hours of work. Recraft is already used by teams at Netflix, Oglev, HubSpot, so you know you can trust the output. Check out the link in the description where you can start creating with Recraft for free today. Thank you so much to Recraft for sponsoring this one. Now, let's get back to the video.
So, let's go ahead and actually install this on our computer. To do that, I'm going to use LM Studio. I'm going to head on over to lmstudio.ai. Download it for Mac. Obviously, download it for whatever operating system you're using. And we'll go ahead and run the installer. Drop that here, and we'll open LM Studio. Local AI on your computer. Let's get started. I want to be shown everything, so I'm going to select a developer, but you choose whatever you want. And we can see that as soon as we open LM Studio, it suggests GPTOSS 20B. And this is the model that most people are going to want to download and use, the 120 billion parameter model. Again, you need a pretty hefty computer to use. So, let's go ahead and download GPTOSS 20B. Keep in mind it is over 12 GB, so make sure you have the space for it.
While that's downloading, I'm going to actually continue into LM Studio here. And then down in the bottom right corner, I'm going to click on my settings. And then I'm going to click on model search, 'cause I want to test 120B on my computer as well. I have a really high-end Mac Studio that I've never really pushed it to its limits. So, I'm going to grab this model here as well. It is 64 GB, so it's going to take a while, but uh, might as well get that download going as well.
All right, so our 20 billion parameter model is downloaded, so we can go ahead and use that now. And then up at the top, we have "Select model to load." We'll go ahead and select the GPTOSS 20B. And now we essentially have a model that's as good as ChatGPT offline. Just for good measure, let's ask, "How many Rs are in the word strawberry?" It thought for less than 1 second. "The word strawberry contains three occurrences of the letter R." Good job. And we can actually see the thought process as well. "Count R letters in strawberry." And then "R appears three times." Pretty fast, too. 74 tokens per second.
We can also see down at the bottom here, we can attach images or PDFs or things like that. We can turn on additional integrations like the JS code sandbox and RAG v1. We can also install more integrations straight through MCP servers. And we can change the reasoning effort between low, medium, or high. Here up in the top right, we've got our little wrench icon. If I click on this, we have more settings. I could give it a system prompt, mess with the reasoning effort over here, change the model temperature, change the sampling settings, give information about structured output, and speculative decoding over my I can click on program here and turn on additional integrations.
Let's test a coding prompt. I'm going to go ahead and click on new chat and let's have it create a Vampire Survivors style game in JavaScript. I'll say, "Create a Vampire Survivors clone using JavaScript so I can test it in the browser." Under reasoning effort, I'm going to turn it on high. And since it is JavaScript, let's turn on the JS code sandbox. I'm assuming that will help with JavaScript. And one last step before I submit this. I need to click on the settings up next to the model here and make sure that we set our context length to the higher end. It shouldn't need to use this many tokens 'cause it's writing code, but we do want to crank it up to make sure that it can actually continue to write the code. I probably don't need to set it all the way to the 131,000 tokens. I'm probably good setting it at like, let's go 20,000 tokens. And then I'll reload to apply changes. And now I'll submit my prompt and it starts thinking.
And this was really quick. It took maybe, I don't know, 45 seconds or so to generate all of this. But it generated a ton of code. We have two files we can play with: an index.html and the main.js. So just for the sake of this demo, let's just create a new folder here. I'm just going to call it temp for now. Inside our temp folder, I'll create our index.html file. So we'll go ahead and copy the text here for our index.html. Create a new document inside of our temp folder here. Paste our text in, and we'll save this as our index.html. We'll create a new document here. Copy our main.js. Paste this in here. And we'll save this as main.js.
Now, theoretically, we should be able to open our index.html file here and have our Vampire Survivors game. Now, it's not doing any shooting, but there are enemies coming at me, and I'm running away, and more enemies are coming at the screen. And you can see I've survived for, you know, 45 seconds. This basically just created a simple game where I'm dodging the characters. So, yeah, it didn't really make a Vampire Survivors game, but hey, it did it really fast. It did it with an open-source offline model, and I'm excited to see what the bigger model does once that's finally done downloading.
All right, it finally finished downloading the 120 billion parameter model. Took quite a while. Again, this is the model that requires about 80 GB of RAM. As you can see, I've got 256 GB of RAM and 208 GB of potential VRAM. So, theoretically, I should be able to run it. I don't know how fast it's going to be, but let's give it a shot. Up at the top, I'm going to switch this model to our 120B model. I'll create a new chat here. Let's click on our settings. Make sure that we crank up our context length again, just so it doesn't run out. For our prompt, once again, I'm going to do, "Create a Vampire Survivors clone using JavaScript so I can test it in the browser." Reasoning effort, I'm going to switch it to high. We'll leave the JS code sandbox on.
And it actually ran pretty well. It generated about 35 tokens per second. And it generated this code here for me. And it actually made my life easy. It put it all in one file. So, I don't have a separate HTML and JavaScript file. So, I'll go ahead and copy all of this text here. Create a new text file. Paste the whole thing in. We'll save this one as vampire.html. And when I open it, I get this. I mean, he's shooting very, very rapidly, but it's a much closer clone to Vampire Survivors than the 20B model managed to make, right? I've got the bad guys coming at me, and it's just automatically shooting weapons. I mean, the weapons are a little too aggressive. It makes it a little too easy. Nobody can even make it close to me. But, I mean, one shot from an open-source model, you know, a couple more prompts and we've got a solid Vampire Survivor code.
Honestly, I still can't believe that this is capable of doing this with just one prompt. Open source, completely offline. It's pretty much as good as the best state-of-the-art models out there right now, except it's open-source, can be run completely offline. You're not sending any data to anybody else's servers, and it's really, really good. Like I'm really impressed with what we can do with this open-source model, and I'm even more excited to see what happens as people get their hands on it and they iterate off of it and they fine-tune it. It's only going to get better and more uncensored and have more and more capabilities as people start playing around with it. So super exciting to see that we can get the equivalent of some of the best ChatGPT models for free and offline right now.
And if this wasn't enough news for you, well, it looks like we might be getting more this week. Sam Altman went to X today and said, "We have a lot of new stuff for you over the next few days. Something big but small today." Referring to the GPTOSS we just talked about, and then a big upgrade later this week. Is that GPT-5? I don't know. Could turn into a really, really exciting week. This is already pretty dang cool. I'm here for it. And whatever happens this week, I will be covering it all in Saturday's AI news breakdown. Every Saturday I make a full video where I demo all of the newest AI capabilities that we have for the week and talk about all the latest AI news. So, if you want more videos like this where I show you how to use the tools, but also keep you looped in on all the latest AI news, make sure you like this video, subscribe to this channel, and I will make sure more stuff like this shows up in your YouTube feed. Thank you so much for hanging out with me and nerding out with me today. Thank you to Recraft for sponsoring this video. Yeah, go play with this new OSS model. It's totally free to use. All right, that's all I got for you. Hopefully I'll see you in the next one.