Transcription
So, we've been hearing quite a bit about Grok 4 since it released a few days ago, but this is just the beginning. First and foremost, a few people have been wondering what happens when the massive compute behind Grok 4. They used 10x the amount of compute on doing RL reinforcement learning with Grok 4 than they did for the previous model, Grok 3. That's due to the colossus giant data center that's, uh, that Elon Musk has built.
One question that people had is, what happens when we take that much hardware, that much raw power, and we use it to train, for example, an AI video generation model? This thing would be next level. It would be much, much bigger than anything we've seen in the past. No other AI video model would come close.
We've seen a number of people building with Grok 4. Here's Danny. We've covered a little bit of his stuff on this channel before. I've reached out to him to see if he wants to come on and do a tutorial. Danny, check your DMs, please. 6.9 million views. But Danny's done tons of other games using Grok, like the infamous Mech and Cheese and whatever this is, the Heretic Hexon Cologne, which looks incredible.
But Elon Musk is talking about launching almost like a new company or a sub-company, which might be interesting to keep our eyes on. He's saying, "We are creating a multi-agent AI software company, XAI, where Grok spawns hundreds of specialized coding and image video generation understanding agents all working together and then emulates humans interacting with the software inside of virtual machines until the results are excellent." So, you're basically creating like a little simulation of a human sitting there on a simulated computer interacting with your software, whether that's a video game or whatever. And he's saying this is a macro challenge and a hard problem with stiff competition. Can you guess the name of this company? He's going to name it. I'm, I'm worried that he's going to name it Macro Hard. And as soon as I say it, I realize he is, isn't he? He's going to name this thing Macro Hard. No relationship to Microsoft. This thing is going to be Macro Hard.
He's saying, ultimately, it's just going to be input bitstream into feedback into output bitstream. Tesla already does this first two steps in the car and has also been able to generate real-world video for a few years. I believe they're calling it the Dojo. Just need to widen the aperture a little bit for a far broader variety of bit IO patterns. So if you think about it, if a car is driving, it's got cameras on it. That's kind of like the input, the, uh, video input. And then it's navigating and changing lanes and accelerating and braking. So that's kind of like the, the, the feedback. It's taking actions based on the input. And now we have robo-taxis rolling around. So certainly they have been doing that for a while with Dojo. I believe they generate, I think using the Unreal Engine or used to be the Unreal Engine. They would generate a bunch of video, kind of scale it down, right? Because the video cameras are like a little bit more janky, a little bit less resolution, right, than something than you can make in the Unreal Engine, which just looks incredible. So, you almost want to like compress it a little bit so it looks more like a dash cam or whatever. So, just a little bit lower fidelity. So, the stuff that Tesla's been doing is not hugely dissimilar from what he's talking about doing here. So, as he's saying, widen the aperture. So just making it a little bit more general, so to speak, for a far broader variety of bit IO patterns, right? So something that can understand the desktop, browsers, video games, videos, etc. And traditional coding would be like painting with brushes. It would be silly to automate painting with brushes using a robot hand. Just render the pixels.
After talking to the XAI team late one night about this, I thought, damn, we're probably in a sim right now. Which it does certainly seem that sort of the eventual conclusion, the eventual finality of this technology somehow, like we're going to build a simulation. It's becoming more and more obvious how useful that would be, especially if the things inside it could be, you know, very smart, almost conscious beings that thought that whatever they were doing was real and really, really important. So if we had beings identical to us and identical world with our exact our thoughts and everything else, just going about our business like that would be the dream world, you know, to build to get data. But let's continue.
So somebody suggested Macro Hard because Elon in the past has floated that name for some reason, and, uh, there's a winking emoji by Elon Musk, so he's going to name this Macro Hard. I just know it.
In other news, XAI has announced Grok for Government, a suite of Frontier AI products available to United States government customers. XAI has also signed a contract with the US Department of Defense. So, of course, they have the contract with Department of Defense and now the General Services Administration. So, basically, Grok and all the various AI services will be available to every federal government department.
So meanwhile, this Meter organization, Meter, Meter, probably Meter. I read about this stuff a lot, so often I don't know what the, you know, pronunciation is, so let me know if, um, so I, I guess Meter is the way you would pronounce it. That's probably the most logical. But they do a lot of study about how AI is progressing. So for example, you might have seen this chart where it's the time horizon of software engineering tasks that various LLMs can complete. So not that long ago, it would be measured in a few minutes. Right now, we're getting to the point where it's an hour and a half for the 03, for example. So, meaning that a task that would take a human being an hour and a half, these models are beginning to take care of those tasks. They're able to run and finish tasks that would take you and I an hour and a half to complete. Now, this is where they predict the AI has a 50/50/50 chance of succeeding, right? So this line represents that 50/50 chance, meaning that it'll do it half the time. Like we expect it to be able to do it half the time. This is the chart that's kind of famous. So the length of tasks AI can do is doubling every six months. So that's the Meter organization, and they're putting a lot of research out like this.
So Nikola Jočić, so he's an AI safety researcher. I believe he's part of the Meter organization. And so he's saying, "I'm a Meter eval researcher validating Grok 4 on our time horizon benchmarks. As an experiment, I'll try live tweeting in this thread as I conduct the eval. This is all raw impressions. Please don't take it too seriously." I retweeted this shortly after he posted it. So you guys see in real time as he's adding more information to it. If you're not following me on Twitter, why not shoot me a follow at WesRothMoney? There will be cake.
But Nikolai is saying, "First, I'm running Grok 4 using various scaffolds on a dev suite to check which scaffold does best." So basically, for a lot of these LLMs, they've really kind of come to life if you have the right scaffolding around them, which basically means some tools that you give them. You kind of give them a little scaffolding around it so it can start doing stuff for you. So they're testing a few different ones, but here's the interesting thing about Grok 4 specifically. So one thing, sometimes it goes off and writes dozens of pages of reasoning for trivial decisions like running a bash command. So, not great, obviously. Now, this was interesting on a dev suite of nine difficult tasks. Each task runs twice to kind of get a baseline. Grok 4 modular or triframe. So the various scaffoldings, the two different ones they're trying. So looks like they run one modular, one triframe. So it gets 7 out of 18 runs correct. Compare this to Claude Opus 4, which gets three out of 18. Does this mean Grok 4's time horizon is longer? Not necessarily, but I'd say it's around 50% likely." So, he's saying that it's possible. It's, it's, it's likely maybe, let's say, that Grok 4's time horizon is longer, meaning its ability to kind of have this long-term coherence and stay with a subject is better than Claude Opus 4.
So those results have not been published yet, as far as I can tell. Let me know if they have published them somewhere, but, you know, on the main page, all the stuff that I can see, you know, Claude 4 Opus, the results there are not available. Grok 4, they're not yet available, but it is possible that Grok 4 will be somewhere near the top. I'm guessing maybe better than 03 and other models on sort of the long horizon tasks. Again, we, we'll see. As Nicholas is saying here, the sample size is very low. So this could be just a statistical fluke. Uh, so, again, it's too early to tell. Take this with a grain of salt. But there's a chance that Grok 4 performs very well on these particular benchmarks or these metrics. The reason this is interesting is again because of how it was trained. They took the Grok 3 model, the same amount of compute that they used for the Grok 3 model. They ran it with 10x the compute. It did very well on the ARC AGI 2. One of the researchers from ARC AGI said that it's showing non-zero levels of fluid intelligence. So, there are interesting things that are happening here. I think we're going to need to see more data and really we're going to need to see kind of like the next iteration of Grok and also all the other models that will be dropping soon if they also demonstrate some of these abilities like a better long-term coherence, more of, you know, quote unquote fluid intelligence. By the way, that might not be the right word to use. That's just one person with the used fluid intelligence versus kind of crystallized intelligence. So this ability to quickly figure out how to do tasks you haven't seen before, kind of quickly learn on the job, if you will. When the sort of next generation of models start dropping from Google, from OpenAI, and, you know, Grok 5, we're going to get a much better sense of, is this a fluke or or is this really happening? So stay tuned for that. That's going to be very interesting to see.
Also, Grok is coming to the Tesla models, the new ones. It's going to be going live on July 12th, 2025, so a couple days ago. So the models that will have Grok available are S3XY or Cybertruck, and you got to have premium connectivity or Wi-Fi connections. But it sounds like XAI and Tesla will have a partnership, which, by the way, there have even been talks of a potential merger. Elon said that he currently does not support the idea of XAI and Tesla merging. Somebody speculated saying that, well, the two companies collaborate closely, right? As we can see here, there's an overlap between Tesla and XAI. And certainly his new announcement here also hints that there's a lot of overlap, right? So there's sort of Elon Musk, there's Tesla, there's XAI, they're all separate entities, but man, if it was a Venn diagram, they'd be pretty closely overlapping, right? It seems that currently there might be an investment. So Tesla might invest in XAI. XAI was valued at 80 billion. I think now it's going to be valued at something like 200 billion, which begs the question, will Elon Musk be the first trillionaire on this planet, at least as far as we know, or at least the first sort of private citizen to be a trillionaire?
But I think for AI progress in general, the really interesting thing here to kind of keep an eye on is what rolls off the production line of Elon's sort of AI machine with the Colossus center, and he's ordering a power plant to be delivered from overseas apparently. So, it's going to be really interesting to see what kind of AI progress he can produce with those things. If he creates an AI video model with all that compute, how's that going to compare to, for example, VEO3 by Google? If he spins up this Macro Hard or whatever he chooses to name it, how good is that going to be? Because again, if he can push other companies around just because he's got the most compute, I mean, that's going to send a pretty strong message out there in terms of like what's required to be at the top of the AI leaderboards.
Anyways, if you made it this far, thank you so much for watching. My name is Wes Roth, and I'll see you in the next.