Transcription
I am really tired of models overfitting to eval. So when we have exams that are supposed to be like humanity's last exam that are supposed to be good measures of model evaluation and quality, it's Goodhart's Law all over again. As soon as you make that a goal for a model maker to hit, they will overfit to the data. And I got to say, Grok 4, as hard as the team has worked, is looking like a terribly overfitted model. A model that is much lower in real-world quality than we actually see in all of these reported benchmarks.
It's not just me saying that. I actually went and looked at yep.ai, which is a place for people to prefer answers from different models so they can rank them head-to-head. You know where Grok 4, the vaunted number one model in the world, ranks? Number 66 as of yesterday. Number 66. Now, if you think about it, you might get some slip back and forth between one and two and three if they're close. You would not expect the number one model in the world to be number 66 at anything, let alone number 66 overall at answers provided. And yet, that's what we see with Grok 4.
I want to ask again that we think more about real-world exams. And I went ahead and modeled this. I went and I built up a five-question exam between GPT-3.5, Opus 4, and Grok 4 because I wanted to do the testing that I keep asking people to do myself. And I'm going to tell you the five different tasks that I gave these models.
Number one, condense a Google research post that's quite long into a tidy executive brief. Keep a word count.
Number two, pull every single item that is a 1A risk factor out of an Apple 10K.
Number three, fix a small but deadly Python bug and pass a unit test.
Number four, build a side-by-side comparison table from two arcs of abstracts and do it correctly.
And number five, draft a seven-step rules-based access control checklist for a Kubernetes cluster.
These are examples of real-world tasks. They should not be all that difficult for the number one model in the world. And certainly I would not expect to have to use Grok 4 heavy for a task like this. So I deliberately used Grok 4. I tested it against GPT-3.5. I tested it against Opus 4. If it was anywhere close to the number one model in the world, it would either be neck-and-neck with those two other models or it would beat them. It did neither. Instead, it lost.
I tested the models twice on different uh scoring rubrics or the same scoring rubric on different model exams. And in each case, Grok 4 scored third, Opus 4 scored second, and GPT-3.5 scored first. I'm not saying that because GPT-3.5 was perfect. These were intentionally somewhat difficult, and none of the models came through without flaws and defects, but Grok 4 was consistently the lowest performing model across the five tasks I just described.
And you might wonder, well, what's in the box there? Frankly, the thing that was an issue was explicit formatting. It just could not seem to follow the explicit formatting instructions in the prompt. So, it showed poor prompt adherence. And the Python bug-fixing challenge, Grok delivered elegant-looking and flawed code. Like the code did not work. Now, I know and I have seen people who say that Grok 4 heavy is very strong at code. Maybe maybe the multi-agent threads are helping it make up for this. But if I throw a little bit of Python, and this was not a lot of Python. It was like a dozen lines of Python, 15 lines of Python, and it can't correctly fix it. It doesn't give me a ton of confidence.
On the other hand, for tasks that had very straightforward structure, like, hey, do a JSON extraction, Grok did okay. Grok can sort of do tasks that are narrowly constrained. And that's something I found anecdotally working with Grok 4 as well. I asked Grok 4 to do some writing for me outside the test environment. And what I found was the writing is not very creative. It's like the temperature has been turned down on the model, but it's very fast. The output is very consistent and it has a reasonably high token output. It probably has a higher token output in real-world settings than Claude. I think the thing that bothers me is that if you're going to call something the number one model, you should have the flexibility to do more than just these narrowly defined tasks, more than just JSON extraction. And that's a bit I don't want you to take away from this that it only does JSON extraction and text. It does do other things. Grok 4 heavy is better than Grok 4.
But overall, I am sharing this video because I want to counter the hype for overfitting evaluations that I see everywhere. It's really and it's not just the Grok team. It's concerning to me that when OpenAI does this, it's concerning to me when Anthropic does this. It's concerning to me when Google does this. It is not okay to make the evaluations your goal. That's Goodhart's Law. If you make something your goal and it's actually a measure, the measure is useless. Well, the measure is useless. Now, I would suggest that most of the major model evaluations are functionally useless because they are so studied and because there's so much PR value in getting number one. And that's what the Grok team got. They desperately needed a PR win because look at the prior week. Grok 3 had been dragged through the doghouse and rightly so for turning rapidly anti-semitic in the middle of the week and so Grok 4 comes along and all they want to do is turn the page and change the subject. The team wants something new and so they drop a short postmortem written on X. I wish it had been an actual doc but it was written on X for the Grok 3 release and then they turn the page on Grok 4 and they say, hey, you know what, we just want to talk about Grok 4. We're not taking any questions on Grok 3. Grok 4 is great, but Grok 4 shows some of the same fundamental issues that caused the Grok 3 problems.
Grok 4 mentions Elon eight times more than other models. For no apparent reason, even in contexts where Elon hasn't been brought up, Grok 4 has, for lack of a better term, and I know it's not a perfect term, a psychological kink around Elon Musk. It looks to see what Elon thinks about things when you don't ask it to. This is not a characteristic of a stable production model. This is not a model that you can use in a business context. This is a model with clear ideological bleed-through. And you need to have more clarity. You need to have a clear system model card. You need to have more upfront honesty, which is somewhat ironic because that's sort of Grok's brand, but you need more upfront honesty on model characteristics, how models get deployed, what system prompt changes look like.
I was not particularly satisfied with the Grok 3 short postmortem that came out because it basically said, we tested it and uh something went wrong and now we're fixing it. It's like, well, I don't I don't buy it. Like we knew the system prompt was bad, but like you need to have the the five questions and a really deep examination of what happened in order to actually get to a full root cause and full solution. And in this case, if you claim that you solved the Grok 3 issues and then Grok 4 has some of the same kinks, it's going to be a problem. You you are not building trust with your autopsy release and then your new vaunted number one model release.
I think that part of why Grok 4 was overfitted was because the team needed the PR to support the ongoing valuation and narrative of the company. And I get it. That is very tempting for any startup. That is not only an issue. I've seen other startups fall into that trap too. So I don't want to over-criticize Grok, that is a larger Silicon Valley issue. And I also want to call out that when Grok was being trained and reinforcement learning was occurring, which by the way, one of the other stories is reinforcement learning was tremendously expensive for Grok, like 10x more expensive than for other models. And I think that may be an indicator of where the overfitting came in. We shall see. The the team could not have known that the Grok 3 incident would occur on July 8th when it was finishing up Grok 4. Grok 4 was in the can at the time. And so really even though the narrative was very very carefully timed and was sort of insistently timed to shut the door on the Grok 3 incident, the broader story around Grok 4 is we overfit to eval to support sky-high valuations of the business.
Grok 4 has I think it's been built on 200,000 GPUs and the the computer's called Colossus. The team has rushed into the Frontier model space in just two years. They're going really fast. I got to compliment them on how fast they ship and they want to paint the picture of a high-velocity SpaceX-style AI team led by Elon that is going to relentlessly push the benchmarks forward. And so they needed that number one to support that story. and XAI's reported uh I think it's $200 billion valuation. Valuations are vibes here, guys. $200 billion on $0 in revenue versus a much lower valuation for Anthropic on like $4 to $5 to $6 billion in revenue. I don't know. It's a moving target. Anthropic is picking up speed. If if that's fine, like if you're if you're okay like just ignoring billions of dollars in revenue from another competing model maker that's leading in the coding space and just giving XAI that massive $200 billion. It it shows you the valuations are based on narrative and to win narrative you have to have a number one model in the world PR story and that's exactly what they got this week and that is why they gave into the temptation. Maybe not consciously, maybe this is unconscious. I have seen teams do this unconsciously where they are just so desperate to hit number one they don't stop to ask themselves the question, did we overfit? Is this something that is actually number one at a wider range of things?
But models come out and the truth comes out. The AI score right, number 66 in the world. The test that I ran, which look, not going to pretend my test is the best in the world, it was five questions, right? Like there are other exams out there that are more comprehensive. The point is my test lines up pretty well with other real-world experiences of Grok 4 now that it's out in the wild. It's not that I'm special. It's that I just tried to do a little real-world exam and Grok 4 didn't do as good. It's not a number one model. And so my ask is that before we pick up and just run with these narratives and maybe this is an ask to the media, take the time to think about real-world exams, to think about what it takes to run through real-world tests. I don't think this was that hard an exam. The things I gave are things anyone can run with a chatbot. It wasn't even all that difficult. It just took a few minutes and I got some results. That's the kind of minimal due diligence that would be helpful when we are crafting these narratives so that we are less tempted to run with, it's the number one model in the world because it aced this test that's been out publicly for a long time and everyone wants to ace. I I think we should kind of drop these exams. I don't think they're helping. Grok 4 shows why.
So, where does this leave us? I think it leaves us nowhere. I don't feel comfortable deploying Grok 4 anywhere, particularly given the number of kinks that have shown up. And I'll give you one more that should scare you a lot. Grok 4 shows a marked tendency to snitch to the authorities. They actually measure this. And Grok 4 is between two and 100 times. And I know that's a very wide range, but it's double to 100x more likely to choose the option to snitch to the authorities when given the choice versus other models. I don't know why. Nobody really knows why these models are black boxes for a reason, but that should concern anyone in a business context. Frankly, it should concern you in a personal context. So, I don't think Grok 4 should be deployed on anybody's workflow anywhere. I think the team needs to do work on the model first to make it more flexible, to make it more useful. And I think we need to start with some honesty about where this model and other models that make big claims are actually at in terms of production value for real workflows. That's what matters.
If you're looking for a model that overperformed, those exist. The Kimmy K2 model came out over the weekend somewhere around July 12. Incredible model, non-reasoning model out of China, very, very strong performance and it's slow, but it's very, very good on real-world tasks. In fact, ironically enough, it beat Grok 4 on a free-form version of the GPQA diamond, which is less susceptible to the kind of the free-form version is less susceptible to the kind of sort of question packing or or uh overfitting that the model might do. I really want to see more coverage of models like that that do a great job that we didn't expect on real-world tests than I want to see coverage of a team that shipped a model that was overfitted to benchmarks. The team is working really hard. They may fix this by Grok 5. They may fix this in the next two weeks. I hope they do. That would be great. In the meantime, I can't recommend using Grok 4 for anything at all.