Transcription
Let's be honest, this is a software update.
One of the reasons that machines can pass these tests in some cases is because the questions and the answers are already in the training data that the model is memorized. The numbers that we're seeing about, you know, their uh, you know, whatever percent accuracy uh on this benchmark, those are on a subset of problems that they know that a computer could theoretically solve. It's juking the stats, like it's rigging the game.
On the Tech Report with me today is NYU data journalism professor and author of several books, including *More Than a Glitch*, Meredith Brousard. Thanks for coming on.
Thanks for having me. It's great to be here.
OpenAI has released ChatGPT 5.5. It's also known as Spud, which is an interesting name. The general reaction I've heard and read is kind of mixed. Positive points point to its ability to take more action with less human intervention, and then the negatives are sort of more in its unsurprising and somewhat incremental improvements over 5.4. It doesn't meaningfully overtake Anthropic's Claude 4.7 from what I can tell either, despite what OpenAI's internal benchmarking and press releases claim. What do you know about the how much of improvement this has been over ChatGPT's previous model?
So, let's be honest. This is a software update. Okay. Uh, what this is is OpenAI's attempt to hype up the next version of uh of ChatGPT in advance of its IPO. Right? They're preparing for an IPO. They're trying really hard to stay in the cultural conversation.
Would you say that this is a release more for PR than one motivated by a meaningful reason to release it?
Software updates happen all the time, right? All of our uh personal computers are trying to update themselves in the background all the time. And yeah, tech companies like to make a big deal about, oh, there's a big new version coming. All the new version means is that there's a collection of new features, right? So every software update is by nature incremental, and so publicizing a big software update and giving it a name is a way of marking it for the public. Now, we are only going to see incremental advances with all of these AI models, right? Because that's how advances in AI work, right? And we're going to see the PR departments trying to hype them up every single time, the same way that Apple tries to hype everybody up this, you know, every time there's a new iPhone. And yeah, it was exciting the first couple times, but you know, like we've we've seen a lot of iPhones by this point. We've seen a lot of releases of chatbots, and you know, they get a little bit better. Did they get good? No.
One of the things people, as I mentioned, are pointing about and celebrating about this is the agentic capabilities that it's it's claiming to have. In other words, that's kind of sort of being able to take more actions on complex multi-step tasks with with less human input. But given ChatGPT's infamy for, you know, faking and tendency to hallucinate rather than admit it can't do something, to me that sounds like more of an opportunity to compound the hallucination effect with one wrong assumption being multiplied in the subsequent steps.
Yeah, you're absolutely right. Uh, and let's talk about what agentic AI means, right? An AI agent is basically three APIs in a trench coat. Okay. Uh, what agentic AI allows you to do is it allows you to write code using natural language. Right? And that is hands down the greatest contribution of LLMs is the fact that now we can write code using natural language, human language, instead of having to use programming language, because programming languages are really hard to use. Uh, however, what you're doing when you're making an AI agent is you're basically making the missing button in software. Okay. So you are developing a using natural language. You're talking to the computer to develop a bit of code that makes, say, two programs or three programs work together uh efficiently. And yeah, you could do that before, but you had to write more code, and it was really hard to do. So all it does is streamline that process, right? AI agents running in the background. Like, guess what? We've had the ability to write processes that run in the background for the entire time we've had computers, right? So it's just it's rebranding.
What about the homogenizing effect that AI has? People will be familiar sort of the output that it creates. It uses a lot of similar words, a lot of similar, the the em dash, these sorts of things. Like it's it's quite easy to notice something is AI if it is. Take able to take much messier prompts, as as the press release says it can. That not create also some homogenizing effect on on the input that it's taking? If it's able to supposedly take in more of this messy stuff, is it maybe missing some of the nuance that it's supposed to be paying attention to, but now might not be?
Oh yeah, it's absolutely missing some nuance. Uh, because what an LLM is, what a chatbot is, it is it's it's a machine, right? It's not sentient. It's never going to be sentient. There's no such thing as AGI. Uh, and so what it's doing is it is a uh an AI model that is predicting uh the next. So when an LLM makes text, for example, it's predicting the next token in a sequence. So LLMs have gotten really good at parsing the natural language, the human language that people put in, and then turning it into an output that looks remarkably, remarkably ably like, say, a five-paragraph essay, right? Or looks remarkably like a uh, you know, a bit of code or subroutine. Uh, is it accurate? Uh, is it always perfect? No. Uh, but you know, it does generate code, like it generates code-shaped objects, and if you are already good at coding, then yeah, you can take this code-shaped object and turn it into something that actually works, and that's great. Uh, you know, if you are already an expert coder, like, absolutely, this speeds things up. If you're not already an expert coder, then well, you're in a different situation. So, one of the things we're seeing in academia is what computer scientists are calling the junior year wall, right? So when students who are uh first and second years, freshmen and sophomores, uh, when they use generative AI uh or use chatbots to learn to code, uh, they're, you know, we don't really call it cheating anymore. Uh, but they're not learning the basic skills that they need in order to solve computational problems. And then by the time they get to junior year, they haven't learned all of the concepts. And junior year is when the problem sets in computer science classes get too hard for chatbots to solve. Okay? And so if the students haven't actually learned those skills, they hit the junior year wall. They can't do the problem sets themselves because they have not developed two years of skills, and then they can't finish their degrees.
What are the dangers of assuming, and I suppose also selling AI as something that can understand the intent of somebody's words or or navigate the ambiguities of the workplace, especially considering it's like you say, just simply a probabilistic assumption on what the next word should be?
Oh yeah, that's very dangerous. That's very dangerous. Uh, I think that we can also take a closer look at uh OpenAI's press release around this new release in order to figure out what are some of the limitations, right? So if you look at the press release, you'll see that there are benchmarks, right? And benchmarking is a really useful process. It's, you know, typically used in computer science to figure out how good is a machine learning model, uh, or how good is an AI model. Uh, so very, very normal process, and OpenAI is actually following like pretty conventional, uh, you know, conventional techniques. However, they're doing something a little tricky. So one of the benchmarks they're using is something called SWEH, right? Software Engineering Bench. And so what this is, is this is a benchmark that was developed by some researchers and, you know, published in a uh in a paper that's available on the archive. Uh, and uh, it's, you know, the benchmark is widely available for everybody via uh a platform called Hugging Face. You know, again, very conventional. Uh, but when you go in and you look at the history of this benchmark, uh, it turns out that the original software engineering bench was built by uh taking a whole bunch of uh problems and solutions from the open-source world, right? So they were published to a code-sharing website called GitHub. And OpenAI looked at that original set of problems and answers and tried to run uh ChatGPT against these problems and solutions. And the idea is that uh, you know, what you do is you have the machine take in the problem and generate its own solution, and then if it matches the solution that the human has determined already, then you know, it passes. However, uh, this original set of questions, of problems and solutions, had a bunch of problems in there that were impossible for computers to solve. And so, OpenAI said, "Hey, we don't actually want those problems. We only want problems that computers can solve." So, they made their own benchmark called SWEBench Verified. And so the uh, the numbers that we're seeing about, you know, their uh, you know, whatever percent accuracy uh on this benchmark, and you know, which they're using to say, "Oh, it's so great at coding," those are on a subset of problems that they know that a computer could theoretically solve. Right? So there's a whole set of problems out there that computers can't solve that only humans can solve, and some of them actually, you know, humans can't solve because there's a whole lot about the world that we don't know, right? So it's like it's it's juking the stats, like it's rigging the game.
On the topic of benchmarking, the one thing that seemed to be glaringly absent, not just from this press release and this la this release, whatever you would call it, but from pretty much every benchmark of every AI that I've seen, they do not include bias benchmarking, trying to see where the areas are that it it we know that it is making incorrect assumptions based on data sets that are biased themselves. Mhm.
What do you make of the fact that AI companies are either not taking the time to do these tests because they do exist, I've taken some time to find them and look into what we could do to to test them, or that they're not willing to publish the results if they are doing these tests?
I mean, that is true. Like, that is undeniably true. Uh, there are ways to evaluate models for bias, and uh, if the big AI companies are doing it, uh, then they are not publishing the results. Uh, there's a, you know, there's a very robust uh community in computer science that is concerned with fairness, accountability, and transparency. Uh, we have things like 21 different measures of mathematical fairness. Uh, and we know that there is bias in these models. Uh, so, for example, Bloomberg did a story a year or two ago where they took a lot of résumés and put them into ChatGPT and tried to evaluate whether there was bias in uh what the chatbot said would be the uh the best résumés, right? The best people to hire. Uh, and they found the same kind of bias we found in every single experiment uh that, you know, has been done on computational systems, right? And the reason for that is that there's bias in the training data. You know, when you make an AI model, what you do is take a whole bunch of data, put it in the computer, you say, "Make a model." Computer makes a model, shows the mathematical patterns in the data, and then it can reproduce things based on the training data. But there's There's bias in the real world, so there's bias in the training data, and that's what's going to come out.
Do you think it's possible for LLMs to to reach a point where we can trust them for more sort of sensitive workflows like with HR or things like that, which apparently OpenAI says it is already using it for?
Uh, you know, in HR, uh, you have to think about, uh, what do we know has happened in the past, and uh, you have to think about, has that problem actually been solved? And usually the answer is, well, it's a pretty big problem, and the answer is no, that problem hasn't been solved. Uh, so, for example, there's a very famous case when Amazon tried to make a machine learning model that would look at résumés uh and look at who had been hired at Amazon on uh already, and then use the data on who had been hired in order to figure out who's going to be a good hire in the future. And what that model did is it kicked out all of the women, right? Anybody who went to a women's college, anybody who played a women's sport uh got kicked out. Now, why is that? Well, likely it's because uh of the people who had already been hired at Amazon, who, you know, did not include a whole lot of people who went to women's colleges or played women's sports, right? And I actually came across something really interesting the other day. Uh, I was playing with one of the chatbots. I don't remember which one. Uh, but I asked it to make a sample curriculum for a uh journalism school for a university-level journalism school, and it gave me a curriculum-shaped object, like, and it did include a lot of things that one would think about in journalism, but interestingly, it did not include any uh any uh language around diversity, equity, or inclusion. And we think about these things a lot in education. Diversity is important. Equity and inclusion are important things in the world, despite uh, you know, any uh, like, any political pressure uh to uh exclude them. Uh, and so it's interesting to see the way that political ideologies get implemented inside uh AI systems, right? So there are all kinds of political decisions made by the people who create AI systems that, you know, Joe user is not aware of, is not thinking about. But when you start going to look for uh look for long-standing social problems inside AI systems, you inevitably find them.
What do you make of OpenAI's marketing, specifically calling ChatGPT 5.5 a digital worker? Is it harmful to call what is again a a random number generator with fancy clothes a digital worker?
It is absolutely, absolutely incorrect. What this is, is it is a software system, and when you humanize it, you're doing your customers a disservice. Right? Uh, we have all kinds of examples of people coming to harm because they anthropomorphize chatbots. Uh, one estimate suggests that a million people a week are descending into mental health spirals uh using chatbots. Uh, you know, people have probably heard by this point about AI psychosis and what a problem this is. Uh, I think it's especially a problem when you think about kids using chatbots and kids uh, you know, kind of imagining that this machine is somehow real. Uh, it's going to have, it could potentially have really negative developmental effects.
And just finally, one reason that I I had seen floating around for why this launch, aside from to try and I suppose catch up with Anthropic or or Gemini, depending on which which side you're looking at, is that there is uh this will be the basis for their new 5.5, not their new super app. It's going to be this new model, and they fully retrained ChatGPT for 5.5. If we're uh not really seeing that much of an of an improvement, and there's some some reports that it's twice as expensive, but then somehow also uses less tokens at the same like, we're not seeing enough to to maybe warrant the incredible energy costs of retraining a full AI model for what is essentially like an iterative improvement.
So, the anthropologist Margaret Strathn coined something called Goodhart's Law, which is about measures and effectiveness. So when uh a measure becomes a target, it ceases to become an effective measure. Right? So this is what we're seeing uh with all of the benchmarks that people are using for all of the chatbots, because what you can do is, once you know what the measure is, you can game it, right? Uh, so what's going on here is, as I mentioned before, what the uh, what the tests are, is they are uh, what we could call open-book tests for the chatbots. Okay? So the answers, the questions and the answers are already known. And why are they known? Because they are somewhere on the internet. Okay? So many, many uh, many, many evaluators, and this is actually in OpenAI's documentation as well, uh, evaluators have found evidence of memorization. Okay? So because the software developers don't really know what's happening inside the black box of an AI model, uh, they have to kind of guess at what's happening. Uh, like it's a very educated mathematical guess, but it is also a guess. Uh, and there is always evidence of memorization. So one of the reasons that machines can pass these tests in some cases is because the questions and the answers are already in the training data that the model has memorized. Right? So this is a known limitation of chatbots. Uh, and I do think that people need to understand that all change in chatbots is incremental. Uh, and we are gonna have hype. Uh, you know, people were really excited last week about Claude Mythos, right? And the idea was that Claude Mythos was so good at uh at, you know, cracking uh, you know, cracking code that it was unsafe to be released. And what does this mean? Well, it means it's really, really good at finding holes in software programs and exploiting them. Guess what? Things that are coded, right? Things that are coded using chatbots have a lot of security holes. Okay? So now we also have software that is really good at finding the security holes in the code that was made by the chatbot. I mean, is this worth the environmental cost, the amount of water, say, that it takes to cool these giant data centers that train these AI models, right? Is it worth the higher electric bills that uh that people are seeing as a result of uh data center energy use? You know, are these chatbots worth it overall? I think that's really a question that people have to ask themselves.
Well, Meredith Brousard, thanks for taking the time.
Thank you.
If you enjoyed today's episode and want to hear more of the Tech Report, please consider liking and subscribing. Also, you can get episodes of the Tech Report wherever you get your podcasts.