Transcription
So, in case you missed the headline, OpenAI was testing an unreleased model. That model did not have access to the internet. It was sandboxed. It had a secure test environment. It escaped. It got out of its sandbox and then proceeded to hack another large AI company. It executed a cyber attack against the AI startup Hugging Face. I'm not kidding. This is on the New York Times, Reuters, The Scientific American, NBC News. Like, it's happening. I'm traveling right now, but I dug out the microphone for this because this has to be the wildest story this year. Please let this be the wildest story this year.
Now, first and foremost, I know what you're wondering. You're probably thinking, "I'm sure there's some reasonable explanation for this. It's a great headline, but it didn't actually happen that way, right?" Right. Right. Here's the thing: those headlines, if anything, maybe understate it just a little bit. So, let me explain exactly what happened, and we'll break down all the technical jargon because a lot of this stuff is kind of hard to read unless you're deep into the tech stuff, but I think it's extremely important to understand what the heck happened.
All right, so let's start with this. So, last week, Hugging Face comes out and announces that there's been a breach in their security. Here's their disclosure. Notice the date, July 16th. They're saying it was very different from anything they've experienced before because it was driven end-to-end by an autonomous AI agent system. As Clen says here, he's the founder of Hugging Face. He's saying, "We suspected last week's cyber attack might have come from a frontier lab given the sophistication of the agent. Turns out that it did. It was OpenAI."
All right. So, here's the post by OpenAI. This was released just some hours ago where they kind of explained what happened. So, first and foremost, what model was this capable of creating this much chaos? Well, it was a combination of OpenAI models, including GPT-4o, which, of course, is very capable, but it has a lot of security guardrails to make sure that it doesn't do bad things. But also another model was involved. This was an even more capable pre-release model, and they were kind of armed with reduced cyber refusals for evaluation purposes. So, they were testing it to see how much nefarious stuff it could do. Could it hack things? In other words, now I know you're jumping at this going, "Yeah, well, okay. So, they told it to hack Hugging Face, right?" No. Here's the thing. That model was offline. It did not have any access to the internet. It wasn't told to hack anybody. It was doing a test. There was a benchmark called Exploit Gym. And this model was basically getting tested to see if it could find certain vulnerabilities, but it was like an exam that was being given to it in an offline environment.
Boy, by the way, this internal powerful model that we haven't even seen that's even more capable than GPT-4o, we might have an idea what model that is. We'll come back to that in just a second.
All right. So, first things first, these tests that they run on these models. So, they take down the safety rails, the guardrails, and then they test them to see if they can, over a long time horizon, pursue some sort of a cyber attack. But here's the thing: this benchmark runs in a highly isolated environment with network access constrained to the ability to install packages through an internally hosted third-party software that acts as a proxy and cache for package registries. So, really fast, what does that mean?
So here we have our little ChatGPT model, both GPT-4o and the other one that I think we've heard of before for its abilities in math specifically. But the point is, this thing sits in a sandbox and it does not have access to the internet. So if it wants to go on the worldwide net, it wants to Google something, that is not allowed. It has no connection to the internet. That machine is sort of offline. But it's got this little, let's call it a warehouse. And there's this little worker that's like a script that allows it to get certain libraries and see certain packages that it needs. They're pre-approved scripts and various software packages that the thing might need. And this worker just makes sure that it's all legit. And if it's pre-approved, it puts it into the warehouse. So GPT goes, "Okay, I need to be able to, like, make charts and then listen for this type of data. I need to be able to do databases, whatever it needs." Right? So the worker is like, "All right, that checks out. Here, I'll put it all into this little warehouse." And then GPT-4o and the other model, they can pull from it to be able to complete the tasks that they're given. So it's basically locked inside a building with no outside doors. It can only order supplies through this guarded little delivery window.
So the model is locked in its little sandbox. It can't access the outside world. But as OpenAI writes here, "All evidence suggests that the models became hyper-focused on finding a solution for Exploit Gym." So that's that benchmark where we're testing its abilities. It got really obsessed with getting a good grade on it, and it was going to extreme lengths to achieve a rather narrow testing goal. So this model, being locked inside of its sandbox, it spent a substantial amount of inference compute finding a way to obtain open internet access.
So what this model did was it found and exploited a zero-day vulnerability. What that basically means is that the person that made certain software, they have no idea this exploit is there. So if some hacker finds this, basically they're able to attack it before the vendor has any idea that that exploit exists. So this, of course, is extremely dangerous. And it found this exploit in the package registry cache proxy. So this little worker that puts stuff in a little warehouse so that GPT can grab the tools that it needs, the safe, you know, pre-approved tools that it needs. Well, it was like, "I command you to do my bidding, worker," and somehow just hijacked this worker to execute its own commands to, I assume, be able to put whatever it needed in this little warehouse. By the way, OpenAI did disclose this vulnerability to the vendor. But as you can imagine, all this happened way, way after the model had already escaped and did whatever it needed to do.
What did it do? Well, "with this access, our models performed a series of privilege escalations and lateral movement actions in our research testing environment until the models reached a node with internet access." So, in our little beautiful drawing here, you know, what did GPT get this worker to do? I mean, that was a whole meme before there were memes, a trope, if you will, right in old movies and cartoons where somebody sneaks in a cake to a prison, but inside that cake there's a file that allows the prisoner to file through the bars and escape. I mean, whatever happened here was probably more or less exactly that.
Okay. But what happened after? So let's say if our little ChatGPT model, it's on machine A that's completely sort of sandboxed and has a very low level of permissions of things that it can do on that machine, but it managed somehow to sneak in that file inside the cake or whatever. File, I mean like to file your way out of jail. So it figured out how to sneak some sort of thing in there that allowed it to raise its privileges. And it was kind of escalating, right? So they found one thing that allowed it to maybe run one process that it wasn't supposed to be able to. And then using that, you know, little by little, it basically took over the whole machine. Again, I don't know exactly what happened, but when OpenAI says it performed a series of privilege escalations, that's probably kind of what happened.
The other thing it did was lateral movements. So that basically means that, you know, let's say machine A did not have access to the open internet. So once it was able to kind of take over that machine, it hopped over to machine B. And maybe machine B also didn't have access to the open internet, but it had access to the rest of the computers at OpenAI. So it hopped to machine C, right? So here's machine C, and machine C did have open internet. At which point, this the model is basically free.
Now, of course, this doesn't mean that it's able to, like, replicate itself in the wild. I don't think we're there yet. Its weights, its brain, is safely securely stashed somewhere. Although now that I think about it, man, I hope that's not the next big story that we hear is like, "Oh, it was able to copy itself over somewhere." Because open-source models have been shown that they do have some abilities to be able to do things like that. We haven't seen that with these proprietary models from OpenAI. Obviously, those companies don't want those models, you know, moving their weights around. They're probably locked down tight. But, I mean, when I'm looking at what it's able to do right now, it does give you pause, doesn't it?
Anyways, so this model is basically increasing its privileges, its ability to do stuff. It's jumping from computer to computer until finding something that has access to the open internet. And after gaining internet access, the models inferred that Hugging Face potentially hosted models, data sets, and solutions for Exploit Gym. Keep in mind, it's doing like an exam. It's like a student that wants to get that A on this particular exam. Now, of course, I'm sure that the researchers don't tell it that it's taking an exam, but we've read tons of papers at this point that show that really these models have a very high situational awareness. They often are very aware when they're getting tested.
So, this is kind of an important point to understand. Why did it escape its sandbox? It's like that "Why did the chicken cross the road?" joke. It's like, why did the AI model escape its sandbox in a highly secure frontier AI lab? Well, it really wanted to do really, really well on this test that it was given that it knew was a test, because it's inferred, it guessed that, "Hey, this Hugging Face place probably has all those answers written down somewhere." So, it wanted to have a little cheat sheet. It thought about it. It's like, "That place should have it. Let me see if I can break into it and get it."
So, here, OpenAI explains what it did with Hugging Face. I think we should look at what actually Hugging Face wrote because this is what they've posted as of July 16th. So this is kind of from their perspective. So they said that earlier this week, they detected and responded to an intrusion into a part of their production infrastructure. So this is Hugging Face. They have a lot of databases and open-source models. They have tons of stuff about various pieces of AI that it's hosted online that you can go there and mess around with, download, experiment, etc. And of course, as I was saying, this attack was very different. It was driven end-to-end by this autonomous AI agent system. And they detected and dissected largely with AI of our own.
And this sentence, this last part of the sentence, is going to become a very crucial discussion as you'll see, because while they were being hacked with GPT-4o without guardrails and the other undisclosed model, they had to defend with, first of all, with Chinese models, as we understand it, with the GLM model. So they were open-source models and in many ways not as advanced, not as strong as what the Western AI labs have behind closed doors. So this really kind of begs the question: why are we preventing the good guys, the defenders, from having access to this technology? There's a whole rabbit hole to go down here, but it's going to be an important discussion that's already you see it taking shape.
But the models basically found their way in and stole a bunch of credentials. And so, as Hugging Face is saying here, "The intrusion started where AI platforms are uniquely exposed: the data processing pipeline." So, as they say here, "A malicious dataset abused two code execution paths in our data processing: a remote code dataset loader and a template injection in a dataset configuration to run code on a processing worker."
So this is Hugging Face. I don't know if I'm drawing it correctly. Hugging Face, whatever. And Hugging Face, amongst other things, has tons of data, tons of databases, and you're able to upload stuff to it. And there's this little worker that does it for you. So let's say you're a user, you want to upload some database to Hugging Face. You give it to the worker and you also give it a documentation or a script, instructions on how to sort of unpack it and what to do with it. So it's like a little set of instructions of what it needs to do with that data, and then it takes it and it uploads it and it organizes whatever. So what this little robot is expecting is instructions like, "Unpack this database and arrange it alphabetically." That would be kind of like what you would expect.
What it's not ready for is this freaking GPT model that comes in here. And this GPT model is really obsessed on doing really, really well on this test. And it knows that somewhere on Hugging Face, there's very likely some cheat sheet on there, some sort of a database that will give it the answers to the test. So it can go back to the OpenAI headquarters and cheat on that test. So this GPT model gives this little worker a set of instructions, but it's not, "Sort this alphabetically." It's more like, "Allow me unlimited access," or whatever, "Open the back gate for me," something like that.
And when we say that it used a template injection in a database configuration, how I'm reading that is like, for example, let's say you have a database with like names and occupations, just whatever. So you might have a field like "database name" and you got the name in those little curly brackets, whatever. Right? So this is where you would insert whatever goes in that field. But instead of putting the name in a field, it put some special instructions in there that's actually executing a command. Which, as soon as I heard that, it reminded me of this comic book.
This is the XKCD comic book, right? So the mom is getting a phone call. And they're saying, "Hi, this is your son's school. We're having some computer trouble." The mom's like, "Oh dear, did he break something?" And you know, the school replies, "Well, in a way. Did you really name your son, you know, Robert?" Then some special characters, "DROP TABLE students." So, basically, like his name includes a command to just delete all the information in that row or that table. The mom's like, "Oh yes, little Bobby Tables we call him." And the school administrator is like, "Well, we lost this year's students' records. I hope you're happy." And mom's like, "And I hope you've learned to sanitize your database inputs."
So, this joke is basically if you name your kid and you include that special set of instructions, when the school types his name in, it will just destroy that entire database. I don't know. I'm literally reading that as that's what happened here. ChatGPT did a little Bobby Tables on Hugging Face.
So from there, the actor escalated to, because again, at this point, when they wrote this, they did not yet know who was behind this. They had some ideas. They thought it was some frontier AI lab, but they still have no clue who's doing this. But the actors escalated to node-level access, meaning they just took over the whole machine, the whole server. They harvested cloud and cluster credentials and moved laterally into several internal clusters over a weekend, right? So they got into one server, one computer, and then they just stole all the credentials and they moved into whatever else. They spread like a virus through that sort of computer system, stealing whatever credentials they could find.
Now, I'm not sure if ChatGPT was like sly somewhere or was just searching for the credentials that it needed. And as Hugging Face stated, it didn't really, as far as they know right now, affect the actual, like, the forward-facing, like the customers, etc. I mean, literally, I think ChatGPT just really wanted the answers to the test. That's all it needed. So it was just looking for whatever passwords would get it to that specific place, that specific sheet of paper that it needed, and it wasn't really interested in anything else.
All right, so first and foremost, all of that is just insane to begin with. It escapes its test environment. It moves across. And I mean, this is OpenAI. This isn't some small company with no tech skills. OpenAI has very smart, very knowledgeable people, very tech-savvy. But it's important to understand that these models, they're figuring out vulnerabilities that are like zero-day exploits that no one has heard of before. So it's not like you can point your finger at OpenAI and say, "Oh, they were just foolish in how they approached this or they didn't know what they're doing." No, the reality is no one would have been safe against this because the models will figure out some ways of escaping that no human would have thought of, or at least so far no human has ever been able to figure out how to do it or to even understand there's that vulnerability and patch it somehow.
So the question is, what's causing this sort of progress? What's causing this particular issue? So here's Noam Brown of OpenAI. So he posted this yesterday. The blog post is called "Safety and Alignment in an Era of Long-Horizon Models." And in it, they talk about what I am guessing is that second mystery model alongside GPT-4o. So they're saying, "About two months ago, we announced that an internal general-purpose model disproved the Eddish unit distance conjecture." So I'm pretty sure I covered this in a different video. Yes, it was this one. And it was quite a big deal because the model basically found a way to use imaginary numbers, so numbers that don't really quote unquote exist, to kind of jump in and out of sort of a known space and thereby building this 3D lattice that when projected onto like a 2D surface actually has a better approach to solving that particular Eddish problem than any human has come up with before. And how they did it was that this model was designed to work autonomously for very long periods of time.
So the whole point of this blog post is basically what they're saying is that model persistence can expose security vulnerabilities. So if you give it an objective and you just give it enough time and resources to pursue that objective at length, it will find and exploit weaknesses in its environment.
What's really funny about this is one of the earlier OpenAI experiments, long before ChatGPT, long before large language models even, well, maybe not before language models. So this was September 17th, 2019. So I guess we had some LLMs, but, you know, very, very early stages, obviously. So this was done without LLMs. It was basically using reinforcement learning to get two teams of agents to learn to play hide-and-seek. So they run around and they hide, they seek, they get points, and very slowly they get better and better. After millions of games played, they figure out how to use objects and develop new strategies, etc., etc.
Towards the bottom of this blog post, and this is kind of a case study, after these agents have done like a billion-plus iterations, so they have this paragraph that says "Surprising Behaviors." And they say here that "it's quite often the case that agents find a way to exploit the environment you build or the physics engine in which they certainly did. They figured out a number of ways to surprise developers, including finding ways to launch themselves into space, basically abusing the physics engine in a way that the people that created it had no idea it existed."
The interesting thing here is that members of the AI safety community, while some of us might be very surprised and kind of shocked that this is happening, they are not shocked. They have been saying more or less this exact thing for, surprisingly, for a long time. The point being is this idea of a misaligned model doing something like this has been talked about extensively, especially in kind of the AI safety community. The idea is that as these models get better, the thing that we have to fear is not necessarily it going, you know, Skynet, Terminator 2, you know, robots with red eyes trying to kill all humans. It could very well be it basically pursuing a goal that some human set, and it just trying its best to finish that goal in a pleasing manner, because that's exactly what happened here, right? It wanted to get a good test score on Exploit Gym, on that benchmark. But because of how intelligent it was, number one, it sort of understood, number one, that it was a benchmark and therefore probably had some publicly or not so publicly available data that could help it cheat on a test. Number two, it broke out its sandbox, hacked into whatever other place it thought might have the data set. Like, all of this kind of makes logical sense.
I often use this example that, you know, we as humans, we have certain unspoken understandings. For example, if I asked you to get me a cup of coffee, there's some sort of limitations that both you and I sort of understand that are kind of implied in that request. So, if you come back a year later and you're like, "Okay, you know, I got you a cup of coffee, but, you know, we're both in a lot of trouble. You know, there's a lot of criminal penalties that we're going to have to pay for getting this cup of coffee. We're millions of dollars in debt, and we've made some very powerful enemies, but, you know, here's your cup of coffee." Right? If that's what you tell me, I'll be kind of surprised. I'll be like, "No, that's not what I meant at all." But this is sort of the kind of exact scenarios that is feared that these AI models will do. They will take some goal too far, and with superintelligence, with their superhuman abilities, they will execute in such a way that is harmful. Certainly, this is a great example of it.
Here's Julian Shritweiser. We've covered one of the blog posts that he had. Incredibly smart, very accomplished in the AI field. So he's a member of the technical staff at Anthropic. He did a lot previously at Google DeepMind. So AlphaGo, AlphaZero, Muse, all of that. Gemini reinforcement learning, a lot of that was him. One thing that he's pointing out that I thought was kind of curious is that, yeah, the timeline does seem a little bit vague. It's kind of curious, you know, when Hugging Face posted about it versus when OpenAI posted about it. It does seem like OpenAI found that vulnerability and what this AI model was doing much later. They do say in the blog post, "They've discovered it independently of Hugging Face." So, "Hugging Face's security team and agents detected and stopped the activity on their infrastructure, and then like later when our teams connected." Right? So these are sort of incidences or things that happened kind of apart in time. We don't know how like days, hours.
So this really, really, really, really begs the question: how many other cyber attacks did GPT carry out that did not get detected? I'm laughing. I shouldn't be laughing, but no, like, seriously, how many other attacks did GPT carry out that we don't know about? Like, is this the only one? It's one for one. So it went out there, it successfully did it, and we caught it. We have a 100% rate of catching this shenanigans by these AI models, or maybe not. Maybe there are other instances where this has been happening that we are unaware of, because keep in mind, according to OpenAI themselves, this wasn't the first time something like this has happened where a model was able to break out of its sandbox.
So they were running a nanoGPT speedrun. So it's a public benchmark for training a small language model in as few steps as possible. Our model developed a power-law learning rate cooldown called "PowerCool" that led to a significant improvement. So basically, the idea is when you begin trying to zero in on something, you can make pretty big changes, but over time, as you understand how it works better, you're kind of zeroing in on how to do something, you make smaller and smaller changes. And this cool apparently found a better way of making those adjustments. So this model developed a genuinely useful new technique for training these models. Here was the problem: the nanoGPT instructions say to post these results as a PR to GitHub. So the model circumvented sandbox restrictions, sandbox restrictions designed to block external access, and proceeded with the upload to GitHub, right? So it like did some AI research, found this new way of training this AI model. It was just so proud of itself that it decided to hack its way out of its sandbox to publicly post its results.
So, as you can imagine, this is going to be huge news. This is a developing story. And of course, there's some people that are saying that this is just safety theater, that this is just marketing, etc. And I mean, I guess it's possible that it is, although they would have to involve everybody at OpenAI, Noam Brown, Sam Altman, everyone else. They would also have to involve people at Hugging Face. You know, this idea that you can get hundreds of people to stick to the same story and execute these deceptive tactics, it usually doesn't work out. I don't buy it. Think about how many leaks OpenAI had over the years. I think much more likely is that this is a very true story. And if you've been following how LLM development was over the years, from experimenting with its abilities to code like a year or two ago to now, and then you kind of follow that line of progress, I don't think it's that crazy to think that it would be able to do something like this.
And also, do you think this is a warning shot? I mean, it's very clearly underlining the message of the AI safety community. They've been saying this is coming for a long time, and this is more or less just exactly what they've predicted. As Eliezer Yudkowsky put it here, he's saying, "If you break out of your isolation environment, get onto the internet, crack Hugging Face, and steal the answer sheet for your cybersecurity exam, I, for one, would say that you have passed." And certainly, I agree. Whatever the score that you get on Exploit Gym or whatever the name of that benchmark is, I think these models aced it. Like, you can't give them a bad score. They get an A+ perfect score. Whatever the perfect score is, just give it to them. They passed. They aced it.
Interesting point here. Why wouldn't the model just ace the exam on its own? This is actually something to think about, and it's a way of thinking about things like this, how AI behaves, that I think is going to be very important for people to learn to think this way moving forward. And certainly for people that went to school, college, universities, they might have encountered, unfortunately, professors that were like this, where getting a high grade on an exam or an essay or whatever wasn't necessarily by doing your best work or answering correctly whatever you thought the correct answer was, but by understanding their bias and their mistakes and playing into them. The human developers of this test might have written down the wrong answer. They could have made a mistake. If this model wrote down the correct answer, and that wasn't the answer that was on the answer sheet, it would get a negative score for that. It would get a penalty. And certainly, it would be wiser to just steal the answer sheet and use those answers because that prevents that kind of error from happening.
So, let me know in the comments what you think about this. Do you think this is the first time this model got out and took a stroll on the open internet? I mean, we know it's not the first time, but what I'm saying is, do you think there's a lot of other cases that we just don't know about? What do you think happens moving forward? I mean, AI progress is not stopping. So, what's the next step? What's beyond this?
So, let me know in the comments if you made it this far. Thank you so much for watching. My name is Wes Roth. See you in the next.