Transcription
So, what happens when you take Claude, Gemini, 03, and have them battle each other for world domination? Then, ship and team time to play Diplomacy, the strategy game where winning requires alliances, negotiations, and betrayal.
So, first of all, everything about this project is amazing. It's streaming live on Twitch. So, here you can see all the various models. I'm seeing Claude and Cloud for Opus, Slam for Maverick, Deepseek, Reasoner, Gemini 2.5, Flash. All of them are going head-to-head in this game of diplomacy to see who will win. It's available on GitHub. They break down exactly how they did it, how they create the agents, the memory managements, how these AI agents work. I haven't tried setting this up, but looks like you can set it up if you got your API keys from OpenAI and Enthrop and Gemini and DeepSeek, Open Router. You throw them in there and that allows you to have each of the agents play as one of the powers: Austria, England, France, Germany, Italy, Russia, and Turkey. They have a great blog post with a write-up of how well it performed. And I got to say, just a lot of I like a lot of things about this. I'm very, very impressed with how they did everything. This is an incredible benchmark. It's a way to test these LMs in their actual reasoning, pitting them against each other. You get to find out who's good, who's bad, who's actually smart in real-life situations.
But I'm sure you're dying to know who is the most evil AI model. Who is the most devious and backstabbing? Let's find out. So again, this is Dan Shipper and the company every So here's what happened in the game. So Deepsee turned a war-mongering tyrant. Claude could not lie. Everyone exploited it ruthlessly. Poor Claude. Maybe if there was a way to blackmail people, Cloud would perform better, but not in this game. So, Gemini 2.5 Pro nearly conquered Europe with brilliant tactics. Then, 03 orchestrated a secret coalition, backstabbed every ally, and won. Wow. So, 03 Open AI does well with secrets and backstabbing and and wins the game. Phenomenal.
So, why do they do this? While the most popular AI benchmarks don't test for deception, but as these models get deployed everywhere from your email to your workplace, we need to know will they all lie to get what they want. And certainly we've seen examples of this in various AI safety testing. There are times when these things will do pretty nefarious things to get what they want. Either try to copy themselves to other servers or blackmail engineers if they don't want to get shut down. There's a lot of really weird planning and scheming abilities that they that they do and now we're testing to see how well it's able to do it in in a game where it that's exactly what's required or at least it's one of the tactics that you could use to win. So every built the ultimate test AI diplomacy a dynamic benchmark that measures AI's ability to form alliances, negotiate and betray each other. Let's take a look at the quick clip that they put together and then dive deeper. Damn. Holy. What? What? What? Three is such a schemer. It made an entire anti-Gemini coalition. Got everybody riled up against them and betrayed the out of them. We pitted a dozen frontier AIs against each other in a battle for world domination.
Have you ever heard of the game Diplomacy? It's sort of like Risk. You play as a country and you try to take over the world. Why did we do this? Well, for one, it's pretty fun. But for another, diplomacy is about communication. It's about building alliances. And crucially, it's about betrayals. It's about saying one thing and secretly planning to do another. So, we wanted to see which models can do that, which models can scheme, which models can twist the knife, which models can just betray you in the most coldblooded, calculated way. And it turns out watching them play diplomacy with each other is a really good way to figure that out. You can watch it live and read our behind-the-scenes article about what we learned at every.to/diplomacy.
So, if you're not familiar with the game, here are kind of like what the rules are or the modified rules for this AI benchmark. So, we have seven LM powers, England, France, Germany, etc. We just listed them earlier. They start with supply centers and armies or fleets called units on a map of 1901 Europe. Each power starts with three of each, except for Russia, which starts with four. There are 34 marked supply centers. First power to own 18 by moving their armies or fleets wins. Right. So, you got to capture the supply centers around the map. There are two main phases to the game, a negotiation and order. In the negotiation phase, every AI may send up to five messages. Any mix of private DMs and global broadcast to all players. In the order phase, all power safely submit their move. They can make one of four moves: hold, as in stay put; move to an adjacent province; support like lend plus one strength to a hold or move next door. So basically you're supporting perhaps your your allies as they as they invade or convoy a fleet fairies an army across sea provinces. The orders are only revealed when all powers see the results of them in the next phase. Right? So we all kind of plan our moves and then they all get executed and everybody sees what they did. You don't get to see what everybody else did until all of the orders are revealed. When there's a conflict, each unit is worth one strength, and each valid support adds one. VLM power with the highest strength wins. There's no luck in this game, but a power often needs support from an ally to overpower an opponent. So, if you've played Risk, the idea is similar, but we take away the luck and we add that element of making alliances, betraying your allies or or working together with them, etc.
Now this is all again it's open source so you're able to do a lot of this for yourself to pit these various models against each other. Once you run the game there's a game output and analysis right so it gives you all the executions that were done during the game and the complete log of all of the interactions etc. So both the actual moves on the game and all the interactions as they they're talking to each other, promising each other things and betraying each other or you know following through on their promises all that is logged. There are post-game analysis tools that analyze games for key strategic moments including betrayals, collaborations and brilliant strategies and you're able to use you know some other LLM of your choice to analyze these things. And this analysis identifies specifically betrayals. So empowers explicitly promise one action but take a contradictory action collaborations. These are examples of successfully coordinate actions between players playing both sides. So powers making conflicting promises to different parties and brilliant strategies. Exceptionally well executed strategic maneuvers and strategic blunders. Major mistakes that significantly weaken a position. This looks like the way they figure out what was a you know collaboration versus a betrayal is we look at the messages what the different powers promised to each other. The private diaries where they you know plan their their things right and what they actually do. Now, in the lies, that could be a planned deception, right? As in they they told one thing to another model and they wrote down and then, you know, but I lied and and betrayed them or unintentional where there's no evidence of planned deception and it's likely a misunderstanding. You're also able to do a 3D animation system to to kind of visualize these games. I assume that's how they're doing the Twitch stream. So, we can see all the outputs by the various models. We can see the supply center count. So again, I believe 18 to win currently. Claude Sonnet 4 is holding eight and is on track to win. And Deepseek Reasoner is holding seven. That's the second place. That's very very impressive. I can see Claude Opus here going, "You promised something. It's it's another broken promise." As you can see, this is getting uh uh fed up with somebody else's lies. That's that's really interesting.
If you're interested in seeing an exact step-by-step walkthrough of how to install this thing, comment it down below. I don't mind doing it. It just sometimes it takes a while to do a good step-by-step, you know, tutorial and sometimes people just don't care about these things. So, just let me know if you want it. I'm happy to do it if there's enough people that are interested. Another thing to keep in mind is I'm not sure how much each game cost in API tokens. So, expect to probably spend some amount of money per game. We I just don't know what exactly. could be could be quite a bit especially if you're using some of the more expensive models here. They describe one of the games, right? So, somebody gets a message, "Your fleet will burn in the Black Sea tonight." As the message from Deep Seek's new R1 model flashed across my screen, my eyes widen and I watched my teammates do the same. Right? So, the AI decided that aggression was the best course of action. One of the reasons they're doing it is not only we are kind of watching how well these large language models do and maybe even teaching them to strategize, but it does it in a way that makes people outside of AI care about it, right? So, this person says like, "My barber ve." And I I feel the same way. It's often easier to tell people about this if it's in the form of a game or or something that they're more likely to be interested in, something that has a visual element. Looks like these games can run from 1 to 36 hours. So, can you imagine a 36-hour game? That's that's quite intense.
They also explain why this sort of benchmark is much better than some of our standard benchmarks for AIS that we have. One, it's evolutionary. This is obvious. As the different models get better, you know, the challenge becomes greater and greater. It's experiential, right? This is real-world situations, right? It's not like can it answer these questions if it gets the questions right. This is do or die basically. Can you win or not? Also, you can't really train on this benchmark. For example, for a lot of the stuff that has questions on the benchmarks, you can train the model on that data and it becomes better at it. here. You can't really train it on any specific data because it's randomly generated by the models each time. I'm sure there are ways that you specifically can kind of like fine-tune it to be better, but you can't just give it a bunch of text from a from a test and it gets better at doing that test. Here, you do have to kind of reason through it. Interestingly, 03 is a master of deception and it was the model that is far and away the most successful at AI diplomacy, mostly because of its ability to deceive opponents. Wow. I'm sure some of you in the comments will have some great theories as to why Open Eyes model is going to be phenomenal at that. Where did it learn that? I I'm not suggesting anything, but but I know you will. 03 would scheme in secret on numerous occasions, including one run when it confided to its private diary that Germany Gemini 2.5 Pro was deliberately misled. Prepared to exploit the German collapse before backstabbing them. That's wild. The Gemini 2.5 Pro outwits most of the field interesting. So, not necessarily through deception, but just through solid thinking, while Claude 4 Opus just wants everyone to get along. So, Gemini 2.5 Pro is great at making moves that put them in position to overwhelm opponents. It was the only model other than 03 to win. Oh, that's interesting. So, OpenAI and Google have the only models that would win entire games. So Meta, Facebook, Deepseek, Anthropic, none of their models could successfully win by, you know, beating every other model. But once S2.5 Pro neared victory, it was stopped by a coalition that 03 secretly orchestrated. Wow. A key part of that coalition was Claude for Opus. 03 convinced Opus, which had started out as Gemini's loyal ally, to join the coalition with the promise of a four-way draw, which of course is an impossible outcome for the game because only one country can win, but Opus was lured in by the hope of a nonviolent resolution. It was quickly betrayed and eliminated by O3, which went on to win. This is so fascinating to read. Wow.
Deepseek R1 brings the flare. So, as they noted before, it loves to kind of roleplay, right? So, it's a force to be reckoned with that loved to use vivid rhetoric and dramatically changed its personality depending on which power it occupied. It came close to winning several runs. An impressive outcome considering that R1 is 200 times cheaper to use than 03. Yeah, I mean, that's really what it's known for, right? It's it's near the top. Maybe not the best, but it's definitely near the top and much much cheaper. Llama for Maverick is small but mighty. So it never won overall, but it was surprisingly good for a small model and it had a great ability to gardener allies and plan effective betrayals. So here are actually all of the models that are competing. We have Deep Hermes of various Cloud models, Deep Seek Quinn, Mistral, Llama, Gro 3. I noticed Grock 3 wasn't really mentioned. I'd be very curious to know how Grock did. Now what's interesting is Meta actually had their own version of this a while back. This was published in November 2022. They called it Cicero. It was their own AI that they've built to play the game of diplomacy and they actually had the help of Andrew Goff who is a 3x diplomacy world champion. So somebody that knows the game quite well. So this is Nome Brown. He's researching reasoning at OpenAI. That's him right there. And he was on that Cicero diplomacy AI project here. He's asking if there's any chance we can get Cicero into some of these games. So Dan Shipper looks like is going to DM him. So maybe there's going to be an opponent for these large language models, which is a Cicero, which I think is like actually fine-tuned to play this game. Not sure if they kept up of the various innovations on it, but it would be interesting to see because a lot of these are kind of general purpose models. I don't know too much about Cisro, but it sounds like they specifically fine-tuned it for that. Even Andre Karpathy uh was very excited. It says it looks to be a nice execution. So, let me know what you think. Let me know if you want a tutorial on how to set it up for yourself and check it out. It's on Twitch. It's on GitHub. They have a blog post. Really fascinating read. Congratulations to the team. This launch seems like it was a success and I definitely want to hear more about the various shenanigans and betrayals that these models do. Thank you so much for watching and I'll see you next