Transcription
I'm scared. Good morning. We have a banger for you today. We're going to launch ChatGPT agent. But before jumping into that, I'd like to ask the team to introduce themselves. Starting with Yash.
Hi, I'm Yash. I work on the agent team and before that, I used to work on operator.
Hi, I'm Jing. I work on agents research, previously on deep research.
Hi, I'm Casey. I'm a researcher on agents, formerly operator.
Hi, I'm Issa. I'm a researcher on agent, formerly on deep research.
So, we started launching agents earlier this year. Uh, we launched deep research, we launched operator, and people were very excited about this. People could see that now, uh, AI was going off to do complex tasks for them. But it became clear to us that what people really wanted was for us to bring those capabilities and more together. People wanted a unified agent that could go off, use its own computer, and do real complex tasks for them, that could, uh, seamlessly transition from thinking about something to taking actions, to using lots of tools, using the terminal, clicking around the web, even producing things like spreadsheets and slides and, and much more. And wanted people want to be able to do this over a long time horizon and, uh, sort of for universal tasks. So the team has been working super hard to bring that together. And today we have ChatGPT with the agent. Um, it's probably easier to show it to you than to keep talking about it. It is one of the, uh, feel the AGR moments for me to watch it work. So, let's take a look.
Awesome. Thanks, Sam. Hello everyone. Very excited to share ChatGPT agent with everybody. And as Sam said, let's just dive right into the demo.
Okay, so we are on ChatGPT as we all know and love. And to turn on the agent mode, you just click the tools menu and select agent. You can also just type agent in the composer bar, and it'll take you to agent mode. Um, Edward and I have a wedding to go to later this year. Uh, it's for one of our mutual friends. Should we, should we have the agent plan it? Yeah, let's do it. I need an outfit. And don't forget the gift. Okay, great. We won't forget the gift. Um, it's a little bit of a longer prompt, so I have it copied in my buffer. So, I'm just going to go ahead and paste it. Um, okay. So, let's see. Let's see what it says. Our friends are getting married later this year, as I said, Minia and Sarah. And we want the agent to help us find an outfit that matches the dress code, uh, propose a few options, nice mid-luxury, taking into account venue and weather. We also want it to find us some hotels, and as Edward said, don't forget the gift. Um, so let's see, and send the prompt away.
As Sam said, agent uses a computer. Uh, so in the beginning, it sets up its environment. It, it, you know, it'll take a minute or two, or not really five seconds to set up its environment. And in this case, as you see, it understands the prompt. It's asking for me for a clarification. I'm just going to let it just continue and work anyway. Um, I think it got confused by saying, "Oh, where's the, um, what exactly is the time of the date of the wedding?" I think it'll figure out using the website. Okay, cool. So now it's kicked off. It's starting to process the prompt and it's opened up a browser. And to walk you through what's happening, here's
Yeah. So, as mentioned, we gave the agent access to its own virtual computer, and the computer has many different tools installed, and it can choose which to use as it's working through the task. So, in ChatGPT, you can see a visualization of the agent's computer screen, and you can see overlaid its chain of thought in text, and that's what it's thinking as it's working through the task and deciding what to do next. We gave the agent access to two different ways to browse the internet. First, we gave it a text browser, and this is similar to the deep research tool. And this is what lets it really efficiently and quickly read many web pages, um, um, and search for them. And we also gave it access to a visual browser. And this is similar to the operator tool. And this is what lets it actually interact with the UI of a web page. So it can, um, drag things. It can use the cursor to click around. It can open UI components. It can fill out forms and enter text in text areas. It's very flexible. So those two tools are very complimentary. And then we also gave it access to its own terminal so that it can run code and it can also generate and analyze files like slide decks and spreadsheets. And then through the terminal, it's also able to call APIs. So both public APIs and APIs to access your private data sources like Google Drive, Google Calendar, GitHub, SharePoint, and many others. Um, and only if you explicitly connect them, similar to deep research connectors. And then it also has access to the image gen API so it can create nice visuals for, um, slide decks and other things as it's working through its tasks.
How is it deciding which tools to use here?
Yes, we train the model to move between these capabilities with reinforcement learning. This is the first model we trained that has access to this unified toolbox: a text browser, a GUI browser, and a terminal all in one virtual machine. To guide its learning, we created hard tasks that require using all these tools. This allows the model not only to learn how to use these tools, but also when to use which tool depending on the task at hand. At the beginning of the training, the model might attempt to use all these tools to solve a relatively simple problem. Over time, as we reward the model for solving problems correctly and efficiently, the model will have smarter tool choice. For example, if you ask a model to, uh, find a restaurant with specific requirements and make a reservation, the model may typically just start a deep research in the text browser to find some candidates, then switch to the GUI browser to view photos of food, uh, check availability, and complete the booking. Similarly, for creative tasks like creating an artifact, the model will first search online for public resources, then switch to the terminal to do some code editing to compile the artifact, and finally verify the final outputs in the GUI browser. With this, we truly feel like we brought together the best of deep research and operator and added some extra sparkle.
That's right. Yeah. So to put this project in context, I want to give a bit of history. So a few months ago, we shipped operator in January, and this was our agent that lets you do online tasks like book reservations and, um, send emails. And then two weeks later, we shipped deep research, and deep research is a tool that lets you do in-depth internet research and output high-quality, um, um, research reports. And after launch, we realized that actually these two approaches are actually deeply complimentary. Um, for example, operator has some trouble reading super long articles. It has to scroll, it takes a long time, but that's something that deep research is good at. Conversely, operator, uh, uh, deep research isn't as good at interacting with web pages, interactive elements, visual, highly visual web pages, but that's something that operator excels at. So, uh, yeah, we felt these approaches were complimentary, and then we, we were also looking at some customer feedback. So, for example, one of our most highly requested features for deep research was the ability to log into websites and access authenticated sources. That's something that operator can do.
Been waiting for that for a long time. Yeah. Um, another thing is that we were looking at the prompts that people were trying for operator, and we saw that they were actually more deep research type prompts. For example, plan a trip and then book it. And so, yeah, we, we really feel like we're bringing the best of both worlds here. And on a personal note, we've all been friends for a while, and it's really exciting to be working together.
So, speaking of matches made in heaven, how is the wedding planning going?
It's amazing to watch. This is an example of a task I hate doing. This can like ruin like, you know, multiple hours for me as I get sucked into these rabbit holes. So, just watching this as you guys have been talking, click through this and just like do the whole thing is really quite remarkable.
Yeah, totally. Um, looks like it started off by figuring out the weather. One of the cool features, um, is that, you know, as some of these tasks may take a little bit longer, you can just go back and see what it was doing. So, that's what we're exactly going to do. Looks like it went through the website. Use the text browser. Interestingly, for that, now it's looking through the suits for Edward. I think it'll find something good. Here you can see it switched over to actually a visual browser to make sure the suit will look really good on Edward. And now looks like, yeah, it's got chugging along, figuring out what to do. Um, and still on suits and now probably getting to the gifts section. Um, okay, cool. So, this is going to take a while. As Sam said, these tasks sometimes can take a long time. So, it's going to continue doing, hopefully much faster than we will do. Um, should we do something else while it's doing it?
I think the team really wanted the, uh, stickers, some stickers for the for the launch. Should we do that?
Yeah, cool. All right. So, we have a team mascot, which is one of our colleagues, Bunny Doodle. Really, really cute, tell you. Um, and we're going to try and bring, um, get some laptop stickers for everybody. One of the favorite features for agent is given that trajectories can take 15 minutes, 20 minutes, 30 minutes depending on the complexity of the task. Um, a lot of times the, you might need to help the agent, agent might need to ask you clarifications, confirmations, and things like that. Um, so I love to use it on the go. So I'm going to use my mobile phone to actually send the query this time and then see how it goes.
Okay, so let's see. Okay, so we are on ChatGPT. Uh, I have already selected the agent mode. I've also inputted our, uh, cute mascot, and I'm going to quickly paste a query. So, query says, make some swag for the team, one by one laptop stickers, and order 500 of them. I'll also say I like Sticker Mule, which we have used in the past, and send it off.
Okay. So, just like it was doing on the web, it's going to take some time, think about like what it's doing, and it'll kick off, kick off the query. And as it's going, it'll take some time to kick it off. Is it? Oh, there we go. So, it'll start working on it. Looks like it's starting to create the anime art. It'll probably use the image that Isa referred earlier on to create, hopefully, an anime art. We'll see how it comes out.
While that's going, anything else we want to do?
Oh, yeah. I also need a pair of shoes because my shoes got damaged.
How did they get damaged?
Uh, by the rain in SF.
Yes. Cool. All right. Uh, well, let's get Edward a pair of shoes as well. So, oh, can you also find us, um, a pair of men's dress black shoes in size 9.5? 9.5.
So, one of the key capabilities of the model is being able to interrupt. I think, you know, as trajectories take long time or whatever time, it's really important for us for it to feel very multi-turn, so the users can interject, user can direct it, user can give it more guidance, less guidance, whatever we want to do, and that's what we're doing here. We essentially, the, the model was chugging along, figuring out all the things that we had asked before, and in this case, we essentially said, hey, can you also, uh, get us a pair of men's black shoes? And now it's thinking, and soon enough, hopefully, it'll take that into account and keep going, uh, into its trajectory. There you go. So, it says, acknowledge the interruption. It said, "Okay, cool. I'll also research men's black shoes in size 9.5." Um, and then it'll probably get on its way. Um, but maybe Issa can tell us a little bit more about how that works.
Yeah, sure. So, as you can see, the agent is very collaborative, and this was really important to us when we were training the model and building the product. If you were asking another person to do a task for you that would take them a really long time to complete, you'd probably give them some instructions to start, and then they might ask you some clarifying questions, and then they'd start the task, and maybe realize, oh, they need more clarification from you, or they need your permission to sign into something or do something on your behalf, and then you might realize, oh, I forgot to mention this thing, or, uh, what's your status? How are you doing? Can I help redirect you if you're going down the wrong path or something? And so similarly for these really long-running agentic tasks, it's very important that both the user and the agent are able to initiate communication with each other so that, um, the agent is able to most effectively help you with your tasks. And so this is something that we actually trained into the model. We trained it to be able to ask clarifying questions, not every single time like deep research. Um, we also asked it, we also trained it to be interruptible, as Yash just showed. And also sometimes it will ask you for clarification and confirmation mid-trajectory.
Yeah. And part of working with agent is that, well, sometimes it'll make mistakes, and that's why we felt it was important to train the model to ask you for confirmation at the last step of important steps. So, for example, maybe before it's going to send the email, um, it'll ask you to take a look at the draft and whether it makes sense and whether there are any embarrassing typos. And if there are, then you can either ask it to fix it, or you can directly take over the browser and jump right into the, um, agent's environment and correct it yourself. And that way, it feels collaborative, and you can, um, really work with the agent.
Should we look at maybe one more demo? We've got this, uh, sort of fun tradition in live streams of using, uh, using our newest models to sort of evaluate themselves or do something kind of meta. Anything like that we could do?
Yeah, let's do it. So, um, I think people would love to know how good the model is. Yes. So, this is a prompt we previously gave the agents yesterday. So, basically, it asks the model to pull its own evolution number from our Google job connector and make some slides. So, we want to keep it simple, like no introduction, no conclusion, just present the results with the charts. As you can see, now the model is connecting to the Google Drive API and, uh, then search within the API. Right now, it looks like the first result is very relevant. So, it's reading the first result. Now it's reading the first result, uh, in details. Let's accelerate this, uh, replay. So, then the model might read from the result again and write some code. So, here you can see that the model is using the image generation model called image generation tool to generate some decorations for the slides. And let's see what's the first slide the model made. So, here the model is writing some code that will be compiled to be the final slides. So, this is the first slide the model made in this demo, which looks okay, but it's not polished enough. One of the key features in reinforcement learning is that the model will review its own results and refine the results to, to deliver a good final result. Let's see what's the finally what the model give us. We can click skip, and then the model gives us a good PowerPoint file. So, it's a real PowerPoint that you can download and open it in any software. Let's open it in, uh, in the office. So, let's present the slides the model just generated.
First are two intelligence benchmarks. Humanity's Last Exam is a benchmark that measures AI's ability to solve a broad range of subjects on hard problems. We evaluate the models with two settings: with and without tool use. We can see that the agent mode's raw intelligence is already pretty nice, and with access to all tools, nearly double the performance to 42%. When evaluating models on Humanity's Last Exam, especially with the Bing ability, we have a two-layer decontamination that ensures that the model doesn't cheat on this benchmark. Frontierms is a benchmark that measures advanced mathematical reasoning ability of models. Different from our baseline of Mini and 03, which use Python with function calling, we get the agent model all available tools like a browser, a computer, and a terminal. The agent achieves new state-of-the-art of 27% on this benchmark with the help of all these tools. Next, we evaluated the model on two agentic benchmarks. Web Arena is a benchmark that measures web agent's ability to solve real web tasks. The agent model improves over the previous 03 model that powers the core. Browse comp is a benchmark we introduced earlier this year that measures the browsing agent's ability to search and find, uh, how to locate information. The agent model significantly outperforms 03 and deep research on this benchmark, achieving a 69% pass rate. Finally, we care about how the users will benefit from our model in the real world. Spreadsheet bench is a benchmark that measures the model's ability to edit spreadsheets derived from real-world use cases. Here, the agent model with the LibreOffice and the computer tool can already solve 30% of the task. When we give the model access to the raw Excel file in the terminal, which further boosts the performance to 45%. Finally, we evaluated the model on an internal banking benchmark. The bench, this benchmark evaluated the model's ability to conduct first to third-year investment bank, banking analyst tasks, such as like putting together a three-statement financial model for a Fortune 500 company. In this benchmark, the agent model significantly outperforms the previous deep research and all three models.
As you can see, this model is one of the most powerful models we've ever trained. It's not only good on benchmarks, it's also capable of reasoning, browsing, and tackling real-world tasks at a level that we cannot imagine three months ago.
That's right. Um, as Edward said, um, we think we've trained a very powerful model. And a lot of the power comes from its ability to browse the internet. And as we know, the internet can be a scary place. There are all sorts of hackers trying to steal your information, scams, uh, phishing attempts. And agent isn't immune to all these things. Um, one particular thing we're worried about is a new, uh, attack called prompt injections. This is where, let's say you ask agent to buy you a book, and you give it your credit card information to do that. Agent might stumble upon a malicious website that asks it, "Oh, enter your credit card information here. It'll help you with your task." And an agent, which is trained to be helpful, might decide that's a good idea. We've done a lot of work to try to ensure that this doesn't happen. We've trained our model to ignore suspicious instructions on, on suspicious websites. We've also have, uh, we also have layers of monitors that kind of peer over the agent's shoulder and watch it as it's going, um, and stop the trajectory if anything looks suspicious. We can even update these in real time if new attacks are found in the wild. That said, though, you know, this is a cutting-edge product. This is a new surface, and we can't stop everything. And so that's why I feel it's very important for the audience to be aware of the risks involved in using agent. And, um, we encourage users to be proactive in kind of thinking about how they share their information. You know, if it's highly sensitive information, maybe don't share that. Um, maybe, uh, use our features like takeover mode to directly input your credit card information into the browser instead of, um, giving it to agent. We feel like we've built a very powerful product, but again, it's important for our users to understand the risks involved.
Yeah, I really want to emphasize that I think this is a new level of capability in AI. It's a new way to use AI, but there will be a new set of attacks that come with that, and society and the technology will have to evolve and learn how we're going to mitigate things that we can't even really imagine yet, as people start doing more and more work this way.
Before we wrap up, should we check in on some of the tasks you kicked off?
Yeah, let's do it. Um, okay. So, I am going to open a new tab and make sure that we can see the progress of our, um, stickers as well. Okay, let's see. All right. So, sounds like stickers are ready. Let me see what it actually. Okay. So, cool thing. This is sort of the end-end result of the, took about seven minutes. Highly likely figured out everything. We'll go back and look at the trajectory and see how it did. But at the end result, it looks like it's added to the cart. This is the subtotal. I can just go ahead and look at it and then figure out, uh, I can just take over at this point, as Casey said, to enter my credit card information and then place the order really quickly. The model is asking for confirmations, etc., as it's supposed to do. Let's just quickly browse through the trajectory and see what it actually did. Oh, it looks like it generated some stickers. Oh, look at that. That's what it generated. You see the sticker. Cool. So, yeah, that's the task. I think I can at this point finish up by myself, or I can ask the model to actually go ahead and do it for me as well.
Let's check on the wedding.
Okay, great. Looks like it just finished in the nick of time. Uh, okay, cool. So, in this case, as, as we said, we were looking for hotels, dress, uh, suits, and also shoes. So, it's come out with a pretty comprehensive report. It looks like wedding venue, date, when it is, with the Zola links, dress codes. It figured out like what the suit recommendation should be, where you can buy. Now I can go ahead and buy myself, or I can ask the agent to go and buy for me. Um, also figured out footwear hurdle options. It actually looked through all the, oops, sorry, it looked through all the availability. You can see actually it gives screenshots of what it checked. In this case, we use booking.com, and it's able to do that. Also has gift suggestions, etc. And next step, I can ask it, as you said, the agent says, "Hey, if you need assistance purchasing any item or have any further adjustments, let me know," so we can do that.
Uh, and I want to show one last demo which we didn't really run live, but I think it's really cool, and especially because the folks who are getting married are really into MLB. Um, so we asked the agent, uh, to go and build an optimal itinerary for visiting all 30 MLB stadiums in just in case you're thinking of a sabbatical, and then design the optimal route, prioritize Hello Kitty nights, and whatnot, and present a final plan as a detailed spreadsheet. I'll really quickly run through this. Um, I think it's just so fun to see. So, again, like as we have shown throughout the, the live stream, it uses a multitude of tools, uses container, the terminal, use, using the browser, working through all the details. It'll probably use again back to the browser, figuring out Hello Kitty nights and then sports stadium and whatnot. Oh, let's see. Did I miss the? Oh, there you go. App building a map using code to actually build it out. And then overall, we get like a pretty solid result, I think, at the end. It takes 25 minutes to work where there's a season start and whatnot. You have a spreadsheet that you can quickly view inside, just right inside ChatGPT. You can map the journey. Cool looking map, I guess. And that's it. So, this is ChatGPT agent. We hope you really like it. And over to Sam.
Amazing work all of you and to your teams. This is, I think, uh, really something that's going to help people get work done, uh, and have more time to do the things they want to do. Um, I think it's, it's really amazing how much you've brought together to deliver this experience. And watching the agent sort of use the internet, make these spreadsheets, make PowerPoints, whatever else, uh, and do all this work is, is quite amazing. We're going live today for Pro Plus and Team users. Pro users will get, uh, 400 queries a month. Plus, and Team users will get 40 a month. Uh, the rollout should be finished by the end of the day for Pro, and very soon for Plus and Team users. We'll try to be live for Enterprise and EDU by the end of this month. As Casey mentioned, although this is an extremely exciting new technology, there are new risks. Uh, people learned how to use the internet generally pretty safely, although of course, there are still scammers and other attacks. People are going to need to learn to use AI agents. And society's going to need to learn to build up defenses against attacks on AI agents as well. So, we're starting with a very robust system, lots of warnings. We will relax that over time as people get more comfortable with it, but we do want people to treat this as a new technology and a new risk surface and use all of the caution that Casey talked about. Um, but that said, we hope you'll love it. Uh, this is still very early. We will improve it rapidly, and we're excited to see where it all goes. So, congrats again. Thank you very much. Hope you enjoy.
Hello. Hello. Thank you everybody for being here. Give me one second to get things rolling. Let me do the pop-out chat here. One second.
All right. So, first and foremost, thank you so much everybody for for being here. I'm not feeling too great this morning, but I do wanted to, um, check this thing out. So, I asked the guys from the Civic podcast to, um, help me out this morning just because I'm not, I think I'm coming down with something. So, I think what we're going to do is we're going to try to merge our streams and test this thing out. So, hopefully they can kind of, um, carry me a little bit today because I am, I am not 100%. But, um, give me one second so I can, I can get this, um, chat going. One sec. And we're going to be, they're going to be joining us pretty soon here.
All right. How does this improve my AI girlfriend from Pain Elemental?
Oh man, that's an excellent Doom reference. That is the actual name of the actual, of the actual mob in there with the actual avatar. Phenomenal. That was the name of it. Pain Elemental, right? Because the big red floating things that kind of look like it. Who, who knows what, what those were called? Who remembers? Um, and then, yeah, Pain Elemental. Terrific. Terrific. Um, well, obviously because the AI girlfriend can go and get you Instacart or DoorDash or what, you know, you can, uh, uh, get her to go and, uh, put on a Netflix, order food, and actually, now that I think about it, a lot of things because there's a lot of stuff eventually that you're going to be able to do with that, including make phone calls. Like, I mean, you're able to do something like Google Voice. So these agents potentially can actually call people and do all of that stuff. So everybody, I'm seeing so many familiar names. So everybody, thank you so much for being here. Um, The Remarkable N, Donut Raider. Let's see what else we got. I saw Little Spooji is apparently number was number one for a while. Um, I got a apparently this cool little leaderboard that tracks everybody's experience, which I'm very, very excited about. So we can kind of see, I don't know exactly what it does yet, but I, I like that there's like a little, the standing thing, uh, which I think is pretty, pretty cool. So, testing, testing, testing. All right, give me one second. Which face am I going to put on the thumbnail?
Never cross streams. That's a good one. Uh, give me one second. All right. And one thing that I got to check to see is do we have, do I have access to it yet or not?
All right. So, here's what I think we're going to do, just because I am feeling a little bit out of it. So, I think what we're going to do is the SVIC podcast, they, they have a stream up and running, and, um, I'm, for those interested, I'm streaming using something called OBS. It's an open-source software. Uh, it's very good, but it, as far as I know, doesn't really allow for multi-guest thing. So, what we're going to do is we're going to merge streams. Somebody said, "Never cross streams." We're going to have to cross streams here. Uh, something you're never supposed to do, but we're supposed to cross streams with the SVIC podcast. So, we're going to do this, and everybody's going to get sort of redirected with no issues hopefully. So, there's no action required on your part. But what that's going to allow us to do is for all of us to test these agents out. Hopefully, it's going to be available to us. Um, and also we'll get some people in here that, um, hopefully will help me talk today. So, let's, uh, yeah, but, but we'll, everybody's going to be in there. So, let's just keep chatting. But, um, oh, the point about OBS is they're going to be using Riverside. That's going to allow me to log in or, you know, join the stream so we can all talk. So, let's do it. Um, Xbard, thank you for the super chat. If I'm correct and meeting itself is compressible, I create an AI-driven semantic compressor VQ VA pipeline. You know, the whole, I'm, we got to talk, we'll talk about, um, all this stuff because there's so much about the idea of compression, and it's a whole rabbit hole that I do want to talk about at some point. But yes, it's like a Twitch raid. So, let me just verify that we are good to go. And, yeah, I think if there are no problems, then we're going to be, um, just transferring over there momentarily. Stay tuned. I'll jump on in like 30 seconds. I just need to reconnect and let's continue our conversation over there. We're going to be testing the thing. Everybody, thank you so much for being here. Um, I do have the best barber. That barber is me. I just use one of those things to, um, shave my head every once in a while. But a link to the other stream. You know what? That's a very, very good idea. So, the link to the other stream is that. Let me pin this message. If it doesn't transfer you over, just click on that one. Okay, guys, I'll see you there in just a quick second. I'm going to end the stream, and that should hopefully, um, Google willing, YouTube willing, get us transferred over there. Let's chat there. Thank you so much. Stay tuned, and away we go.