Transcription
All right, OpenAI just finished their live stream. They have a bunch of new agent functionality, specifically through the API. Let's watch it together, and I'll give you my thoughts.
We're excited to launch a bunch of new tools that make it easy for developers to build reliable and useful agents. Now, when we say agent, we mean a system that can act independently to do tasks on your behalf. I like that definition; so, a system that can act independently and accomplish things on your behalf. I think that's a really good way to explain it. Usually, all includes stuff like it can do things autonomously and has tools, and it has memory, and obviously, built around a core model, but I really like that definition as well. The first is uh, operator, which can browse the web and do things for you on the web. The second is deep research, which can uh, create detailed reports for you on any topic you want. By the way, operator is kind of barely usable; it's okay, but doesn't really work a lot of the time. Deep research, on the other hand, is phenomenal. I use it all the time; it provides a lot of value to me, and there are a lot of real-world use cases that I'm using it for day-to-day. Now, the feedback for those has been fantastic, but we want to now launch those tools and more in the API to developers. So, we've spent the last couple months going around talking to developers all over the world about how we can make it easy for them to build agents, and what we've heard is that the models are ready. So, with advanced reasoning, with multimodal understanding, our models can now do the kind of complex multi-step workflows that agents need.
Yeah, and I think that's a key and maybe even underemphasized in this video: the models are ready. Look at Manis; Manis is an incredible project built on likely Claude, but the core intelligence in these models is good enough, and now we need to build out the infrastructure around the models, the tool use, like mCP servers, local environments, so it can actually write code, execute code, test, and create directories, and that's another thing Manis does really well. And so the architecture around the core model is really what needs to be built out now. But, on the other hand, developers feel like they're having to cobble together different low-level APIs from different sources; it's difficult, it's slow, it often feels brittle. So, today we're really excited to bring that together into a series of tools, uh, and and a new API and an open-source SDK to make this a lot easier. So, with that, let me introduce the team.
Yeah, hi, I'm Elan. I'm an engineer on the developer experience team. I'm Steve. I'm an engineer on the API team. And I'm Nick. I work on the API product team. So, let's dive into all the stuff that we're launching today. Like Kevin mentioned, we have three new built-in tools, we have a new API, and an open-source SDK. Uh, starting off with the built-in tools, the first tool that we're announcing today is called the web search tool. The web search tool allows our models to access information from the internet, so that your responses and the output that you get is up-to-date and factual.
Yeah, I mean, that is the most basic tool that any AI system needs is the ability to search the web, because then they go from being a static, completely frozen-in-time source of information to being actually able to have real-time information, to learn, to get better, and and all of this requires being able to search the web. Uh, the a web search tool is the same tool that powers ChatGPT search, and it's powered by a fine-tuned model under the hood. So, this is the fine-tuned GPT-4 or 40 mini that is really good at looking at large amounts of data retrieved from the web, finding the relevant pieces of information, and then clearly citing it in its response. Um, in a benchmark that measures uh, these type of things, uh, which is called Simple QA, uh, you can see that GPT-4, so that's really interesting. I did not know that the web search tool had a custom fine-tuned model under it, and here, look at this. So, this is Simple QA, simple question and answer accuracy, and even cutting-edge models, GPT 4.5, GPT 4 01 03 mini, they don't score very well, but when you have these models enhanced with search, they become much better. Boro hits a high score, a state-of-the-art score of 90% ENT. So, that's the first tool. Steve, do you want to tell us about the second one?
Yeah, the second tool is actually my favorite tool, and this is the file search tool. Now, we launched the file search tool last year uh, in the assistance API as a way for developers to upload, chunk, embed their documents and then do really easily do uh, RAG really easily over those documents. Now, we're really excited to be launching two new features in the file search tool today. The first is metadata filtering. So, with metadata filtering, you can add attributes to your files to be able to easily filter them down to just the ones that are the most relevant for your query. The second is a direct search endpoint. So, now you can directly search your vector stores without your queries being filtered through the model first. Nice. So, you have web search for the public data, file search for the the private data that you all right, both of those very useful tools, web search and private search, and now you can have metadata. So, for example, you can likely tag your files or directories and just more easily search them. And one of the launch partners of the agents SDK was Box, also the sponsor of today's video; they're doing some really cool stuff with the agents SDK, enabling enterprises to search, query, and extract insights from unstructured data stored in Box. Let me tell you more.
Introducing Box AI from Box. Every business sits on top of an immense amount of unstructured data, and yet the true potential of all of this data remains largely untapped. The problem is analyzing all of that unstructured data is really, really difficult, until now. That's where Box AI comes in. With Box AI, developers and businesses can leverage the latest breakthroughs in AI to automate document processing workflows, extract insights from content, build custom AI agents to work on that content, and so much more. And Box AI works with all of the leading model providers, and so you can always be sure you're using the latest AI with your content. Use it to extract key metadata fields from contracts, invoices, financial documents, resumés, and more, to automate workflows. You can also ask questions of any of the content you have within the Box ecosystem, such as sales presentations or long research reports. And if you're a developer, leverage Box AI's API to build really cool automations and applications right on top of your own content. Box AI handles the entire RAG pipeline for you, do all of this while maintaining the highest levels of security, compliance, and data governance that over 115,000 enterprises trust. Unlock the power of your content with intelligent content management by Box. Thanks again to Box. Now, back to the video.
And then the third tool that we are launching is the computer use tool. The computer use tool is operator in the API, but it allows you to control the computers that you are operating. So, this could be a virtual machine; it could be a legacy application that just has a graphical user interface and you have no API access to it. If you want to automate those kind of tasks and build applications on that, you can use the computer use tool, which comes with the computer use model. So, again, really cool. Computer use is useful, and it really makes me think about Manis again. Obviously, I've just gotten done testing Manis like crazy, and so I'm thinking about it a lot. And so Manis takes all of these different things and puts it all together in a really nice way, and that's what we're seeing here is a framework, an API where you can essentially build all of these things yourself. So, it's giving you the ability to control a computer; maybe you want to spin up new environment per session, very similar to what Manis does, then you go out and search the web for all the up-to-date information about whatever your task is, and then you can store all that information locally; you can write notes, and you can write code and have all of that stored in files on your local computer, on your containerized environment, whatever it is. So, this is the same model that is used by operator in ChatGPT; it has soda benchmarks on uh, OS World, web ARA, web Voyager. Early user feedback on the Kua model and the tool has been super, super positive. So, I'm really excited to see what all of you built with it.
All right, so those are the three tools, um, and while we were building these tools and thinking of getting them out, we also wanted to take a first-principles approach at designing the best API for these tools. Um, we released chat completions, I think in March 2023 alongside GPT-3.5 turbo, and every single API interaction at that time was just text in and text out. Since then, we've we've uh, introduced multimodality, so you have images, you have audio; we're introducing tools today, and you also have products like 01 Pro, deep research, operator that make these multiple model turns and multiple tool calls behind the scenes. So, you wanted to build an API primitive that is flexible enough; it it supports multiple terms, it supports tools, um, and we're calling this new API the responses API. And to show you the responses API, I'm going to hand it over to Steve.
Cool, let's go ahead and take a look at the responses API. So, if you've used chat completions before, this will look really familiar to you. So, the completions endpoint is the standard on the web for AI right now. Any model that you're using through an API likely is the OpenAI standard, and that is the completions endpoint standard. You select some context, you pick a model, and you get a response. It's pretty simple; it's pretty simple, and it's always hilarious. So, maybe not, I don't know. Um, so to demonstrate the power of the responses API, we're going to be building sort of a personal stylist assistant. So, let's start off by giving it some instructions. You are a personal sty, you're only typing in front of like 50,000 people right now, don't worry about it. Cool. And we'll say uh, we'll get rid of this, and we'll say, what are some of the latest trends? So, nothing really different from what you would be doing with the completions endpoint; the the only difference is you can see right here we're using responses instead, but they're going to expand on that. Let's watch. Jokes in the context, jok from the let's see what it says. Okay, cool, great. Um, but no personal stylist assistant is complete unless it understands what its users like. So, in order to demonstrate this, we've created a vector store that has uh, some, you know, like some entries, almost some diary entries of what people on the team have been wearing. Um, we' that's not weird at all; it's weird at all. I would just let it happen. Uh, we've kind of been following people around the office and kind of like understanding what they what they've been up to. So, we we we uh, we yeah, there's a whole there's a team; there's a team on it. Yeah. So, go ahead and add the file search tool. All right, so this is now new; you can insert tools directly in the responses API call, and as we're seeing here, they're using the type file search tool, vector store IDs; this is where you can actually maybe specifically call out a vector that you want to use, and then they have fil, so these are the metadata filters I believe, and uh, I'll copy in my vector store ID. Yep. Okay, and here I can actually filter down this the files in this vector store to just the ones that are relevant to the person that we want to style. So, uh, in this case, let's start with Elon; we'll go ahead and filter down to his username. Yeah, so that's the metadata filter that they were alluding to earlier, and we'll come back here and we'll refresh and we'll say uh, can you briefly summarize what Elon likes to wear? I often ask ChatGPT this question, yeah, but it never knows, and now it can actually tell you what the cool. So, Elon has a distinct and consistent style characterized by Miami Chic. That's really awesome. Um, so the fil, and you could see right there, so file search call, what does Elon like to wear? Elon's clothing preferences, style summary, fashion choices, and yeah, so you can see the actual call right there in line. Tool is a great way to bring information about your users into your application, but in order to be able to create a really good application for this personal stylist, we want to be able to bring in fresh data from around the web, um, so that we have both the newest information and also stuff that's really relevant to your users. So, in order to demonstrate that, I'll add the web search tool. You can also add data about like where your user is. So, let's try with somebody else. Kevin, are you happy going to be taking any trips anytime soon? Let's say Tokyo. Okay, cool, Tokyo. So, I'll put in Tokyo here, and we'll swap in Kevin, and the responses API is really cool because it can do multiple things at once; it can call a file search tool, it can call the web search tool, and it can give you a final answer just in one API response. So, in order to tell it exactly what we want, let's give it some instructions, and it'd be good if I knew how to code well. Great, you say you're an engineer here. Yeah, well, I'm in training. So, uh, what we want the we want the model to do is when it's asked to recommend products, we wanted to use the file search tool to understand what Kevin likes and then use the web search tool to find a store near him where he can buy something that he might be interested in. All right, so that's pretty cool. You define the tools; it's interesting that they're still just using GPT-40, uh, and then in the instructions you explicitly call out how you want the API to use the tool; so, use the file search tool to get the user preferences, then use the web search tool to find stores near them. All of this information is likely stored in the file search tool. So, you can see there's information about KW, Kevin Wild. Let's keep watching. So, let's go back and say uh, find me a jacket um, that I would like nearby, and what the model will do is it will uh, issue a file search tool call to understand what kinds of things Kevin likes to wear, and then it will issue a web search tool call to then go and find uh, stuff that Kevin would like based on where he is. So, the model was able to uh, just in the scope of one API call find a bunch of Patagonia stores in Tokyo just for you, Kevin, which which go, it actually corresponds to Kevin's preferences; he's been wearing a lot of Patagonia around the office. So, um, but no personal stylist assistant would be complete unless they could actually go and make purchases on on your behalf. So, in order to do that, let's demonstrate the computer use tool. So, we'll go ahead and add this; we're using the computer use preview model and the computer use preview tool, and we will ask. All right, so that's pretty cool; you do have to specify display height, display width, because of course, if it's supposed to click around, it needs to know how and what the bounds of its environment are. But yeah, computer use now through the API. Me find my friend Kevin a new Patagonia jacket. What's your favorite color, Kevin? Uh, let's go with black. Black; can't have too many black Patagonia jackets. And what the model will do is it will ask us for a screenshot, and we have a Docker container running locally on this computer, and we will go ahead and send that screenshot to the model; it will look at the state of the computer and issue another action, click. That's really cool. So, it's operating system agnostic; I believe it's browser agnostic; it should just work as long as you provide it with the display height and width, because then it can overlay coordinates on top of it and try to guess where the cursor actually needs to go, drag, move, type, and then we will execute that action, take another screenshot, send it back to the model, and then it will continue in this fashion until it feels that it's completed the task and then return a final answer. So, while this is kind of going and doing its thing, we'll hand it back to Nun.
Yeah, awesome. So, these are some really cool tools and a really flexible API for you to build uh, agents and and you have you have amazing building blocks to to do that now. But for those of you who have built more complex applications, like say you're building a customer support agent, it's not always about just having one agent that's sort of the personal style uh, stylist; you also have some uh, agentic application that's doing your refunds; you have another thing that's answering customer support uh, FAQ queries; you something else that's dealing with orders and billing, etc. And to make these applications easy to build, we released an SDK last year called Swarm, and Swarm made it easy to do agent orchestration. This was uh, supposed to be an experimental and educational thing, but so many of you took it to production anyway, so uh, you're like forcing our hand over here, and so uh, we've decided to take Swarm and make it production-ready, add a bunch of new features, and we're going to be rebranding it to be called the agents SDK. So, I hadn't actually used Swarm; I know a lot of people have tried it out; I didn't actually use it, so this is kind of new to me. Elan built uh, a Swarm and helped build it, so I'm going to have hand it to him to tell you more about how it works.
Yeah, thanks, Nun. Yeah, so uh, in my time at OpenAI, I've spent a lot of time working with enterprises and builders to help them build out agentic experiences, and I've seen firsthand how pretty simple ideas can actually grow in complexity, like when you actually go to implement them. And so the idea with the agents SDK is to keep simple ideas simple to implement while allowing you to build more complex and robust ideas still in a pretty like straightforward and simple way. So, um, let's take a look at what Steve had before in the demo but implemented using the agents SDK. It's going to look very similar at first; we have our agent defined here; we have some instructions, um, and we also have both of the tools, file search tool, web search tool that we had before. Is this using like responses under the hood? Yeah, so by default this is using the responses API, but we actually support multiple vendors; anything that really fits the chat completions um, shape can work with the agents SDK. Nice. So, um, during the practice runs, we actually we actually accidentally ordered like many, many Patagonias, so I'm sorry; I understand what's the problem; we're helping you here. Uh, we want to return some of them, uh, and so to do that, I could usually just add in like a returns tool and like add more to this prompt and get it to work, but the problem with that is you start to mix all of this business logic, which makes your agents a little bit harder to test. And so this is the power of multiple agents is you can actually separate your concerns and develop and test them separately. So, to do so, let's actually introduce a like an agent specifically to deal with the sorts of uh, like returns. So, I'm going to load mine in and great. So, we still have, and this all feels very familiar; it feels very similar to how LangChain is. So, the agents SDK is essentially a code framework to build out multiple agents and allow them to work together. Now, obviously, I'm quite biased towards LangChain, not only because I've been using it for a while and I do think they're best, but I'm also an investor. So, it's interesting to see this; I love competition, and I love the fact that and they're about to say this; this is open source now. Let's keep watching. Our agent from before, but you can see there's also this new agent, the customer support agent here, and I've defined a couple tools for it to use, the guess get past orders and then submit refund request, and um, you might notice these are just regular Python functions, as this is actually a feature that we uh, people really loved in Swarm that brought over to the agent SDK, which is we'll take your Python functions and look at the type inference or look at the type signatures and then automatically generate the JSON schema that the models need to use to perform those function calls, and then once they do, we actually run the code and then return the results. So, you can just define these functions um, as as they are. Now, I've given them um, now we have our two agents, right? We have the stylist agent, and we have the customer support refunds agent. So, how do we interact with both of them as a user? This is where the notion of handoffs comes in, and a handoff is actually a pretty simple idea; it's pretty powerful, and it's when you have one conversation where one agent is handling it and then it hands it off to another where you keep the entire conversation the same, but behind the scenes you just swap out the instructions and the tools, um, and this gives you a way to triage conversations and like load in the correct context for each part of the conversation.
Yeah, so this feels again very familiar; it just allows you to define multiple agents, have each agent be very specialized in the things that it can do, the description of what it is, the tools it can use and so on, and then you can have kind of a manager agent, a dispatch agent; they call it a triage agent, and that just allows that agent to coordinate between other agents. So, what we've done here is created this triage agent that can hand off to the stylist agent or the customer support agent. So, enough talking, let's actually see this in action. So, I'm going to save and do you know um, I think we may have ordered one too many Patagonias; can you help me return? I don't understand; I I know; I'm so sorry; I can get you one later. So, what just happened here is it started off by transferring; remember we're starting with the triage agent um, to the customer support agent, and this is just a function call that I'll show show you in a second, um, and then the customer support agent proactively called the past orders function where we can see all of Kevin's Patagonias. I think you'll be okay. Um, cool. So, to actually see what happened behind the scenes, usually you might need to add some debugging statements by hand, but one of the things that the agent SDK brings right out of the box is monitoring and tracing. So, I'm going to go over to the tracing UI that we have on our platform um, to actually take a look what just happened. So, these are some of the previous runs that we've had; I'm just refreshing the page, um, and we can see the last one, uh, and this last one you can actually see exactly what happened; we started with a triage agent, which um, we sent a request to, made a handoff, and then switched over to the yeah, and I must say that this UI is very clean; I really like it; just very easy to see what's going on. And if you want similar functionality to be able to trace your agents, not only in OpenAI agent SDK but in other platforms like LangChain as well, check out my friends over at AgentOps; they're awesome; they're not sponsoring this video. Let's keep watching. Uh, we can see what the original input was, was, and handoffs are first-class objects in this dashboard, so you can see not only which agent we actually handed it off to, but any that it like it had as options that it did not, which is actually a really useful feature for debugging um, afterward. Once we're in the customer support agent, you can see they get get past orders function call with any input params. Here there were none, um, and then the output is just again just all of Kevin's very monotonous history, um, and then finally we can get to the end where you get a response. And so these are some of the features that you get right out of the box with the agents SDK; there's a few more you uh, we also have built-in guardrails that you can enable; we have lifecycle events, um, and importantly, this is an open-source framework, so we're going to keep building it out, um, and you can install it like very soon or right now, so you can just do pip install openai-agents, and we'll have an one for the JavaScript coming soon. Um, but to close this off, so very cool, open source.
Yeah, definitely. Thank you to OpenAI for open-sourcing this. Let's actually perform the the refund. So, uh, you know uh, you know what? I'm sorry, Kevin; get rid of all of them. Oh, what am I going to wear? Kevin's going to be cold. Yeah, let's see. Now, I believe because it's open source, you don't actually need to use an OpenAI model, which is really nice. I like to use multiple models; some models are better at certain things than others. I do believe because it is a full OpenAI project that most likely it's going to work best with OpenAI models, but that's one of those things you're probably just going to have to test yourself. A lot of them, there we go; takes a while to return so many Patagonias. And so what what happens under the hood? How do you how do you debug this? How do you understand more about what's going on? Yeah, so that we can all do back in the in the tracing in the tracing UI. So, this is a pretty nice straightforward way to build out these experiences. Yeah, and Joel, awesome, pass back to you.
I'm so excited for all of you to have access to all of these tools, uh, and before we wrap up, I wanted to make two additional points. First, we've introduced the responses API, but the chat completions API is not going away; we're going to continue supporting it with new models and capabilities. There will be certain capabilities that require built-in tool use, and there'll be certain models and agentic products that we release in the future that will require will require them, and those will be available in responses API only. Responses API features are a superset of what chat chat completion supports, so whenever you decide to migrate over, it should be a pretty straightforward migration to you, and we hope you love the developer experience of responses because we put a lot of thought into that. The second point I wanted to make was around the assistance API. We built the assistance API based on all the great feedback that we got from all of our beta users, and uh, you know, we we wouldn't be here without uh, without all the learnings that we had during the assistance API phase. We are going to be adding more features to the responses API so that it can support everything that the assistance API can do, and once that happens, we'll be sharing a migration guide that makes it really easy for all of you to migrate your applications from assistance to responses without any loss of functionality or data. We'll give you ample time to move things over, and once we once we're done with that, we plan to sunset the assistance API sometime in 2026. We'll be sharing a lot more details about this uh, offline as well, but yeah, that's it for me. I'll hand it over to Kevin to wrap us up.
Awesome. Well, we're super excited to announce the the responses API and the idea that we can bring take a single powerful API and bring together a whole bunch of different tools, from RAG and file search to web search to Kua and our uh, operator uh, computer use APIs. Now, you can count on us to continue building powerful new models and bring more intelligence to bring more powerful tools to help you you build better agents. 2025 is going to be the year of the agent; it's the year that ChatGPT and our developer tools go from just answering questions to actually doing things for you out in the real world. We're super excited about that; we're just getting started; we know you are too, and we can't wait to see what you build. All right, so that's it. 2025 is definitely the year of the agent, especially this week, between Manis and now we have OpenAI responses API endpoint, and just so much more coming, I bet. So, if you enjoyed this video, please consider giving a like and subscribe, and I'll see you in the next one.