Transcription
So, entropy has released a list of really interesting updates for the agenda tool, calling them "not a lot of people are talking about or pay attention to," but I think it is a really big deal if you're building agents, particularly for long-running and complex tasks. And this is what I want to talk you through today: what it is and how you can apply it to your agent.
So, what entropy has, I can almost consider as two calling 2.0. If you're not familiar with two calling concept, two calling is a foundation of agents. It transforms large language models from outputting pure text to outputting a specific JSON that can be used to invoke an API or functions. So, that enables large language models to take actions in the real world.
And the mechanism of tool calling or function calling hasn't been really changed for the past two years. Basically, you will give large language models a list of tool definitions. It has title, name, description, and most importantly, the parameters that the agent needs to generate to run this function. Then, when a user asks the agent a question like, "What's the weather in Paris?" it will send both the user question to the schema to the large language model, which will return a tool call text, looks something like this, where the model decided to call `get_weather` function with parameter `paris`. Then, on the server side, we will run the function with this specific parameter, get a response, and then send all the previous messages plus this new tool response back to the model. And in the end, the model returns a synthesized message.
So, when you look at this process, it is actually kind of manual and basic, but because everything is happening automatically, it feels kind of magical. This is exactly what is happening when you message your cloud bar, and it is showing typing. Behind the scenes, it is doing this ping-pong round trip to decide what tool to call, output parameters, run function, get response back, decide the next step.
But this tool concept we introduced two years ago also has a lot of problems and limitations. The biggest issue is efficiency. So, for more complex tasks where an agent will run multiple different tools in a row, we're basically purely relying on the large language model to generate the parameters for each function, and this can lead to some non-deterministic behavior, as well as loads of waste in the token context window.
Let's just take one basic example. Let's say your agent has access to a list of Gmail-related tools, and you ask it, "What are all the emails we have from Bob in my Gmail inbox?" What happens is the agent will first try to call the function `search_email` with parameter `Bob`, which can return a list of different email IDs, because that's how their API endpoint works. Then, to get each email, the agent will need to run `get_email` tool with ID one, then do it again for the other email ID, repeat multiple times until it gets all the information. And here, we are relying on the large language model to regenerate the ID exactly the same. And also, some of the tool's response results is gigantic. Like, for the `search_email` function, it returns us a huge amount of metadata that we don't really need, even though we just need the ID to be able to invoke the next function. But all those information just stay and eat up our context window unnecessarily.
And this type of situation can get even worse in some other situations where the function tool call parameter is actually complicated. Let's say you give your agent a task to write a blog about AI news, and it can run the web search first, which returns a list of URLs. And for each URL, it will run the `web_fetch` tool, which will return HTML alongside a whole bunch of noise in the HTML as well. And in the end, if you have a specialized tool to write a blog, where it will take raw content and output a well-written blog, then the large model will need to manually recreate all the raw content from each web fetch function to pass into this `write_blog` tool. And this is going to be very expensive and with a whole bunch of tokens unnecessarily.
And this problem is not going to be solved by just having models with bigger context windows, because even though today's models have a 1 million context window, the actual effective context window is just somewhere between 120K to 200K. So, you almost always want to optimize what goes into the context window.
And this is what Entropic's new advanced tool call release is so interesting. They released a list of different improvements to its tool call capability to solve exactly this problem, and I will talk you through one by one.
Firstly, let's talk about programmatic tool calling. This is one of the most important releases here. The main idea here is very similar to a paper called "Executable Code Actions." So, instead of using a large language model to be the glue to take a response from one action and pass it to the next action, what if we can just give a model an environment where it can access any tools the agent has access to and just let the large language model output a piece of code to run multiple functions? It will know how to use code to pass the result from one function to the other. It can even use things like `for` loops, conditionals, to achieve more complicated workflows in a very deterministic and token-efficient way.
And this paper was released when we were still on like GPT-4 era, but the improvement of tool use was already significant. And Cloudflare also had something similar released last year called "Code Mode." They found that large models are much better at writing code rather than outputting JSON like this for calling the tool. So, getting a large model to perform tasks with tool calling is almost putting Shakespeare through a month-long class in Mandarin and then asking him to write a play. So, their point of view is that large models are just going to be so much better at writing code rather than outputting JSON and reasoning about it, deciding the next step. So, they also have "Code Mode," where it has functions to convert any API to a TypeScript. So, the agent can write code to invoke those API tools.
And this Entropic programmatic tool calling is exactly the same concept. So, instead of just getting the model to run one tool call at a time, see the response, and decide the next action, they can just get the model to have access in an environment where they have direct access to all the API tools. Then, the model can just write a piece of code like this to invoke those API tools, so that they can contain a whole bunch of noise and the context within the function rather than exposed in the context window.
And in their experimentation, they compared two methods side by side to get the same model to perform some puzzle-resolving tasks. And you can see the context window of programmatic tool call is so much smaller than the traditional tool call, and GPT-4.5 is able to proceed much further on this task completion compared with the previous one. And the way they designed this makes it very easy for you to enable programmatic tool calling in your existing agent setup without restructuring the whole agent architecture.
All you need to do is, in your large language model response, making sure you include this `code_execution` function. So, `code_execution` function is a sandbox where the agent will use to write and execute the code. Then, for each tool that you pass into the large language model, you can pass this new parameter called `allowed_caller`. Pass this `code_execution` tool as one of the callers here. With this, the programmatic tool call is already automatically activated.
It will firstly return a response that looks like this. So, when you pass the large language model a function for `query_database` with a request, "Query customer purchase history from the last quarter and identify the top five customers by revenue," the large language model will firstly return a response of a piece of code that it has written. And inside this code, you can see it will try to call this function with certain parameters, as well as another one with tool use for this function and input parameters. So, you can take out this response by filtering out all tool use and caller to do the actual database query. And once finished, you can send back all the past conversation history with a new user message of tool response. Then, the large language model will just take your new tool results, feed it back to the piece of code you written before, run it, get a response, and synthesize, and in the end, get a final response back.
So, the change on the agent side is fairly minimal. All you need to do is just adding this `code_execution` tool alongside `allowed_caller` parameter for the functions and update your agent runtime to be able to extract those types of tool use. And this comes with a bunch of benefits, because with code, large models can use `for` loops to just batch process tasks that can be run in parallel. And you can also use conditional loops to set up more complex structures, as well as some deterministic filtering to making sure only relevant information is returned. It fundamentally reduces all the model round trips to just the minimum amount of large language model calls. So, you can reduce 30 to even 50% of token consumption and run agents much faster. This is particularly good for use cases where you're processing large data sets that needing aggregations or fairly deterministic tool call sequences that can be represented in code.
And if you're interested in learning more, we have step-by-step examples and tutorials in AI Builder Club about how it works and how you can set things up properly for both programmatic tool and all the other techniques that we're going to talk about. Meanwhile, we have courses and content to cover AI coding, building production agents, and weekly workshops where industry experts and myself will come and share the latest learnings of building large language model software. And most importantly, we have this community of top AI builders who are building and launching AI products now. So, if you want to follow and learn what myself and other builders are reading and learning every day, this is a great place to be. I put the link in the description below so you can join if you're interested.
But yeah, this is the first part of programmatic tool call. Next, let's talk about dynamic filtering for web fetch.
Dynamic filtering is a subset feature of this programmatic tool call. So, Entropic releases dynamic filtering capability for the `web_fetch` tool. The problem it tries to solve is that the `web_fetch` tool traditionally dumps a large HTML page into the cloud's context window. But most of that HTML content is irrelevant, and this leads to a lot of wasted tokens in the cloud's context window and reduced accuracy. And what dynamic filtering does is that it will add a layer in the middle. Instead of returning the full raw HTML, it will run a code that filters out only relevant content and extracts only those relevant ones into the context window. And from their testing, this dynamic filtering method is able to reduce on average 24% of token consumption.
To activate it is also pretty straightforward. You just need to be making sure you're pointing to the special version of `web_fetch` tool. You will see in the API response, you will automatically see those code execution steps happening in the middle to extract only specific relevant keys. So, in the end, it returns back only the relevant content back to the large language model.
Meanwhile, OpenAI also introduces the tool search concept. So, the problem it tries to solve is that the way agents use tools or APIs is not really scalable very easily. Loads up context window in your agent regardless of whether this tool is always needed or not, and this didn't feel very optimal. And also, why so many people are dropping the concept of API instead of moving towards the skill plus CRI version because it's just so much more token-efficient? However, agent skills are not a complete replacement for APIs, because APIs also come with a lot of benefits, especially around type safety. Since large models will know exactly what the input schema is and will try to follow that exact input schema, which you don't really get from the skill plus CRI easily. That's why Entropic released this tool search concept.
What that means is, instead of loading up hundreds of tool schemas into the context window, you just have one tool called `tool_search` tool that can be used to retrieve relevant tools, and that takes only around 500 tokens. So, for many scenarios, it can lead to up to around 80% of context window optimization.
And the way it works is that you can give your large language model call a `tool_search` tool, and then for each tool API, you can start setting this `deferred_loading` data. Once the `deferred_loading` is true, then this tool won't be visible to the agent by default. Instead, the large language model can use this `tool_search` tool to retrieve the information dynamically. And in the large model call response, you'll just see some additional steps show up in the middle where you will see it will try to invoke this tool. And this works with APIs as well. For the API server, it is actually very flexible. You have this one config called `default_config`, and if you set `defer_loading` to be true, that means all the tools from this API server by default won't be visible. But you can set specific actions within this API, like `search_events` here, to be `deferred_loading` false, which means only this specific function will be always visible. So, this is a really good method if your agent has more than 10 different tools or APIs and use this to significantly reduce the amount of token consumption.
And lastly, tool use examples. So, this is trying to solve another problem that for some of the complex tools, you found large models, even with APIs, will struggle to know how to use this tool. Think about a tool you might have for customer support called `create_ticket`. It has many different properties that the agent needs to fill in. Even though the schema will clearly describe what kind of field needs to be put in, but it's not entirely clear how to fill in each field. For example, for `due_date` here, it's not clear which format you want the agent to output for `due_date`. And also, there might be some correlation between one field to another. Like in this escalation, you might have different SLA hours depending on the `level` field here. And all those things can be probably achieved before by adding more description and prompts to indicate how to use a tool. But now they have a more specific field for you to adding those information in called `input_examples`.
So, when you define a tool, you can also pass in an array of examples of how this tool will be called. So, it can be used as an example for the agent to follow. And this is particularly useful for complex nested structures where valid JSON doesn't imply the correct usage, or your tools have many optional parameters, and large language models just keep forgetting or not filling certain optional parameters when it should. In his own experimentation, using tool use examples can improve accuracy from 72% to 90% on complex parameter handling.
So, those are a few new use cases and patterns that Entropic released about tool calling that I thought was really exciting and interesting. And again, if you want to learn more, there are step-by-step tutorials in AI Builder Club you can read and follow. And every week, we have weekly workshops to talk through those latest learnings from both myself and industry experts. And if you want to follow and learn what I'm reading every day about this constantly changing AI field, this is a great place to be in. I put the link in the description below so you can join if you're interested.
I hope you enjoy this video. Thank you, and I'll see you next.