Transcription
One of the most exciting launches of the past month was Manis, an agent that could operate your computer. Part of the in-depth, kind of like, technical investigation behind it revealed—and and and—the creators revealed this on Twitter as well—that it takes inspiration and uses this idea of Coda. So Coda is basically, instead of doing tool calling, you're actually going to have the agent write code that can call those tools.
To be a little bit more explicit: when you have, uh, off-the-shelf agent that most people think of, the way it works is it runs in a loop. You give it a bunch of tools; it decides which tools to call; it calls those tools; and then you keep on going until it's finished. The way it does that tool calling is it uses the tool-calling functionality that a bunch of the closed-source models support. So they have something called tool or function calling: you pass in a bunch of tools; it'll generate JSON schema; it will generate the right one to pick, maybe multiple; and then you actually execute those and call those.
The idea of Code Act is a little bit different. Rather than generating that JSON for which tool to call, you actually write code. Like this: even if search_web is just an arbitrary function that you define or arbitrary API you want to hit, you still call it as if you're writing code. The intuition for why this works is kind of twofold. So, one: LLMs are really good at writing code; that's one of the main things that they've been trained for; they're arguably better at that than writing JSON. Two: a lot of the code writing that gets done is there's a lot of thought that goes into like what steps the code gets executed in, what functions call in what order, what the results are. And so when an LLM writes this code to do this function calling, it can actually chain a lot of these function calls in a deterministic way. So in the normal, kind of like, tool-calling paradigm, you call one tool, you get back a response; you call another tool, you get back a response. This code calling allows you to save this results variable and then reuse it, and this can all be kind of like planned by the LLM in one single LLM call.
To make this really concrete, I want to show an example of using Langchain Coda, a new Python package that we created to do this. This is built on top of Langchain, and it implements the ideas from the Coda paper. The basic idea is that you will prompt it to write some code; you will then execute that code, even if that code involves calling kind of like arbitrary functions that aren't necessarily in your coding environment; and then you basically run that in a loop until it decides that it's done. So let's take a look at the example.
In the example here, we define a bunch of tools. So you define these functions that basically take in inputs and return stuff, and they're nicely typed. You then define your code sandbox; this is just a super janky local one; this will use the local eval function; you should not use this in production; this is not safe, but this gets the point across. And this basically will run the code. You'll notice that we're also taking in the existing, kind of like, variables that exist, and then we're returning any new variables that are created. This is so that in the future we can use this. So the type signature of this evaluation function should be: it should take in the code string and a list of the locals, which is this dictionary of variables, and it should return, uh, result, and this is basically some string and then a dictionary of any new variables.
Once we have that, we can then create the agent. It's pretty simple; we're just going to import the chat model we want to use; in this case, we're going to use Claude Sonnet; we're going to import this create_coda function from Langchain Coda; and then we're going to import this checkpoint so that we can use this for follow-up conversations. And that's it; we'll get our agent this way, and we can then interact with it.
So let's check out an example of this in practice. I'm in the Langchain Coda repository; let's look at examples; in this math example, and we'll see this is all the same code that we had defined in the example in the, um, in the README. The only thing that I'm going to do before I run this is I'm actually going to open this up in Langchain Studio so I can visualize it. So I defined also this `langchain.json` thing; this just points to the agent in the examples folder; and then once I have that, I can run `langchain dev`; it'll pull it up, and I can start interacting with it in the browser. Before I interact with it, I just want to show the architecture of the agent; it's super simple; it's got this call model node and then the sandbox node, and it basically iterates between them.
So let's ask it a question. The question to ask: let's get this nice long math question. So this is going to involve calling a lot of the tools that we define above, and if we did this with a normal tool-calling agent, it would take a bunch of iterations; it would take a while. But let's see what happens with Coda. So let's submit this. We can see it start to iterate; we can see the models start to generate some output here; it generates—you'll notice that it will generate some code in this backtick backtick backtick Python code block. So we put this in the prompt—I'll show the prompt in a little bit—but we put this in the prompt. After it does that, it then goes into the sandbox; it then basically returns some variables; and then it calls the model again. And so that was fast. So let's, uh, let's take a closer look at what happened by going into LangSmith.
So here we have the full trace, and here we can walk through in a little bit more detail of what exactly happened under the scene. So we call the model for the first time. When we have this model call, we have this—so this is the system prompt; I'll show where this is created—and then we have the human message that we pass in; here's the LLM output that the AI does; and so you can see it writes this Python code block, and it does a few things. So it creates a bunch of intermediate, kind of like, variables; it prints out some stuff—so this is this is what the result of the the sandbox is—and I'll show this, but it prints out some stuff, and then it does some more stuff, and it prints out some more stuff. Let's go into the sandbox, and let's see. So this is the script that's passed in; so we pass in this stuff; I don't really care about inputs, um, so let's hide this, um, and then for the output, it gets back this thing. So it gets back a human message; this is all the print statements. So if we look down here, we can see that we print out like, uh, yeah, the baseball lands final position. So here, uh, we get this back. Now it actually calls Chat Anthropic again. So what is it doing here? I actually didn't expect this. Let's see what's going on; this is why LangSmith is great. So here, uh, it's analyzing these results; so it actually sees from the printed-out things, the intermediate results, and it actually sees that it got something kind of nonsensical. So okay, so this is now realizing this nonsensical; it's actually doing some math, but then it's going to redo the math here to to recalculate. So here, if we go into sandbox, we can now see that what we get returned from the sandbox this time is this string here, and then the final call to Chat Anthropic is basically just returning to the user the initial or the final result.
So that's kind of what's going on under the hood, and you'll notice that in this initial thing that it's generating, it's using a bunch of these tools. So `multiply` is a built-in tool, or sorry, `multiply` is a tool that we pass; um, we have, uh, we have, we have what else? We have `divide`, um, `multiply` again, `subtract`. So it's using these tools that we define and pass in in the example code right here. So `multiply`, `divide`, `subtract`. So this is a pretty simple example because it's using things that honestly it could use the built-in library for, but the point is that you can define these specific tools that can call APIs, and then the LLM can just write code to call those functions. And so if we take a look under the hood of what's going on in Langchain Coda, it's pretty simple; it's just a single file. We see that we have this `create_coda` function; it takes in a model to use; it takes in a list of tools; it takes in this `evaluate` function; this is maybe the most important one; this is the function that actually runs the code, and remember this is the code that has these tools injected to it as context; and then it takes in a prompt. If you don't pass in a prompt, it will use the default one. This is what the default one looks like: it basically says, "You will be given a task to form; you should output either a Python code snippet that performs that provides the solution to the task or a step towards the solution; any output you want to extract from the code snippet should be printed to the console." So this is important because this is what's passed back to the user, and then code should be output in a fence code block because we're going to parse that out. So that's the backtick backtick backtick Python; and then, uh, it says, "In addition to the standard to the Python standard library, you can use the following functions," and this is where the tools are inserted into the prompt. So we need to let the LLM know the tools that it has access to; so it has access to the Python standard library, but then also any that are passed in. And so if we go back to our LangSmith trace, we can see in the system message all of these functions that are defined right there.
So this is pretty much it for Coda; it's a pretty cool technique; it allows you to do tool calling in a different way than the standard way that the the model labs will will kind of like provide off the shelf. One benefit is that you don't have, like, some of these models just don't support the traditional or the the the fancy tool-calling technique where they output things in in JSON, and it's parsed; that's that's generally something that's more part of the API, but all the models are good at writing code. So with the right, kind of like, prompting and parsing and and code evaluation, code execution environment, you can actually do some more complicated things than you could with just the normal function calling. So Langchain Coda is really interesting; I'd encourage you to check it out; I encourage you to try it out for some problems. Thanks for checking out this video.