Transcription
I'm going to tell you how to build effective agents. Anthropic, the company behind the Claude family of models, just dropped a bunch of information about how to build effective models, and I read through it. It's actually really good! We're going to go over it together, and I'm also going to give my thoughts after building a bunch of agents myself. So let's get into it.
This video is brought to you by Vulture, the easiest way to power your generative AI startup with the latest Nvidia chips. Check out Vulture; I'll drop a link in the description below.
So, this is the blog post that came out just about a week ago: "Building Effective Agents."
Now, the first thing it starts out with is saying that you don't need complex frameworks to build agents, and that's true. Look at custom GPTs from ChatGPT. Those are effectively agents. You choose a personality, you choose a role, you give it tools, you give it memory, and that constitutes an agent in my mind. But that's the most basic form of an agent.
I personally use Crew AI; I'm an investor in them, so of course I use them. I also think they're the best agentic framework. If you need anything more sophisticated than just defining a single agent, agentic frameworks are really, really powerful. These agentic frameworks are getting more powerful by the day.
Specifically, what they say here is that consistently, the most successful implementations weren't using complex frameworks or specialized libraries. Instead, they were building with simple composable patterns. Now, as these agentic frameworks mature, you're going to want to choose one of those. You don't want to reinvent the wheel each time; you don't want to try to figure out what the best patterns are each time. That's the point of a framework. That is why we've had frameworks for code forever.
Next, they go over what an agent is. A lot of people have different definitions of what an agent is. I personally think an agent is essentially the core LLM, the intelligence wrapped with memory tools and the ability to collaborate with other agents. Anthropic says some customers define agents as fully autonomous systems that operate independently over extended periods, using various tools to accomplish complex tasks. Others use the term to describe more prescriptive implementations that follow predefined workflows.
At Anthropic, we categorize all these variations as agentic systems but draw an important architectural distinction between workflows and agents. Workflows are systems where LLMs and tools are orchestrated through predefined code paths. Agents, on the other hand, are systems where LLMs dynamically direct their own processes and tool usage, maintaining control over how they accomplish tasks.
Now, the best agentic frameworks blur the lines between workflows and agents. There are certain parts of agentic systems where you want more structure and you want to use predefined code to help these agents along. Then there are other parts where you just want them to think and create and work with other agents to come up with the best solution. I really think, as I said, the best agentic frameworks blur this line or make it easy to use both in a single use case.
Next, they go over when to use and when not to use agents. This first sentence I could not agree more with: "When building applications with LLMs, we recommend finding the simplest solution possible and only increasing complexity when needed."
So important! That not only applies to agents; that not only applies to code. That is a life lesson. When building any system, start with the simplest implementation possible and only add complexity when necessary. Agentic systems often trade latency and cost for better task performance, and you should consider when this trade-off makes sense.
The more sophisticated and more complex your agent usage is, the more tokens it’s going to use, and it’s going to take more time. Now, solving that latency and cost is also something that you can work on with your agentic system. This is why I'm also an investor in Groq, because they have really cheap, insanely fast inference speed. So all of a sudden, when you plug that into an agentic system, the latency and cost part of the function becomes less critical.
So they double down and say workflows offer predictability and consistency for well-defined tasks, whereas agents are the better option when flexibility and model-driven decision-making are needed at scale.
When and how to use frameworks? They talk about different frameworks: LangGraph from LangChain, AI Agent Framework from Bedrock, Rivet, Vellum. Now, I don't know why they didn't mention Crew because Crew is basically the biggest framework out there, but fine.
They also talk about why frameworks should be used. They offer a layer of abstraction, they come with a bunch of built-in tools, and it's basically a predefined golden path. You don't have to think about the best practices as much. Of course, you have to learn the framework, but you just don't have to figure out a lot of the ancillary issues as you go.
Now, of course, there are downsides to using agentic frameworks. They often create extra layers of abstraction that can obscure the underlying prompts and responses, making them harder to debug. It also makes it tempting to add more complexity when it's not necessary, just because it's so easy to do so.
So we're going to go over some examples of what an agentic system looks like, starting from something pretty simple. Anthropic specifically talks about when a problem is this simple: their core models, their base models, have all the necessary functionality to solve it without needing any kind of additional framework.
So here's what we're seeing here: you have the prompt, then you have the LLM. The LLM, through the Claude family of models, has the ability to get search results, so retrieval, has the ability to use tools, and has memory. Then you have the output. After it decides what it needs to use, then it gives you the output.
Now, these base models are getting better and better, and really what that means is the model providers are baking in more and more agentic functionality directly into that base model, which is good.
Now, I didn't cover this much, but there was a release by Anthropic. By the way, Anthropic is completely on fire lately! They released something called the Model Context Protocol, and it is a framework for allowing LLMs to interact with third-party tools. It's essentially a definition for how to do so. Here, they describe it as allowing developers to integrate with a growing ecosystem of third-party tools with a simple client implementation.
Let's look at a workflow now, and this is a workflow for prompt chaining. So let's read a little bit about it. Prompt chaining decomposes a task into a sequence of steps where each LLM call processes the output of the previous one. You can add programmatic checks on any intermediate steps to ensure that the process is still on track.
So why would you need this? It sounds cool, but why would you need it? Well, the best way to explain it is when you have a situation in which it's complex and can be broken down into really modular pieces. If you try to get a model to just accomplish this multistep task all in one go, you're basically not going to get as high of quality.
So the trade-off here is to do each step independently, feed it to the next prompt, and then allow it to build off of that. As it says here, you're trading latency for quality.
Here's the example that they give: generate marketing copy, then translate it into a different language. These are two obviously different steps. So first, you would generate the marketing copy, then you would pass that to another prompt and allow it to do the translation separately. It's not trying to do both of those tasks in a single go.
Another workflow example is routing. Routing is incredibly powerful. You can have very specialized agents waiting to accomplish their task, and they can be very different in what they're able to accomplish. The cool thing is you can send a prompt and have a router decide which agent is most appropriate for whatever the task is.
What kind of tools does it require? What kind of expertise does it require? What kind of model does it require? Of course, because you can have different models for different agents. As it says here, routing works well for complex tasks where there are distinct categories that are better handled separately and where classification can be handled accurately either by an LLM or a more traditional classification model algorithm.
Now, one example of routing that they give here I think is absolutely brilliant, and so much so I invested in a company that does exactly this. Here they say routing easy common questions to smaller models like Claude 3.5 Hau and hard unusual questions to more capable models like Claude 3.5 Sonet to optimize cost and speed.
So if you want to give the right prompt to the right model at the right time, routing is a good way to do that. The company that I'm referring to is Not Diamond. They basically take your prompt, decide which of dozens of different models is most appropriate based on cost, latency, and quality, and then they will tell you which one to use.
So big cost savings, big latency improvement, and just overall quality improvement as well. But of course, you can create a simplified version of a routing algorithm yourself just using a model.
Another workflow is parallelization. If the order of operations doesn't matter at a certain step, you can have multiple agents working in parallel to decrease the latency of the completion of the task. I hadn't thought of this, but they described two different variations of parallelization.
First, sectioning: breaking a task into independent subtasks run in parallel. Simple. But then they also have voting: running the same task multiple times to get diverse outputs, essentially chain-of-thought reasoning. This is a very basic explanation for how the thinking models work. You come up with a bunch of different variations, and then you figure out which one is best, and then you continue from there.
So again, you can create this yourself. These are all very simple patterns in theory. Of course, they get more complex as you actually build them and productize them, but in theory, they're pretty simple.
So when would you use parallelization as a workflow? They give two examples for each type of parallelization.
First, sectioning: implementing guardrails where one model instance processes user queries while another screens them for inappropriate content or requests. This tends to perform better than having the same LLM call handle both guardrails and the core response.
Now, if you remember back to the LLM jailbreaking game video that I made, where essentially the game was there was this crypto wallet, and if you convinced the LLM to send money out of the wallet, you would win whatever was in the wallet. Really cool video, by the way! I'll drop it in the description below. They did not do that; they had one model handling all of it.
But as I kind of suggested in the video, if they had another model just double-checking everything—so one model to actually handle the sending and receiving of crypto, and then another model to actually be the guardrails to make sure it's not sending crypto—it would have been much more powerful.
Next in sectioning is automating evaluations for evaluating LLM performance, where each LLM call evaluates a different aspect of the model's performance on a given prompt. Very straightforward. You have one model generating the prompt and one model evaluating it.
Then for voting: reviewing a piece of code for vulnerabilities. When you have one model creating the code and another model evaluating that code, you get much better quality code. This is something that we've all seen. The first generation of code from a model is sometimes okay; sometimes it's not great. But if you run the code, give it the errors, and then allow it to fix its errors, it becomes better much more quickly.
What I've also noticed about models in general is that they're actually much better at evaluating things than generating things. That's not a hard and fast rule whatsoever, but generally, what I've seen is models are much more accurate at evaluating something than generating that thing.
Then, of course, evaluating whether a given piece of content is appropriate.
Then another workflow you have is orchestrator workers. This is something that I've used quite a bit with Crew AI. All of these patterns are available in Crew AI, of course, and most other agentic frameworks that you might want to use.
Now, what I really like about the orchestrator pattern is that it's able to get a result from an LLM, be the orchestrator, decide what to do with that result, and then potentially send it back for another iteration or send it off to another agent to complete the next step.
In their definition, in the orchestrator workers workflow, a central LLM dynamically breaks down tasks, delegates them to worker LLMs, and synthesizes their results. This is probably the workflow that I use most often.
I'll tell you specifically what I built with this workflow. I uploaded a PDF, asked an agent to create a bunch of truthful questions and answers, and then I had a reviewer agent make sure all of those questions and answers were accurate.
What I found is if I created 30 questions and answers, two of them might be hallucinated, meaning they either didn't exist or they were just wrong. So I had another agent check it. Evaluations, then I had the orchestrator agent send it back to the question and answer generator agent to generate two more and then send it back to the orchestrator. This all happened very seamlessly; it was really cool.
So when to use this workflow? There are two examples that they give: coding products that make complex changes to multiple files each time, and search tasks that involve gathering and analyzing information from multiple sources for possible relevant information. That is obviously the one that I created.
Then another workflow: evaluator optimizer. Again, another super common pattern for AI workflows. One LLM calls a response while another provides evaluation and feedback in a loop.
As we can see in this diagram, we have the prompt, we have the LLM call generator that generates a solution, the evaluator evaluates it, and either accepts it and gives it or rejects it and says generate another one. This is exactly what I did with the question and answer, except it used evaluator optimizer and then it also used the orchestrator. So it was both of those patterns together.
A couple of examples where this evaluator pattern is useful: when you have a clear evaluation criteria and when iterative refinement provides measurable value.
First, an example: literary translation, where there are nuances that the translator LLM might not capture initially, but where an evaluator LLM can provide useful critiques.
By the way, one of the most powerful ideas—and I've said this already, I'm going to say it again—one of the most powerful ideas in all of AI right now is the evaluation pattern. The idea that the first generation of a response to a prompt is not usually going to be the best. That is literally what the thinking model does: it generates a bunch of different outputs, votes on the best one, then iterates on it. It's just this really long generation-evaluation-iteration cycle.
Then a second example is complex search tasks that require multiple rounds of searching and analysis to gather comprehensive information, where the evaluator decides whether further searches are warranted.
Next, Anthropic goes on to just describe more about agents, and I want to pull out a couple of key sentences and ideas from this. First, let me show you this line: "Agents begin their work with either a command from or interactive discussion with the human user."
Now, that might not always be the case. I guess technically an agent will always have to start with a user because even if you set it to be completely autonomous, you're still setting it to do so. But still, the way that they're describing it is you're either saying, "Okay, go do this thing," or programmatically kicking off that agent to go do a thing.
They also talk a little bit about human-in-the-loop, which is another powerful idea for agentic frameworks. At which point is it critical that a human reviews the output or a human makes the decision? Also, are the agents capable of including a human in the loop at the right time?
Once the task is clear, agents plan and operate independently, potentially returning to the human for further information or judgment. Agents can then pause for human feedback at checkpoints or when encountering blockers. The task typically terminates upon completion.
So when do we use agents? Agents are good for open-ended problems. By the way, everything we've talked about previously has been workflows. Now we're talking about agents. Agents can be used for open-ended problems where it's difficult or impossible to predict the number of steps and where you can't hardcode a fixed path.
The LLM will potentially operate for many returns, and you must have some level of trust in its decision-making.
The following examples are from our own implementations of agents: a coding agent that resolves SBench tasks—by the way, if you haven't seen the interview with the SBench team, it's really interesting. I'll drop it down below—which involve edits to many files based on a task description, or our computer use reference implementation where Claude uses a computer to accomplish tasks.
So here's the high-level flow of a coding agent: we have a query from the human, basically, "Here's what I want you to do." The interface where the agent is clarifying and refining what the human wants it to do, so basically defining it much more thoroughly.
Then you send all the context to an LLM. Within the environment, the coding environment, you could do a bunch of things. These are all just tools, essentially: search files, return paths, write code, status, test results, and then we complete and display the results to the human.
Now, here's the important part and something I already touched on: these building blocks aren't prescriptive; they're common patterns that developers can shape and combine to fit different use cases.
The key to success, as with any LLM feature, is measuring performance and iterating on implementations. That is probably the most important thing in this entire document. What I have found more than anything, especially because everything is so early right now, is that you just have to test it, test it, test it, test it. Use observability tools, use agentic frameworks when you can, when it's necessary, and just test a lot of different things.
Run a bunch of different tests, benchmark it. A lot of these agentic frameworks, including Crew, include benchmarking as part of their core functionality. So definitely just test everything you can, and you're going to discover different patterns that work well for you and for the tasks that you need accomplished.
I want to say thank you to Eric Schluns and Barry Zang, who wrote this. This is a great starter article for how to think about agents. I want to do more educational material around agents, so let me know if you want to see that in the comments down below.
If you enjoyed this video, please consider giving a like and subscribe, and I'll see you in the next one!