Transcription
If you are a developer building agentic systems, this is a must-read blog post from Anthropic. There is a lot of hype around frameworks for building agentic systems and agents themselves. We have some really great frameworks like Crew AI, LangCraft, and AutoGen, but in most cases, you don't need them.
The most successful LLM implementations for agents use simple composable patterns rather than complex frameworks and specialized libraries. This is kind of the main takeaway of this blog post as well. You want to start with the simplest possible solution and only add complexity when needed. Most of the frameworks have a lot of abstractions, which you don't want in your production systems.
In this video, we are going to look at the blog post, some of the architectures that they have suggested, and I'll also show you code examples of how to implement those. In the process, we will explore different patterns, understand when to use them, and most importantly, learn how to avoid unnecessary complexities in our agent designs.
First, let's clarify what exactly is meant by an agent, because this term gets thrown around a lot in the AI world. For me, the definition of an agent is extremely simple: you have an LLM that has access to a number of tools, and it can decide when to use which tool. This meme really captures the essence of agents.
But as the blog post states, some developers think of agents as fully autonomous systems that operate independently for long periods, using various tools to accomplish complex tasks. Others see them as more structured systems following predefined workflows. Anthropic put this into two categories: one is workflows, and the other one is agents.
Workflows are systems where LLMs and tools are orchestrated through predefined code paths. These steps are clear, ordered, and predictable. Think of this as a traditional state machine, where you have the current state of the system based on some external input or feedback. The system moves to another state. Now, these states, plus the paths that the system can take, are well-defined well in advance. Most of the systems that you're going to build using LLMs today will actually fall into this category, and that's exactly what I usually recommend when I'm working with companies.
Now, agents, on the other hand, are more like autonomous problem solvers. They dynamically direct their own processes and tool usage, maintaining control over how they accomplish tasks. They can adapt their approaches based on the situation. This is more probabilistic in nature, where the system can plan what it's going to do and which tool it's going to use, rather than following a predefined path.
The problem with agents defined in this way is that if something goes wrong, it's extremely hard to debug. On the other hand, workflows can capture really complex systems and they are relatively easier to debug if something goes wrong. So, in general, your first option should be to use workflows rather than agents. But there is a place for agents in our current systems or applications, and we will talk about those later in the video.
Now, the next section is about when and when not to use agents, and I really like this part, which resonates with my own experience. The main recommendation should be not to build agentic systems at all if you don't need them, and most people and companies actually don't really need probabilistic agentic systems. A simple workflow is more than enough.
If you are building agentic systems, they will add additional latency and cost, which is going to be a trade-off with performance. So these trade-offs need to be considered when you are working in a business setup. A workflow, on the other hand, can be used to build very complex systems, especially if you need predictability and consistency for well-defined tasks.
Now, agents are extremely flexible, and if you need more model-driven decision-making, then they are definitely an option. But as they say, for many applications, optimizing single LLM calls with retrieval and in-context examples are usually enough. So just to sum it up: don't use agents if you don't need them, and if you think you need them, you need to make sure that you exhaust the possibility that a simple state machine is not enough for your application.
But what about all these different frameworks? You can use Crew AI, LangCraft, and AutoGen to build very complex systems. Now, in general, if you're just starting to experiment, these are great frameworks. You should use them for experimentation purposes because these frameworks make it easy to get started by simplifying standard low-level tasks like calling LLMs, defining and passing tools, and chaining calls together.
They also introduce extra layers of abstractions and hide the underlying prompts and responses, which makes a huge difference in the performance of your agentic systems. But if you want to put these systems in production, you actually need to understand what is going on under the hood. In most cases, you don't really need this abstraction or the additional complexity that frameworks like LangCraft or Crew AI would actually add to your application.
That's why their recommendation is to start using the LLM APIs directly. Many patterns can be implemented in a few lines of code, and we will look at some examples. It's a lot easier to do than what you would think.
Another problem that I have seen with these frameworks is the breaking changes. Because things are moving so fast, none of them are mature. There's new research, new techniques, and sometimes additional changes or updates will break your existing code bases. So it's even hard to keep up or maintain your code bases if you rely on these frameworks.
But keep in mind this recommendation is for developers. If you're a non-coder or you don't already have a lot of coding experience, then my recommendation would be to start with frameworks because it will make it easy to begin. But later on, as you gain experience, the recommendation would be to start directly using these different APIs from the LLM providers and build systems around them.
A good example is Swan from OpenAI or the Transformer agents. If you go through the code, these are repeated composable patterns that they use, and the abstraction is kept to a minimum.
In the rest of the video, we're going to look at different common patterns that the Anthropic team recommends or they have seen in production. The first thing that they talk about is the augmented LLM. When you're using or building these agentic systems, either workflows or agents, the way they describe it, you cannot use a simple LLM itself. You need to augment the LLM with different abilities, and these abilities are retrieval tools and memory.
Now, here's an input to the LLM. The LLM can use something like a retrieval system, for example, a RAG system, to get factual information, or it can interact with external systems using tools or API calls. These are going to be function calls. Then, in order to keep track of everything that it has seen so far, all the actions that it has taken, the LLM also needs to have access to some sort of memory. That is why they are calling it the augmented LLM.
So that is a very important building block when it comes to building agentic systems. Something they highlighted when implementing the augmented LLM is that you need to tailor these capabilities or augmentations for your specific use case and ensure that they provide an easy, well-documented interface to your LLM.
The way I recommend to think about it is the implementation of functions or methods in a programming language. The best practice is that you want to make sure that your function performs a single operation. If you add a lot of complexity in a function, it kind of beats the purpose. Similarly, if you're building an agentic system, each agent is supposed to perform a single function, and the LLM needs to have access to well-defined tools that will help it execute that single function or a single specific task, rather than giving it a number of generalized tools.
This is extremely important to keep in mind when you are building systems. These are not magical systems, and you want to make sure that the LLM or the agent has to make the least amount of decisions possible.
Now, let's talk about different patterns. The first pattern is a workflow or a state machine, and this is called prompt chaining. Prompt chaining decomposes a task into a sequence of steps where each LLM call processes the output of the previous one. Essentially, the output of the first becomes the input to the second, and so on and so forth.
Now, sometimes you can have these programmatic checks or gates, so you can actually check the output of a subsystem and make sure that it is executing what you want. If it fails, you can repeat that task. Think of this like breaking down a complex essay into an outline. Then, based on that outline, you generate a rough draft, and then based on the rough draft, you will generate the final version. Each step kind of builds on the previous one.
So when to use this workflow? It's ideal for situations where a task can be easily and cleanly decomposed into fixed subtasks. The workflows follow predefined paths in order to execute or accomplish a task. The main goal is going to be to trade off latency for higher accuracy by making each LLM call an easier task.
Examples of use cases would be content generation, as I said, first generating an outline, then expanding each section. Code generation is another example; you can create a high-level design, then implement specific components. Translation is another example where you can first translate it literally, then refine the translation for language flow.
The second pattern is routing. This pattern is like having a smart receptionist who knows exactly which specialist should handle each request. In the front, you have a router LLM, which is extremely smart, and then it passes on the task to a specialized LLM depending on the nature of the task. These could be different LLMs with different system prompts, and depending on the user input, this router selects which LLM and which system prompt to use.
Routing works well for complex tasks where there are distinct categories, and different LLMs or different models are better for each of those categories. The router itself can either be an LLM or a traditional rule-based classification system, so it doesn't really need to be a generative AI system here.
Some examples again are customer service, where you can direct different types of queries to appropriate departments. Content moderation is another example, so you can route different types of content to specialized checkers. Code reviews are another example, where you have different aspects of the code reviewed by different experts, and then the router can decide which expert to choose.
The next one is parallelization. The idea here is to execute the task in parallel. You can break down the task into subtasks that are independent. This will be an example of sectioning, which breaks the task into independent subtasks. You run those through independent LLMs and then aggregate the results.
Another example would be voting, where you run the same task through multiple different LLMs. The idea is to get diverse perspectives or outputs and aggregate them to get the final output. The voting example kind of relates to ensemble models in traditional machine learning systems. If you're familiar with ensemble systems, the random forest is a perfect example of a system like this, where you take your input, run it through multiple different trees or models, and then combine those results.
The idea is that these different models are looking at different aspects, which aggregates the results, and usually ensembles give you better results compared to a single LLM or a single model. The problem is that you're making parallel calls, which will add cost when you run them.
So when to use this? It's effective when you can divide your tasks into subtasks that can be parallelized, and that will increase the speed, or if you want to have multiple perspectives to solve your problem. Sometimes it actually helps a lot if you provide a lot of information to an LLM. It can hallucinate, but if you break it down into subtasks and independent LLMs are looking at the subtasks, then you can potentially get better results.
A closely related pattern is orchestrator and workers. In the orchestrated worker workflow, a central LLM dynamically breaks down tasks, then delegates them to workers and synthesizes the results. If you look at this pattern, it looks very similar to parallelization, except there's this orchestrator right in front of the parallel section of your pattern.
You want to use this pattern when there are no predefined subtasks. If the orchestrator has to dynamically come up with the number of subtasks that need to be executed in order to achieve our goal, then you want to use this setup rather than something like parallelization, because it expects that you will be able to have predefined subtasks that you can run in parallel or run the same thing through multiple LLMs.
It's a lot more flexible compared to simple parallelization because the subtasks are not predefined but determined by the orchestrator based on specific input. This is like having a project manager who breaks down complex tasks depending on the input, assigns work to specialized team members, and then synthesizes results into a cohesive solution.
Examples again include coding products that make complex changes to multiple files each time, so you don't know which files to change ahead of time, but depending on the input, you will change them. Search tasks that involve gathering and analyzing information from multiple sources for possible relevant information are also examples.
The last one is the evaluate-optimizer. In this workflow or pattern, one LLM call generates a response while another provides evaluation and feedback in a loop. Think of this as having a writer and an editor working together. The writer writes something, the editor gives feedback, and based on that feedback, the writer has to improve it. The process goes on for a predefined number of steps.
In traditional computer vision, GANs or Generative Adversarial Networks were a classical example of this evaluator-optimizer workflow, where one system generates an image, and the other figures out whether it's fake or not or assesses the quality. Based on that feedback, the generator will try to generate better images.
Now, when do you want to use workflows? They are particularly effective when you have clear evaluation criteria. If you can break down your task into predefined paths that the system can take in order to accomplish a task, then workflows are a perfect example. Workflows can also contain a hybrid approach, so it doesn't really have to be all an agent system within LLMs. It can have a traditional machine learning component or even a traditional software component along with this generative aspect.
So this should be your first thing to consider whenever you're building any agentic system. You want to make sure that there are deterministic steps involved, and if you can come up with a state machine to accomplish that, you should use workflows.
So far, we've been talking about workflows or state machines, but then there are agents. There are definitely use cases in which you want to use these probabilistic systems, which can make independent and automated decisions. For these systems to work, you want to monitor their progress, and usually, you want to have a human in the loop because those are the best kind of agentic systems that you can deploy in production.
Essentially, this is what it looks like: a human, the LLM, the LLM does an action, the action is executed in the environment, it gets feedback from the environment, and an action is executed through tools. Then the LLM will decide whether it needs to stop or, sometimes, as I said, have a human in the loop. It will show the results to the human, who observes those and provides feedback if needed or steps in if it needs to stop the execution.
But the main thing is that the LLM is autonomous, and its ability is to make a decision about when to use certain tools or when to take certain actions and when to use them. They can be used for open-ended problems where it's difficult or impossible to predict the required number of steps or where you can't hardcode fixed paths. This is where it's kind of really hard to evaluate as well. Sometimes they will just fail, and you won't even know the reason for failure since the LLM has to make a number of different calls in order to execute a given objective.
It could mean higher costs and potential compounding errors. Now, I am describing these issues; it doesn't mean that you can't use autonomous agents. There are a number of different applications, but I still feel like we are in the process of having these systems matured. So you really need to keep an eye on these systems if you're going to be running them in production.
Based on my own experience, here are a couple of examples where they show that agents can be helpful or useful. According to Agent, they resolve sweet bench tasks involved editing many files based on the task description, and then it has to run them, get feedback from the environment, and make the edits again if it doesn't pass the test.
They also refer to their computer-used reference implementation, where Cloud uses a computer to accomplish tasks. Different aspects of an agentic system include a human who interfaces with the LLM. You get the query from that; sometimes the LLM might refine the query based on user feedback. Then it starts interacting with the environment itself.
Here's a coding example in which it has to search for modified files and things like that, but it could be anything. It might be like accessing weather information or stock market data. So it goes and gets feedback from the environment, refines its tasks, executes it again, and when the LLM thinks that it has completed the task, it will come back and show it back to the user.
Okay, so these were different patterns that Anthropic recommends to look at if you're building agentic systems. You can combine these to build more complex systems, and this is not an exhaustive list. But let's look at some code examples of how you would potentially implement some of these workflows without the need for specialized libraries or complex frameworks.
Now, for this, we're going to look at this cookbook from Anthropic, which goes over basic workflows, the evaluator and optimizer, and orchestrator workflows that we discussed. The main file every one of these notebooks is using is this: it is basically calling in LLM, which is the Anthropic API client. That's the main LLM execution call, and then there's another function that extracts the XML tags and generates structured output. But these are the two utility functions that each of the notebooks is using.
So let me walk you through the example code in the notebook, which makes it extremely simple to implement these workflows in your own setup. If you want to run them, you will just need to clone the Anthropic cookbook repo, and you'll have access to these notebooks.
Now, in terms of implementation, this basic multi-LLM workflow implements prompt chaining, parallelization, and routing. For prompt chaining, you provide your system prompts and then execute them one by one. As we looked here, you have the first LLM call, second LLM call, third LLM call, and the implementation is essentially a for loop where you have system prompts for each of the LLM calls. You run the first LLM call, get the result, and feed that result back as input to the second LLM call, and so on and so forth.
Now, for the parallelization, you use multi-threading where you get the user prompt. The user prompt can be decomposed into subtasks and run those, or it can be the same prompt that runs through different LLM calls in parallel. This is an implementation of how you can potentially implement a parallel workflow.
The last one is the routing. In the routing, the main selector prompt basically gets the input, and then depending on the specific needs of the user, it can select a specialized LLM, pass those to that specialized LLM, and then get the results from that LLM and generate the final response.
So we are essentially looking at this router LLM, which is the selector prompt that I just showed you, and then you have these parallel pathways. But actually, not parallel pathways, because the router will select one of the available LLMs to generate the final output. This notebook also goes through some example use cases. I highly recommend looking at this notebook to get a better sense of how those simple implementations can be used for specific use cases.
Now, this second notebook shows an example implementation of the orchestrator and workers workflow. You want to use this specific pattern if you don't have well-defined subtasks and you need to create those subtasks dynamically at runtime. In this case, they have the orchestrator prompt, which is a bigger LLM that will generate a number of subtasks depending on the user input.
Then the worker prompts are parallel executors that will execute those subtasks. You also have this passing function for subtasks that are generated by the orchestrator. Whenever a user query comes in, you first pass this to the orchestrator. The orchestrator will generate a number of subtasks, then they will go through the workers, and each of the workers will execute those subtasks to generate their own results. At the end, you synthesize everything using a specialized synthesizer.
Now, the example use case in this notebook is marketing variation generation. So here's the orchestrator prompt. You want to make sure that you customize your orchestrator for the specific task that it is supposed to do, or you also want to customize your workers because those are the ones that are going to be executing those subtasks.
This last notebook implements the evaluator-optimizer workflow. You want to use this when you have clear evaluation criteria and you have a value from iterative refinement. In order to execute this or implement this, there's a generator function, then there is an evaluator. You need to have a very clear idea of what the evaluator is supposed to do, and then you just have a loop that will go through this loop of generation, evaluation, feedback, and whenever we pass based on the evaluation criteria, that's when it's going to stop.
The example use case that they have presented is an iterative coding loop. So you have the evaluator prompt, which will evaluate the code, and then there is a generator prompt, which will generate the code for a given user task.
The takeaway from this blog post and my own experience is that you want to start simple and only add complexity when needed. Frameworks are amazingly helpful if you're building a proof of concept, but usually, you end up customizing a lot of your code for your own application. Sometimes it's better to just start with a greenfield implementation and build your own custom solutions. This will save you a lot of engineering efforts, a lot of time, and a lot of compute costs as well.
Now, this suggestion or recommendation is specifically for developers. If you are a non-developer and you're looking for a quick way to implement, frameworks are the best way to go. But again, if you're going to deploy anything in production, you need to understand how they work and what the underlying implementation is, because that is going to save you a lot of headaches later on.
Anyways, I hope you found this video useful. Thanks for watching, and as always, see you in the next one!