Transcription
Google recently released a white paper titled "Agents," where they share how to effectively build agentic systems. We're going to look at key insights from this paper. This is especially useful if you're building systems on top of agents.
People use different definitions for agents, but now there seems to be a convergence or agreement happening in the community. So first, let's see how Google defines an agent. According to this white paper, an AI agent can be defined as an application that attempts to achieve a goal by observing the world and acting upon it using the tools that it has at its disposal. Essentially, it's an LLM system that has certain tools to interact with the external world.
The cognitive architecture that Google proposed in this white paper suggests that an agent is supposed to have three different components. The main component is the LLM or the model at its core. Then, the model has access to some tools, for example, web search or interacting with different databases. Finally, it has an orchestration layer where it can do planning and reasoning, take actions using these tools, and update its memory, keeping track of both the actions and results in the short term as well as the long term.
Think of agents as autonomous systems that can act independently of human intervention, especially when provided with proper goals or objectives that they are meant to achieve. In general, if you have a clearly defined problem and a specific predefined set of steps that you can take, my recommendation would be to use a workflow. There is an article from Anthropic titled "Building Effective Agents," which describes this concept of workflows. I'll also have a very detailed video on the topic, which will be provided in the video description.
The idea here is that if you have predefined steps that the system can take, then you don't really want to rely on the probabilistic nature of the reasoning and planning of the agent. I think where agents actually shine is when there's an absence of explicit instructions from the human or no predefined steps that you can take to achieve something. This is where the reasoning capabilities of agents can be extremely helpful.
Before looking at some of the key insights, let's understand the difference between agents and models. An agent is not just an LLM with function calling or tool usage; it's a lot more than that. The first and foremost difference is knowledge cut-off dates. Models are limited to the training data, while the knowledge of agents can be extended through connections with external systems using the tools at their disposal. For example, a web search can expand the knowledge that the agent has access to.
Second, an LLM, by its nature, is a single inference or prediction based on the user query. You need to have a whole infrastructure that can manage chat sessions if needed, but by its nature, the LLM is a single input-output system. On the other hand, the agent has the orchestration layer, so it can implement and manage chat history and even multi-turn conversations. It can look at the previous conversation it had with the user, update its plans, and actions accordingly.
Similarly, models do not have any native implementation for tool usage. On the other hand, agents have the ability to interact with the external world using tools. The last component is that models have no native logical layer implementation. You can use prompting techniques like Chain of Thought or REACT to provide complex prompts that will guide the model, but natively it does not have that implementation. However, by their nature, agents use one of these reasoning frameworks to do the planning, execution, and reasoning on top of the inputs and outputs.
So we're going to look at Chain of Thought, REACT, and Tree of Thought to see how these different frameworks are used by agents. The key difference is that agents have access to tools and can reason when to use certain tools. Here is a quick example: the model has access to different tools, which can look up flight information, execute code, or perform calculations when needed.
Based on the user input, it will first come up with an internal thought process. There are different frameworks that we are going to look at in this video. Based on that, it will take action. The action could be using a tool, analyzing the result, updating its internal thought process based on the observation, and generating final responses. This can happen in a single or multi-turn setup.
Now, for the reasoning, the key component is the frameworks that are used. Whenever we talk about agents, people discuss reasoning capabilities or planning, but how exactly are they implemented? In this white paper, they discuss three different frameworks. The first one is REACT, which is a prompt engineering framework that provides a thought process strategy for language models to reason and take action on the user query, with or without in-context examples.
The REACT framework is probably one of the first and most famous frameworks used for building agents. The idea is that whenever you receive a user query, the LLM will first break it down, create a plan, and then, based on the tools it has access to, execute the plan, observe the outputs, and update the plan if needed based on the observations.
The other one is Chain of Thought. You are probably familiar with this; people usually ask the LLMs to think step by step. Before some of these reasoning models like O1, this was a very famous technique. This enables reasoning capabilities through intermediate steps, and that's where that thinking step by step comes in. It's actually a family of techniques, so there are some sub-techniques like self-consistency, active prompting, and multimodal Chain of Thought that you can utilize to build agentic frameworks.
The last one they discussed is Tree of Thought. This is a relatively new one, and some of the reasoning models utilize this. It is well-suited for exploration or strategic look-ahead tasks. You basically create multiple potential solutions and then explore which one is the best.
Just look at an example of Chain of Thought again. Here, you have a user question, then there's an internal thought process. For example, "I want to book a flight," so it decides to use the flight tool, gets the input from the user prompt or query, executes this function call, gets the observation, updates its internal thought process or creates another thought based on the observation from the tool, and then generates the final output to the user.
Now, all of this is happening at the cognitive architecture, which Google refers to as a framework for reasoning, planning, and decision-making. According to Google, this cognitive architecture gives the model the ability to not only be reactive to user input but also to be proactive.
The second most important component of any agentic system is the tools that are at its disposal. Essentially, the LLM is frozen in time and is limited to the amount of training data that was used for training. But the real power of agents is in their ability to interact with external data sources, and that comes through the usage of tools or functions.
Google buckets these tools into three different categories: extensions, functions, and data stores. The first one is extensions, which bridge the gap between an API and an agent in a standardized way, allowing the agent to execute APIs regardless of their underlying implementation. This sounds very similar to the concept of model context protocol that was recently introduced by Anthropic.
Let's look at a quick example. Do you want to use the Google Flights API? If you want to fly from Austin to Zurich and you provide a prompt like, "I want to book a flight from Austin to Zurich," then the agent can easily figure out that the origin is Austin and the destination is Zurich. That information can be provided to the Google API.
However, if you're prompted with something like, "I want to book a flight to Zurich," the agent has no clue about the origin of the flight, and that's why it will not be able to make the API call using a simple function call. This is where the extensions come into play. They teach the agent how to use API endpoints by providing a few short examples and teaching the agent what arguments or parameters are needed to successfully call different API endpoints.
For each API endpoint, you're going to add a few short examples. That way, the agent will know when to use an extension and what the different inputs are that are needed. The second set of tools is called functions. These are very similar to functions in any programming language, where a function is basically a self-contained module of code used to accomplish a very specific task.
In terms of implementation, this looks very similar to what an extension is. The only difference between a function and an extension, according to this white paper, is that functions are executed on the client side, whereas extensions are executed on the agent side. The model just generates input outputs for a function that executes or implements a very specific operation that needs to be taken care of on the client side. The results are returned to the agent, which will use those observations to update its plan and take some actions.
To recap the difference: in the case of extensions, these are a few short examples where the agent can execute them. In the case of functions, these are actual functions where inputs are coming from the LLM. Then, on the client side, we execute those functions and send the results back to the agent or the LLM. Functions give you a lot of flexibility because the developer can offload certain tasks on the client side.
One use case could be security because this gives businesses or developers a lot more control over how the agent accesses sensitive data or performs specific operations on the data. The third family of tools is what they are calling data stores. This is basically expanding the knowledge of your LLM. The idea is that if you have private documents or certain websites that came online after the training cut-off of your LLM, then you can use data stores, which will format these different types of datasets into a standardized format that can be used by the agent.
One example of this is RAG, or retrieval-augmented generation. In RAG, you standardize all your information in a vector store using an embedding model. Whenever there's a query related to the information in your vector store, you use the embedding model to compute embeddings for the user query, then do retrieval on the vector store or database. After that, the retrieved contents are passed on to the agent to generate the final response.
In my humble opinion, RAG is probably one of the most important and practical applications of generative AI and agents in general. It's used in industry; everybody is trying to build their own retrieval system. If you're interested in that topic, make sure to subscribe to the channel. I build a lot of content covering different RAG techniques. I even have a course on advanced RAG techniques if you're interested. The link to that course is going to be in the video description.
RAG is so important that most of the agentic frameworks you see will have dedicated tools for retrieval. That is why even Google, in this white paper, is considering RAG or retrieval or data sources—whatever you want to call it—as a specific category of tools.
Here's a very nice table that recaps the different tools you have and their use cases. You have three possible tools. If a developer wants the agent to control the interaction with the API endpoint, then you want to use extensions because that is going to be executed on the agent side. This is used for leveraging native pre-built extensions. For example, there are different APIs that are pre-built, and you just need to provide some examples of how to use those.
Then, extensions are a very good use case. Anything that you want to be executed on the agent side should be implemented as an extension. On the other hand, function calling is going to be done on the client side, and that is for security and authentication reasons.
Another consideration here is time constraints. If you want to do async operations or batch processing, then you want to use function calling. The last one is if you want to interact with data sources and use those to expand the knowledge of the agent, then you want to use data sources, which is basically RAG implementation. That again is going to be executed on the agent side because the agent is going to be making use of these knowledge bases or data sources to extract information.
There's a section on enhancing model performance with targeted learning. You could use three different approaches for targeted learning. One is in-context learning, where you provide a very specific prompt, tools, and a few short examples at inference time. This will allow the model to learn on the fly from the provided examples. This is a very popular technique, and frameworks like REACT or Chain of Thought can make good use of this.
However, the problem with this is that it's kind of frozen in time because you provide these few short examples once based on the user prompt. If you want to dynamically populate the model's prompt, then you could potentially use something like retrieval-based context learning. In this case, you provide a few short examples specifically based on the context in which the agent is being used, and then you can dynamically populate the prompts based on the retrieved context.
Both of these are relatively easier to do and are a lot more flexible. In both techniques, the model is learning on the fly. If you want a lot more permanent learning, then you can use techniques like fine-tuning-based learning. In this case, you will need a larger dataset that will have specific examples for inference on how it's supposed to use different tools and how to interact with different scenarios. This is going to be a lot more permanent, and your prompts will be a lot simpler.
If you don't want to add a lot of few shots and your context window is very limited, then you can use something like fine-tuning-based learning. They also have an example of how to get started with agents in LangChain. I have created a number of different videos on different frameworks and different LLMs on how to build agents, so I'm not going to cover that part.
One of my goals for 2025 is to cover a lot of technical content that will include how people are building and using agentic systems in industry. If that's something you're interested in, make sure to subscribe to the channel. There are also a lot of exciting things happening, which I'm going to share in a few weeks.
I hope you find this video useful. Thanks for watching, and as always, see you in the next one.