Transcription
Hi everyone, welcome to this episode of hands-on with AI agents. Today, we are in for a treat. We have Socrates here with us to discuss agent ops, and yeah, we have a master class waiting for us.
Hi Socrates.
Hi Sida, and thank you for having me here. I'm Dr. Socrates Kartakis, a GenAI black belt for Google. Today, we will talk about how you can operationalize agents. So, how you can go from idea to production. But let's crack on. We will start by analyzing the different ops definitions. We will talk about DevOps, about MLOps, and we will land into generative AI and operations. Then, we will continue on deep diving more on the GenAI operations by showing the key operations like evaluation, model selection, etc. And then we will focus mostly on how you can operationalize agents with a very robust and efficient way. And lastly, we will share with you different resources and artifacts that you will be able to start using.
That sounds fantastic. But could you tell us a bit more about why MLOps and GenOps before we talk about agent ops? What's the backstory there?
Perfect. Let me clarify for you all the different definitions that we have in the market and to clarify for everyone how we separate everything, if it is okay with you. So, generally in the ops domain, right, the foundation of everything is DevOps. In DevOps, we follow best practices in terms of development; that means that we use repository pipelines, automatic testing, evaluation of our code, etc. So, what happened the last few years is we embedded the notion of MLOps. MLOps is an extension of DevOps that helps us to productionize machine learning solutions. And what is the difference between the two? Machine learning is a non-deterministic environment. That means that we cannot really predict sometimes what is the outcome, and we need to perform different operations and different technologies like evaluation of the models if we want to move to production.
Now, with GenAI, we observe new terms in the market. The one is the foundation model operations, in which we have the providers and the ones who build the models, like Google DeepMind, to operationalize the way that they move everything to production or in terms of the big, these huge foundation models that we have. But in GenAI space, we need to productionize applications; right, it is not anymore I productionize just one model; most probably I'm using a model in order to build a clever and AI-based application. Under GenOps, we have different domains. These domains, first of all, and the foundation of everything is how we operate with our prompts; so sometimes we heard that like prompt ops, and now we start having more other, we start having other domains as well like agent ops and RAG ops. Overall, we are using the notion of GenAI ops as an umbrella to hide all the new categories of operations that will arrive into the market because, as you know, GenAI is a quite new domain, and many technologies will arrive soon. The main point, contact point of application development and model development is the foundation models, and as I said already at the left-hand side, we have producers that they create the different models and finetuners that they retrain even the models, and at the right-hand side, we have the consumers who are responsible to use the different models to build applications.
So, essentially this means before we jump into agent ops, we have to talk about MLOps and GenOps because we derive the concepts from those; is that right?
Exactly. We need to talk about them; we need to give the foundation information and knowledge to the audience about these domains, and let's start with MLOps, if it is okay with you.
Absolutely.
I used to have this slide with all my customers first, why? Because most of the times we used to focus on technology, but actually whenever we want to operationalize something, we need to focus on people and processes and then to derive the technology. Likewise, MLOps is the combination of people, process, and technology in order to operationalize machine learning solutions efficiently. And most of the times we need to build MLOps platforms to reduce time to value. That means that we want to move more solutions faster into production with a very secure way. For example, we need to support private cloud, and how we can achieve that by standardizing our repositories, by standardizing our CI/CD pipelines, by standardizing our infrastructure and also enabling templatization. Most of the times we want to achieve around 80% templatization in order to make sure that everyone in our environment uses exactly the same principles. By the end of the day, we achieve reduction in platform, people, and processes cost.
But let's see the different environments that we had in the traditional MLOps platforms. The first environment was around cloud architects and security teams. This team is responsible for the whole infrastructure on the cloud for the whole business. Then, the next team is the data engineering team, or we can say the data lake/data mesh kingdom, in which the data engineers they used to ingest data to the cloud and they used to prep, pre-process the data, catalog the data, make them available to others. We have data owners who are responsible also to share the data with other stakeholders, and we have business stakeholders that most of the times they log into visualization dashboards and check the results.
The next environment is the data science environment. In the data science environment, the data scientists they used to experiment with different models, and they collaborate with ML engineers using an MLOps platform to productionize their ideas. At this stage, I need to, to, to say that we extend the platform team with MLOps engineers that they know the needs of machine learning that are unique because this is a non-deterministic environment, and they help the architects and the security teams to build the right platform for both data scientists and ML engineers.
Lastly, but not least, we have the machine learning governance environment. This environment is a centralized place that we store all the models. We store all the data, the performance of the models, the artifacts that we use to produce the different models. And the product owners, they have a single place, a single point of truth in order to understand what is going on with their own products. Also, the auditors can go and check with a single, with a single view what is happening with the artifacts, who has used what, and they can assess the whole let's say interaction with the data easily. Easily.
These environments can be translated to architectural designs. An example architectural design is like the following. At the left-hand side, at the left-hand side, we have a Google Cloud project in which the architects, security teams, and the engineers they used to create the infrastructure, for example, VPCs, networking layers, IAM roles and policies, sometimes based on Terraform, and to create all the other environments that we have at the right-hand side. What are those environments? At the bottom layer, we have the data lake project. If you have multiple data lake projects, this becomes data mesh. Over there, data engineers and data owners, they have access. They create ETL pipelines and make the data available to others. In the middle layer, it is like a computational layer. We have the data scientists to conduct experimentation in a sandbox environment because they might want to try new research papers, solutions, algorithms without really interacting with production data. And then we start having the development, staging, and production projects in which we start productionizing our machine learning solutions and models. In the development environment, data scientists they collaborate with machine learning engineers in order to automate the Python notebooks that they have created into experimentation environment. That means that they create automatic pipelines to perform pre-processing, training, post-processing, and to generate one of the models. Then, the models are stored inside the central place inside the governance project. This place we call it model registry. Model registry is very important because it stores all the details of all the different versions of the models, and the product owner can approve which model will continue to staging and production environment. In staging environment, we conduct different tests like stress tests, integration tests, end-to-end tests, and after we are happy with that, there is another manual approval gate that we have the product owners to approve. Yes, we can move now everything to production, and then the model is, is promoted into production environment. At this stage, I need to say that there are two different ways to productionize machine learning solutions. Either you productionize the models as pre-baked artifacts or you productionize the source code of the pipelines in order to perform the same thing into production in the governance project. The only thing that I need to add is that we store also repositories, CI/CD pipelines, and code artifacts, most probably in addition to the model registry as well.
So, if you step 100 ft backwards, this all is basically four different buckets. The first one being infrastructure, then data, and third one being your environment, and the fourth one is the governance around them. Is that right?
That right?
Exactly. This is right, and these are the fundamental environments that you need to have in place. Anyways, even if you want to productionize applications, it is exactly the same.
Thank you for breaking down this complex but seemingly very important architecture for us, so where do we go from here? Do we switch to GenOps?
Yes, it is time to talk about GenAI and to see the differences and what are the additional components that we need over there. Before jumping into this, what is very important to understand is we have different personas. Most of the times the GenAI applica, the GenAI developers they used to consume models that somebody else has created. That means that they are mostly the consumers of foundation models. On the other side, we have providers and finetuners who used to train models, right, or either from scratch or retraining the models based on different algorithms that we will see another time. The consumers, they need mostly to focus on prompt engineering, context retrieval, and agents. As we will see later on why I introduce you this because the providers and finetuners, they need to follow the MLOps practices as we described already. But for consumers now we have the practice of generative AI and operations, or GenAI ops, in that they need to use in order to productionize their applications. So, if I return back to the landscape that we saw already, we need now to extend it, and we need to add the generative AI application layer in which we have also new personas here. We have prompt engineers or testers who are responsible to focus on creating the different prompts, and we have GenAI engineers that they collaborate with them in order to use effectively and optimally the different foundation models, but also because we talk about application development and most of the times we need a front end, we need a website with a chatbot, for example, we need to have DevOps and app devs to help with the development of this.
Are you ready to see more details about Generative AI in operations?
Yes, absolutely.
I think now we're going to talk a little more on GenAI specific processes and workflows. Let's deep dive into those with the second section. As you know, in the market, we have more than 20,000 models, right? This is this is an outstanding number of models. So, the first thing that a company needs to do is they need to filter the models that they need to use because they need to approve the different terms and conditions. They need to check the EULAs as we used to call them, and they need to select, for example, 20 models. Most of the times they select up to five models that they can use widely to all the different use cases. These models can be selected based on leaderboards. But of course, leaderboards they show just, you know, the high-level picture of the accuracy of a model. For every use case that arrives to the business, we need to conduct a very deep evaluation for the particular use case using our own data because the a good score in leaderboard does not mean that it will work, the model will work correctly also in for our use case. So, we need to conduct a deep evaluation of the top three models that we can find, and then we need to select the best model not only based on the performance in terms of precision, but we need to check other business metrics like latency or cost, and after the combination of precision, cost, latency, and other parameters that we might have, we select the best model for this particular use case that we need to implement. Implement.
Let's dive a little bit further to the different steps. At the beginning, we have a product owner who brings the use case in. Then, the product owner collaborates with AI engineer and prompt engineers. That might be the same person, right? Don't, it does not need to be a different person, but it needs to have two different hats. And after the discussion that they have, they say, okay, we can solve this problem using GenAI. Let's try some of the models that we have. We have. Then, AI engineer brings the models together with prompt engineer who creates the first prompts for evaluation of the models. They select the top three models. Now, these first prompts that we create or the prompt engineers they used to create, we need to store them somewhere, and here we introduce the notion of prompt catalog that we have in the market. Prompt catalog is a place that we store not only the first prompts that the prompt engineer produces for the first tests, but later on this prompt catalog will host thousands or hundreds of prompts in order to test our models properly.
Will this prompt catalog also include versioning of the prompts that we can leverage of?
Yes, for sure. This prompt catalog needs to store the whole lineage of all the prompts because the team needs to know if I change this prompt then does this GenAI application becomes better or worse, for example, if I change this instruction prompt. So, we need to have complete version control. Some customers they used to implement that using Git technologies. Some other customers they combine multiple technologies like cloud storage or BigQuery in order to achieve the same thing.
Now, after we have selected the top three models with a few numbers of prompts, the next step is to start extending our prompt catalog and start creating some hundreds of of prompts. We need to select the right, of course, metrics that we already have done this. I'm not going to stack into details about evaluation because we will cover this in a, in a separate video. But after we have created this golden record with prompts and answers, if we have them, then we pass all this data into the three different models. We generate the results per record, and then we aggregate the results of the performance for all these three different models. At this stage, you can have also prompt testers who are responsible to extend the prompt catalog with more prompts in order to cover more edge cases. But this process is only for precision, right? As I said earlier, we need to check also speed and cost at least. How can we do that? We need to have a reference table with all the models that we, we have. This speed and cost are static data, right, that we can have for all the different models, and combining those with the precision, we can come to the decision of which model is the best for us. I will give you an example. Gemini Pro is one of the most powerful models that we have out there, right? But the Gemini Ultra is also really good and but it is much faster and with a lower cost. So, for some business cases, we might prefer to use Ultra instead of Pro. This is the example that I can give you.
I have a question on the prompt testers. So, if you go back one slide, you mentioned the prompt testers will be able to, um, basically personas who can, uh, extend the prompt catalog. So, is this a space where we can also use GenAI basically to generate synthetic prompts for evaluation?
Oh, very, very good point, actually. Yes, in most of the cases the customers they don't have this golden record as I said at the beginning, right, and they don't have labeled data. How can they do that? Creating synthetic data is something that they used to do, but not straight away, probably. Most of the times first they, they have a human evaluation that they go, label data manually, and then as they have more and more examples, they can start introducing GenAI in order to create synthetic data with the same way. You don't need to use only GenAI for synthetic data; you can use GenAI to evaluate your records if you want. If you heard in the market, we have the LLMs as judges or LLM evaluators. Instead of having human evaluators, we can have LLMs to play the role of the evaluators.
Yes, that makes sense. Thank you, and please carry on.
We have done now the experimentation, right, up to this point; that means that we have selected the right model. What should we create in order to have the application up and running? We need to create a back end. What is this back end? After we have selected the model, done the initial prompt engineering, we have stored all the test results, we need to start building, first of all, guardrails. What are those guardrails? Most of the times we need to filter the input, the inputs to the LLMs, the output to the LLMs. For example, if, let's say, have a customer that logs into our chatbot and starts asking irrelevant questions to our business, then we need immediately not to call the LLM but to return back to the customer and to say we cannot answer this, this question, or if the LLM tries to expose him information that it is not necessary or tries to answer with a very toxic, toxic way, we need to filter out the output. Output. Additionally, we can implement caching mechanisms. What are the caching mechanisms? Most of the times we don't need even to call the LLM. We can store the most frequent answers into a different table and to return those back to the users. This is very, very useful especially whenever you want to create natural language to SQL queries, for example, you can store and you can make the solution more precise. The next part that we need to implement is about automatic context retrieval. What is that? In evaluation, we have a static data set, right? We have an input, we have an output, but in real use case, whenever we talk to a chatbot, most of the times the chatbot needs to get details about the customer that we are talking to. It needs to get details about the history of operations and interactions with the customer. Here, we need to implement RAG; we need to implement agents and tools in order to get this context at real time. Now, the next component that we need to build is the rating mechanism. Okay, we have the end users, they interact with our system, but somehow we need to exploit this interaction in order to evaluate more cases for our use case. That means somehow we need to have a translation, if I may say, whenever a user says, oh, I like this interaction with a chatbot. How can I translate that to my tests and to my prompt catalog prompts in order to test further my models? And the last piece of the back end is we need to have monitoring. Monitoring is very important. We need to check continuously about toxicity, hallucination. We need to integrate grounding mechanisms. That means to make sure that whatever we answer is based on whatever we retrieve from from our context. And in case that we have too many bad answers or toxic answers or hallucinated answers, we need to start again the process of evaluation. And of course, this is the back end. Nobody sees the back end. Most of the times the end users they interact with the front end. The front end is a website, is somewhere that they can log in, interact with a chatbot, interact with a, I don't know, a different type of environment. It depends on the use case. So, more or less, then users they see a nice beautiful web page that they hide the access to the back end with a very nice and smooth way. And of course, we can use the interaction with end users to continuously improve our test sets, and every rating that they will do through the front end, we need to translate and extend our test sets.
But do you want to see now how the back end and the front end can be incorporated to an architecture design and how we can productionize that with a very robust way?
Yes, I'm eager to see how this builds upon the architecture that we previously discussed.
Perfect. And I have the answer for you. So, this looks quite similar to the MLOps architecture design that I showed you earlier. But now in the middle, instead of having the MLOps dev, staging, pro projects, now we have the GenAI application.
Development project, then the GenAI application staging project, and the production one. In the development project, we have the prompt engineers and JI developers to conduct their experimentation, as we saw earlier, right? To check and to get the right, uh, models and to select the right models that we have available. By the way, all the prompts that, uh, they create, we need to store them somewhere centrally. And where should we store them? At the top layer here, I have the data lake project.
In the data lake project, we store the prompt catalog that later on we can use in the staging environment for automatic evaluation and even in production to get the feedback from the end users and to improve and increase the prompt catalog with new prompts. The next step after the experimentation and the selection of the model is to create the back end. As we discussed earlier, this back end needs to have access to the data lake. In the data lake, we can have a RAG implementation. That means that we translate the different data to vectors. We can cover this in a different, uh, video. We're not going to go deeper into RAG, or we can have a connection with agents and different data tools, for example, APIs that they get access to different databases like BigQuery. With that way, we implement the real-time context retrieval. And we need also to implement guardrails; guardrails in GCP. The easiest and best way is to use Model Armor that helps you to implement all the guardrails that we mentioned earlier, and at the same time, we can have a front end that we can deploy. Cloud Run is one of the most common ways to deploy the front end.
Now, after we have implemented the back end and the front end and we have tested that manually into the dev environment, then through the CI/CD pipeline, we promote everything into the staging environment. In the staging environment, we can run automatic tests as we had earlier in development. But this time, you can have the prompt testers to interact with, play with a playground environment with the solution and try to add more tests to the GenAI application. In that case, whatever prompts they create, they used to store into the prompt catalog that we have into the data lake. After we have passed both the tests of the GenAI testers, the prompt testers, and the automatic tests, then we promote everything into production, and then users they interact with the front end.
At the bottom layer, I have also the governance environment that we talked about earlier. In the governance environment, again, we store the repositories, the CI/CD pipelines, but we store also all the metadata of the evaluations of the tests that we have run, and it is the single place that both product owners and auditors can log in and check the performance of the models and the artifacts that were used. There's a significantly more number of boxes than what we previously had. Just kidding. But it's great to see the different processes involved, the personas, the back end, front end, and everything mapped into one single architecture.
Now, now we know that this is what you actually need to productionize your GenAI ops deployment. Um, and I really like how you spoke about the guardrails and Model Armor. Basically, it is a way to, let's say, input; if you pass input prompts, it is a way to actually prevent prompt injection and check for prompt injection and in the outputs. So, the LLM generates it; basically, it desensitizes your data for PII and stuff. So, yeah, I think that's super crucial, like you pointed out.
Exactly. And because you mentioned that we have so many different boxes and it looks a little bit complex, that's why what we have done for you is we have created this end-to-end starter pack that you can get our source code that is based on Terraform and other implementations and modules, and you can start using it in order to see how you can simulate such an environment and productionize your solutions with a very robust way.
Absolutely. Thanks for the call out. There's a separate video on the agent starter pack where you can quickly grab these templates and be up and running with your GenAI ops production workloads.
Exactly. Now we have set the foundations of GenAI and GenAI operations. Are you ready to jump into agents and to talk about agent ops and how to productionize them?
Absolutely. I've been waiting for it. Let's do it.
Perfect. Section three, agent ops and operations. Let's talk about this. We will start with the very basics because this is what is missing from most of the people; right, they talk about agents here, agents there, but actually they don't know what are the agents. If I can give a very simple definition, informal definition for everyone to understand, an agent is nothing else than a prompt that instructs a model how to call different tools. That means that now we don't have just a model that we interact and we get answers based on the generic knowledge that it has. But now we can start interacting with code and specifically, and specifically, in the first example that I've just mentioned, we have a model that we interact; we get answers. In the second example with the tools, what we have is we go to the model and we say, "Oh, what is the weather in London this time," and what is happening is the model needs to understand that it needs to call a tool with a parameter of London and to get the answer of the weather and to reply back to us, or we can have a different tool to send just, let's say, the Google ticker acronym in order to get back the stock price. So we can have different tools that we will see and we will focus in the next few slides about this. About this. But let's see how we can build an agent to make that simpler for everyone to understand.
The first thing that we need to do is we need to define the different tools. To define the different tools, we need to define different functions. What are those functions? A function is a declaration of a function. Get stock price, for example. This is a function responsible to return the stock price. Then we have the definition of the parameters of the type of the function. For example, the parameter is ticker, and this parameter has the acronym of the specific company. So what we have created here is just a function declaration. It's just pure text that we will pass into the model, and the model can understand when this function can be used. Whenever we create multiple of these functions, then we wrap them in a list, and this list of functions is a tool. Tool. So the next step is to have now the tool that we have created combined with a selected model. For example, here we are using Gemini 2.0/00001. And say the tools that you have is just this one function that we just described. The instructions that I give you is that you will answer about the stock prices and you will call the API, and this is now the core of an agent. The core of an agent is a combination of the available tools to the model, the instruction how to use the tools, and the foundation model itself. But now, by having the core, how does it work, right? How do we execute this agent? If I may say this core of the agent, we have the end users to give just an input query. This input query might be, "Okay, what is the current stock price of Google," for example. This query goes to the core, and then this core is responsible to understand if I need to call a function and with what parameters. In this question, in this example, we see that the model and the call will reply, "Oh, you need to go to call the get stock price with parameter ticker and string value. Okay, I need to call it." But in the place that we have implemented our agent, this runtime, we need to call the actual function. We go, we call the actual function, and then this responds back with the answer. As we can see here from this API, we get all the details of a stock price. We fit this output back now, step four, into the core, and now the core has enough information to answer the question of the stock price. So the core creates now the final response that is this phrase here: "Google's Google stock price is currently this amount reflecting to XYZ." This final response is what the core reconstructed. So as you can see, it's a flow of multiple steps. The core understands what is the function that we need to call. The runtime calls the function. We pass the results back, and then we have the final response. Does that make sense?
Yeah. And I'm assuming that we don't have to do all of these steps within the agent, right? The agent will implicitly take care of the function call, the parsing parameters, and everything. And everything.
Exactly. This is the magic that the agent needs to be responsible for in order to call the model, identify the functions, call the functions, return the results back, and return the final response, as you said. So we can imagine that this box at the bottom is just a container that simulates how an agent works.
Yeah, fair enough. There's just the inner workings or mechanics of the agent, but we don't have to necessarily implement all of these steps. These will be abstracted away from the developer.
Exactly. And later on, we will see also the agent frameworks that we can use in order to make our lives even more simpler. But now we talk about ops, and one of the most important operational processes is evaluation, as we saw earlier in the GenAI world. This time we have some extensions here. What are those extensions? We don't have just the input prompt and the final response, but we need to have data about what is the right tool to call for this response, what is the right parameter of this tool, and what is an example response. By having this data, now we can run more evaluation tasks around agents, but before I jump into the evaluation tasks, it is very important as a prerequisite to say that during the tool development, the developers who are responsible to create these tools, they need to create code unit tests; that means that they need to test that their tools they work. Now, by having this a prerequisite, the next step is to conduct the evaluation based on the data that I mentioned earlier. This evaluation, the first evaluation that we run is about tool selection. So we need to check the success rate of the tool selection. We need to check the success rate in terms of the parameters creation for the function calls. We need to check if the model needs to call a tool or not. And what is the success of this? Here, this is an example of metrics that you can use. In a separate video, we will talk in more details about evaluation and tool selection. Another evaluation that we need to run, of course, is whatever we run for GenAI applications earlier. That means that we need, based on the input query, do I get the right answer? Is the answer grounded? Is that based on the context that we received? Yes or no. So we return now back to the foundation. And last but not least, we need also to evaluate operational metrics, right? How fast the agent operates? What is the cost? What is the latency? And after we have all these five steps ready, now we have a proper evaluation mechanism for everything.
If we go back a bit in the previous section, in the GenAI ops, you spoke about the prompt catalog as a way to store prompts, extend and version it. But here, like you're explaining, we have different parameters in addition to the prompt. We have the tool, we have the tool response, and the parameters. So how will this work into a prompt catalog, or do we need some other special kind of capability to store all of this?
Yes, if I may say this is just an extended version of the prompt catalog, introducing also the notion of tools, tool calling, and parameters. If we do that, then we can evaluate the agents easily. So we can repurpose the existing tool prompt catalog to hold all of these. Of these.
Exactly. Exactly. Thank you. But we saw about evaluation. How can we optimize now the performance of an agent? One very important part that I want to talk here is about the tool design and the function declarations that I mentioned earlier. So the first thing that we need to have is a proper description of the functions, proper description of the parameters. If we don't have this, then we don't have an optimal agent. Second, we need to select the tools with the right way or the functions in order not to have overlap or not to be contradicting or not to do the same task. Why? Because then the model will confuse which model it needs to use. So we need to have tools and functions that are responsible to do a very specific task, and it is completely different than the other functions as well. Do you think if I give the agent 100 tools, would it be able to work efficiently? Like what's the limit? Let me answer with a question here. If you give to a human agent 100 tools and you say, "Now go and build a car," what will happen?
Chaos.
Exactly. So we need to think of agents like microservices that they have access only to specific tools that they will help them in order to perform specific tasks. It is exactly the same case. Now, how many tools you can incorporate, you need to run evaluation to answer this question because it is per use case. Case. Case.
Sounds good. But one of the most important topics is about tools, right? This is what I told you already. So we have different types of tools that we need to incorporate. We might have code tools like simple Python or other language functions. We might have APIs in our private cloud, in our VPC, or we might have public APIs that everybody can use. Can use. Behind the scenes across all these different tools, we might have different data sources, different computational elements that we need to link. Now, for a business and for a company in order to start using these tools and not recreating all these tools again and again, we need to have something that we call it a tool registry. What is a tool registry? It's a centralized catalog that we store all the tools that we have created in order to make them available to the different agents that we want to build. So in a tool registry, what we need to store is metadata about not only what is the declaration of this API, but how we can call it, who is the owner, what is the performance of this tool, what is the version of this tool, and more or less more agent-specific information over there. Tool registry in Google Cloud, we have Vertex AI that it has API Hub, and we can use it together with a metadata store like Firestore in order to implement this combination of the two. So here, with the help of a tool registry, are you saying we're defining an interface on how we communicate with the tools instead of saying, "I'm going to write three different Python functions which are going to call three different APIs, and I'm going to have a separate authentication for each of them, a separate way to call each of them." We're going to put all of these three tools into something called a tool registry, and we're going to define our terms of communication with the tool registry. Is that right?
Yes. Yes. Exactly. And then the tool registry is responsible to handle all the authentication, authorization aspects. So you don't need to think about this. You have a central place to configure all this for your users. Now, one more thing I want to talk about is where the tools are located, right? We might have data tools. Data tools are like tools that they call a database or BigQuery database or another type of database. That means that we have these tools located into the data lake. You might have tools like API tools. Those might be located into a production environment for an application, or might be another agent as a tool that might be located into the GenAI production environment, or you might have just simple code tools that might be located into Code Artifact somewhere into the governance environment. So what I want to say is the tool registry helps us a lot to have a central place to store all the details for all the different tools across all these different environments. The key question is how we can achieve that, right? That right. So one solution that we can suggest is we can translate all the different types of tools into APIs. For example, you might have code tools that are located into a repository, and whenever I push a new code into the repository, I can translate immediately these tools into containers that we can deploy into Cloud Run and to expose them through API endpoints. Or let's say if I create a stored procedure in a database, I can wrap it around an API and store it also into the tool registry, as we mentioned earlier. Now, the key consideration is how we can fully automate that. We need to incorporate CI/CD pipelines, and for the data tools, we need to make sure that both the authentication of the APIs and the data governance is aligned and in place because we don't want users to access data that they don't need to have access to.
Can you talk a bit more about the code tools? I understand the data tools are something, the tools that access databases and fetch data and information, but what are code tools?
Code tools might be computational tools, right? That I want to create a calculator, and I have created the addition, division, multiplication, etc. So if I move then these tools and to make them available to everyone, nobody needs to recreate those tools. They are available. Of course, this is just a dummy example. Imagine something more complex, right? That we might need to calculate, to calculate, I don't know, to call a machine learning model and to do forecasting, for example, right? In that case, it is really critical to have reusability in place. So the question is, is it fair to say these are Python functions or maybe any language like Java functions or Node functions that basically does something deterministic and you want to wrap that around and then provide that as a functionality to the agent?
Exactly. Yes, you're right. And the key question is how we can enable our code to translate those tools into APIs. And the first thing that we need to do is to standardize our repository structure. This is one of the fundamental parts in all the ops environments, right? To standardize the repository structure. Here I have an example from one of our customers. You don't need to follow exactly this, but I wanted to show you how we standardized that in the past. So here we have a folder, tools, and in this folder, we have API tools, code tools. It might be just a single folder with all the tools, but each tool has its own subfolder. This subfolder has a test folder for all the unit tests for this particular tool. It has the Python implementation, for example, here, like the main function, the location of the main function. You might have requirements.txt if you want to translate that to a container easily or an image configuration if you want to fully automate the creation of containers. So with that way, we have a module that can be translated from code into an API tool. By having now the tools ready, what we need to construct is an agent. That means that we need to select a model, create a prompt, and link it with the tools. So we have a different folder in which each agent, because we might have multiple, they have a separate folder. Within this folder, we have a folder for evaluation. This is the process that I described earlier. You need to have the Python code to run this evaluation. You can have the configuration.json just to say where is the location of the evaluation data, of the prompt catalog, for example. You need to have a monitoring implementation. Monitoring is a real-time evaluation. If I may say, then you have the implementation of the agent itself that creates this core and creates all the function calls and everything that I mentioned earlier. You have the instruction prompts. This is where the prompt engineer used to focus and to change it in order to improve the performance. And you might have some configuration for the agent that might be necessary for more complex environments. You might have a deployment folder with all the Terraform code to deploy this agent or Python code, depends on the solution that you have. And lastly, you need to have tests, tests in terms of the agent, and integration tests. These tests are different than
The tool test. Of course, it is for the complete interaction of the agent. And this particular uh directory structure is an example, right? I mean if if I have an agent that does only one thing, like let's say for example a weather agent, and I have only one tool call, I don't necessarily have to follow this, but it's I think based on the complexity of your agent and best practices of how to separate different tools, kinds of tools that you have. Yes, you're right; in that case, you have a more simpler example, right? I have seen also repositories that they have just four files: the one is the the toolp that has all the tools in one single file as you head. The second one is the instruction prompt uh that uh as we described earlier. The next one is the agent implementation, and the last one might be the the context retrieval if it is necessary. So, with four files, you can do everything. It depends on your scale. It depends on the size of the agency. It depends on uh how complex is the solution that you want to build.
But now, by having standardized the repository, we have everyone to know how to develop new solutions. We can create templates even, but the most important thing is we can have a CI/CD pipeline to productionize everything with a very streamlined way. That means that uh let's imagine that we have someone that creates a new tool. We need to create a feature branch, create the new tool, push to the main branch. This will activate the CI/CD pipeline that does validation of the repository, the structure. It runs the different tests that I mentioned earlier. It deploy it deploys the different custom containers. It builds in the different custom containers. It deploys the agent, and then we have prompt engineers and AI engineers to start manually testing the agent. After they are happy with that, we have a manual approval gate here that we say yes, now we can move that to the staging environment. Then the CI/CD pipeline continues, deploys everything into the staging environment. In terms of the agent, we run now automatically the different evaluations and tests. Here we can have also prompt testers, as we said earlier, or agent testers, if I may say, and after we are happy with that, another modular approval gate. Yes, this agent works well. Let's move it to production. Production. Now, across all the CI/CD pipeline, we need to store metadata. Metadata in terms of the performance, metadata in terms of versions, metadata in terms of where the code is, what data we used, etc. And we need also to register our tools into a tool register in order to make them available to the whole business and to avoid recreation of those. Okay, but I think it is enough now with the theory. Let's see a little bit of architecture design. Correct? Yes, let's do it.
So earlier we saw this architecture design from GenAI Ops, right? How can we extend that in order to accommodate the agents? What we need to have is uh, first of all, in the back end, we need to implement the agent part to talk with uh the different tools that we might have available in different places. But also, we can have agents and tools separately as services, right? So we have two options. Either we integrate the agent into the back end or we have an agent as a service. Then we we promote these agents and tools as a service and the back end, if it is necessary, to staging and prod with the way that we set, and along this journey we store in the tool registry that is located into the governance account all the details about the tools that we are using. These tools might be located into the production environment of uh GenAI app or it might be into the data lake. As we can see here, we can have data tools that they are accessing, for example, BigQuery or other databases, and we need to extend the repositories into the governance account in order to accommodate the agents and tools as a service uh over there, and over there we have also the artifact registry as well.
So all of this what we spoke about right now fits well for a single-turn agent. But what about uh, you know, memory for long-term persistence or multi-turn conversations? How do we do that? Very good question. Up to this point, we have seen just a single-turn interaction with an agent. That means the the core of the agent decides one function, gets the results, and gives us the final response. This is not how it works in the reality. In the reality, we can have multiple loops until we get the the the right information to generate one answer. And this picture here depicts exactly this. So the core gets a a new query. Then it starts checking, okay, I need to call a function with these parameters, get uh back the results, and then it says, oh, I need to call now a different function in order to get more parame more context in order to answer the solution, to answer the final response. So uh in this interaction, we have intermediate responses, and we land into a final response only if the agent has enough information to answer the the question or if we have reached a maximum number of iterations. Why is that? Because uh if we sometimes I don't know leave the agent without any control, this might land to an infinity loop or to let's say 30 different steps that might uh, you know, destroy our whole application application along these lines, and along this interaction, one of the most important, as you said, is the memory. Memory keeps track about all the different events that happen within one agent. This helps us a lot because then the agent does not need to ask again and again the same questions in order to get more context, but everything is stored there and can be reused later on and for evaluation and for testing, for experimentation, this is the perfect component that we can use for that, and as I said, evaluation right uh this is one of the key components in general, how we can evaluate a multi-turn interaction. First of all, we need to conduct whatever we saw in the single turn. That means that uh does a single turn work correctly? Yes or no. Then we need to check if the sequence of the different function calls is correct. So now we don't have just one function call, but we have multiple function calls. Are those in the right order? Are are those enough in order to generate for us uh the answer? Have we reached, for example, the maximum number of iterations until we get the answer? The next step is we need, of course, to do the evaluation as we know with the input query and the final output, as well as we saw in the GenAI foundations, right, GenAI Ops foundations, and the last evaluation is we need to check the topic. What I mean is sometimes the agents might deviate from the topic that we used to talk about. So that means that we need to check, okay, do we have a a persistence on the topic that the agent used to talk about? Do they deviate? Do we have relevance on on the question that it asks? And this is the last big evaluation topic. As I said also earlier, this is just a small list of metrics that we can use. We will have a different uh evaluation, a different video for evaluation in order to deep dive more on those.
Absolutely. I think this is a very important point that you made. um for multi-turn evaluations, for for multi-turn agents, uh evaluation might not work as it normally would, like, for example, let's take the case of a customer support ticket agent, and let's say a user is prompting, "Hey, I can't log into my account," and then the the agent responds with, "Okay, I'm escalating it to the human uh who can help you with," so the next interaction from that specific user who is asking about, let's say, "Hey, what's the status of my order," might not work as expected. The tools might not be called because there's an escalation in progress. So in these kind of tricky scenarios, your evaluation strategy changes drastically based on the memory, based on the context, and everything. Everything. Yes, you are completely right, and this becomes even trickier if you want to have evaluation of multiple interactions across the time. So what I mean is you might have an end user that comes one day and talks through the chatbot with the customer service, and after three days it comes again and starts talking again about the similar issue or a different issue. How we can evaluate that? How we can exploit that? In that case, we introduce the notion of the long-term memory. Long-term memory is for every completed interaction or every topic that we have completed with our users. We need to store this into this interaction from the short-term memory to the long-term memory that is persistent and most of the times is located into the data lake with that way. If the customer returns back after uh 3 weeks, then we can retrieve the data from there in order to start answering questions without uh uh you know having to ask exactly the same questions and again, or if we have a very long process of uh human escalation and approval. Instead of uh you know, stucking in a specific topic and and not being able to continue with the agent interaction, we can use also the long-term memory to achieve the same thing. Thing.
Now, one difference between short-term memory and long-term memory: Short-term might be just for one day that we keep it. Long-term is more persistent. Uh we need to make sure that we are quite careful with the data that we store. In both cases, it is good to store everything in long-term memory into the data lake because you can have data governance, and if we have PII data, for example, we can handle who has access to where. Otherwise, we need to start removing the PII data before we offload everything into the long-term memory, or we need to start doing other activities like we need to summarize the short-term memory and then to store that into the long-term memory. Different approaches, this depends on the needs of the business business, and to implement those, most of the times short-term memory is located close to the agent. It is based on Cloud Trace or Cloud Logging. Then, to have more persistent storage for long-term memory, we might use BigQuery, Firestore, Cloud Storage, or might start using graphs. Graphs, why? Because later on we'll see with multi-agents. Graphs are very important to also link the relationship across the different interaction of multiple agents. Do you want to see how we extend the architecture design in order to accommodate uh the memory component? Yes. And I think this is the final architecture for this session. So let's sum everything up and put the multi-turn agents in. Perfect.
So for the memory components, we need to store the short-term memory into the production environment close to the agent. As I said, for the long-term memory, we store it into, most of the times, a data lake, and also what I have seen with most of the customers, they used to uh ingest the data of long-term memory into the RAG system in order, for example, if a customer comes after 3 weeks, right, and they want to talk about, "Oh, what's happening with my mortgage that I talked three weeks earlier," right, you don't need to get the whole interaction, the whole history uh and the whole long-term memory, you can take only the memory that is related to the mortgage. So that's why the combination of long-term memory with RAG is very very important. I think it's a totally different um topic to talk about what what you're saying; ideally, it's a sharding mechanism but for memory. So that's a whole new domain that we can talk about for days. For days. Yes. Generally, how we orchestrate the memory is a huge topic and quite innovative that currently we have customers implementing those. But what is next? Next is you said already, right, single agents is what we talked about, but most of the times we now in the market we have the notion of multi-agents. What are the multi-agents? So we need a little bit to parallelize multiple agents like uh microservices as I mentioned already. That means that uh we will have the small agents that are responsible for specific tasks. That means that uh somehow we need to orchestrate all these different agents in order to complete a a use case or a longer a bigger task. So this is where uh multi-agents come into play. And to give you an example, you might have, let's say, an agent responsible for the bookings, an agent responsible for checking the account, an agent responsible for um checking issues uh with with the bookings. And somehow when the user comes to the chatbot might say, "Oh, I want to book a new trip to Greece, for example." In that case, we need to redirect the user to the right agent. But how we can do that? We have different ways to route among the different agents. We might have a router agent as I explained already, and you can send the user to the right sub-agent in order to interact with. We can have an agent that calls in parallel multiple agents. We might have a sequence of agents. For example, I implemented an auditing workflow, right? That I want first to talk with this agent, then with that agent, etc., etc. We might have a circular flow, or we have we might have a more innovative but scary sometimes dynamic flow that every agent can interact with all the other agents in the system. As you can imagine, this increases the complexity. All that we learned today about evaluation, optimization optimization, and everything works here, but we need now to perform everything at scale. And if we start thinking agents as tools, then this will give you a better idea how you can adapt the valuation to this multi-agent scenario.
The last piece that I want to show you about this is multi-agent systems; they work very well in big enterprises in which we have multiple teams that they implement these microservice agents to the different environments. All these agents they use different tools that we might have a central tool uh uh tool registry as I mentioned earlier to use them across, but also another component that we used to have is an agent template catalog or an agent catalog. An agent catalog is responsible to catalog all the agents that are available in our business. An agent template catalog is a place that we can store the code for different templates of agents that we have in order to boost the development. So these are now new components that you can have in this complex environment to to help you build uh faster all the agents that you have. One last thing that I want to talk before we we sum up and we close, as we started, is uh uh as you can imagine, to implement an agent if you use all the foundations, function calling, using uh different foundation models, it is not an easy job, right, or it can be simplified how by using frameworks. So the agent frameworks is nothing else than an easy way to develop uh agents, and they integrate models, tools, memory, and other components in a single orchestration that uh you know we don't need to create everything from scratch all the time, but we need to focus only on the important part that is the prompt engineering, for example, it is the tool development, and uh the the design of the agent that we need to. So this is the final architecture design for the Agent Ops. Uh this is the the landscape of uh the different teams. One last question that I want to address is how is that link with MLOps that we saw at the beginning, right, and I want to scare you a little bit. Is that okay? You could try.
So here I have a an architecture design, a giant that integrates all the different components. At the bottom, we can see the GenAI application development that we can have the agents, then we have the data lake, then if we want to fine-tune or train our own foundation model, we need to have MLOps, so here is the MLOps layer, and then at the top I have a central governance environment, AI governance environment, and at the left we have the platform team, so at This picture we see application layer, data envelopes, AI governance, and the landing zone from the platform team that uh gives us the picture that I started with the presentation, right, so this is the complete uh set of environments that we need in an enterprise level, and uh the last piece is because we have now all these different foundation models available to us, this layer of MLOps can be abstracted if we use, for example, Gemini, if we use other models from other vendors that they already have done this process of MLOps for us. So, finally, after all this journey, we we arrive to the final section that is the summary and the resources to do a quick recap of what we discussed in this video. We started off with DevOps and MLOps practices, and from there we evolved into GenAI Ops, which is basically where we introduced model evaluation, prompt catalog, and other processes around how you can do your GenAI Ops better, and from there we evolved into Agent Ops, which is a whole new domain of how do you build agents? What about evaluation, memory for agents, and how do you store agents? What about agent evaluation, like all of these concepts around agent um agent ops? So why don't you give us the resources of how do we get started with this, Socrates? Sure, because I know that uh most of the people that they are following us, they need to they want to read more rather than watching videos. I have prepared a set of uh blog posts for them that uh you're able to to get uh the complete material from there, the diagrams and everything in a text format that you can read more details and uh with your own pace, but also we have uh the last uh uh link that is the end-to-end starter pack, and actually this lands to our call for action. What I want from you is to go check the starter pack, start using that, and start your journey to productionize your own agent. With that, thank you very much from from my side, and thank you Sitha. Yeah, thank you very much, Socrates, for this master session, and for all of you watching, thank you very much for joining us today and leave us comments about what you'd like us to talk about next. Stay tuned for more videos in the series.