Transcription
Great. I think we can get started. Uh, we have 30 people now joined. These are real ones. So we can get started anytime. Uh, that's what I expect. This is a smaller one because so that we can ask questions. So let's get started. All yours Maria Mera and Pamela. I will come in and out. But let's get started.
Thanks Mahesh. Welcome everybody. Thrilled to be here with both Mahesh and Pamela. Thanks so much for joining us um learning more about this Microsoft uh build session on Agentic Rag that Pamela gave. I am Mera Emerling. I'm the CPO at Glaze. I have built and scaled AI and SAS platforms across 90 plus countries serving public sector enterprise and commerce. I'm really focused on frontier technologies and turning those into trusted modular systems um delivering real world outcomes. So, Pamela, we're so excited to have you here. Uh, would you like to start off with a quick intro?
Uh, yeah. Hello everyone. My name is Pamela Fox. I'm at Microsoft where I'm a Python cloud advocate. So, that means my job is to help Python developers be successful with Azure and Microsoft technologies. Of course, for the last few years that has meant everything AI. So, I spent a lot of my time working with Azure OpenAI. I have really focused quite a bit on Rag. uh, which I know you all you you all have tried out uh, you know, the way of getting questions asking questions and getting answers that are grounded on data. I really like that use of LLMs because it's that we're using LMS to get accurate answers instead of getting you know made up answers.
Great. Thank you Mahesh. Uh, yeah, I think most of you know me. I worked at Microsoft and I love build. I keep watching build videos and I thought uh it would be great that Pamela can come. So I'm the one who is responsible for all of us you to go through rag again but uh I also teach a course on maven and most of the people who are in the call are through the course in the course we talk about rag a lot then we talk about fine-tuning then we talk about how can you scale with MCB and all so maybe we should get Pamela back for MCB but that's my little introduction before this course I'm also starting my own company and before this I worked at Microsoft Meta and Google AWS So pretty much every large company but I'm I'm really excited about learning what's new in Azure. So I will leave it to Pamela and Mera. I will keep coming in and out but all yours.
Sounds good. Thanks Mahesh. So last week we talked a lot about rag retrieval augmented generation which allows us to pass more context or really the relevant context right to large language models through prompts. Today I'm really hoping that this session is going to clarify what is a gentic rag? Why do we need it? So let me start by opening up with the question, what is a gentic rag and why do we need it? Pamela, maybe you can unpack that for us.
Yeah, that's a it's a really good question, especially the way you worded it. So when you say agentic rag, you know, it actually has many different definitions. Generally, when we just hear the word agentic, what does that mean? Well, that usually means that we're equipping the LLM with tools, with things that it can do besides just, you know, text prediction, right? So, an LLM like on its own, an LM all it can do is, you know, really generate text or, you know, suggest suggest uh tools to call, but it can't really do anything itself. So, you know, when we make something an agent, we give it tools. We say, "Hey, here's some tools you can use, right? Um, and with rag, one of those tools is searching for data." That's like the very like, you know, basic rag. And in some ways, all forms of rag are agentic because we're equipping an LLM with a tool where it can search stuff, right? Um, but a lot of times when people really talk about agentic rag, they mean uh using more tools, more steps, and more like reflection and iteration in order to improve the result and get to a better answer.
We just do the most basic rag uh like the one I did here um in this application, right? I've taken the user's question and uh we can actually see the the process here, right? So here's the user question. Yeah, the most basic one would be we've got the user question. We use it to search the search box and it's the search engine and then we get back the answer. Right now, if we're going to make that agentic, there's there's many points at which we can uh you know empower it with more tools and more ability to improve the response. Uh so what we talked about at build is specifically agentic retrieval. So the idea that okay if we get a question how can we then um do a better retrieval based off that question right so maybe we should like break that question into uh you know into multiple queries and and send those off right there's other forms of a dentic rag that we didn't talk about at build but I have talked about in other sessions which is doing reflection and I think you're going to actually do maybe some reflection in one of your uh in the the class assignments and reflection would be like okay after we get back the response. Let's look at the response. Let's reflect on that. See if things improved. If not, maybe we should go to a different query. Maybe we should go somewhere else to find an answer. Maybe we should just rewrite our answer. Right? And so the idea is like how can we, you know, add more steps to our process, you know, generally using LLM for those steps in order to improve our retrieval like be able to pull from more places and in order just to improve our answer, which you know, generally means some sort of reflection. So I think that those are the the two biggest kinds of agentic rag that I've seen is doing a gentic retrieval and then doing a gentic reflection here.
Okay, so this is like the very standard one, right? We get the user question, we get back search results and then we send that directly to the LM and say, "Hey, get back the response, right?" And then we get back the response. This is like the most basic rag um that we can do. And then what we talked about at build was adding uh aentic retrieval on top of that. Uh, so we go here let's see aentic retrieval. Okay. So now I'll ask a question here with aentic retrieval. So this is when we talk about aentic retrieval we're saying like all right we get back we get a query. We get also the previous conversation. We send that to our query planner and the query planner is a model uh and it's it's an LLM right? So the LM says okay based on this query and this conversation history these are the possible search queries that I suggest. So it can it can do multiple queries and it can execute those queries in parallel and get back the results merge those results in a clever way and then those results we get back. So the the idea here of gentic retrieval is that we can potentially answer questions that are harder, right? like what we might call like a multihop question um that could go across um and you know like bring together kind of disparit topics into a single uh result. And here you can actually see when we um when we do this in the application with a gentic on we see you know the question which is asking about some health benefits. We see the subqueries. We see how many results it got from each subquery and how long they took and they did get done in parallel. So even though this looks like a long time they are all done in parallel. So that's nice. And then those all get merged together using a merging algorithm. And then we pick the top and then we send the top ones to the LM in order to get back the result. Right? Um, so this is just going to you know improve our answers for hard questions that uh you know go across multiple topics.
Here we go. Okay. All right. So now you can this has the nice little I had fun making the diagrams for this one. Uh, so yeah, so we you know, we add in the step right? So we get the answer, we evaluate the answer, we see what the score is. So this is using LM for evaluation. If the score is low, then we're going to use an LM to reflect and decide like, okay, should we do a search? Should we do a new answer? Do we do a web search because we actually need results from the web and then display the answer. Now, this answer adds a huge amount of overhead because here's another LM, here's an LLM, here's a search step, right? That really increases your latency by so much. And what I found was that it wasn't really improving things. Uh, now there's definitely some questions where it could improve things, but given the amount of overhead here and the fact that most of the time it wasn't improving things, we didn't merge this change. And that and that's something you'll that's something that's hard about rag is that you'll find these questions that are hard and you can think of ways that you could improve the rag answer quality for those questions. But do you want to use those techniques all the time? Like, is it worth the extra latency, the extra cost? Is it going to generally benefit things all the time? Right? Um, and we did when we implemented this in the actual repo, we did manage to like turn this all into a single LLM call to try and do as much reflection as in possible at once, but it still um just wasn't good enough. So, we did not we did not bring reflection into our repo. I do all this stuff in the in the open so that people who want to can try it out because it maybe it didn't work well for my evaluations but um, you know, maybe for somebody else's use case, you know, then they might find that it works well right? So if somebody wants to see how how we did it, you know, they can go and they look in code, you can kind of see the prompt right? So the prompt's kind of interesting. You can um, you know, see uh here's a prompt uh this is a prompty file which is a way of defining your prompts right? And we said like okay, you need to categorize the relevance of the response, the groundness response and the correctness and then he these are all like the possible tasks you can do. So we were trying to do as much as much as we could fit into the single reflection step um to try and figure out what um, you know, what would be the next step and what what you add in your evaluations.
I think this is a one of the interesting question Mira had but what you how you evaluate rag or the quality of all the enhancement you do on top of rag. Yeah, that's a great question. Um, I've got my uh evaluations here that I can I can show you. Um, so generally the approach to evaluation is that you know, first you're going to set up ideally you set up some ground truth. Ground truth can be really hard to set up because you want to have like a lot of qu like at less and their answers and either you going to use a human to write what is the ideal answer or you have to use you know a a program in order to come up with those answers and it's uh it's just tricky to do that. So, uh, you know, I have a script that attempts to generate some ground truth for a data set. Um, but it's definitely not perfect and it doesn't reflect what necessarily what users would really, uh, want. So, ideally, when you're doing evaluation, you start with some version of ground truth. Uh, and then as you are using your ra application in production with actual users, you want to be updating what it is you're actually evaluating to make sure it reflects what users are going to ask, right? Because users are your best, they're the best. Yeah. They're the best indication of what your system's going to get asked, right? So, um, yeah, so once you have users, you're you're going to want to improve your ground truth. But, uh, that's the first step is I do the ground truth and then I I evaluate all the questions, right? Because one of the things that people will sometimes do is that they'll like go in and they'll like just try their like favorite sample questions, right? Like, you know, we've got like these three example questions here and you might tweak the prompt and then try them be like, "Oh my god, it's better." Right? Now the thing is it might be better for these three questions but it's not necessarily better across the board and I've constantly had this experience where somebody will be like oh I found this improvement and then I go and I run a bulk evaluation and it is not an improvement right? So it's super important that you are always doing bulk evaluations and not getting biased by your like favorite sample questions right and it's also important that when you're doing evaluations you do want to set the seed this is a technical note but you want to have as much rebuildability as possible and the seed is a way of having slightly more reproducible evaluations. Still not perfect, but uh, yeah, that's one of the things that is really hard about evaluation is that how much it can change based off of if you change anything about your application, you know, it it throws your whole evaluations off. Um, so these are the evaluations that I ran the night before the build session. Uh, so generally what I did, I just set up um the application with different parameters. Like you can click on here and actually see the parameters I use. So this was like okay I'm going to use a gentic retrieval. I'm going to set the seed of one. Um, I'm going to set the max sub queries which is how many queries it could plan to 10. I was using temperature of three and a top of uh temperature of.3 and a top of three. Right? So this I generally what I do is I set up the parameters and then test uh against uh all the uh ground truth questions. we get back the answer and then the answer we send to LLMs to judge the groundedness and the relevance and then I also look at the length of the answer the latency of the call you'll definitely see here that using a gentric retrieval increases our latency significantly we go from about 3 to 5 seconds to you know 10 to 13 seconds and that makes sense because we've added another um another LM call to the mix that answer you can only get from A mira go ahead.
This is great. Uh, thank you for laying the foundation. We can get into Q&A mode. Please raise your hand. Mira already had some questions. So Ma will be asking one question. Yeah, absolutely. No, so the key takeaway um really automated evaluation can only go so far. So, the domain expertise comes in just from your last bit. um for teams putting these systems into production, what is your advice on bringing the human evaluation um into the loop effectively without you know slowing things down on that side? And also another question was um in terms of for PMs and working with technical teams on these systems, how should they think about balancing the relevance and the groundedness with the latency and the cost, especially when deciding to scale that up into production?
Yeah, good question. So, um, yeah, the first question was uh, yeah, productionizing with human-based evaluation and uh, it's a really good question. I think that like the Azure AI foundry has a mechanism where you can actually like have humans go through results and do like a thumbs up thumbs down. So if you had a domain expert, they could go through and like thumbs up and thumbs down. This UI here is just something I wrote myself, just a command line tool. U, but I think in the Azure like in AI.azure.com, azure.com. If you go to like the evaluations tab, you'll see various um options options there. Uh, so we've got the automated evaluation. So I would manual evaluation on top. Yeah. Well, oh that's right. The manual evaluation. Yeah. So I would definitely look into this manual evaluation here. Um, and and setting that up. One thing that's a good practice that uh we're not doing yet is uh at least in this sample is adding like a thumbs up thumb down. And that's you know, you see that all over the place like if you're in like co-pilot, right? Like on the answer you'll you'll see um, yeah, like this one, right? So that I think is is a really helpful thing to do because then you're getting the signal from the user themsself. And then you could add you could add both of these. But definitely if you're getting thumbs down, I would certainly send those for evaluation. And I also feel like if somebody is thumbs downing something, it's they're also kind of not exactly giving you permission to sample it, but you know, they you know, they're they're giving the system a signal that they want this to be looked at. Now, go ahead.
So at build uh what I saw was there was a demo wherein uh for similar questions the agent was uh coming out with the exact same wordings exact same answer. How was they doing that? Well um there's two possibilities. One is that they could have been doing caching. Uh, so one thing you can implement is you know doing your own caching based off of a user question. For example, if I had a cache for these sample questions, when I clicked on it, instead of going through all that work to get the answer, I would be checking in the cache for a key uh with this question. And if it's there, I would just return back the results. So that works as long as nothing has changed about the system since uh you know, since you started building the cache. Uh, if something changed about the system, you would want to bust the cache, which means remove everything from the cache. Uh, it also works as long as you don't have any user specific data. So for this application, we do actually have the possibility of adding user specific uploads and their own private index and stuff. So that's that's also what makes these evaluations so hard is that we can try and like have as many things hold as many things constant as possible. Um, but you know something gets rounded up or down and then boom our answer is different.
So one question uh Pamela right I think what you are seeing in market because I know you talked to a lot of customers you're customerf facing so wanted to know like is out ofbox rag solving a problem in industry and when people should switch to more expensive rag solutions like Microsoft because there is an easy way I can just build a chrome db out of box rag system but obviously the hard part of making it work accurate and serve my customers the higher quality relies on me then and I need to hire engineers or I can just rely on Azure and just get to Azure search engine and then say yeah Pamela has run all the eval only best things are here so when are people good or for this group when should they make that call based on your experience like get to more sophisticated tooling or even go from normal rag on Azure to agentic rag what are the indicators of that or is that a choice So I can just say half of them go here and half of them can go here.
Uh, I mean I think it depends on uh, you know, what kind of data you're working with right? If you are working with rag on documents then uh then I think you really want the the best searching possible because rag on documents is is the hardest. If you're doing rag on like already structured data then you may not need as powerful of like searching and and rag solutions uh because you've already got structured data like let me see I think I have got mine I see if I have mine running right now yeah so this is the rag on Postgress right so if we're doing rag on a a Postgress database where it's like actually on the rows of the database then uh it's just an easier thing to do because we're dealing with such structured data that you know it's it's easier for the LM to answer it's easier to search yada yada right so there we don't necessarily need all the functionality of Azure AI search and we can get away with just using Postgress with like a PG vector um and just like the built-in full text search right um because there you know the results from the database they're they're very structured and straightforward um but with this you know with this one here we're dealing with uh chunks from documents it's it's hard to search these chunks and and get good results. So then, you know, then I always recommend getting having a really really good search. Uh, and basically has the best search that I that I know of. Um, and the reason is because going back here um because it it has this full this is a full search stack um that it supports and not all of the databases make this search uh search stack easy, right? So when you do a search on Azure AI search, it does it can do this full hybrid with reranking which means that it takes that text query it does a keyword search using you know best-of-class keyword search BM25 which is the the best kind of search you can do. It also does the does the vector search using your vector embedding model and then it you know combines those results together. So we've got the merge results and then what's really nice is the semantic ranking model. Um, and it's the same ranking model used by Bing for ranking their search results when you do a query on Bing. So, it's a very good model. Um, and it will actually rerank the results to put like the best ones at the top and even give you scores and say how relevant it thinks, you know, each answer is the query, right? So, here we've got scores ranging from like 0 to four, right? So, you can even cut stuff off and say like, hey, if the score is one or less, I'm just not even going to look at it. So that to me is really important, especially when we're involving vector search because vector search can so often give you noisy results. Like vectors are amazing. They do amazing things. They get you amazing results, but they also get you noise. And you don't want to send noise. You don't want to send anything distraction distracting to an LLM because then an LLM will it'll say, you know, it'll it'll use that as an excuse to answer in an ungrounded way, right? So, particularly when you're using vector results, I I really think you need a way to remove distractions, remove noise. Um, so this is what I recommend anytime somebody's doing rag on documents is that you want you want to have this full stack of hybrid plus reranking and setting a like a threshold for the reranker so that you're really getting the best retrieval. Uh, so whether you do that with a eye search or you do that with another database um, you know, that's that's up to you. What I find is that if you can get good retrieval, you can get good results, but you need that good retrieval first.
Semantic ranker. Yeah, I see a question. Is semantic ranker a service or a tool. So with Azure AI search, it is a feature built into Azure AI search where you just say, hey, I want to use the ranker when I do this query and it'll use the ranker and it's got that ranking model uh built into the system. Uh, it's it's not an LLM. It's a kind of model called a cross encoder model. Yeah, you can see this first result cross encoders and reranking, right? So, um, it so it's it's built into Azure I search. However, if you were doing something separate and you wanted a standalone reranker, you could use um there's the cohhere like has a standalone model uh coher rank, right? So, they actually offer a standalone ranking model. There's also a like um there's ones you can just download from uh just from hugging face. So there's an example um for Python if you're doing uh PG vector uh where you could just pull in cross encoder from hugging tools right? So this is the MS Marco mini LM and it'll predict the ranking scores and then you can uh rank according to it right? So this is basically what it's doing behind the scenes is writing a cross encoder model re-ranking them giving you the new list and uh, you know, AI search is doing it in a fairly performant way. But um, yeah, you can set this up uh in conjunction with other systems as well. Yeah, let me share very I should post links in the chat. Here we go. Place this one. Do this one.
One more question on that is can you quickly show us if we want to get started with Azure search and use what you just showed. Is there a repo that you go to which is your favorite repo and people can bother you later when it doesn't work? Yeah, I get lots of issues filed. Uh, so yeah, the main repo is is this one. That's if you want this full end to end solution that we're looking at. Um, if you do just want to get an idea for like um, how uh, just how this stuff works, just just like seeing the basic code. Um, I also just have a you know, this this repo here which is just notebooks that I use you know, when doing presentations you know, where you can just see you know, what is what does a rag look like with Azure a search. um, you know, where we we get the question, we get the documents, this is the one that's using semantic right? We're doing a vector search, we're doing a text search, we're doing a semantic ranker that's all we just have to specify you know, what we want when we you know, call the API get back that those results and then we send it to LM so this is the simplified version of it and then if you need like a full end-to-end full stack rag solution with lots of lots of features then you can check out the other repo yeah so answer like just take that one call that Azure search call and that will be at some point in SDK will be agentic true and that's all I think we need to do and then you get everything what Pamela was showing on the slide from Azure and you need not to worry about how it is implemented or getting some external models and doing events they have done the work just call Azure search and pass aentic rags true and that's pretty much it and I'm happy to put a lab together because Pamela was kind enough to show us so we'll put a lab together for Azure search and how can you do what you did with Chrome with using Azure search and agentic rack happy to take one of Pamela's lab here and we will put it in our labs so you can try it and if you run into issues we will obviously help you mira back to you.
Yeah um I guess to wrap up one last question so we talked a lot about agentic rag and how it works and what it enables but when teams try to implement it where do you see um them typically going wrong. Are there any patterns or traps that you've seen um that look different from the more traditional rag setup?
Good question. So, so the question is like generally what are the struggles with rag? Yeah, generally like where where things fail for you or what is still not working Pamela in this rag world because I don't know like sing the planning is the solution re-ranking is the solution but what's still open for people to not just like think every question any developer ask or VP ask you just say agentic rag uh, yeah, then you have to define what a gentic rag is whether you mean retrieval or reflection and yada yada it really it all comes down to retrieval, right? Like because if you can get the answers to the LLM, it it can give if you can get the the right context to the LM, it will very happily answer. So, you know, even though we think of rag as being an LLM thing, the quality generally comes down to retrieval. Um, so, you know, a lot of what we do is like try and help people understand like how to like how to debug retrieval. Like that's why we always have this like thought process here where you can like look at it and you can see the results and see like get a feel for you know um, you know, what it's getting back and what's not getting um, so, yeah, so the question is like how do you, you know, get something get get everything retrieved that you want right? So I uh, you know, one big question right now is like multimodal documents like how to properly get back images um, I'm actually working in another tab I'm working on a new approach uh this is my multim multimodal tab where I'm working on a new approach to multimodal um ingestion, right? So getting back, you know, figures and tables and all that stuff. Um, there's also like getting back metadata, right? Like you you here we've just got the chunks, right? But what if there was actually some like interesting metadata about this document like the date that it was made or something like that? How do we get back that data? Like do we stuff it into the chunk? Do we put it on a field? Like do we Yeah. Um, something like that. uh, and then the ability to answer you know questions that are like summaries of the whole of the entire knowledge base that is something we still cannot do even with our agentic retrieval like so with our agentic retrieval we're better at answering like complex like kind of multi questions right but if we asked a different question like let's say we asked like you know give a highlevel summary of every every role in the whole company with a bullet point for uh for each role, right? This would actually be a really hard question to do um because it would need to basically look at everything in the like a huge amount of the knowledge base, right? It would have to retrieve hundreds and hundreds of chunks just to give this highle summary, right? Um, or you can also think if you're doing rag on a book and you just ask for the entire plot of the book that would be hard to do in fact because you'd have to basically retrieve like the whole book. So there's been various approaches for this. There's graph rag and then there's lazy graph rag. I think those are the two most well-known ones. Um, graph rag is way too expensive. So I don't know anyone who's actually practically used it. Uh, lazy graph rag is supposed to be like cheaper and more possible. So we're looking into that one. But we've also talked about stuff like just doing um a summary index, right? So like you know in addition to having our chunk index which is how we normally answer questions maybe we also have like a summary index and that that summarizes things at particular like levels of granularity so that you eventually could answer a question like this. Um, but yeah, that kind of question is really really hard and that was Mera's question mirror I think that was your question is when do you think about like with groundedness and relevance like what are the tradeoffs right? So exactly that was your question and yeah, because here like if we look at this like 10 seconds like okay well that is and to be fair this is 10 seconds time to the last token um when you're thinking about user latency you ideally want to look at time to first token but when I'm doing evaluations I don't do streaming right Uh, but when I'm in the in the UI here, I'm actually doing streaming. So I often look at both like the answer length, but also like latency and answer length, right? So in this case, the answer length didn't actually increase that much, but latency did. So that means that it is affecting the um even the time to first token. Um, because part there's latency and then there's perceived latency. So one thing that some people do is that like you know, as soon as you get search results, you could start showing the search results, right? Like give people the feeling that something's going on. Give them something to poke around. Um, that's not something we do yet here, but I know other frameworks do that, right? So, like showing if you can like reduce the perceived latency, then you can take more steps and you'll still have happy users because they'll feel like something's happening, right? The thing you really don't want is to have a loading bar for 10 seconds straight because that's boring and then you go and like go on TikTok or whatever, right? So um, you can think of ways to reduce that perceived latency but then at a certain point uh, you know, some people might want the answers and the interest thing is like I talked like I talked to customer build who was like oh yeah like we're fine with like 30 second latency and then I talked to another person who was like oh my god 3 seconds is like way too slow and I was like okay well we there's different users here right? So I think it's going to depend you once again it's going to depend on your users what they expect how much like if it's worth it to them to ask like to wait for the better result. Um, the other thing you can do is like oh this is what I was thinking of doing like reflection right? We're talking about how reflection like just took it took too long. But one thing we could do is that when we could like really shorten this and say when we generate the answer we could ask it to also assign a confidence score and like an evaluation at that point. So in that same call, we could ask for both an answer and a quick reflection, right? We could parse that out and we could show that in the UI and say like, okay, this one has a confidence score of 70%. Right? Um, or, you know, reflection, right? So that way you could actually do it all in that same request. And that would be the way to like radically cut down while still adding a feel like some sort of reflection. And then people could decide whether to like, you know, maybe retry the question or something like that. Um, and that's just a general good good practice with AI stuff is to do it give as much transparency as possible. There's this really good um hacks design toolkit. This is you all should check this one out. It's really good. Um, and what I my favorite thing is if you go to examples, let's see, we go examples, right? And you'll see all these like actual examples from um, from real LLM powered apps, right? So make it clear how well the system can do, what it can do, right? Highlighting things, uncertainty highlighting. There's just so many cool ideas here, right? So, yeah, uh, yeah, I don't have a clear answer there. I think that's, you know, a decision you have to make about um, you know, whether you're seeing better results and whether the increased latency is worth it. Um, but you can look into ways to reduce perceived latency and uh, yeah, just figure out what works for your audience.
Great. Thanks for walking us through all that, Pamela. Um, I realized I stacked a few questions in there and it's late on my end, so thanks for keeping me on track and getting that in there. Um, Mahesh, did you have any other questions?
No, I just want to thank Pamela. I want to thank Mira for getting us all together, organizing this event and making it possible for all of us and doing all the research. Thank you Pamela once again. I wish you all the luck for your next build event or your next customer call where you can say yes it works all the time and the latency is awesome uh for you for getting best accuracy and we will do uh what we will do in our community is we'll put a lab for Azure search with this tool. So at least we know that Pamela came and we can put this recording and then say this lab happened because Pamela came to our community. So thank you for helping us on board to Azure search and I think everybody get $150 with Azure, right? Uh, that's like if you're getting started with it, right? Um, yeah, there's a certain amount that you get for for Azure free. Um, Azure AI search does have a a free tier. Unfortunately, in the free tier, you don't get the ranker agent, right? So if you do free search, you're okay. You're only allowed one free search and it doesn't support the semantic ranker. I guess that probably means it doesn't support agentic retrieval. So I guess I should update this doc here. But yeah, this is like our doc about how to like, you know, reduce your cost when trying this stuff out. Um, oh nice. Anybody who want to adapt, we will be happy to put our Azure key and give you that access. At least we'll do one demo with a real example on contracts and show you a difference between out of box Chrome DB the answers you're getting and with Azure search what you're getting getting. We'll do that lab, we'll do the demo. Again, thanks a lot for everybody joining and making it so interesting.