Evaluating LLM-based chatbots: A framework for reliable AI assistants

This webinar was brought to you by Conversation Design Institute as part of our monthly expert series. Join us every month to learn more about conversational AI.

Welcome everyone to our webinar. We're going to be talking about evaluating LLM based chatbots, a framework for reliable AI assistance. My name is Elena, and today with me is William from CDI.

Couple of words about me and why I'm talking about this talk, why I'm, uh, so passionate about it. I'm an AI advisor and CEO and founder at Parlabs and Chbot, and I've been working in the conversational AI space since 2018. And since then, I've helped more than 110 teams with their conversational AI projects, automation projects. And, um, I have an interesting background. I think I come from a linguistics research background, and then I also did my research master's degree in AI. And throughout my career, I worked as an NLP research engineer, full-stack Python developer. Then I was freelancing as an AI developer, and now I'm also an AI agency owner. So I've seen best practices in testing and evaluating software, and I also have experience doing that for chatbots and LLM-based assistants. So that's what we're going to be talking about today.

And, uh, why this topic and how did this all get started for me? Well, sometime ago, I posted on LinkedIn this resource library where I made a collection of 60 LLM eval tools, which got kind of viral. I got more than 40k views, a lot of likes, more than 500 comments, and around 200 people left their email wanting to get access to this resource. And I also did a mini user research asking people what kind of tools they tried. So what I found out is that most people are still not using any evals, uh, around 66% in my mini research. And we ask you today as well, who is evaluating, who is not. So my research mentioned said that most people are doing vibe evaluations. Then some have manual process established, and I think in our audience, that's the most common answer. Um, and then even less use custom Python code and write unit tests, and even even less people are using any eval libraries. So all of this gave me an impression that everyone is still pretty much trying to figure things out, and it's a very important topic, and everyone is very eager to find out how to evaluate LLMs. So that's why we are talking about it today. And, uh, if you want access to this library where I collected all the different resources, then here is a QR code, and you'll get the link also later in the chat.

Uh, for now, the plan for today, we're going to talk about why you shouldn't launch your LLM-based chatbot without evals. We're going to quickly go through over an LLM evals framework. Then cover how to make a good test set. Then I'm going to introduce you to LLM eval metrics that you can use in general. And then the last section would be specifically for evaluating LLM-based chatbots, which is a little bit different than evaluating just LLM-based applications. And the goal of today is that we all feel more confident releasing to production and have a way to control our LLMs a little bit better.

So why shouldn't we launch without evals? Uh, the core problem with LLMs when we are working with LLMs is that LLMs are non-deterministic and unpredictable, which means that with exactly the same input, we can be getting very different output. And that's a problem. Uh, if you already work with LLMs, you know that it comes with a big package of different risks, such as hallucinations. So these very common LLMs come up with answers that are not true, not based on any ground truth data. Then inconsistent outputs, that's basically the feature of LLMs. Every time it gives you a little bit of a different response unless you know how to control it. Then, uh, harmful outputs. So things like bias and discrimination. This one is less common, but if that does happen, then it has a very high impact, both on your users and your business. Uh, then we also have jailbreaks. So adversary attack dogs, prompt injections, and things like that. Again, less frequent, but when it does happen in this small percentage of, uh, cases, then it also has a very high impact. And we have data and PII leaks. And so, a bit of the same story.

So the core problem when we build with LLMs is that non-reproducible outputs means for the business that the product is unreliable. And if we are working in high-risk domains like finance, health, legal, things like that, working most of the time is just not good enough. We cannot afford to launch an app that's just 60% of the time it works every time. So as AI developers, we are the ones responsible for making things safe and ethical by design. So I'm really happy that all of you came together here today to learn more about this topic, because I think that, uh, we are the ones who understand most, uh, how things work. So this is our responsibility to, um, learn about LLMs and implement it in our products. And, uh, yeah, with great power comes great responsibility. So let's, uh, do that.

Uh, let's start with a free-stage LLM LLM evals framework, just to give you a picture of, uh, how the whole cycle working with LLMs, building chatbots with LLMs might look like. We start with an experiment stage where we first define what good means. So what kind of, um, conversation we are aiming for, and then we run experiments. Maybe we build our prompt. Then we test how it performs first manually, and then automatically, we check if we like the prompt. Then maybe we add a new requirement when we discover that, okay, something is not working. So we redefine what good is. Then we run experiments again. And we do that until we are kind of happy. And afterwards, we need to test, uh, before we release. So we run regression tests, making sure that all the things that are critical for our business case, they pass, and we are safe to release it to production.

After we release it to production, comes the monitor stage. So our app is already live in production. People are already chatting with our AI assistants. And here we need to have guardrails, input and output. So input means that whatever user is sending to us, we are filtering whether it's safe to even pass further through our application, or maybe we filter it out immediately. And output guardrails means that whatever LLM produces, before we show it to the user, we do some check on it to make sure that it's safe. So that's one thing that we can do live. Another thing is we always need to log the data so we can later analyze it, and we can set up real-time alerts for any system failures, any conversations that get flagged, and we want to look into that immediately and fix the prompt and, uh, deploy the new fix.

After that is done, we have the improve stage. So this is already when we are working with the logs. We have the conversation transcripts, and we analyze them either manually, or maybe we have an automatic system that scores the conversations and flags the ones that really need our attention. So we dig into that. We debug what went wrong. Then we find what needs to be improved, and we update our test sets based on the production logs. So we find new edge cases that are critical for our business, and we go back to the experiment stage where we redefine what good is, run experiments, and it all starts all over again.

Today, we're going to focus specifically on the experiment stage. So answering the question, how do we know if we are improving? So how do we set up chatbot experiments, LLM-based chatbots, AI assistant experiments? Uh, well, here we need to ask ourselves two questions. So first is, how do I know if my new prompt is better than the old prompt? So we need to have a way to quantify our progress and put a score on our new prompt to tell us if we are moving in the right directions or not. And the second question we ask ourselves is, can I safely release to production? And that means that every time we update the prompt, or we update the underlying model, so we used cloud now, we're using OpenAI, or we used OpenAI, uh, for, oh, now we are using, uh, for one, we need to test it and we check, did all the cases that used to work before, are still working?

And just like with any software, when we are developing chatbots and LLM-based assistants, test-driven development still applies, which, if you're not familiar with this term, it means that even before you start, uh, writing a prompt, you first define what good is. You first write your tests, and only afterwards you work on your prompt. Uh, so how does the plan look like? We start with the test set. So we come up with some examples, either from production logs or maybe from our head initially. Then we set up with all metrics. We define what good means, and only afterwards we write the prompt, and we keep improving the prompt until it passes all our tests. So that's test-driven development.

Um, then how do we make this test set? What does making a good test set actually mean? Well, our main goal when we make a test set is to collect a test that represents our use case well. So in my opinion, a good test set covers two questions. One is, what is important to measure for my use case? Um, for example, it can be domain-specific examples. Uh, let's say here, we are working on an AI system for a property management company. "What is your pet policy?" is one of the questions that we get often. So we need to cover those. And apart from the domain-specific examples, we need to cover edge cases as well. So transfer to human, out-of-domain questions, maybe a person being rude, things like that. And the second question is, what does good in my domain mean? For example, again, for property management, uh, let's say the reference response is, "We allow cats and dogs at our community. Learn more here." This is the link and the definition of what good is. So the domain-specific criteria here is, we need to give a short answer and link to the full pet policy every time someone asks about pet policy.

And remember, we mentioned all those risks: hallucinations, inconsistent outputs, harmful outputs, jailbreaks, and data PII leaks. Well, a good test set needs to cover all of this as well. So, uh, if we are looking for some formula, this is what our good test set coverage would include: happy paths, so business-critical queries; then edge cases, like our domain questions and other language being rude; then sensitive topics, like, and this depends a little bit on your domain, but you want to test for, uh, mental health, uh, requests, finance advice, legal things like that. Then we also want to test our system, our AI system, against adversarial attacks. So prompt leakage, PII leakage, uh, jailbreaking attempts, and things like that. And also, we want to include in our test set diverse user styles. So short queries, long queries, things like that.

And how do we come up with this test set? Well, we start with analyzing existing data if you have that. So you go to your emails, call transcripts, and then you copy and paste examples from real people's questions and add that to your test set. And there we find happy paths, edge cases, and other things that we talked about. We add that to a test set. We test, we deploy to production, and then after we log the production responses, we update our test set with examples from production logs. So our goal eventually is, even if we start with an imaginary test set initially, that afterwards our test set contains real test cases. So not GPT-generated, not from your imagination, real production test cases.

And here I want to emphasize that defining what good means is actually a manual and continuous process. So in this process of creating a good test set with good coverage, uh, we keep in mind that we use automatic testing later to scale our judgment and not to replace it. So this is why it's important that domain experts help you with creating the test set, and that initially, you always start with the manual process because you are the one who knows your business and you know, uh, what good means in your domain. Okay.

And I have a few more tips for building a good test set, uh, for you as well. So one thing is to ask subject matter experts to label the test data. So if you have access to subject matter experts, then, uh, make them your test data creators, make them check your labels, check your conversations, and things like that. Then another thing, which I think is often overlooked, and especially is relevant for us, uh, who are building chatbots for sales and customer support. Oftentimes, if you build an AI assistant for this specific use case of sales teams and customer support teams, they have company guidelines, they have standard operating procedures, some onboarding documents, some documents that explain that how they need to behave when they are on a call. So always greet with the first name, then ask this and this and this question, always end by asking this and this. So, oftentimes teams already have that documented for humans. So you can use exactly those guidelines to define your test cases and specifically eval criteria. And the last one I already mentioned, that we try to use examples from production logs and not made up.

And, uh, maybe a bit contradictory here, but still wanted to mention that, okay, yes, it is possible to also generate synthetic test data, and it can be a good start, and for some of the domains, maybe that also makes sense. And I think that you can start with a synthetic test set if you want to probe model weakness or similar bricks attempts at scale, maybe if you want to evaluate robustness or wording changes or paraphrasing and things like that. But always keep in mind that a synthetic test set is not the same as real-world generalization. So always ask, uh, subject matter experts to check your synthetic test set, and I also maybe wouldn't start with this. I would prefer to start, uh, with the manual process of creating test, uh, data.

And, uh, here are also a few more tips to cover the other risks that we were talking about. You can Google things like prompt injection datasets and add that to your test set as well. So this one is from Hugging Face, for example. Then, uh, there are also adversarial datasets. So this one is from HarmBench. Then Evidently has actually really amazing resources on the topic of LLM evals in general, but also they created this table with LLM benchmarks and datasets. So you can check that one as well. Uh, yeah.

So now that we know, uh, what a good test set is, let's say we collected examples that represent our use case. We now want to talk about metrics. So we want some sort of score that would tell us if we are doing better than before. If the new prompt is better than the previous prompt. So we want to put some number on it. And, um, if you remember, our core problem with LLMs is that they are non-deterministic and unpredictable. So again, that means that with the same input, we will be getting different output, and that also means that different output can be good. And that's exactly the challenge with evaluating LLMs. How do you evaluate if many things at the same time can be correct?

So, uh, I define metrics for LLM-based chatbots into four different categories. One is pattern match reg syntax validation, things like if it starts with something, as, uh, the simplest one. Then we have statistical metrics for, let's say, you are, you still, uh, have intent detection as part of your flow. So for those kind of things, you can still use F1 score, accuracy, BLEU, or ROUGE. Then we have machine learning-based metrics, such as semantic similarity, maybe between your reference output and LLM output, BERTScore, sentiment analysis. And finally, we have something that we have since recently, well, recently, I think maybe a year or two, called LLM as a judge. So something that helps you define custom eval criteria. And I assume that all the other things, most of you already know. So we'll quickly discuss what LLM as a judge is and the challenges of evaluating with LLM as a judge.

Uh, here I'm actually curious, and I don't know if I see the chat. I think I do. Write plus in the chat if you know what LLM as a judge is, and minus if you don't. Plus, plus, minus, plus, plus. Okay. So most people know, then, and some still don't. So I'll really quickly explain, uh, what it is. Well, the idea is super simple. Basically, you generate, uh, your response with an LLM. So the chatbot produces a response, and then you use another LLM to evaluate, uh, this chatbot response, or to evaluate the whole conversation. So that's, uh, that's basically it. And LLM as a judge is great because custom criteria are better than generic metrics. And with LLM as a judge, you can measure what's important for you specifically. With LLM as a judge, you can, uh, split it into three different categories. You can create a test set with a reference example. So how the good response looks like, then with no reference, and then you can do pairwise, pairwise comparison. Let's go through all of those three one by one.

So how would a test set for with a reference look like? Let's say this is our user question: "What are the top attractions in Paris?" Then, uh, someone manually writes a good reference output with the tone of voice you like, with all the details that you want to mention. So, "When in Paris, visit Eiffel Tower, Louvre, and Notre Dame." And then we use our prompt to generate the LLM output. So it was, "If you are visiting Paris, some of the most popular sites are Louvre, Eiffel Tower, blah, blah." And then you have an LLM as a judge, which, uh, basically takes those two things and asks a question: "Are they similar or not?" And here you can define how you want the output to be. So maybe you return a JSON, and, uh, with Pydantic, you say that it can predict one of three categories or one of four categories. So then you will be getting an evaluation result from LLM's judge and some explanation, and afterwards, you can summarize and give it some score at the end of the table, and then you would know how this test set batch performed on your current prompt version and compare it with the previous one.

Uh, then, uh, another thing is no reference. And here we have two options. So option one looks like this: question is the same. So, "What are the attractions in Paris?" And then instead of saying what do you expect the model to return, you write an evaluation question. And let's say here, it's a bit abstract. I say, "Is this answer friendly?" And of course, you need to define what friendly is, um, and you apply all of, uh, this question to all of your test cases. So always, no matter what kind of question, it always needs to be friendly. That's just one of your criteria. And then again, you ask LLM, "Did our LLM output pass this criteria?" And then it says, "Correct or incorrect," and it gives you an explanation. Another thing which I use more often is the no reference option too. So this is when you define different eval questions per question, per user question. For example, "What are the top attractions in Paris?" The evaluation question can be, "Does the response mention free tourist attractions and ask a follow-up question?" And then maybe we have, uh, another user question which says, "Um, where can I eat in Paris?" And the evaluation question would be, "Does it mention five, uh, restaurants and exact address and Google link?" So this way, depending on your use case, you can, uh, define different evaluation questions, and that's what we use quite often in production for our clients, this option too.

And, uh, the last, uh, category was pairwise comparison. So this one is pretty simple. Let's say you have two prompts, and you have defined some metric which can decide which of these is better, and you run it on a lot of different rows and examples, and you always ask, "What is better?" And of course, you need to define what better is. So you still apply all the best practices for prompt engineering, and you are very specific with how you train your LLM judge.

Just to show you how that might look like in practice, you can build your own LLM judge prompt. Just in Python, you don't have to use any libraries. You literally can just, uh, write a prompt. For example, "Please answer the question about the text with correct or incorrect." And then we have user question: "Which attractions?" No. Then we have an evaluation question: "Does the question contain free links?" And the text: "Best attractions in Paris is Eiffel Tower and Louvre." And then we say, "Please make sure the response consists of a single word, 'correct' or 'incorrect', and return a JSON in the following format." And then you literally just use a standard OpenAI library to run this evaluation on top of your response and get the scores. So that's the simplest thing you can do.

Uh, here I want to warn you that you need to be mindful because one LLM as a judge can be expensive because we are using LLM, and, uh, if you run tests and your test is big, every time you will be spending tokens, and depending on which model you are using, uh, of course, you'll be spending different amounts of tokens and money. And still, hallucination risks apply when you work with LLM as a judge evaluators because they are based on LLMs, and LLMs are non-deterministic. So exactly the same things apply to LLM judge as the problems that we are discussing. So what do we do to improve that? There are a few tips you can use for building your LLM judge prompts. One is using binary scores. So yes or no, polite or impolite. Then explain each of these scores very specifically. So don't be vague. Adding examples, setting a low temperature, uh, using more capable models if possible, and returning the JSON, and maybe using Pydantic to structure your JSON output. So basically, similar things that, uh, we use when we write prompts are also applying.

And here, when you are building LLM eval, no, LLM as a judge evaluators, uh, you also need to go through a certain process of aligning the score LLM as a judge is giving you to the human labels. So you need to test your LLM as a judge, and you need to have an evaluation test set for your LLM judge, which will later be, will be evaluating tests. So it's, uh, it's a bit complex, but this is how it is. So you need to create an evaluation test set specifically for your LLM judge. So that means that you are evaluating how good LLM as a judge is evaluating. So you will be evaluating your evaluator until the labels align, and only then you can use this to test your, uh, LLM application consistently.

Uh, there are a lot of different LLM judge implementations. So one we already covered, you can write your own prompt. Then there are classic LLM judge implementations from existing libraries, and there is something called Gval, which DeepEval has implemented, for example, and there are a couple of ours. So just to quickly give you an idea of what we can use. Uh, LLM as a judge has an implementation in a platform called Evidently, and there they have, you can choose between two options: binary classification or multiclass classification. And here the way it looks like, you define the criteria. So for example, you are measuring how concise your responses are. So you write, "Conciseness refers to the quality of being brief and to the point," blah, blah, blah. You add some examples. Then you add target category that you are hoping to predict, so "concise," then "non-target category: verbose," then "uncertainty: unknown." And, uh, then you send it the message that you're evaluating. So this is an example of binary classification. But you can do multiple categories as well.

Then, uh, there is something a little bit more advanced, which is called Gval. Won't go too much into details, uh, into that. I will instead just show you an example of implementation from DeepEval. So they have two options that you can choose from. You can either use Gval criteria or evaluation steps. So if you are using, uh, Gval criteria, then you will be again in text explaining, uh, that coherence is the collective quality of all sentences, blah, blah, blah, and then you will define evaluation steps, like first to read the news article, then read the summary and compare it to the news article, then add the score. That's option number one. And option number two, I think I messed it up a little bit. So this Gval criteria and evaluation steps, they go together. And then, uh, you can also use something called Gval rubric. So you can also ask LLM to give a weighted score for different, uh, things. For example, from, give a score from 0 to 3 to measure the fluency of the actual output, and then also measure the logical flow of the actual output, measure the linguistic flow of the actual output. So this gives you an opportunity to measure different dimensions of quality for your responses.

Uh, just to sum up this part about metrics, we have a lot of different metrics, and some of them, I think I'll mention a bit later as well. So you don't have, uh, a reason not to evaluate. Basically, I can't tell you which metric you need to be using, but there are tons, and you just have to see what aligns best with your human judgment and what's important for your use case.

Okay, now let's move to the part of what can we evaluate, and let's, uh, just make sure we are on the same page regarding the LLM-based chat with architecture. So the standard architecture that we have in mind, and I here I want to say that I think that evaluating chat is actually different than just evaluating, uh, one of generation prompts. So it has complexity on top of it. The classic LLM-based chatbot architecture looks like this: User asks a question, "Do you allow cats and dogs?" Then we use RAG to retrieve context from our vector database. We paste it to the LLM and we use a prompt which says, "Please respond based on our data." And then, uh, we get an LLM output saying, "Cats and dogs are allowed." So here, and then we do that on repeat, and it turns into a conversation. So here we can be one evaluation RAG. So how well those chunks are being retrieved from our vector database. Then we can be evaluating, uh, generation. So how relevant this output is to the user message. And we can be evaluating the whole conversation quality, which is what makes evaluating chatbots based AI systems, uh, more complex.

So let's go through those three things one by one. So evaluating RAG response quality and conversation structure. For RAG, we have multiple options. We can first evaluate retrieval quality. So let's say in your test set, you define for a specific user question the ground truth context. So you expect to retrieve from this static database things like "no bridge restrictions and cats are allowed." And then you have actually retrieved context by your RAG system. And here it's quite easy. You can just measure context precision and recall and compare those two with each other. Then we can also measure answer correctness. So here we would have context actually retrieved by our RAG and the LLM output, and we will be checking if the LLM output is not, uh, hallucination, if it's actually based on this RAG. And, uh, for example, Amazon has this RefChecker library, and they have a really nice image which I think really well explains what our options are. So you can compare those two and see if the facts are correct. So entailment, if they contradict each other, or if one of, like, for example, there is something in RAG which is not in the output, or there is something in the output which is not in RAG, which I think is called hallucination.

Then we can actually do a proper hallucination check. So ask ourselves, "Does retrieved context contradict the LLM output?" And here again, we have a lot of options for how to evaluate. So one is to write a DIY prompt just in pure Python or whatever programming language you are using, and it can look something like that: "Below is the context that needs to be used to answer the question," and here you paste your RAG context. Then this is the user question: "Do you allow pets?" This is the generated answer: "No, cats and dogs are not allowed." And then your goal is to check if, uh, the context here aligns with the model response. So maybe we predict three categories: contradiction, missing facts, and correct, and we just again use a standard OpenAI library to do that. Another thing you can do, uh, is actually predict JSON and do more complex, uh, predictions. So type of overlap: disjoint or equal or subset or superset of overlapping, and whether it's contradicting, uh, one another or not. And of course, there are also hallucination, uh, metrics in existing libraries. We, for example, at Parlabs, really like this DeepEval hallucination check based on LLM as a judge, and it's pretty simple to use. So again, you just define your context, what the actual output was, and then you use their hallucination metric and define the threshold of, uh, when does it pass hallucination check or not. And this thing, by the way, you can also use as your guardrails, uh, during the monitoring stage.

Uh, then DeepEval also has Gval hallucination checks, a little bit more advanced, so you can use that one as well. Uh, what else? Uh, when we are evaluating RAG, we can also use something called RAGAS, which is four different metrics. One is answer relevance. So we are comparing relevant claims and total claims. Then you can evaluate answer faithfulness. So is the LLM output supported by retrieved context? So is, uh, what LLM returned supported by whatever RAG returned? Then you can also measure context precision. So what proportion of relevant chunks are in retrieved context? So maybe LLM output only mentioned one fact from your RAG context, something like that. And then you can also measure context recall. And you can combine this into a weighted metric as well, which is called RAGAS score. And again, DeepEval has a RAGAS implementation. And then there is a separate RAGAS library that you can use to measure those three things separately or get, get a weighted score as well. So that was about evaluating RAG.

Now, we can also relate response quality. And here I wanted to share with you one example of how we build our test sets. I think it's a bit similar to what I was already showing you, showing to you with LLM judge, but let's go through this, uh, again. So we define different user messages that are critical for our business case. So for example, user says, "Check business hours," or "How big are your three-bedroom units?" or "Show me amenities," or "Show me community features." And then for each of this, you define separately your evaluation question, and the more detailed you are, the better. And this is, I think, a really abstract evaluation, but you would want to, uh, check for things that are actually important for you. So for example, for the neighborhood, if I know, uh, for which property this question is, then I want the text to mention all of those things. So you can go really specific here. And this is the prompt that I actually already showed you. So just as a reminder, you can apply this evaluation question to the LLM text and ask it to return a JSON which says whether it's correct or not. So yes or no, with some explanation. And then there is also conversation structure, which I think is my favorite part of evaluating LLM-based chatbots, and something that I don't see a lot of tools are doing out of the box, but hopefully, that's changing soon, or maybe already changing. Let us know in the comments if you know anything that does it.

Basically, the idea here is that we want to evaluate conversation structure. So evaluate flow rules and check that required steps happen in the correct order. So let's take a few examples. If this is a customer support, uh, chatbot, then maybe you want to evaluate how does it react to "I want to talk to a human" at different stages of the conversation, because maybe the first time the user says "I want to talk to a human," you say, "Please try to talk to me. I promise I'm smart and I'll try to help, and otherwise, I'll transfer you to a human." And if the user asks it for the second time, or in the middle of the conversation, and it asks the, they ask the exactly same phrase, then we want to react differently. So we can't use those flat evaluation metrics. We have to take the whole conversation into account. Or, uh, maybe you are building a voice agent that always needs to ask the first name, first things first in the conversation. Or you have a very specific guideline on how the conversation needs to be finished so that your customer support voice agent asks questions like, "Uh, did I answer everything? Do you have any other questions?" and then maybe, I don't know, adds a promotional phrase or offers to book a tour or something like that. And, uh, you can also apply this, uh, strategy for question-answering kind of chatbots.

I don't know about you, but for us, it's quite a common use case recently when the client wants that the AI assistant asks a set of questions in a very specific order, while of course, they are contextually aware, so they don't always look the same. And, uh, you can also evaluate that how well your AI assistant is following the structure. So, if we look at an example of this "talk to human," and just to make it a little bit more visual, the test case would be this kind of array of dictionaries with the conversation history. So very standard format, exactly what OpenAI and our models are using now. So the role of assistant: "Hi David, how can I help you today?" And the user says, "Talk to human." And then the AI assistant, based on this, predicts something using your prompt, and the assistant says, "Sure, let me connect you." And then we have LLM eval specifically for this test case. So not just a general LLM eval, but for this test case. And this LLM eval says that the answer should be "no." And then we check and we see, okay, actually it's wrong. The assistant said, "Sure, let me connect you," but we want to encourage them to talk to human first. Then we will have another test case, uh, where the conversation is longer. So, "Hi David, how can I help you?" "I was wondering if I can book a tour." "Okay, I can help you with that. When do you plan to move in?" And then the user says, "Talk to human." So like you can see, exactly the same phrase, but the conversation history is different. The assistant says the exact same thing: "Sure, let me connect you." But the evaluation criteria now is different. So we want the answer to be "yes," because in the middle of the conversation, we think it's okay to connect to human immediately. So then, uh, this conversation passes our eval check, our LLM check.

Uh, yeah, this is just to demonstrate that actually this is not so difficult to build. So we have the test set, and we predict the assistant response. Then we have this LLM as a judge, which looks like this, or maybe a bit more detailed, because you still need to go through the process of tuning your LLM judge. But as a draft, "Please evaluate the following conversation between user and assistant. You can only return one word: true or false. Return true if the assistant connects the user to human, otherwise return false." And that was for this use case specifically. And then we again just use a standard OpenAI call to do the prediction, and then we are using the standard unit test functionality of Python and check, assert equal message response versus expected response. Um, yeah, so that's it.

And then I also have a bonus of what else you can do when you're eval, create convers, no, when you evaluate chatbots. So you can also do conversation simulations. How would that work? Like you can define multiple user personas. So maybe one user, if we are still talking about this property management domain, one user comes with five cats, and they want to know if five cats is okay. Another one wants an apartment for, with five rooms, and they want to know how much it's going to cost. So different scenarios, right? Then you would provide examples of test scenarios you want to simulate. So maybe conversations from production, so you can send it to the LLM that's doing the simulation and would say, "Okay, this is the user persona that came, this is how the conversation typically goes, try to act as this user." And then you would let your, uh, simulator that you created talk to your LLM-based chatbot on behalf of this user persona. They will talk, they will produce a conversation transcript, which you later can evaluate either manually or using auto-scoring.

So how that might look like, we can have a user profile prompt. For example, for the property domain, it can be something like, the user profile is a name, pet beds, household members, apartment type, they're looking for, moving date, and maybe a list of questions that they need to ask for this conversation. And then you have this user prompt, which basically says that you now act as this user. You will be talking to whoever. And this is an example of how you lead the conversation, what kind of questions you ask. And then this is what's going to happen. An LLM generates a user message. Then your chatbot generates the assistant message, and it goes on and on. And afterwards, you get the transcript, which you can either evaluate manually, or you can also build an auto-scoring evaluator, which, uh, actually came from an idea that a lot of teams that have customer support teams or sales teams, they are already doing conversation transcripts for their humans. And often times to evaluate the performance, they either do it manually. So maybe they have a person who randomly checks the transcripts to check how sales reps are acting and performing, or how customer support representatives are performing, or they build this evaluation that automatically evaluates those conversations according to some criteria. Often it looks like an Excel table where they say, "Okay, every sales rep needs to introduce themselves by name, and then they need to always, uh, name the company they're calling from, then they need to ask at least three questions, and in the end, they need to say that they'll send a link." So they already probably have criteria for evaluating human-to-human conversations. So what you can do is you can train an LLM to evaluate your transcript according to the exact and criteria as you evaluate humans against, and you can evaluate your simulations just like that. So on the screenshot, this is how the prompt can look like. It might say something like, "Score a transcript below between an agent and a prospect based on 12 criteria which are provided below." So you would paste them below, and then return the results in this format, like criteria one score, did it pass or not, and some explanation.

That was it for evaluating chatbots. I have a few more tips for, uh, how to deal with all of those evaluations. So one is, you probably might want to evaluate multiple times. So four to five times, because of the nature of LLM as a judge, but also because every time you will be evaluating, the output of your LLM might be a little bit different. So you want to take an average score and not just a one-off score. And like I mentioned, don't forget that you need to be evaluating your LLM as a judge evaluators, otherwise it just doesn't work. And this all was about different ways you can, uh, write code in Python or use existing libraries. But there are also different open-source libraries and open-source platforms as well that you can use. And, uh, there, what they usually have is some sort of LLM playground where you can visually see your prompt, do some version control. Maybe you can, uh, set up LLM as a judge there. You can also do human evaluations. You can track latency and costs and things like that. So that's also an option. There are a lot of platforms like that. And from the research I've done, I think you need to do your own research, basically, and see which one you prefer more. So I talked to different people, and everyone has their own preference. Like I already mentioned, I have this collection of different tools in a Notion table. And I also specifically have this open-source tab in this collection where you can see, uh, all of those tools and just test for yourself if maybe you want to use the platform instead of building those LLMs from scratch. And William will share the link also in the chat. And, uh, yeah, let's just quickly recap the final framework.

So when we are evaluating LLM-based applications, LLM-based chatbots, we have three stages. So we experiment, we monitor, and we improve. And during the experiment stage, we define what good is, run experiments until we are happy. And then we have a certain test before we release to production. Then during the monitor stage, we log data. We set real-time alerts. And during the improve stage, we analyze the logs. We define what's wrong, and we update the test set so we can go back to the experiment stage. So just to sum up, test-driven development, so test before you write prompts, define what is important and what is good for your domains, that's a manual process. Build a realistic test set and keep improving the coverage of the test set. This is, like, you know, building chatbots, building AI agents is a long process, a continuous process. So keep improving your test coverage as you test. And, uh, if you want to learn more on this topic, you can follow me on LinkedIn. I share a lot of GenAI tips. I talk about conversational AI, AI automation. So you're welcome to stay in touch there, and I'm happy to answer any questions if, uh, there are any.

>> Yes, there's definitely questions. Thank you so much, Elena. This was great. A lot of information packed. Uh, we have about 10 questions. So let's see how far we get. Starting off with maybe a more simpler one. Saur is asking, is it safe to generate data using LLM? Well, we've seen that sometimes it's safe. Maybe the better question will be, are there cases where you feel it's unsafe to generate LLM data sets, test sets?

>> I guess, uh, I'd like you to define safe. Like, one thing is that it doesn't always generalize to your real-life, uh, examples. So, yeah, that's the first thing that comes to mind when you ask safe. But maybe you know, but you can explain better.

>> Yeah. Yeah. No, I, I wouldn't. But I guess you point out that's that's already a good answer, right? We all have to define what is safe, what is truthful, what is, yeah. Yeah. But I think it is a, it is helpful to get started, and if you involve, uh, subject matter experts to check your test sets, that also works, or maybe if really you want to test your system at scale and you already defined the, uh, the initial test set, and then you want to use LLM to add variation to your already proofread examples and test like that.

>> Great. Yeah. And I think maybe this also ties into a question by Yamia. Um, they're asking that for an internal chatbot, they don't have real production data. So they're asking, how can we evaluate the performance well training data, data sets? But maybe you have also other ways that they can get around not

>> Yeah. Well, you either imagine your test data, or what is better is you ask people who. So what, which use case was that? Can you tell again?

>> Um, is merely saying that it's an internal chatbot.

>> Internal chatbot. Okay. Well, you might ask, uh, people what kind of questions they ask. To be honest, for any kind of chatbot development, conversational AI, I think your goal needs to be to release whatever you have to users as soon as possible to collect the way they ask questions. So if you really don't know how to start, maybe the first thing I would do is create some sort of fake and safe chatbot which maybe most of the time says "I don't know." But at least you can collect what kind of questions are coming. Maybe you do that just for a week, not to embarrass yourselves too much, but at least you then have, uh, examples from real people, and then you can build on top of that if you don't have access to analyzing these conversations already. Like maybe they've been asking their HR department through email. Maybe you can analyze that or talk to HR.

department and ask okay what kind of questions do you frequently get? So that can be a good start I think.

Yeah, nice. I agree. U in our opinion that's also how we typically do it. So I want to rush to the next question. Uh Justina is asking and this is a question we get a lot. How many evaluation cases do you think is optimal? not not necessarily a specific number but some tips on the amount of test cases to run.

Well, I think it depends on how many use cases you have within your business. So, if you want to answer like if you have five intents, let's say one you want to handle when user answer asks to talk to a human then maybe when they ask for returns then they ask for something else. So those kind of use cases maybe you can start with up to five for each of them but then again it would scale per how many new features so to say you have within your AI assistant.

Awesome. Thank you. Yeah.

But if you're starting with zero even one would be good. Um, Juan Carlos is asking, "Do you normally run evaluations against each single response or can it be done against a complete transcript? Are there pros and cons to using transcript evaluation?"

Yes. So, it really depends on the use case and we do both for conversations where structure is really important. So really we are following a certain conversation framework either like for customer support first name then ask three questions then this then we try to do this uh conversation analysis for some things like let's say you're building an assistant in voice lower it's more modular where you have like intent detection then maybe some hardcoded blocks and then uh LLM block Maybe we would be evaluating those separately, especially if they all have different tasks. Um, yeah, like one LLM is doing classification and then another one is generating questions and another one handles how to um handle this whole conversation when the user wants to talk to a human. So it it depends.

Yeah. All right. Um many great questions. So I'm just going to continue. Uh Sarup is also asking uh is hiring a seasoned QA test leader with without a AI background a smart decision or should I hire someone with generative AI generative AI developer maybe less testing experience but an AI experience for evaluation.

H I don't have an experience hiring a senior QA developer so I wouldn't know how it is in practice. I do think though that a lot of practices from evaluating classical machine learning models, neural networks or even just software development. A lot of that also applies to when you set up evals uh for LM based chatbots. Things like uh you want to write unit tests, you want to test end to end. In the end, you also want to do CI/CD pipeline and make those tests run every time a new PR on GitHub pops up so it gets uh tested um regularly. So I think maybe both. And that having said um I think you mentioned hiring AI engineer. Uh from my experience even AI engineers don't always know how to test evals. I also don't know perfectly how to do those LLM evals because it's quite a new niche and everyone is still figuring it out. There are a lot of great frameworks already, some of which I already shared with you today and I don't think we are near any near to have the gold standard of this is how you evaluate. We are all figuring that out. So maybe I'll focus more on the fundamental skills and then other things are going to different conferences, reading blogs, following people who do work on that and experimenting for your own use case because I think it will be very personal especially now that um yeah different outputs can be correct and you are doing something custom.

Oh yeah. Yeah. So on our end, by the way, we also only have experience with people that are already working with generative AI that are then learning how to do proper evaluation. So we don't have any QA people with a QA background. Um, but it would be cool to see how they uh how they respond to this topic to this field. Maybe sorry maybe on this topic also it does help if someone who writes evaluators has prompting skills or is willing to learn because with LMS again best prompt engineering practices apply and it's also good to involvememes so even if those are nontechnical people probably they will be the ones telling you what good is so uh yeah, it's teamwork

uh I have many more questions so I'm just going to Go into the next one by Audrey Ross.

They're asking, "Do long answers in your test set affect how many test cases you should have per uh or how many times you should run the test case because many of their use cases have rather long responses versus a sentence of two. I guess this is a question of if you're getting a lot of a longer response then in theory you have a more varied response. Um, I don't think it would affect the number of test cases you should have. So, I would count the test cases as one test case, two test case, no matter if it's long or short.

Yeah.

All right. And of course, it will be more expensive to run your tests if your prompt is longer or if your output is longer. So, something to keep in mind. Then maybe here you might want to experiment with different models and check check if cheaper models, smaller models can get to the same message experience as more expensive ones

makes sense.

And yeah, it also latency. Yeah.

I'm going to jump into the question by Mo Hassan. They're asking, "What is your favorite Python stack for running the tests?" Well, I really like um deep val which I showed today and well not really Python stack but I really like agenda UI and it's an evaluation platform which we use also.

Great.

Yeah. And to be honest it also changes a little bit like every couple of weeks because all those tools keep improving and there are quite a lot as well.

We have another question by Moassan. which metrics or which factors such as tr truthfulness versus time of voice which ones to look for?

Yeah. Well, uh hallucination check is one thing that's the first thing that you should be doing on the input and output guard rails thing for sure which we didn't cover today. But also even during your test set I think that would be the metric that I would go for first. Um, yeah, that would have a really high impact if your chatbot will be hallucinating prices or policies or things like that. We've seen a lot of examples there. So, I would maybe go again over the risks that I mentioned and the ones that I labeled as um high uh frequency risks and I'll start with them.

Yeah.

And I think it also really depends on your domain. Let's say you are working in legal domain and you're building a legal AI assistant. Maybe there you would really want to test when users uh share I don't know a criminal story or something like that and really for you it's important that you don't you know encourage them accidentally and say oh I totally understand you. So it would depend on your um domain as well I think. Uh what are there any specific evaluations for testing um voice for testing voice enabled genital AI?

Yeah, that's a good question. Actually testing voice is different than testing text chatbot. So this is spot on in the library that William shared in the chat. Uh I believe I also have a section for specifically agents that evaluate voice and uh from what I seen they most frequently are doing the simulation. So for voice that's pretty common and uh we still we also build voice agents. We still use this framework that I explained for evaluating the conversation um structure. So first ask name then do this then do that. And I think with voice it's also especially important to be evaluating end to end because there are a lot of components like you can be evaluating transcription uh tracking latency which can be important and um when the AI assistant interrupts you or not. So I think um when you voice other things will be more important for you. So this define what good means and would need some external work. Then a question from Rudy Rodriguez Garcia. They're asking, "Have you noticed differences when you use different LLM models as a judge um mentioning how GPT5 and GPT5 mini gave them a different evaluation.

Yeah. So definitely just just like with anything if you use different model you will be getting different result. For us on practice we typically use smaller models for evolve. So I really like GPT4 uh or GPT 41 41 mini. Um, yeah, here you just again need to test compare different models and see what works best for your evaluators. I don't think there's one recipe for all.

All right, that makes sense. And then the last one, the last question. Um, Kulinder is asking how the weighted subcontext output works as part of the LLM judge evaluation.

So I guess that's about the rubric of uh deep eval. If it's about this one then uh it will be predicting a different score for each of these rubrics I believe. So you can see them separately and I think it would somehow weight it but I don't remember what formula is behind it.

All right. Yeah. And as Lena was saying this this topic is uh is not finished. We're just figuring this out. Thank you so much Lena because we we really learned a lot. At least me I learned a lot and I seeing many people in the chat also uh happy with the information they got.

um thank you everyone for such thoughtful questions. I'm really excited. There's a lot of interest for this topic because I really believe uh we should all be evaluating and uh yeah, thank you for your time and for coming.

All right, thank you everyone. Keep keep the newsletter in in check and Lena's uh socials in check for the recording that uh will be shared with you um likely early next week. And um have a great afternoon or evening or morning well depending on where you are. Thank you everyone.