Transcription
In this video, we're going to walk through setting up offline evaluations in LangSmith. To do this, we're going to introduce a new core concept: this is the LangSmith data set. And we're going to walk through how we can create a new data set, what data sets are made up of, and how we can add new examples to data sets over time.
Then, we're going to use this existing data set to run experiments on our application and test and see whether our application is improving as we tweak different things about it. This can include changing the prompts or swapping out different models. Cool.
We're going to start in this evaluation section, uh, in this data sets and experiments pane. And let's actually take a look at the new data set button here, uh, and what it means to create a new data set. You can create a new data set based off of an existing CSV file that you have, uh, and this will populate your data set with pre-existing examples. You can give your data set a name and description, and for us, we're going to use the key-value data set, which just means the inputs and the outputs are dictionaries. You can also define schemas for your inputs and outputs, and this is really useful because you know which fields you can expect, uh, and then can write custom evaluators or tests, uh, that will make use of those specific fields. Cool.
So let's click into this particular data set that I've already created called PolyGroundTruth QA. This is an interesting data set, uh, and it's also a very common pattern because it's a data set of ground truth, golden examples. What that means is, over time as I've created this data set, I've added examples that I think are the gold standard or a great representation of how our application should be responding to particular inputs. An example is just an input and an output, and if I click into this example here, I can see this input that our application gets, and then I can see the output that our application returns. The output will contain information such as the actual content of the AI response, and it will also contain some other metadata, such as, uh, the token usage and stuff like that. Cool.
So how did I get these examples into the data set? There are a few ways to add examples to a data set. You can add an example manually by specifying the input and the output. This is pretty time-consuming, uh, but it is a great way to make sure that your data sets examples are all, uh, the gold standard. You can also add AI-generated examples. This is a pretty cool feature where, uh, let's say I've manually added these nine examples, and now I want to generate 10 more. It can use these existing examples and leverage an AI prompt to generate, uh, more examples, and then you can manually sign off on the ones you like and add them to your data set. Cool.
There is another way to add examples to your data set, and we're actually going to navigate back to the tracing project, uh, that we've been using, which is LangSmith onboarding. Here I have a trace that I ran a little bit ago. It asks, "What sport are you the best at?" And this is the AI resp resp, and let's say myself as a developer, as a user, I'm looking at this example, and I actually really like it. I think it is up to the gold standard, and I think it belongs in my testing data set. I can directly from this trace add this example to a data set. And so if I clicked here on runnable sequence, uh, I can add this to a data set. It will show me exactly what I'm adding, and I'm going to go ahead and select PolyGroundTruth QA.
Cool. So now if I go back to the data set, I'll see that I have 10 examples here. Now that I have all of these examples in my data set, let's walk through how we can actually use this data set to test our application. We're going to navigate over to the experiments pane. When we run an experiment, we have access to the raw input and the output that we have from our gold standard data set, but we're also going to hook up our most recent version of our application to run over the input, and this will generate a new output. The goal of an experiment is to compare this new output with the reference output from our data set, uh, get some metric of how we've performed relative to that reference. Cool.
So I'm going to hit this plus experiment button. We can run experiments over data sets in the playground interface as well, uh, over a specific prompt, but in this case, let's show how we can run it over the SDK. Cool. We have this walkthrough that walks us, uh, through the exact steps that are necessary to run an experiment. We need to install some dependencies, set our environment variables, which we've already done here. It walks through creating a new new data set, and we don't actually need to do that since we've already created a new data set. This example shows us how to hook up our LLM application to our experiment, and we will have our application locally as well. And then we can write an evaluator. An evaluator takes a run, uh, and an example, so the run is the new output, uh, from running our application over a particular input, and the example is the existing example from the data set. We can then use fields from both the run and the example to to, uh, generate a score for that particular piece of feedback. We'll show an example of this in a second. Finally, we'll run the evaluation with the evaluate, uh, method from the SDK, and you can see that we have specified our data set. We've specified the evaluators that we want to run, uh, and then we also, uh, specify our application.
Cool. Let's pivot over to the code and show how we can actually run this evaluation. This application code should look really familiar. This is what we've been using throughout, uh, in this case I'm using gpt-40 mini as the model, uh, and this is the same traceable application that uses Poly, gives it some facts about Poly, and then we ask Poly to respond, uh, to a user's question using the fact. Cool.
So just like we just saw in the LangSmith UI, uh, we've defined a custom evaluator here. This is a really simple one, and it's just a toy example, but all we're doing is we're asking, uh, or basically we're measuring to see if Poly responded at all, uh, if the length of the response is greater than zero, and if this is the case, we return a didRespond key with a score of one; otherwise, the score is zero. So we create this evaluator, and now we're going to pass this evaluator into our evaluators list. We're going to specify our data set, and we're going to run our latest application from our code, uh, which looks the exact same. I'm going to give the experiment prefix here of GPT-40 mini, and this is because I've also run some experiments with other models, and my goal here is to compare how my application performs using these different models. Cool.
So we've just run our experiment. Let's go ahead and take a look at it in LangSmith. Pivoting back to LangSmith here, I can see this new experiment pop up with GPT-40 mini. My evaluator, uh, for didRespond has already returned, and I can see that for every example in this experiment, um, I did in fact respond, which is, you know, a promising sign. You might also notice that we have two other evaluators here which, uh, haven't run for this experiment yet but are running or have run for the other experiments. These are called auto-evaluators, and let's take a look at them as we wait for them to run.
In our code just now, we defined a custom code evaluator, uh, and that evaluator we manually attached onto this particular experiment because we were interested in it. We also have the ability to define these auto-evaluators, which will automatically run on every single experiment that's run over this data set. This is useful because for these auto-evaluators, you don't have to add them, uh, every single time you run an experiment. These will always trigger and always give you feedback. There are two types of auto-evaluators: we have LLM as a judge, and we have custom code, and we have an example of each here. If I click into this C code evaluator, uh, we can take a look at what it looks like under the hood. It looks really similar to what we just defined. It takes in a run and it takes in an example, and in this case, uh, if you remember our prompt to Poly asks that Poly repeat the question back exactly, uh, and this is basically checking that it checks that the string that Poly returns, uh, starts exactly with the question from the user. We also have a correctness evaluator, uh, so what this does is it's an LLM as judge evaluator, and that means we ask an LLM to be a judge in this case and give us a score on how correct it thinks our new, uh, application's response was relative to our reference ground truth. And so if I take a look at this prompt that we're passing to the LLM, we can see that we give it three pieces of information: we give it the original input, we give it our submission, which is the latest output generated from running our model just now, and we also give it the ground truth reference to compare to. And we can see we do this variable mapping up here where we map input to input, submission to output, which is what our application generates, and then reference to reference, which is our ground truth output from our data set. We then make use of structured outputs; we ask that the LLM gives us back a score from 1 to 10 that is a measure of correctness.
Cool. And so it looks like these evaluators are still turning away here… nope, actually they just finished. And if I zoom out, I can see I have correctness and didRespond scores as well as repeatsQuestionExactly scores, and these are the three evaluators that we've just run: two auto-evaluators and one custom defined through the code. You can see I've run this experiment three times, uh, using the exact same application but swapping out different models like 35, 40, and 40 mini, and we can compare the, uh, performance of our application, uh, directly in this table, and we can see how it actually looks like 40 mini has performed the best across the board. If I want to take a closer look at how we did on each individual example, I can compare multiple experiments from this view. So let's take a look at 40 mini and 35. We can select a metric to compare by, and so now we can see that for correctness for our inputs and for our reference outputs, uh, I can see how we scored on each actual example, and here comparing 40 mini and 35, I can see that 35 actually did better on this example but did worse on four others, and I can see their responses right here in this view. Cool.
To recap, in this video we walk through data sets, we walk through experiments, and how we can use experiments to get empirical metrics on comparing our applications performance. Thanks for watching.