Transcription
Hi, I'm David from LangChain. Today, I'll be walking through how you can both build and evaluate a customer support agent.
Before filming this video, I built a customer support agent using LangChain and ported it over to LangChain Studio, which is where we are now. LangChain Studio is a specialized IDE where you can interact with, debug, and visualize your agent. The customer support agent I built is designed to assist customers of a digital music store. This customer support agent has access to a SQL database that not only has information on the products and the offerings of the digital music store but also information on customers who have shopped at the store in the past and details on their past purchases.
This customer support agent has two core functionalities. The first is the ability to answer questions about the products and offerings of the store. This is questions like, "Do you have any songs by Amy Winehouse?" "Do you have any albums by Pink Floyd?" The second functionality is the ability to handle and process customer refunds. Each of these functionalities is handled by a separate subgraph in our architecture, and we can see here we have this question-and-answering subgraph that handles the queries related to the products and offerings of the music store. We have the refund subgraph that handles and processes customer refund requests, and we're routed to each of these subgraphs by our supervisor or intent classifier node. So, when a query comes into our graph, the intent classifier or supervisor node determines whether it should be routed to the refund subgraph or the question-answering subgraph. The last node we have here in architecture is called "compile_followup." All it does is clean up the state of our agent and return a nice final output to the user.
Let's actually ask our agent two questions and see an action here. So, I'm going to go ahead and copy and paste a predefined message I have here. The message is, "My name is Mark Phillips; here's my phone number so you can identify me in the SQL database. I want a refund on a past purchase." When we click submit, our intent classifier node will correctly route it to the refund agent. And if we scroll the way down, we'll see that our customer support agent correctly responded and said, "All right, Mark, which of the following purchases would you like to be refunded for?" And we return a list of invoice IDs in the database that are associated with his, his, his name—first name, last name, and phone number. And now Mark can follow up with the invoice IDs that he would like a refund for and walks you right to the SQL database and execute that refund for him.
For the sake of time, I'm not going to ask that follow-up. I'm actually going to ask another question that will be handled by our question-answering subgraph. So here I'll say, "Never mind, do you have any songs for sale by Amy Winehouse?" Great. Now, when I hit submit, you'll see that my agent correctly routed the question over to the question-answering subgraph. And when I scroll the way down, we can see that my assistant returned the following answer: "Yes, we have several Amy Winehouse songs available from our albums *Back to Black* and *Frank*," and it returns 12 songs from the album *Back to Black* and 11 from the album *Frank*. Great. So that's a really quick overview of what our agent looks like and does. Now let's dive into both building and evaluating it.
Before I hop into a notebook and show off how to evaluate this customer support agent in code, I'm first going to touch on, at a high level, why evaluating an agent is difficult and why it's important. It's difficult to evaluate an agent because today's agents take a large number of steps before returning an output, and these steps are not explicitly defined beforehand by a developer. They're instead determined dynamically by an LLM or multiple LLMs. As a result, when you're evaluating an agent, you not only want to make sure it's producing a high-quality output but also that the path or the trajectory that it followed when constructing that final output is optimal. And we can use our customer support agent here as an example. So if I were to be evaluating this customer support agent and it were to receive a query from a user, and let's say the question is, "What songs do you have by Amy Winehouse?" I'd want to make sure that my customer support agent not only produces a high-quality, accurate output but also that it routes the query to the correct subgraph—in this case, the question-answering subgraph—and within this subgraph we call the right tools, and we call them in the right order.
Now, it's important to evaluate both the final output and the trajectory or the steps that the agent takes because you can have a world in which the output is high-quality and the trajectory was very unoptimal and inefficient. For example, you can imagine a world in which we receive a query about Amy Winehouse, our supervisor node or our intent classification step correctly routes the question to the question-answering subgraph, but within the question-answering subgraph we call one tool an unnecessary number of times, and we actually call some incorrect tools. We will still likely return an output that's accurate in quality, but the inefficiency that took place within the question-answering subgraph led to a lot of unnecessary latency and a lot of unnecessary number of tokens generated. And this is also a really, really good illustration of why evaluating an agent in general is so important because let's say you make a change to your agent—it can be a small change like switching to a new model provider or iterating some prompts in the application or a more substantial change such as one that fundamentally changes the core architecture of the agent—you want to make sure that this change would not degrade some part of your application's performance, whether that's latency, whether that's tokens, or the quality of the final output. And you want to make sure you catch this potential regression before you push to change to production and make it live for all your users.
Great. So let's actually evaluate this customer support agent. And to evaluate this customer support agent, let's use three strategies. The first strategy is going to evaluate whether it's producing quality, accurate final outputs. The second strategy we're going to utilize is actually going to make sure that a single step in our agent is behaving correctly. In this case, the evaluation is going to make sure that our supervisor node or our intent classification step is correctly routing to the appropriate subgraph. So, for example, when the supervisor node receives a query about a customer refund, we'd want to make sure this node correctly routes to the process refund subgraph. And if it receives a question about the products or the offerings of the music store, we want to make sure this intent classification step or supervisor node is correctly passing it on to the question-answering subgraph. And the final strategy we're going to use is going to actually look at the entire trajectory or path that the agent follows when answering some query and ensure that it matches the optimal or ideal trajectory we'd like it to follow. So we're going to run these three evaluation strategies using the LangSmith SDK.
Before we're able to run these evaluations, we need to define three things. The first is a golden data set, and this golden data set is going to vary based on the strategy of evaluation we're using. For example, if we're putting together a golden data set for the final output accuracy evaluation strategy, we're going to want to make sure our golden data set has some number of inputs and outputs. The inputs are going to be example queries or questions that a customer will ask our customer support agent, and our outputs will be really high-quality, accurate responses that we'd love to see our agent make in production. Now, if we were instead conducting that single-step evaluation strategy, we'd want our golden data set to still have inputs that are questions or queries that could come from a customer to the agent, but now we want our outputs to be the expected subgraph that the intent classification or supervisor node would route to when it receives that query. Right. So this golden data set will vary based on the evaluation strategy, but at a high level the inputs will be questions from customers to the agent, and the outputs will be the expected behavior or response from the agent. The second thing we're going to define is our application logic. So, for the sake of this demonstration, the application logic we're evaluating is just the customer support agent I've built with LangChain. And the last thing we're going to define here is our evaluators. Our evaluators are also going to vary based on the evaluation strategy we're employing. If we're conducting an evaluation on the accuracy of the final output, we'd likely want our evaluator to be some LLM judge that can assess the accuracy of our agent's output relative to some reference, some reference or ground truth output. If the evaluation strategy we're employing is single-step, right, to assessing whether our supervisor node is routing queries to the correct subgraph, we'd want to maybe bring in some Python code and make sure that whichever subgraph our agent routed a specific query to matches the ideal or accurate subgraph. Awesome. And all three of these components are going to tie together when we actually run our experiment or evaluation with the LangSmith SDK. When we run our evaluation, LangSmith SDK will take our application logic and run it with the inputs from our golden data set. This will generate some new outputs that will then get scored relative to the outputs in our ground truth or golden data set by some evaluators. And so let's actually walk through some specific examples.
So if we were to be evaluating the final response of our agent for accuracy, what will happen when we run our experiment is we'll take the application logic or our customer support agent, run it over the inputs of our golden data set, which are just queries—um, that we defined earlier—queries or questions from customers. These new outputs will be scored by an LLM judge. This LLM judge will basically assess how accurate these new outputs generated by our agent are relative to the reference outputs in our golden data set. The next evaluation strategy is the single-step evaluation strategy. So when we run this experiment with LangSmith SDK, again we'll be running the customer support agent over the inputs in our golden data set, and instead this time our evaluator will be some Python code that just makes sure that in our runs—the runs that were made by our customer support agent—that we routed to the correct subgraph. And what we'll basically do is we'll just double-check that whichever subgraph we routed to in those runs by our customer support agent matches the name of the subgraph in our reference output inside of our golden data set. And our third and last evaluation strategy is trajectory. Again, we'll be running our customer support agent over the inputs of our golden data set, and within these runs we'll make sure that the trajectory the agent it followed or the path it followed matches the path we defined in our reference output, which is optimal and something we'd love to see in production. Great. Now let's actually hop in the code and show off how you can implement this evaluation with the LangSmith SDK.
Great, we're in the notebook. Before I go into the evaluation code, I'm first going to go over very briefly the code that underpins the customer support agent. Now, this video is primarily focused on evaluating the agent instead of building the agent, so I'm not going to walk through this code line by line, but I'll attach this notebook to the description of the video so you all can go deep into it on your own time. As you probably already saw from the architecture and LangChain Studio, this agent has three main components. This agent has a refund subgraph. This subgraph handles queries related to customer refunds, and inside of this refund subgraph there are three nodes. The first node, "gather_info," prompts the customer for additional information if we don't have enough context on the customer to execute and process a refund. The second node, "lookup," is executed when we have enough information to identify the customer in the database but not enough information to actually process a specific refund. So what the "lookup" node will do is take the customer's identifying information—this is the customer's first name, last name, and phone number—and read from the database using some SQL all of their associated invoices or purchases with the music store in the past and return a message to the user that says, "Hey, so-and-so, we were able to find your records in the database. Here are the invoices and past purchases we've seen you've had with the store. Which ones would you like a refund for?" And once this refund subgraph gets that context, we're able to use the last node, which is the "refund" node, and this node writes to the database using SQL to process a customer refund once we're able to both identify the customer and have a specified invoice ID that they would like a refund for. So that's the refund subgraph. Again, not going to go through the code, so just going to scroll straight through here. Feel free to look at it on your own time.
The second component is the question-answering subgraph. This subgraph is meant to handle queries related to the products and offerings of a store, and this subgraph follows the React Agent architecture. If you aren't familiar with what the React Agent architecture is, view the link I'm attaching to the description of this video. But all the subgraph is is an LLM with access to certain tools. These tools allow it to read from the database using SQL and thus answer questions such as, "Hey, what songs does the store have? What albums are you selling? I'm looking for a song by this artist. What, which songs by them are you selling at the store?" And these two subgraphs are pulled into another graph, and the way that we actually end up invoking either of these subgraphs is through this intent classifier or supervisor node that we have at the top of our architecture. And so this intent classifier supervisor node basically looks at the query that the customer asked, determines whether it's about a customer refund or the products and offerings of the store, and routes the query there. And we already covered this, but the last node just cleans up our state and returns a final message to the user. Great. So that's the general gist of our customer support agent. Feel free to hop in the code in the description linked below. And now I'm actually going to dive into evaluating this agent with LangSmith SDK.
Let's evaluate our agent with LangSmith SDK. If you're not familiar with LangSmith, it's a platform that helps you build production-ready LLM applications. Discussing LangSmith in detail is a bit out of scope for this video. Instead, I'll just link some docs and resources below. So if you want to learn about LangSmith, you can. The first evaluation we're going to run will assess how accurate the answer generated by my customer support agent when it receives some query from a customer is. And to run this evaluation, we're first going to need to create a golden data set. And as you can see from my code, we're manually creating this golden data set here, and I've already defined five input-output pairs that this data set will be composed of. In each input-output pair, the input will be an example question that a customer would pose to our customer support agent, and the output is going to be a high-quality, accurate response that we'd love to see our agent produce in production. So in this first input-output pair, we can see the input is a question: "So how many songs do you have by James Brown?" And the output is an accurate response to that question from our customer support agent. So here the response is, "We have 20 songs by James Brown." All right, we'll be sending this to LangSmith as a data set under the name "chinook_customer_service_final_response." "chinook" is the name of the database that our customer support agent is hooked up to. We'll be sending this data set to LangSmith because LangSmith is where we'll be able to view and interact with the results from our evaluation. All right, great. So that's our golden data set, and we've sent it off to LangSmith. Now let's move on to step two, which is defining the application logic that we want to evaluate. So because we're just evaluating our customer support agent in this evaluation, what we need to do is define a target function called "run_or_graph." "run_or_graph" will receive the inputs from our golden data set when we kick off our experiment using LangSmith SDK. And what this target function will do is just pass in the inputs—for the example customer questions from our golden data set—to our customer support agent, denoted by "graph" here. Our customer support agent will take this query or question, generate a response, and all we'll do is return the response. And we'll basically be plucking the state value that we have for our agent called "follow-up," because it's inside of "follow-up" that will actually—should be storing that response that we that we would return to a user. Great. So that's the application logic we're about to evaluate. And now let's define our evaluator. So it's similar to how the target function that houses our application logic automatically receives the inputs from the golden data set when the experiment is kicked off using LangSmith SDK. Our evaluator will automatically receive the inputs from the golden data set, the outputs generated by the target function or my customer support agent, and the reference output from the golden data set. And our evaluator, which is defined inside of this function "final_answer_correct," will take all three of these pieces of data and give them to an LLM judge. And this LLM judge has a system prompt which instructs it to act as a teacher grading a quiz. We inform this LLM judge that it will be given a question—in this case, the question is just the input from our golden data set—the ground truth or correct response, which is just the output from our golden data set, and the student response, which is just the answer generated by my customer support agent inside of the target function. We'll give this LLM judge some more granular details on what denotes a correct student response and what denotes an inaccurate student response, and then we'll actually invoke this LLM judge using structured output. And we'll be using structured output to ensure that our LLM judge returns two things: the first is a Boolean value. This Boolean value will be true if the LLM judge decides that the student response is mostly or exactly correct compared to the reference output or the output from our golden data set, and false if it decides that that response generated by our customer support agent is not accurate or not correct relative to the output from our golden data set. And our LLM judge will also return another field that's a string. The string will just explain the LLM judge's reasoning or rationale for why it decided that our customer support agent answer was correct or not. And so we're just going to be—we're just going to be returning that Boolean "is_correct" value to LangSmith, and we'll be able to see it on the UI after we kick off this experiment. So the last step is to actually run the evaluation, and we'll run the evaluation using LangSmith's evaluate function. And inside of this evaluate function, all we need to do is specify our target function, "run_or_graph," which houses our application logic or our customer support agent. We'll need to specify the name of our golden data set, a list of the evaluators that will be used to score the new outputs generated by our customer support agent—in this case, it's just the "final_answer_correct" evaluator we defined up here—and the name of the experiment. And this will be present on the LangSmith UI as well, which is where we're actually going to jump to now.
All right, we're in LangSmith's UI. The first thing we can note is that our golden data set was successfully created. So if I click into this data set in LangSmith and toggle over to examples, we can see that the inputs and the outputs that we defined in code were successfully sent over to LangSmith. So let's click into an example input and output. So the input is just one of the questions we defined in code: "How many songs do you have by James Brown?" And the output is that high-quality, accurate answer we would like our agent to return when given this question. And so here we have the response: "We have 20 songs by James Brown." So great, our golden data set was created, and now let's go take a look at the experiment that we ran. So if I click in, I can view this experiment in pretty granular detail. I'll see the inputs and the outputs for my golden data set. I'll see the output that was...
Generated by my target function or by my customer support agent when it received the input for my golden data set, and I'll see the score that my LM judge gave this new output by my customer support agent for accuracy. Now, if you remember in code, we were having our Allen judge return a Boolean value, true or false, for whether this new output was accurate or inaccurate. So here, the true or false Boolean value is just showing up as Z or one. One is true, zero is false. So here we can see that when our customer support agent was given this input from the Golden data set in the Target function, it did a really good job at producing an accurate response, and thus our LM judge gave it a one or true for being correct. We can see down here our customer support agent struggled when given this question about who recorded "Wish You Were Here" again, and thus our judge returned false, or that its response was not accurate. We can scroll through and see what our LM judge scored everywhere they output here as well.
One thing that's really nice about this experiment view is we can actually click in and look at an individual evaluator run. So when I click into an evaluator run, I can see what our evaluator looks like under the hood. So just like how it was defined in code, we can see that we added an LM judge, in this case, Chat GPT-4 mini, with a special system prompt. The system prompt instructs it to act as a teacher grading a quiz. And we can see that we give this LM judge a human message that has context on the input from our golden data set, the output from our golden data set (which is referenced as a ground truth response), and a student response (which is just the answer that our customer support agent generated in the target function). And then, at the very bottom, we'll see that response from our LM judge, which includes both the reasoning or the rationale for why it either judged our response as accurate or inaccurate, and the Boolean value for whether it decided our output was correct or not. Great.
So, talking back over to the data set, one other thing that I think is super helpful is viewing our experiment at sort of a high level. So here, I can see that across all my runs, the average correctness score was 0.6. So ideally, we'd like this to be one, meaning all of our runs received a correctness equal to True by our LM judge. We can see that's not the case, and thus we have a good bit of work to do before we release this customer support agent to all of our users. LangSmith will also provide us some pretty nice information on the p99 latency and error rate across the runs in the experiment. Let's move on to our second evaluation strategy: single step.
In the single-step evaluation, we're going to be assessing how well my supervisor node, or my intent classification step, performs when given a question from a customer. Ideally, when we receive a question from a customer, we'd like our supervisor node to always correctly route it to either the question answering subgraph or the process refund subgraph based on the content of the question. To run this evaluation, the first step is going to be again to define a golden data set. I'm defining this golden data set manually in code, and I have five inputs and outputs already defined here. The inputs in my golden data set are just going to be a list of messages. Sometimes this list of messages is just going to have a single message from a customer. This message from a customer will have a question. So, in this example, in the input, it's just going to be a question from a customer that's asking, "Hey, I bought some tracks recently, and I don't like them." The outputs in my golden data set are just going to be the names of the subgraphs or the agents that we would like our intent classifier or supervisor node to route to when it receives this list of messages as context. So again, sometimes the inputs are just going to be a single ask from a customer, and sometimes it's going to be a list of multiple messages. So in this example here, the most recent message is still going to be an ask from a customer. So here, the customer is asking, "Did Prince release any albums in 2000?" But the list of messages also has two previous messages that were exchanged back and forth between the customer and the customer support agent. So the first message is the customer asking, "I want a refund on purchase 237," and the second message is the customer support agent's reply, which said, "I've refunded you for that purchase. It was a total of $1.98. How else can I help you today?" And again, the output is going to be the name of the subgraph or the agent that we'd like our intent classifier or supervisor node to route to when it receives this ask and these two previous messages as context for what already took place in the conversation. So we're going to create a data set with these inputs and outputs and send them off to LangSmith.
The second step to running this evaluation is going to be defining the application logic we want evaluated. And so, for this evaluation, because I'm actually not evaluating my entire agent, I'm just evaluating the performance of a specific step, I don't need to run the inputs for my golden data set, which get passed to my target function. When we run the experiment, I don't need to run these inputs from my golden data set on the entire graph. I can actually take the inputs from my golden data set, which gets sent to my target function when the experiment runs, to just the intent classifier node. And when I get a response back from the intent classifier node, I can use the goto property to get back the name of the subgraph or the agent that it wants to route the query to next. So that's what my target function is actually going to do in this single-step evaluation. Next, I'm going to define my evaluator. So my evaluator is going to receive the outputs, or whatever my target function returns when given the inputs from the golden data set, and the reference outputs, which are the outputs from our golden data set. And my evaluator here is much simpler than the one in the final response evaluator. All it does is a simple check: Is the subgraph that my support agent wants to route to when it received that list of messages equivalent to the name of the subgraph or the agent that we have in our golden data set? We're making this check because the name of the subgraph or the agent in our golden data set is the correct one. So we want to basically gauge that our agent picked the right subgraph to route to. Great. So now we've defined all of these. The last step is to actually run the evaluation with the evaluate function. Again, we're going to specify the name of our target function, which houses our application logic. Again, here we're not actually going to be running the entire graph; we're just going to be picking out that intent classifier node, giving it the inputs from our golden data set. We're going to define the name of our golden data set, the list of evaluators, and the experiment prefix, which will show up in the LangSmith UI. All right, we're back in the LangSmith UI.
The first thing to note is we successfully created our golden data set. We can click into any input/output pair and see that the input is as expected; it's a message or a list of messages that includes questions from customers. Here we just have one question from a customer, and the outputs in the golden data set, instead of it being a high-quality, accurate output, this time it's just the name of the subgraph or agent that we wanted our intent classifier to route to when it received that message or list of messages as context. Let's toggle over to experiments and look at the experiment we just ran under the hood. So in this view, we'll see the inputs and outputs from our golden data set, and then we'll see the name of the subgraph or the agent that our intent classifier node inside of the customer support agent wanted to route to when it received these messages as context from our golden data set. So it's done a really great job; for each input from the golden data set, it routed to the correct subgraph or agent. We see across the board that our evaluator gave it the score of one, which means it picked the right subgraph to route to. Awesome.
Let's dive into our third and final evaluation strategy. In this evaluation, we'll be assessing whether our customer support agent is following an optimal path or trajectory when it's answering a question from a customer. The first step we're going to take when conducting this evaluation is creating a golden data set. As you can see from my code, I'm manually creating a golden data set here, and I already have defined five inputs and outputs that this data set will be composed of. In each of these input/output pairs, the input is going to be an example question that a customer would pose to our customer support agent, and the output is a list of steps that we want our customer support agent to follow in order when answering the question in the input. Let's use this first input/output pair as an example. So in this input/output pair, the input is a question that says, "How many songs do you have by James Brown?" And the output is the trajectory that we'd want our customer support agent to follow when it's answering this question. So when this customer support agent receives the question, "How many songs do you have by James Brown?", we'd ideally like it to route to the question answering subgraph from the intent classifier node, and then within the question answering subgraph, we'd like it to use the lookup track tool before routing to the compile follow-up node, which just cleans up the state and returns a nice answer to the user. We're good to send this golden data set to LangSmith under the name "Chinook Customer Service Bot Trajectory."
Great. So now that we have a golden data set, let's move on to step two, which is defining the application logic that we want to evaluate. To do so, we're going to need to create another target function called `run_or_graph`. When this evaluation is kicked off, the LangSmith SDK will automatically send the inputs from our golden data set to this target function, which we're then going to pass to our customer support agent (which is denoted by `graph`). And what we're going to do is, as this customer support agent is answering this question from the inputs variable in the data set, it streams back the events, and we're even going to be able to stream back the events from our subgraphs because here we're setting `subgraphs=True` and `stream_mode=True`. And as these events are streaming back, what we're going to do is append the names of all the nodes and tools that we utilize to an array called `trajectory`. And so, at the very end, when we return this `trajectory` array, we'll have the names of all the nodes and the tools that we routed to and used when responding to that question from the customer. Great. So now we have our golden data set and the application logic we want to evaluate. Let's move on to step three, which is defining our evaluators. Up until this point, each of the two evaluation strategies has just used a single evaluator. In this trajectory evaluation, I'm actually going to use two evaluators. The first evaluator is called `evaluate_extra_steps`. What it'll do is it'll fetch the outputs, or the `trajectory` array that gets returned from my target function, and the reference outputs from my golden data set (both of these are automatically passed to it by LangSmith's SDK when we kick off the experiment), and it'll just check if the trajectory that my customer support agent took in the target function has additional steps compared to the trajectory we defined in our reference output. And if it did take some extra steps, we'll return it here. And the reason we're checking for extra steps is, if you remember from earlier in this video, we want to make sure our customer support agent doesn't take an inefficient trajectory. So we want to make sure, for example, when we get routed to the question answering subgraph, we don't call one tool an unnecessary number of times, and we don't call a wrong tool. Our second evaluator is going to be called `evaluate_unmatched_steps`. It's also going to automatically receive the outputs, or the `trajectory` arrays that were returned by my target function, and the reference outputs for my golden data set. And this `evaluate_unmatched_steps` function is going to check if our customer support agent called some tools or took some steps that are not in the trajectory we defined in our reference output. And this check is going to be order-sensitive. So if our customer support agent followed the trajectory step one, step two, step three, but the ideal trajectory that we defined in our golden data set is step three, step two, step one, we're actually going to say that our customer support agent had unmatched steps, because even though the names of the steps are correct and match the ones in our reference output, the order is incorrect. And so we're just going to return the number of unmatched steps here under the key `unmatched_steps`, and we're returning the extra steps under the key `extra_steps`. You'll see both these keys and their associated values in LangSmith CI. Our last step is to run the evaluation using the `evaluate` function. We're going to specify the target function here, the name of the golden data set, the list of evaluators that we're using, and the name of the experiment, which will show up on the LangSmith UI. And we're going to hop there now.
All right, we're in the LangSmith UI. The first thing to note is that we successfully created our golden data set. So here I'm in our data set, "Chinook Customer Service Bot Trajectory," and we can see that the inputs and the outputs that we defined in our code were successfully sent over to LangSmith. So I'll click into this example input/output pair from our golden data set. We can see the input is just that question we had defined in code: "My name is Aaron Mitchell, and I'd like a refund on my Led Zeppelin purchases. Aaron also provides his phone number so we can identify him in the database." And we can see in the reference output the ideal trajectory that we defined in code was also sent over successfully. So the ideal trajectory we'd want our agent to follow when it receives a query like this is to first get routed to the refund subgraph, then get routed to the `GatherInfo` node, then get routed to the `Lookup` node, and then ultimately to that last `CompileFollowUp` node. Great. So our golden data set looks good, and it got successfully sent over to LangSmith. Let's click in and look at our experiment in a bit deeper detail. So in this view, as usual, we can see the inputs and the outputs from our golden data set, and then we can see the trajectory that our customer support agent followed when it was given the input from the golden data set in that target function. And so we can see, for this input from the golden data set, our customer support agent followed this trajectory when constructing its final answer, and then we can see the two evaluators that were attached to this new output by the `extra_steps` evaluator function and the `unmatched_steps` evaluator function. So for this input from the golden data set, or this question, our customer support agent did phenomenal. You can see that it didn't take any extra steps, and it didn't have any unmatched steps. In fact, the trajectory that it followed is an exact one-to-one, or perfect match, with the trajectory that we defined in our reference output. Great, and our agent did perfect on these other three inputs as well from our golden data set. So here, no extra steps, no unmatched steps. One input where our customer support agent struggled a little bit is this input where the customer asked, "Who recorded 'Wish You Were Here' again? What other albums by them do you have?" So when our customer support agent was given this question, it not only took some extra steps, it actually took some unmatched steps. And let's look at what took place. So ideally, when our customer support agent receives a query like this, we would like to route it to the question answering subgraph, because this question is about the products and offerings of the music store. We'd like it to first call the `lookup_track` tool; so in this case, it looks up the information on the track "Wish You Were Here" again. And then, to answer this second part of the question, "What other albums by them do you have?", we'd want our agent to then use the `lookup_album` tool; so take the information it got from the `lookup_track` tool, look up all the albums associated with the artist who created "Wish You Were Here" again, and then return it to the user. Our agent didn't follow this optimal trajectory; it got tripped up and instead first called the `lookup_album` tool when it should have first called the `lookup_track` tool. And as a result, it realized it needed to course-correct, and it needed to first call the `lookup_track` tool to get the information on who recorded this song, and then call the `lookup_album` tool and be able to answer that second part of the query, which is, "What other albums by that artist do you have?" And so, as a result, it had—it took three extra steps and had three unmatched steps. Great. But other than that, right, the agent performed pretty well on this trajectory evaluation. We'll just have to make sure going forward we keep an eye on inputs like this and tweak some part of our agent so it does better going forward. Awesome.
So those are the three evaluation strategies I wanted to show off in this video, and this will be the end of the video. If you want to keep learning about building agents or evaluating agents or anything beyond, I'd really encourage you to take the Introduction to LangSmith course we've put together and the Introduction to LangChain course we've put together. You can access both at the URL academy.langchain.com/collections. This will give you a deep dive into LangSmith. We've touched a little bit on it in this video, just from the standpoint of experiments and datasets, and this will be a deep dive into LangChain, our open-source framework which helps you build agent-based applications. So I hope this is helpful, and I'll see you in the next one.