Transcription
In this video, we're going to talk about setting up automations in LinkSmith. Automations are executed live on traces as they come in from your application. Today, we'll take a look at the different types of automations that we can set up, uh, one of which is online evaluations.
Cool. First, let's take a look at these two traces which I ran recently. There are two questions that I asked to our parent named Paully, and we can see that for this one, Paully does a pretty good job. She likes playing soccer; this was in her list of facts. However, for this one, I asked, "Do you like animals?" and Paully actually says that she likes animal crackers. This was kind of a tricky one on purpose. Paully has a fact that says that she likes animal crackers, and we can see in this case that she kind of uh hallucinates and doesn't really answer the right question.
Cool. Let's take a look at automations. Uh, to do that, we'll click on this rules button in the upper right-hand side, and we can see two existing automations that I've already set up. Uh, but let's go ahead and click "Add Rule" and sort of walk through the different kinds of automations we can set up and how we can set them up. So first, we can name our new automation rule. Then we want to set a series of run filters, and the run filters are applied to determine which runs in this table are actually applicable for this automation. You can filter on a lot of different things; uh, a common one might be status, and maybe you only care about traces where status is Success. Uh, so you can filter on that and the fact that it's a root run. There's also this uh sliding bar that allows you to set the sampling rate. So for some automations, you'll want to sample every time, but for other more expensive ones, such as LLMs as a judge, you might only want to sample for some subset of the automations, and this basically just allows you to tweak that. We also have this button that allows you to apply this new rule onto past runs, and so if I do toggle this on, I just need to select a date to backfill from, and it will start this rule automation at that date and uh apply it there.
Cool. Let's walk through our actions. Uh, there are a few basic ones like we can automatically push a trace to an annotation queue, and similarly, we can automatically push a run to a data set. We can also specify LLMs as a judge evaluators or custom code evaluators to conduct online evaluation. Online evaluation, uh, like these automations, is applied to your run at runtime, live as it comes in, and so this is a really great way to get feedback on your application immediately, uh, whether through an LLM or through a code evaluator. Setting up the evaluator looks very similar to what we did for offline evaluation. We can uh use an off-the-shelf evaluator, or we can create a prompt from scratch, and we just wire up various components from our application to the actual prompt that we pass the LLM. The same thing goes for the custom code evaluator, except in this case we only have access to the Run object, uh, and we can do different things like check that particular part of your run matches an expected value.
Cool. You can also trigger webhooks from uh these automations. A pretty common flow might be if you get a certain type of error, you might want to trigger a webhook that will create a Jira ticket or uh ping someone to take a closer look. This is a great way uh to do that through automations.
Cool. We're not actually going to add a new automation here. Let's take a closer look at these two automations that I have set up. One is called "Score Answer Relevance," and the other one is called "Add Relevant Answers to Annotation Queue." So the first uh evaluator here uses an LLM as a judge online to score the relevance of the answer that we provided for the user's question. The second one takes this feedback score, and if it is high enough, it will automatically push that run into an annotation queue for a human to look at. This is pretty neat because these two evaluators are chained in a sense. First, we need this online LLM as a judge to give us a feedback score, and then based on this feedback score, we might add it to an annotation queue. There's nothing special that we did here to allow them to be chained in this sense. For our evaluators that generate feedback, those runs with the new feedback are then run through these rules again to make sure that any uh rules that are triggered off of those specific pieces of feedback will fire.
Cool. Let's take a closer look at "Score Answer Relevance." If I go into this rule, edit it, I can see it's an LLM as judge evaluator, and if I click into the prompt, I can see all that I've done is I've mapped in my input and output, and I've asked the LLM to return me a structured output score of relevance. Uh, this is going to be a number from 1 to 10, and so all of my answers are being scored uh from 1 to 10 on relevance. As my sampling rate here is one, I can then take a look uh at my logs for this particular automation, and I can see it recently ran on these two traces right here. So for my question of "Do you like soccer?" the answer was quite relevant. If I take a look at the run, um, once again, we can immediately see uh that "Yes, I like soccer" is is pretty relevant. However, uh, this other run here, which we also just took a look at, where we ask if it likes animals and Paully says that she likes animal crackers, is not super relevant, and so this got a feedback score of three. Note that this feedback that we get is available under the auto evaluator source, and the key here is provided as well as the score in this feedback pane in the trace, so you can always go back to the trace to see this feedback.
Cool. Now let's make sure that our auto evaluate ref fired as our final step here. If we navigate over to our annotation queue, "Paully Correctness Analysis," and we wait for it to load, we can see that we did in fact get this "Do you like soccer?" uh question here, and now a human can come in, evaluate it, potentially edit it, and then potentially add it to a data set. Thanks for following along.