Transcription
In this video, we're going to talk about annotation cues and Link. Within annotation cues are a great way for both developers and subject matter experts to give feedback on how your application is performing. This feedback can then be incorporated to make your application better over time.
Let's go ahead and create a new annotation cue. We're going to enter a name, which could be maybe "poly correction analysis." Uh, we could optionally give a description, but in this case, we don't have to. And we'll go ahead and select a default data set. This will just make it easier for us to promote traces or examples to this data set from our annotation cue. We also have the ability to configure a number of viewers per run. In our case, one is okay because I'll just be the reviewer. Uh, and you we also have this concept of reservations that allows us to lock a review uh for a certain amount of time for a particular reviewer to look at. Cool. Let's go ahead and create our annotation cue.
And now that we have this cue, let's go ahead and add a few traces to it. So let's now get over to our uh Linksmith onboarding tracing project. And here I just have a few recent traces, and I can go ahead and add these both to the annotation cue. Cool. So I've added these traces to the annotation cue. I did this by selecting in the trace UI and then adding from down here, but you can also do this for individual traces by clicking in and then clicking "add to," which will allow you to add it to an annotation cue.
Let's go back to our annotation cue now, "poly correctness analysis," uh, and here we can see that we have two items in the queue. And for each, we can see the inputs and the outputs. So let's go ahead and maybe look at this other example first. Uh, you can see that we have hot keys bound which make traversing the annotation cue pretty easy. The question in this case is, "Are animal crackers yummy?" And the output in this case is, "Are animal crackers yummy?" We repeat the question, and uh, Polly yes thinks they are yummy according to the facts. This is something that I think is a great answer, and so I'll go ahead and give it a human correctness score of one and also go ahead and add it to my ground truth data set because I think it was pretty good. And now I'll go ahead and click "done" and complete this example. Cool.
Now we see our queue has been reduced to one, and we have this question about pickleball. In this case, I don't actually think Polly does great. I think Polly, uh, here, um, mentions that she doesn't know what pickleball is, but then gives this sort of uh irrelevant answer about animal crackers. I also think uh Polly didn't exactly repeat the question verbatim; she swapped in "I" instead of "you." And so what I can actually do here directly in the annotation cue is edit the output that we received. This will allow me to uh make the changes necessary to turn this into a golden ground truth example. So now, once I've created my edits, I can go ahead and add this to my golden ground truth data set and complete this example. Now, if I go back to my golden data set, I can see I have some new examples, including the question about pickleball. And if I click into this example, I can see that we have this updated content, which I, as a human, was able to intervene and fix.