Transcription
Hello again! This is Leno Tadros.
In this video, we're going to try to help the LLM a little bit. So far, I can ask any questions. These are very powerful GPT models from OpenAI. But what if I tell it, for instance, I would like to find out what the weather is tonight in my city, Orlando, Florida? Or what was the score of the football game last night?
Well, unfortunately, they do not have that information because they were trained with data from several months ago, or some of them are from a year ago. They don't have access to that. They are polite, though; they will tell you, "I don't have that information, but I recommend going to the weather.com channel or the awe.com website to get this information." But I don't have it for tonight because I was not trained with that information.
So, is there any way for us to help it out a little bit? Yes, we can! Let me go ahead and show you how to do that.
So here we go again. Let's go ahead and change the question and put something like, "What is the temperature in Orlando, Florida tonight?" All right, let me go ahead and save that, and I'm going to send this to the LLM directly. Let's see what it will come up with.
Indeed, within a couple of seconds, the output will say, "I am unable to provide real-time weather updates as my knowledge only goes up until October 2023, and I don't have live data access. However, you can check these websites." It's exactly what we talked about; it doesn't have this current information at all.
So, all right, let's go ahead and find out what we can do about that to help it out a little bit. I want to introduce you to a website called Ser API. If you go to serai.com, this is a very famous website that can actually use AI to get you information, like tools to be able to search Google and Bing and other things. It's very nice to have. You can register for free; they will give you some amount of money to play around with just to check it out.
Of course, you can actually pay to get a lot more tokens to be able to use it, but you can definitely do this exercise for free as well. In Ser API, I already registered my account, and when you sign in, the one thing that you'll see there will be a private key.
Don't worry about it; by the time you see this video, I would have regenerated this key, so it's not a big deal. But that's the only thing that I'm really going to be interested in, so I'm going to copy that to the clipboard.
Let's head back to our foundation, AZ Foundry, to be able to create a connection.
All right, where do I create a connection in the system? For that, you'll notice in the project itself, all the way at the bottom, there is a Management Center. Let's click on that Management Center. These are all my current connections in the system. I want to create a brand new connection.
There are a lot of great ones in here for another service, for OpenAI, and Aurei Service as well. We're going to do a video on each and every single one of those in this series. But if you go down a little bit, you'll notice that Ser is one of the resource types that are available in here as well.
I'm going to click on Ser. Let me go ahead and paste that API key we just got from the Ser API website. I'm going to paste it right there, and then we can give it a name. I'm going to call it "Ser connection." You can call it, of course, whatever you want.
We say "connection." There you go, and we'll share it with all our projects. I don't have to enter this in other projects in the future as well. I'm going to say "add the connection."
All right, from that point on, I will have a connection added to my list of connections available for the project as well. Excellent! Let's head over to our project.
All right, we are in a project now. Again, let's go ahead and open up the same prompt flow one more time. This time, let me bring it up a little bit. I don't want to go directly from my input, from my question, straight to the LLM. Before I go to the LLM, I want to help it a little bit.
So, I would like to make a call into the Ser API first to get information about what the current weather or the temperature in Orlando tonight is, or the game from last night—what was the score? Then that information will be passed to the LLM to format it, to do whatever you want with it, and then you will get the output. That is the way we're going to do it for this exercise as well.
All right, so how do we do that? Well, first of all, make sure that the compute session is running, and then more tools will be available. Notice Ser API is one of them as well. We're going to create a video for every single one of the ones that you see in the more tools as well.
But for right now, let's take a look at the Ser API. I'm going to give it a name; we'll call it "Ser search." All right, you can call it again anything you want, and we'll say "add."
This one will take a lot of different inputs in here. The first one is the connection. Hopefully, you already created the connection, so there is my Ser connection that is available in the project. Then you get to choose between Google and Bing; it's up to you. You can try them both to see which one works best for you.
The location here is where the search will be originating from: United States, Europe, Australia, whatever you would like. Also, how many results will come back from Google or Bing? The default is 10, so I'm going to leave it at 10; that's fine.
Then the query—where is the query coming from? Well, the query is going to be our question, right? What is the temperature in Orlando tonight? So I'm going to say this will come in from the input question. There you go.
If you'd like to turn on safety as well, you can turn this on or active so that we can actually make sure that the information being passed in and out can adhere to the safety for self-harm, violence, sexuality, and all of that good stuff. Sounds good? So I'm going to save all of this, and then we need to go back to our LLM.
There is the LLM in here, so I can fix the graph, of course. The first thing is the question: where is the question coming from? It is coming in from the input question. Great!
So we're missing something in here. How can we get the LLM to use this information that we searched using Ser API? Well, I'm going to first of all need to change my system prompt. So you're a helpful assistant; that's great.
Then I'm passing you the question, so I need to come in here, maybe put a comma, and say, "Only answer based on the information provided." That's it! That means I will need to bring in some context, some information. You can call it whatever you want. I'm going to call it in here "information." Some people call it "context." Whatever is pleasing to you, that's fine as well.
Then I'm going to put here the double brackets and put inside here the variable that I want, which is "information." Again, this could be also "context" if you want. Then I need to see that information as part of the input, so I need to validate and parse the input.
Then we get an extra one at the bottom, and here, once it's correct, there is the information. So first of all, I'm telling the LLM that you're getting the questions from the input question, but the information is going to be coming in from the Ser API output.
Excellent! So now that fixes my graph. That means my questions get asked; it goes first to the Ser search, which will go with my API key as that question. It will get some information back from the Ser search, and that is the one that will be passed on to the LLM.
The LLM is not doing a lot of work at this point because it's not going to actually answer the question; it's just going to take this information and maybe format it, do whatever the LLM can do for you. But again, the heavy lifting at this point is happening on the Ser search, but it will definitely work, and the outputs will come out of the LLM as well.
Does that make sense? All right, so let's go ahead and save all of this.
Now, let's ask the same question again: "What is the temperature in Orlando, Florida tonight?" We'll say "run." As you can see, the Ser search is completed, and now it's going to be passing this information to the LLM. It's completed, and it's successful.
I can see the view outputs in here as well. The temperature in Orlando, Florida tonight is 65°. That is great! This is something that's happening right now in Orlando, but the LLM was able to get this information from the Ser API.
I can ask, "What was the football score from the game last night?" and it will be able to tell me. Without the Ser API, the system, of course, will not be able to answer that.
You can also trace it; you can go to the trace, take a look at the flow. It took almost 3 seconds to do the whole thing. The third part—there are no tokens, remember? The tokens for the Azure OpenAI only happen on the LLM.
But it took 128 milliseconds to ask that question to Google. Whenever it got this answer, we passed it on, and we have now almost two seconds to take the information from the Ser, and it was able to reply.
I can actually see the entire stack and the tracing of the questions going in, and then I can see all the 10 different things that came back from Ser to be able to pass to the LLM. So it did a pretty good job for us in here as well.
I hope this was a useful video for you. Please make sure you click on "like" and "subscribe" so that you can actually see more videos when they come out from the system as well. Thank you!