Transcription
Hello, this is Leno Tadros, and in this video, we are going to show how to use a different prompt templating engine than the one we used before, like Jinja2. In this one, we're going to use a Microsoft one called Prompty, and we will see how we can actually bring it into Prom Flow to do the job. Let's go ahead and get started.
All right, here I am in my Azure AI Foundry. The most important thing is I'm not going to start a brand new Prom Flow from here; I'm going to do it in Visual Studio Code so I can use Prompty. The only thing I will probably need from here is in the Management Center. Let's go click on the Management Center at the bottom there, and I want to open up the Azure OpenAI resource that got created.
There are a couple of things here that are important for me. The first one is the endpoint; that is the target I need. And of course, I need the API key. So, I actually moved all of these things—the endpoint and the API key—into my clipboard so I can use them. But these are very important for me so that I can actually use the engine that was created in my Azure AI Foundry hub to be able to use it from Visual Studio Code.
So, let's go to Visual Studio Code and start the process there. We are, I created a brand new folder called PF Demo. It's completely empty; there is nothing in this folder whatsoever. I would like to use Visual Studio Code, but I don't want to write all the code by myself. I would like a cookie cutter that will help me out.
So, there is an extension for Prompty available in Visual Studio Code. If I go to extensions in here, let's look for something called Prompty. Hopefully, we'll find one. There it is! If I click on this guy, it's still in preview, but we're going to go ahead and install it in here. Once it's installed and done, I'm going to close this down.
Now, when I go back to my project or my folder, I can come in here and create a brand new file. We'll call it, for instance, let's say test.prompty. Okay, and once it has an extension of .prompty, the extension will kick in right away. You will notice here it's generating a Prompty file. I can actually right-click on it, and I can do things like create a brand new prompt, add a Prom Flow code, add a semantic code, and a link chain code.
All of these things, as a matter of fact, because there is nothing that will insert in an open file, I'm going to actually create a new Prompty from the beginning so I can get some cookie cutter code. So, let me go ahead and delete this guy. We'll say delete this file. All righty, bye-bye to you.
I'm going to come in here, right-click, and I'm going to say new Prompty. All right, there is a basic Prompty. Maybe I want to call it test. All right, let's go ahead and rename it. We'll call it test.prompty; that's fine. You can call it anything you want as long as it has an extension of .prompty.
Now, this is the template that comes automatically from Microsoft. Our dear friend SE in here, we're going to say Leno Tadros is the author. I can change the name of the prompt itself; we'll call it test prompt. You can call it again whatever you want.
Then there will be some sections in here that are very important. The model in here is very important. I need to tell it if I'm going to go against the type Azure OpenAI or just OpenAI or some other type of model that I would like to use. Okay, so I'm going to leave it as Azure OpenAI. It asks me for the endpoint and the deployment name.
All right, so let's go ahead and get that. I think I have the endpoint. If I copy that from my clipboard, I'm going to go in here, and I can actually remove this line and un-hardcode the string of my endpoint itself. Of course, this is usually not a very good way of doing things. I would rather you leave this as an environment variable and just go and create an environment variable.
So, we'll say plus ENV like that, and in the ENV file, I'm going to go ahead and load the two things that I'm going to need, which are the OpenAI endpoint and the API key. If I come in here, we'll paste those, and they are right there. That is the Azure OpenAI endpoint, and the Azure OpenAI API key has to be called this way.
Then I'll be pointing to whatever I copy and paste from my project for the connection for the Azure OpenAI in Azure AI Foundry. Sounds good so far? All right, let's go back to Prompty. Let me save this file first before I forget.
I'm going to come in here. Notice there is nothing about the Azure OpenAI, and this is something that I need to do as well. So, I'm going to go right after this line. We say Azure, or I'm going to call it API key. Look at the compiler; it's trying to help me out, which is great.
Now, it's going to go get the API key from the environment, which is the ENV file. Azure OpenAI API key. Don't worry very much about this yellow line; it's actually all correct. This is still in preview, so we are good. Again, I can actually go back to my ENV file and enter an Azure deployment called GP4. That's the name that I created called GPT-40, or I can just hardcode it in here.
We say GPT-40; it's up to you, of course. If you know you're going to be changing models instead of having to go into the source code every time and change it, just go to the ENV file and change the deployment name from there. But I'm not going to use anything else other than GP4, so we're good here as well.
Then the model will have some parameters. So, the template comes with 3,000. Actually, that's too much; let me just make it 1,000. But I can actually set a lot of things. I can set the top K; I can set the parameters for the temperature. Let's say temperature, and we'll say this is going to be 0.2, for instance.
Okay, so we can bring it pretty close to how good this is. Also, I can be passing some samples for inputs. This is very similar to how we did the inputs in the Prom Flow itself. So, I'm going to create one called first name. I'm going to call it Loo, for instance. That sounds good to me.
Then the context in here, the sample comes with some information about T and so on. I'm going to delete that, and let me go ahead and grab. I'm going to go to LinkedIn and actually copy and paste my bio from my LinkedIn. So, I'm going to go in here to context and save this guy.
All right, sounds good. This is my bio from there. Then the question itself. So now I have three different inputs: the question. We'll say, for instance, tell me a bit more about Leno Tadros. All righty, good enough.
We'll say save this guy. Notice after you do this section that has all the information, or let me talk about it from a configuration perspective. I'm telling it where to go get the endpoint, what the API key is, what model I would like to use, the temperature, the maximum tokens that can be used, and I'm also giving you a sample of all the different inputs that will need to begin G.
I'm going to give it values as well, so you don't even have to pass it in as parameters when you make the call itself. It's all going to be in the Prompty file. Then I'm going to go ahead and put the system prompt: "You are an AI assistant who helps people find information. As the assistant, you answer questions briefly," and all that good stuff.
All right, this one even is telling it to go ahead and use emojis. That's the template that comes from Microsoft as well. Then I'm going to have a customer, which again, remember first name was one of my inputs, and I passed it Leno. So, I'm going to say, "You are helping Leno to find answers to these questions. Use their name to address them in your responses."
The context is this entire paragraph that I got from my LinkedIn bio in here, and there it is, the context. Finally, the question itself, which is "Tell me a bit more about Leno."
If I've done this job right and my environment is correct regarding bringing in the endpoint and the API key, I should be able—first of all, let me save this file. It's creating the meta file for that, and once it's done, I will be able to run it.
See, there is a run in here. Also, you can run it directly from the command line. You can say, "PF flow test," and pass it the name of the test.prompty in here, and it will run it. So, you can do everything in the command line, and that's important.
Even though I'm going to run it from inside of here, I have to tell you it is quite important to understand that you can run this with code because later on, you might actually want to do LLM ops. You want to do CD with GitHub or Azure DevOps or whatever you would like.
So that whenever somebody makes a change to a Prompty file or configuration files and then pushes it into a repository in GitHub or Azure DevOps or Git, whatever you want, it will automatically trigger a workflow. The workflow will end up running the flow test, evaluating it, and then go ahead and deploy it, creating a container image and then putting it in a registry for Azure Container Registry, and then deploying it to a web app or Kubernetes.
I mean, there is so much stuff that you can do by setting up everything correctly in code instead of doing it visually. Also, notice that the prompt you have something very fun in here. You see it says at the bottom, "Open test chat page, test it, batch run," and so on. That means if I click on test right now, it will do everything in the command line, so it will probably actually be successful and will answer this.
Let me actually run a test first, and then I'm going to open it up in that very, very nice user interface that got created for you by Prompty automatically. Actually, the power of the Prom Flow has that built into the SDK as well. So, let's click on test, and that will open up the command line, as you can see in the terminal, and it will run, passing my bio and the question.
I'm hoping this will end up giving us the answer directly from there. Oh, look at that! It did work. So, it says here, of course, Leno, because we told it to please use the first name in the answer. So, you're quite the powerhouse, blah, blah, blah.
Okay, thank you very much. But at the end, as a Microsoft usual error, it has a lot of information about me based on the context that I gave it already, which is definitely a very good thing. Does that make sense so far?
But look what else I can do. I'm going to go in here. Instead of just testing in the terminal, I can say open test chat page. All the stuff comes from the extension called Prompty, which makes your life much easier. I love this part. Let's click on it, and that will open it up.
It will run it in Prom Flow, and it will use the Local Host 127.0.0.1 to be able to use the user interface. See, the UI is going to bring in the user interface, so it's going to 127. Look at this; my user interface is actually opening up in here. Isn't that great?
This is running using the tracing on my machine in here locally, and I can actually go ahead and set up all my settings. What do you want the chat input to bring the question from? Notice that I have a question, a first name, and a context. These are all the three different inputs.
So, I'm going to say the chat input will be maybe the question. The chat history, I'm going to leave it empty; I didn't have a history for this one. The chat output will be output as well, which is great. How about the Prompty inputs? What is the context coming from?
So, I can actually come in here, and I can copy and paste if I would like to test it without actually using the sample that comes with the Prompty. I'm going to copy that, and I'm going to put the paragraph from LinkedIn about me right there.
Then, what is the first name? Let's say Leno is the first name. And then what is the question? Like, tell me—oops! Oh, you cannot type in here, by the way. This is kind of confusing, but if you put your cursor over it, it will tell you to specify the value of the chat input in the chat window input box.
That means you cannot type in here; you have to come in here, which makes sense. It will say, "Tell me more about Leno Tadros." All right, let's click on enter, and again, it will end up using all the stuff from the test.prompty, all the configuration and everything, but it's a visual way.
Hopefully, it will give me the same answer it did before, and it did! So, this is great. The beautiful thing about this book is that there is a view trace in here. If I click on view trace now, I will get right in the user interface. I can actually debug everything.
I can see how many seconds it took; it was 2.2 seconds and 335 different tokens. If you put your cursor over it, it will tell you the prompt tokens, the completion tokens, and the total tokens: 335. Okay, there are some bugs in there; they're working on it, but this is still in preview.
All righty, but I think it's going to be awesome. So, there is my test, the test prompt in here. If I go to the second one, I will be able to see what the input is. Let me open this up. Oh, let me show you the output in here.
But if I click on this guy, now I can see the whole thing from the input to the output. So, I can see the input is Leno for the first name, the context, the question, and the template itself: "You are an AI assistant." Then it put all the stuff together with the context and the question and the first name, and then the output—there it is.
I'm going to go back finally to the LLM piece. So, if you click on that, that is the answer. All right, I can see what the system message was, what the context was, what the first name was—all of the stuff will be available for me.
So, it will make it very easy to be able to debug this if I would like to find out. Sometimes when I'm running this, I get exceptions; something is wrong with my credentials with my key. There's also this exception tab in here, and this is a great place to click on exceptions and come read the entire stack of why you're getting an error instead of reading it in the terminal or the command line.
So, this would be a much easier way to read it in here. Also, I can see the raw JSON that came back, so you can actually take a look at the trace ID and get a lot of information that you will not see in the conversation itself.
There is the prompt template; there it is in here. I can also see the LLM parameters. I can see the tools; there are no tools. But again, in future videos, we will see how we can bring in outside tools to be able to let the system use function calls automatically, and you'll be able to see which one was used and why.
Very, very powerful. So, I hope this was useful to you to see how we can use Prompty to play in the game of Prom Flow as well, using the engine that got created in the Azure AI Foundry.
If you like this video, please click on like and subscribe so you can actually be notified of future videos as well. Thank you so much, and have a wonderful day!