Transcription
Hello again! This is Leno Tadros.
In the previous video, we worked with the Azure AI Foundry using audio. In this video, we're going to use vision and see if we can actually deal with images and explore what kind of models can help us recognize the information inside an image itself. So, let's go ahead and get started.
All right, I am back here in my portal. Let me go ahead and launch the Azure AI Foundry. I'm going to open up the studio, and at this time, I would like to see if I can go to my playground. Remember, I'm not going to use any special playground; I'm just going to go to the chat playground itself.
I want you to remember that we have GPT-4.0. We are lucky because we have GPT-4.0, which means it's multimodal—it has audio, vision, and text all inside of it. So, I don't have to actually get a specific version of a specific model, like in GPT-4, for instance, to be able to do that automatically because it recognizes this is GPT-4.0.
You notice the attachment in the bottom right of the screen that allows me to attach a file. Let's say, for instance, I already scanned a little receipt that I got from a restaurant on my phone, and I would like to see if it can actually recognize what's going on inside of there.
So, I'm going to go ahead and click on that and find out where my receipt is. I think I called it "test receipt." There you go, "test receipt." Let me go ahead and open it up. It's not very readable, and it actually has a lot of different things in it, so we want to find out what the system can actually do for me.
I'm going to say, "Can you explain the content of this receipt?" All right, let's go ahead and see if GPT-4.0, the vision portion of it, can actually figure this out for us as well. I'm going to go ahead and do that. We'll wait just a few seconds, and hopefully, it will come back.
It will say, "Certainly," and it was able to find out that it's coming in from Dixie Cream Cafe. There's the address that they got from it, the phone number, the date was November 22nd, the table number, the number of guests, the server name—everything inside of there, including everything we've ordered, and the total of $51.39, which is correct. That is what's inside of the receipt itself, even the thank you and the summary at the end.
The receipt shows that the two guests had breakfast, and it was $51.39, including tax. It did a pretty good job with that as well.
All right, so now I would like to use that in a prompt and see if I can actually do something like this with it as well. Let's go back to Prompt Flow, and I'm going to go ahead and create a brand new prompt flow in here. We'll keep it a simple standard flow; that's fine. We'll say "Create," and this one we'll call, for instance, "Receipt Flow."
All righty, and we'll say "Create." We'll wait a few seconds, and that flow is very simple. Usually, it will come back with a joke and an echo, which I don't need either one of them. So, I'm going to click on the joke to say bye-bye to you, and the echo as well will say bye-bye to you as well.
All right, can I actually do this based on GPT-4.0? Absolutely! GPT is very powerful and has a vision portion, but I wanted to show you something a little bit more complicated on this one.
So, let me save this for right now and go back to models. What happens if I try to use a model that has multiple versions of it, and only one of them includes, for instance, the vision portion, like the GPT-4, for instance, not the 4.0?
Notice if I say "Deploy the model," and let's deploy the base model in here. I'm going to look for GPT, not 4.0, but 4. Let's see... yeah, there it is, 4. I'm going to say "Confirm."
There's a lot of different versions in here. One of the things that people miss on this one is if I say "Customize," for instance, and I would like to find out where is the one that is not only text but also has vision. If you bring this down, you will notice that there is only the Turbo 20 24409 in here at the time, of course, of this recording of this video.
That's the reason because of the global standard deployment type; it only has the Turbo available for it. If I change this to make it standard, I will have access to all of them.
So, notice there's a lot of different versions. The one I'm interested in is the vision one. If you don't choose this one, you will not be able to do this with the receipt if you end up using GPT-4 itself, not 4.0.
Okay, so I'm going to say I would like to use the vision in here. We'll connect to a specific AI resource. We'll use, for instance, the West US right now. It is not available in the East US or East US 2 as of the recording of this video, so I'm going to send it to the West Coast. I think this is good enough for me, and we'll say "Deploy."
Then we will give it just a few seconds, and it's successful, as you can see, and we are in good shape.
All right, let's go back to our prompt flow and start using this specific deployed model.
So, where do I start? While we're playing around in here, let's go ahead and start the compute session because it takes a couple of minutes. While it's doing that, we can actually go ahead and change things.
First of all, what will be the input? The input will not be text; it will not be a string. So, first of all, it will be a receipt. That means it could be a picture, a photo, whatever. You can call this whatever you want. I sometimes call it "receip." I'm just going to call it "photo."
You can call it anything you want, and then it's not going to be a string; it's going to be an image right there. This is where we're going to upload the image in here because this is a completion. This is not a chat, but of course, you can make it a chat as well; that's not a problem.
And then it's not a joke, of course. The output is going to be output. Let me put that in here just to get ready for all of these things.
All right, my compute session is running, and I'm good to go. That means more tools will show up in here. Excellent! Let's click on that and look at this thing inside of preview right now as of the recording: Azure OpenAI GPT-4 Turbo with vision.
So, I'm going to add this guy. We'll give it a name. We'll call it "Vision," and we'll call it "Receipt," for instance. That's the name of my step in here. I'm going to set it up to the connection.
Remember now, I have two connections. I have the regular one that I started in the previous videos, which is in the East US 2, but there is this one for the vision for GPT-4 that has to be in the West US because there is no equivalent to it in the East Coast as of this recording.
So, I'm going to click on that. The deployment name will see that it is going to be GPT-4. I'm going to leave the temperature as one to give it as creative as possible. We'll make this 1,000 tokens; that's fine. Nothing else needs to change in here.
Then the system prompt would be: "As an AI assistant, your task involves interpreting images and responding to questions about the image. Remember to provide accurate answers based on the information present in the image."
Then can you tell me what the image depicts? I'm going to change that. That's going to be part of the user. I'm going to say, "Scan the receipt and tell me the date, the vendor name, total amount, and what else do we want from there?"
We want the vendor name, total amount, date, and we'll say "number of people dining." All right, that's good enough for me, and I'm going to be passing the image here as the input as well.
Let's go ahead and validate this. Of course, as you can see from here, the image is going to be the only variable. So when I say "Validate," the image should be the only thing that will be available in here.
And there it is, the image input. Where is that coming from? If I click on here, that would be the input of the photo that I passed in the input.
All right, I should be in good shape now. The only thing left, if I go to the output, there is no echo anymore, so I need to change this to make it the "Vision Receipt Output."
And there is my graph. I will give it an image in the input. The vision will take it and will try to get me the date, vendor name, total amount, and the number of people dining, and the output will be sped out from there.
All right, let me save all of this, and let's go ahead and give it a try.
So how do we try it? We go to the value in here in the input. I'm going to click on the upload, and let's go get the same receipt again. There it is; that is the receipt. It will be right here.
And now, the moment of truth! We're going to go ahead and click on "Run," and we give it a few seconds. Once this has a green check mark, it will give us the output, and we'll make sure that this is working correctly.
Indeed, it took almost like 10 seconds, but if I click on "View Outputs" in here, the receipt provides the following information:
- Date: November 22nd (which is correct)
- Vendor Name: Dixie Cream Cafe (correct)
- Total: $51.39 (that's correct)
- Number of People Dining: 6 guests (that's incorrect)
Please note that the receipt image is a bit crumpled, but the requested information is legible. The number of guests was two; it put a guest in here again. It was crumpled, so actually, the table number was six, but the number of guests was two.
So again, you can actually make a clearer picture of the scanned version and try to run this again, and hopefully, it will give you the two. But the majority of the data in here is correct, and you can see where we are with that stuff at this point.
All right, I hope this was useful. I'm going to stop this recording at this point. In the next video, I would like to deploy this, and I would like to show you how to use this in real life using an API from the outside, like something like curl, Postman, Fiddler, or something else, or from your own application, for instance, from a web app or something like that.
Let's go ahead and do that!