Transcription
Hello, this is Leno Tados again. In this video, let's go ahead and try to use embedding in vectorization using Foundry Prom Flows.
We're going to use the AI search to be able to vectorize a big file—maybe 100 or so pages in it, whether it's a PDF, Word document, or Excel. It doesn't really matter. But we would like to chunk it, embed it, and vectorize it inside of AI search, and then start a conversation in Prom to see if it can actually answer based on that file or multiple files if we want to as well.
Let's go ahead and take a look. So far, we've used the Sur API, we did the LLM directly, and all that stuff is great. But this time, I would like to create a brand new Prom Flow.
Let me save everything first. I'm going to click on Prom Flow, and we're going to create a brand new Prom Flow. This time, there is one called "Multi-Round Question and Answer on Your Own Data," which is exactly what I want. I'm going to go ahead and say clone this one. We'll give it a name; we'll call it, for instance, "Leno Vector Q&A." There you go. We'll say go for it, and that will be the name. I'll come back after this is done.
All right, it only took a few seconds, and it automatically helps you out with the input. Of course, we have the output at the bottom. We have the LLM, we have all the different variants, and so on and so forth.
Let me go ahead and take my face out of there so you can concentrate on the code.
In here, the first item is to modify the query with history. This is the one that will, if I would like to continue in my chat, remember all my answers and my questions from previous questions as well. So, I need to remember to bring in the connection.
Let's go click on Connection. There is my OpenAI collection; it will be at Chap. If you remember, we deployed GPT-4. That's great. We'll keep the temperature at zero; it's fine. Maximum tokens: 1,000. This is good enough for me.
Let's save this one. Also, let me skip the lookup for now. I want to go to the Generate Prompt Context. Let's click on that. This is the one that will actually have the Python code done for me automatically by the Prom Flow that knows exactly how to take the source from whatever came up from the lookup and be able to pass it on using Python into the context of the prompt engineering of the Prom Flow itself. Great! Don't have to touch it.
If I go to the Prom Flow, notice there's "multi-mle" behind each other in there. That's because it will automatically create three different variants so that you can actually run some evaluations to find out which Prom Flow does the best job for you. But for this one, I'm not going to do that. I don't need three variants, so I'm going to say show the variants.
Now, I'm going to get three of them. I'm going to only leave the variant underscore Z, the first one only. Then, if I go down, I can delete the second one in here, and I'm going to leave the third one as well. Great! You can bring them back later on if you'd like to try it out.
But for right now, I have only one prompt, and that's exactly what I'm interested in doing here. The important thing about the variant is that it has a system prompt: "You are an AI system designed to answer questions..." It has all the good stuff, and it will also be coming in from the source citation.
Notice all the chat will be in here. The user for the chat input will be there, and the most important thing, of course, is this part, which has the context, and it's bringing in the variable for the context as well, which I will need to hook up to whatever comes back from the vector database.
In our case, I'm going to use Azure AI Search. In the next video, probably, we'll do F and show other databases as well, like Cosmos DB and others if we would like. Sounds good?
All right, so now I need to come back here and find out from the inputs. They are all correct; they're coming in from the right places. The final one will be the chat with context—that's the LLM itself. Okay, that's where we're passing it all the information from our prompt, including the context.
I'm going to go ahead and set up the connection, and we'll set the GPT-4 like we did before, and we're good. Now, I'm going to go ahead and say save on all of that stuff.
So, what is left for us to do? What's left for us to do here, folks, is the second one is the lookup in here. Of course, if I go to the lookup, I will need to set up the ML index, which I didn't do yet. The query we know is going to come in from the history of the output, and then I'm going to set up the query type—whether you want it to be based on keyword or semantic. There are a lot of different options that you can choose from, and here we'll do that in a second.
But first, I'm going to leave all that. Let's save and then go back to the left side under the models and endpoints. There is the data and indexes. Let's click on that.
There are two things I need to do. I need, first of all, to get access to my data. I have a big file in here. Let me show you where that file is. I think it's important for you to see it.
So, I went and downloaded here. I live in Orlando, Florida, and they have the entire—let me hook it up here all the way at the top—official Florida driver license handbook. It's 104 pages and has a lot of good information that I have no idea about, but I still can drive here for some reason.
All right, so I would like to take this entire content of the 104 pages, and I'd like to embed it and vectorize it and start asking questions regarding things that I might not know about the laws here in Florida, even though I've been driving here for a long time.
So, let's go ahead and see what we can do with that. First of all, under data indexes, before I even get to the Azure AI search, I need to upload this to my storage. So, I would say new data, and it will ask me where do you want to get this from.
So, I'm going to bring this down. I could bring it in from my blob storage if I add it to a blob storage. I can get it from a data with storage URL. Maybe I have a different account that has the data in it. I can upload it from there, or I can just upload it right now from my hard drive.
So, I'm going to say upload files in folder. We'll click on them, we'll say upload files, and hopefully, we can go here to my download. And there it is—the English driver handbook. It's about 6 MB; it's not that big, and it will upload it, and we're good.
Now, I'm going to say next, and now we'll need to give it a name. I'll call it, for instance, "Florida Drivers." You can call it whatever you want. We'll say create this data, and it has been uploaded. We are pretty much in good shape at this point.
Okay, so that's our first piece to download that into the storage area. Now, I need to create the index, and that's a fun one. So, we click on the indexes in here. First of all, I should be able to see my data that has been uploaded—that is the Florida drivers.
Then, I'm going to click on indexes at the top in there, and we say I would like to create a brand new index right there. Where is the data source? Let me bring this up. Notice there's a lot of steps in here. Actually, I could have bypassed the data files that I uploaded, and I could have done everything from inside of the indexes by uploading the files from here.
But again, that's doing too many things at the same time. So, at this point, I really would like to concentrate on one step at a time. The create a new index will be coming from the data in Azure AI Foundry. I already uploaded the data, so I'm more than happy to go get it from there.
Let's go ahead and click on that, and notice there is my Florida drivers. Great! Let me go ahead and click on that. But notice something in here that says an Azure AI search resource and an Azure OpenAI connection will be required to index your data.
I did not have an Azure AI search resource created in my resource group. This is a good time to click on this and go ahead and create one. Let's go ahead and click on that. We'll say create one. That will open another tab here at the top in my browser and allow me to go ahead and create a brand new Azure AI search resource.
Let's go ahead and do that. I am going into the resource group in here. I want to use the same one that I use for everything else, and we'll get the name in here. We'll call it "Leno AI Search." That's good enough for me.
The location will keep it like everything else in East US 2. For the pricing tier in here, folks, if I click on the change pricing tier, notice the basic and standard are currently not available. The load is very happy and very heavy right now.
Remember, we are in the last few days of 2024 in here, and East US and East US 2 are heavily being used for load balancing for a lot of retail stores. So, hopefully, that will come back in January again. But right now, I can use the free one, which might have some limitations, or I can use an S2, which is extremely expensive—$1,000 per month.
So, let me talk faster so I can delete this as soon as possible. I'm just kidding; we'll keep it probably for another 10 minutes, and after that, I'll delete it. Let's go and say select this guy, and now we'll have the standard 2.
All righty, as the clock is ticking, we say let's do the network. We'll keep it public; I’m not caring right now. Let's go ahead and do a review and create that resource, and I'll come back after that resource has been created.
All right, well, that took 12 minutes to deploy the Azure AI search, okay? But it's done now, so we'll say go to Resource. The important thing here is I'm going to be interested in the indexes, of course, which will all be empty right now. I didn't do anything yet.
Let's go back to our prompt in here that I'm creating with the create and vector index data index. Remember, I already loaded the data, so now I need to create a connection. Remember that Azure AI search that I just created does not have a connection with your project.
You can do it from the Management Center, or you can do it while you're creating the index itself. So, when I say, for instance, let me see the data source, that will be the Florida drivers. If I say next right now, it will ask me what is the Azure AI service that you want to use.
If I bring this down, notice there is nothing in here. The only thing that you can do is to go connect to an Azure AI search resource that you have in your resource group, and that's exactly what I'm going to do.
Let me click on that, and then it will show you the Leno AI search I just created a few minutes ago. I'm going to say add that connection. Remember this part; you can do it from the Management Center as well. I find it easier to do it while you're creating the vector index; it will work no problem.
All right, so it automatically comes up with a weird name for the vector index itself—"ashy caller." Usually, there are a lot of weird names that come up in here. I'm going to leave it; I'm not going to change it; it's okay.
The virtual machine, I would like the system to auto-select it. Hopefully, with a decent size, it can allow it to do this pretty fast as well. Excellent!
So, from that point on, let me go in here now. The Leno AI search will show up, and I will leave everything else the same. I'm going to say next. It will tell me the configuration search setting: add vector search to this search resource. That is correct, using the OpenAI connection I already have.
Remember, in the beginning, in the first video in this series, I actually, other than GPT-4, also created the Ada 8 002 for embedding. If I did not add that embedding 002 for Ada, notice the remark in here that says if you don't have one, we will add a text embedding 88002 for you automatically as part of your model and endpoints so that you can actually do this.
But I already added one, so I don't need to do this again. So, I'm going to say next, and then we'll say, yep, that looks good to me. Go for it!
Now, we'll go ahead and get a cup of coffee, and we'll actually go ahead and use the ml.azure.com machine learning to be able to do the chunking, the embedding, and the indexes.
See, there is a step one of three. It's trying to find a machine now that is available, and once it finds it, it will latch onto it to do some allocating, and we'll start the chunking. Then, the second one will do the embedding, and finally, we'll do the index as well.
But usually, this takes between 5 to 8 minutes. We'll come back after this is done. Of course, if you're interested in what's going on during these seven or eight minutes, click on job details, and that will open ml.azure.com. It will show you all the different nitty-gritty about how we're doing the chunking, how we're doing the overlapping, how we're doing the embedding.
I mean, you can intercept and do whatever you want, but again, AI Foundry is for application developers; it's not for data scientists and data engineers. If you are interested, click on job details, and you'll go into that world if you want to.
But for me, I'm just going to go ahead and pause this for the next 5 to 6 minutes until all of the three different steps are in the green, and we got ourselves an index.
Notice after a few minutes, it found the machine finally, and now it's in progress of doing the cracking and chunking. I call it the Benihana effect to be able to cut it into pieces and do the chunking piece, which is very important.
We'll come back after the second step is on its way. Here we go! Actually, the first piece is finished. Now it's starting step number two, which is creating an Azure AI search index. It's not started yet; it's looking for another machine that will do this on, and we'll come back after the second step is done.
Also, while this is happening, I want to give you a tip here: why are we using Azure AI search, and how do I make a choice? There are so many different vector databases. I mean, you can do this with Azure Cosmos DB, you can actually use Azure PostgreSQL, you can do it with Pinecone, etc.
Azure AI search is very good for unstructured data, like data coming in from a PDF or a Word document or an Excel spreadsheet. It has a lot of capabilities. If your data is tabular or transactional data, you might actually be better off doing it in something else.
But remember, when it's unstructured data, Azure AI search is the best for that. Of course, you can do it in other places as well, but if you have transactional data—what I mean by that is structured or semi-structured data—maybe using that in Cosmos DB or PostgreSQL or even coming soon Microsoft SQL Server Vector Store would be great for that as well.
All right, we're done with the first two steps. Now, the final piece after creating the index in AI search, I am going to register the index, and that's an important piece because the Prom Flow is going to look for that registered index.
Okay, and it's done! Notice all of them are done, and if we refresh this, it will say completed. Okay, we're good; we can actually keep going.
All right, so now I have my data index inside of Azure AI search. Let's go back to Prom Flow and take a look at where we are. I'm going to go to the lookup in here. I want to make sure, first of all, that the start compute session is running. This has to be running, and if it's not running, sometimes just by refreshing the page, it should be running.
But if not, I need to click on it to make sure it's running. All right, I started it; it took a couple of minutes. Make sure you save everything first before I go to the lookup, and let's see what we can do for the lookup.
This is the important piece. First of all, I'm going to choose my ML index content. If I bring this down, you'll notice there are two things that you'll need to fill in. The first one is the index type. Let me take a look at that.
The first one, hopefully, will be the registered index. If it's not there, something went wrong with the creation of the index in the Azure AI search. I'm not using a brand new one for Azure AI search or for a pass to my ML index; it is a registered index already.
Let me click on that, and then the second one needs the asset ID for that ML index. If I click on that, it hopefully will see the weird name that we saw earlier—that's the "ashy caller" or whatever—and that is correct; that is the one I want as well.
Okay, I'm going to save that, and now that's the first piece that will go into the lookup itself. The second one is the query coming in from the history output; that's fine.
Then, what type of query do you want to use? There are several that you can actually use: keyword, semantic, vector, or you can use a hybrid between vector and keyword or a hybrid and semantic. I'm going to go with the vector and keyword; it's very, very powerful.
The top key for the probability, I'm going to take the first two that come directly from there to pass it on to the LLM later on. At this point, my lookup is done.
So now, if I say save, remember I'm going to ask a question. Of course, that question will have to be embedded as well. I'm going to actually add it to the history, and then I'm going to look up on this question. That lookup will take my question, embed it, and we will go into the vector store, which is Azure AI search, and we'll come back with some results.
These results will be embedded into my prompt—my prompt engineering, my general generate prompt context that will be generated right here—and then it will be passed in here.
So, where is the answer coming in from the Azure vector store? It will be right inside of this guy that says context in here. So, by the time it gets finally into the LLM, the LLM will get the entire prompt: the system prompt, the context, the user question, the history—everything will be inside of this prompt text.
If I’ve done everything correctly, hopefully, we'll get a good answer from this 104 pages. But I'm not paying for the 104 pages to be passed to the LLM; that would be too much money, right? I just want to do the search first—maybe get like five or six chunks, which is way better than 104 pages being sent to the LLM.
Also, I will get semantic meaning. It means the question does not have to have a direct one-to-one relationship with the words I'm using, so you can use semantic meaning as well. If you're close enough in the sphere of the vector store, it will come back with an answer as well, which is really awesome for that.
Let me save this. The computer is running, and I can go ahead and click on chat. Of course, the demo that I started with the cloning has its own thing going on in here. I'm not going to use that; I'm going to click on the plus to start a brand new session.
I want to figure out a question that I should ask that comes out of this 100 pages. Remember that PDF that I showed you earlier? Let me go down a little bit in here and see if I can find something interesting to ask about.
Ah, there you go! Distracted driver. All right, we can ask it, for instance, "What are the categories of distraction?" Let me take this thing in here; it's on page 42 of this whole thing.
I'm going to go back in here, and we'll say, "What are the categories of distraction while driving?" All righty, let me put it in the question, and let's see if it will do the job.
Say go for it! Drumroll, please! It should now be doing this. It should be taking that question, embedding it, coming up with 1,536 different floats that represent this question, going to the vector store, and finding out the closest thing possible available for all of that.
Not only that, but I told it to bring citation as well. I only have one PDF, but if I had 100 or 1,000 PDFs, it should be able to find it.
Look at that! The three main categories are exactly what we saw in the PDF right there: visual, manual, and cognitive. That's great!
Then it told me the total tokens for generating this is 1,163 tokens, and it took 5.32 seconds to do that as well. The nice thing is I told it to tell me the citation. The source is the English driver handbook.pdf.
Can you imagine this being a medical or a tech document or something that requires compliance? Before you give an answer for an LLM, you have to tell us where you got this information from. I want to make sure this is not hallucination, so being able to tell you which PDF, Word document, or Excel spreadsheet out of thousands of different ones that you have already vectorized is this answer coming from is proof of why the LLM chose that answer.
This is great! Hopefully, this is definitely useful for you.