Transcription
I thought we'd take a look at vector search, right? Which is one of the techniques used in modern chat systems to find data and use that data when answering questions, right? So, we know we talked about retrieval augmented generation in a previous video.
In that video, to try and get the language model to answer a better question about I can't remember what it was I was asking about. I think it was a a football stadium. Let me tell you about the amazing Olympia Stadion in Berlin, Germany, right? I just jammed the whole Wikipedia article for the football stadium into the prompt. Now, that's okay, but what happens if your uh you don't have a Wikipedia article? Or what happens if your company records are thousands or tens of thousands of documents? How do we find a good document to help answer a question? If I say, "Where in our back catalog of documents is the the thing about how me Shawn promised to pay me for doing this?" Um how can I find that, right? The process we use is something called vector search. And it's actually quite cool because it's it's similar in in in sort of in principle to the video we did on Face ID, right?
>> Uh okay, so that's where you had to unlock your face with your phone.
>> Exactly, right? And it's painful. If you want to unlock a face with your phone, you won't let me forget that one.
>> I'm sorry, it's just that Um it in that what happens is you train a a neural network to look at a picture of someone's face or depth map or a lidar image of someone's face and put their face into some embedded space where they are near or far away from other people's spaces. And you can log them in if they're near to what you think they're meant to look like, right? This is exactly the same, but it's now for sentences and paragraphs or words rather than for faces, right?
So, let's imagine that I ask a question. So, I ask a question like, "Why is the sky blue?" Now, everyone knows of course it's because of Rayleigh scattering, right? And this is also quite a common example we used in AI. Um now, let's put aside the fact that most big models could probably answer this because there's sufficient information in the training data. And let's pretend this is a question where we want to seek the answer of this from some data set of paragraphs of text that we've got. Let's say we've downloaded a huge encyclopedia, we've got all these texts. How do we find the bit of text that answers that question so that we can paraphrase it and respond?
So, what we're going to do is we're going to embed this sentence, the tokens in the sentence, using a language model into an embedded representation. So, let's just draw a representation in two dimensions because that's easier. So, this is dimension one. And this is dimension two, and they're completely meaningless, right? And the sentence, "Why is the sky blue?" is over here. And maybe you ask a different question like, "What is an atom?" I might have to ask one of the people from Sixty Symbols cuz I'm not actually sure. Um the what is an atom? And this will come out somewhere else, right? And the idea is that the network is it's been trained on enough text that it's contrastively learning to say, "Look, that sentence is not like that sentence, right?" in this space in the embedded space.
Now, the problem with two dimensions is that you quickly run out of space, right? So, you need to have many more. But the key to this is any paragraph that said something like, "Because of Rayleigh scattering." I think I spelled it right. Apologies if not. The hope is that these two sentences are semantically similar, right? And these two are not. And so, when we embed this bit of text, we come out somewhere over here, right? Which says, "Okay, we don't know that these are the same sentence or that they say exactly the same thing, but semantically the content of this sentence is similar." That should have been purple. How embarrassing. I didn't fix it. Uh this purple embedding is close to this green one.
So, now let's imagine you were building a system where you've got hundreds and hundreds of paragraphs or thousands or millions of paragraphs of text. Some of them might be useful, some of them might not be useful. What do we do? Well, what we do is we store all of these embeddings and we can retrieve similar-looking documents. So, you ask a question, "Why is the sky blue?" It embeds that into a numerical space and finds, let's say, the 10 nearest points to "Why is the sky blue?" Right? Now, some of them might answer the question, some of them might not, but we then use those to look up the original text and we put that into the prompt of our language model as well as well. So, instead of just dumping a Wikipedia page in like you don't know what you're doing, uh you smartly find relevant pieces of text. And this is an unbelievably powerful tool. It's it's it's also very very common, right? If you're inserting things into a large language model, vector search is a huge hugely useful part of that process. It's not practical to stick millions of documents into a large language model. And if you do that, you might not find the right bit of text anyway. Let's let's make the problem a little bit easier. Okay.
So, how does this process work? Well, we train this model using something like contrastive learning, right? Where we basically give it lots of examples of text that is similar and lots of examples of text that's different and teach it to put them in different places, right? This is the same way that CLIP embeddings work when we did our video on CLIP embeddings, but that's for images and text. This is just for text.
>> You can't control what you're going to find on the internet, right? So,
>> Absolutely. This is called vector search, right? So, you might have a database of many many and the text, text one, text two, dot dot dot dot, right? It's a huge lookup table. So, you you find your position, you find these embeddings that are close to it, and then you get the text, right? Um and then you've just got to create this database. And to do that, you just find a load of text, you embed it once, and you store those embeddings, right? So, it takes up quite a lot of memory, but these embeddings are not that big. 128, 256, 500 dimensions, right? Just 500 numbers in a row.
Now, this is quite a nice way of doing it. I know I've been grumpy about large language models and and AI hype and things in some of my videos, right? But actually, this is one of the things that I think is really impressive about large language models. If you never have to get it exactly right, right? Which is a useful thing in this case. So, for example, if I ask, "Why is the sky colorful?" the answer is probably similar, right? And so, but it might move it slightly, but you might still get interesting bits of text. You Or "Why is the sky blue?" but I spell blue incorrectly, it will be close enough, right? And the large language model embed The embedding will be in the right place. So, you can make typographical errors or grammatical or or use poor grammar when you talk to a large language model because the way that it interprets these tokens, that kind of is ignored, right? Or at least overcome. And that's I think quite a useful feature. It's one of the things that people like about these um these AI systems is that they offer this kind of natural language way of interfacing with something like this database. If if I ask you a question, but I foul up my sentence, you'll probably get the gist of what I'm saying, right? Maybe I try and ask ask something in French. My French is not that great. Someone who's French will probably get what I'm saying because I get enough of the words correct even if it's in the wrong wrong order.
>> So, looking at that on two dimensions, that's pretty straightforward. We can see where those X's are, but even if it was just three dimensions, one of those X's might be 100 miles away in the other direction.
>> Yeah. How do you cope with that?
>> Yeah, um okay, good. So, there are a few ways you can measure similarity in an embedded space, right? The the common one is is something we call cosine distance, right? And it's the angle between two vectors. So, if we imagine this is the this is the origin, right? Not not. This here is a vector. It has a D1 and a D2 or an X and a Y. This is another vector. All right, that and I nearly didn't have to bend it to All right, this is the angle between those vectors. Whereas this vector here, you can see the angle is much smaller. So, in two dimensions, the cosine distance will just be what is the angle between these two vectors? What is the angle between these two vectors, right? And we normalize it typically between, let's say, not and one or minus one and one.
Now, in three dimensions, it's actually exactly the same. Do you think I can draw that? I mean, we can give it a go. So, in three dimensions, you'd have something like I'm I'm regretting doing this already and I haven't even started yet. So, you got you got this, and then you've got this coming out here. Um and then you've got a vector in 3D space like this one, and you've got another vector in 3D space like this one, which for the sake of this diagram is more angled that way. It's still the angle between them, right? And which is going to be between not and 180. In four dimensions, I'm afraid I can't draw it on the page. In five dimensions, I can't draw it on the page, but the calculation is exactly the same.
So, why would we do that and not just measure this distance here, right? Because we've got essentially 500 or or or three numbers. They are a point in 3D space. They don't have to be a vector. Why would we measure this angle? Well, the answer is that it then basically degenerates everything everything is of length one, right? They're all on a a big circle like this, right? In 500 dimensions. If you can visualize a 500-dimensional circle. I try sometimes, right? It might be that for just for reasons of network output, this vector is actually really really long, right? It's the same angle as it was before, but what we're saying is the length of the vector is unimportant. The magnitude of this vector is unimportant. We only care about how the direction they're facing. And this essentially it reduces a bit of sort of error and noise in your process, right? You could do it this way, but you'll find that maybe a sentence is just like a bigger version of what it should be, and it ends up putting it somewhere else, which is similar to other sentences.
Is Is it possible though that that you've got two things that are nowhere near each other, but they're on the along the same vector line or more than two things?
>> That's what cosine measures, right? So, cosine is measuring whether two things are essentially along the same line regardless of how long they are. So, if you had a bunch of different lines on here, they would all have a cosine similarity of one or, you know, no angle.
>> So, they are similar
>> Yeah, they are similar, right? And that's kind of the hope. Um these would all be measured as quite different if we were doing it based on just distance Yeah. in through in, you know this dimensional space.
All right, so I've coded this up and when I say coded this up I've used libraries to code up. Just a very simple example to show you how this works. Okay, so I've imported some libraries. Um I'm going to create my embedding network, right? So the embedding network is not a network that predicts the next word in a sentence or predicts what image is interesting. It just turns your sentence, however long it is, or your paragraph into a numerical representation. Uh so I'm going to import that. Um I'm using hugging face to do this and the model I'm using is MPNet-base-v2, apparently, right? It's a standard one, right? So it's downloading all the model and so sorting that all out. So there's obviously there's a there's kind of like a a process of going through and turning it into a vector. Yeah. In actual fact, the process is not dissimilar to what a normal large language model does, right? It's still a transformer network. It represents the sentence as tokens. It goes through all these attention layers. It's just been trained to output lots of numbers instead of what the next word will be.
Um so I've got three embeddings here that I'm going to create. E1 is why is the sky blue? E2 is the sky is blue due to a process called Rayleigh scattering, right? Which I would say is similar in theme to the previous sentence. Um and E3, which is my wildcard, is bicycles typically have two wheels, otherwise they're called a tricycle or a car. And I'm sure the people in the comments will point out all the other non-two-wheeled vehicles that um I decided that wasn't the most important part of this video. So I'm going to embed this. So each of these will go through the network and produce E1, E2, E3, which are embedded variables. So these just think of these as positions on our in our dimensional space that we can measure the distance between. If we add another um code block to this, I can say for example E1 and you can see here it is. These are the embeddings for E1 and they're quite long. They're just meaningless numbers. They aren't useful to humans. They're useful because the distances between or the angles between this embedding and others are useful, not the numbers are useful in other ways.
So if we scroll down, I'm going to calculate the cosine distance between E1 and E2, E1 and E3, E2 and E3. And as far as I know this returns a number between zero and one, where zero is they are the same vector and one is they are very, very angled very, very far apart, right? So let's run this. Um so I'm doing E1 to E2, E1 to E3, and E2 to E3. And remember E1 and E2 are our Rayleigh scattering related sentences. And there we go. So the distance between E1 and E2 is 0.2 and the distance between E1 and E3 or E3 and E2 is 0.94, right? All right, so almost about as different as you can get, really. So imagine you had a bunch of bunch of interesting facts, right? Like your bicycle wheel fact, right? Which is very interesting, in your database with these embeddings and you looked you said we need to answer a question on Rayleigh scattering or why is the sky blue, what text do we use to do that? This will find the nearest text, which is the thing about Rayleigh scattering, not the thing about bicycles, right? And that's how you try and get the correct answer. There's one thing I'll point out is that we live in Nottingham, very famous
>> [snorts]
>> bicycle factory called the Raleigh Bicycle Factory was based here. Maybe that's why this is 0.94 and not and not one. I don't know. Um but my my spelling of both could be wrong. So let's improve
>> [snorts]
>> my retrieval augmented generation process with a slightly more complicated example. Here we're going to download and load up the NIST recommendations for key management document. Now, the NIST recommendations for key management is a great document if you're interested in key management, that is cryptographic keys, right? How long should your key be? What algorithm should you use? Where should you store it? How long should you store it for? How many times should you copy it? Should you encrypt your key with another key, right? These are all questions answered in this document and it's 170 pages long. So let's suppose you had a cryptographic key management question and you didn't want to read the whole document, right? Now you could use the index. But but let's suppose you can't use the index for for for reasons of it makes my example better. Um what we're going to do is we're going to load up this PDF and then we're going to look at here's an example page from my PDF. This is actually just I think the the boilerplate at the beginning of the document. We're going to use something called a text splitter to chunk the whole PDF up into groups of about 800 words or 800 tokens. And we're going to have a bit of an overlap between them. You might you don't want your really important fact to be split halfway, so you just don't you just do a bit of overlap. You could fiddle about with this and change it, right? So I'm going to chunk up my deck. And so I can see, for example, that split 171 is something about exchanging information between internet identities, right? I don't, you know, who knows.
The next thing for me to do is to embed every single one of these chunks into the embedded space and stick them in a a database where I can retrieve them. I'm using a database called Chroma for this. There are various that you could use. So all I'm doing here is I'm passing the embedding function by our network that we used earlier into Chroma DB. And then I add all the documents. And this takes a little while on the GPU. It's now actually processing all of those chunks. There is only about 800 to 1,000 of them. It's not too many. And so now I have a query. What it Now I'm not going to ask about why is the sky blue because I don't think it's in the document. Uh so I'm going to ask what is the current recommended crypto period for a symmetric key wrapping key, right? Now I don't know if there is such a recommendation, right? But it seems like an interesting question one could ask of this document. So I'm going to query it. I use the vector DB. I'm going to do a similarity search. That will be using cosine almost certainly with my query and I'm going to return 10 potential power the closest 10 points, right? And the text for those. So I run that. Doesn't take very long. The look up these these comparisons are points are very, very fast. And then
>> [snorts]
>> we look at the results for doing it. So the Ah, here we go. So the first text says the crypto period recommended use of period for symmetric key wrapping key is that it's used to wrap very large numbers of keys in a short period of time is on the order of a day or week. So I don't know whether that's I mean it's probably right, right? It's in the NIST document, but it does seem like a relevant paragraph that it's dug out, right? It hasn't returned anything on bike manufacture. Um so you could actually build this into a kind of hands-off query and then respond thing.
So I'm going to create a a Mistral 7B instruct model, right? So I'm I'm basically just going to have a little little large language model, which usually is quite a small model. It's not great at answering questions compared to some of the bigger ones.
>> [snorts]
>> Um I'm going to build a small pipeline where I automatically query, retrieve the data, and then answer based on that data, right? So I've created a prompt template that says you answer me a question using only the information provided in the context. Do not use outside knowledge. And then this context is going to be populated with my query, right? It's loading up the the model. It's good thing I chose a 7 billion one and not the 170 billion one. Where is that running? On the cloud?
>> This is running on the on the cloud, yeah. You could run this. It's quite a small model. You could run it on your own graphics card at home.
Okay, this seems to have done something. So create my plot I'm going to create my pipeline, create my prompt template. Um this is mostly using hugging face and LangChain, all right? And then I've got my rag query. So there's also a rag query here that says why.
>> [laughter]
>> so I'm going to delete that. Is it asking you why you're doing this?
>> Yeah.
>> Um so the rag query says it does it first of all in it does a vector similarity search a vector search with the query, returns 10 answers, and then sticks them all into the context, and then asks the language model to answer the question based on that context, right? So that's fairly straightforward. So what is the current recommended crypto period for a symmetric key wrapping key? Run that. Language model responds, hopefully in a minute. The context suggests the recommended crypto period for a key depends on the number of keys being wrapped in the duration of use. For wrapping large numbers of keys, order of a day or a week. Relatively small number of keys, it could be up to two years, right? So we'll let the cryptographers in the chat decide whether that's actually reasonable. That doesn't sound too bad. But the point is I haven't had to look through the document. It's all happened automatically based on my query, which is quite cool.
Now what happens if you talk you ask it nonsense, right? Well, in the prompt I've said don't answer the question if it's not in the context. So if why is the sky blue? blue? and I run that, my hope is it will say I don't know or it will say ah, the provided context does not contain information about the color of the sky specifically what is blue. The contract focuses on cryptography, network protocols, and key management, right? So that you know, you have to think about your prompt, otherwise it will start having a guess.
That I was going to say that's possibly my favorite thing about that the fact that it knows when it doesn't know. That's an interesting point, right? A lot of people ask me when we talk about AI, why don't they just say when they don't know? A lot of the time you're not using retrieval augmented generation or vector search. You haven't got this context of truth. If every time you ask a question, it sourced a data source and answered based on that, it could easily say the data source is conflicting with what you're asking, right? Or I don't know cuz it's not in the data. But what they usually do is rely on their own network weights and training data to answer the question and they have a go.
Um it's easier to get a language model to say no when you're saying compare this with this, as opposed to just answer this with no comparison, you see? So it's about how do you develop these systems? What do you want them to do when someone says something, right? And and it isn't an easy problem to solve. If you train your network to say I don't know to a lot of stuff, it will start saying it to loads of stuff even if it did know, and then no one wants to use your system either, right? So it's not an absolutely obvious system. But for building a system, let's suppose you want to build a system that's more robust. So maybe you're actually looking to put all your company's data records in here and you're looking to find the record that that is is is is you know linked with this theme. This is more likely to give you a response that's correct, but it's also more likely to say there is no such data, right? As opposed to, yeah, here's the data and then literally make something up, right? Which of course is never a good thing. Add up the bytes and check the sum. That's great, right? It's great except that it's quite common to get two bit errors, one that adds maybe 16, you know,