Transcription
Again, episodes about AI, but no, we will discuss vector databases. >> What is a vector in this context at all? >> Vector databases are, in fact, not databases. >> Why wasn't simple search good enough? How can you search for an image by text with ordinary search? >> The phrase "stooped." >> A place is signed where it smells of dog. >> For me, these are like scary words. For this reason, by the way, the idea of building a search by logs works very poorly. >> Will I understand this at a human level at all? >> Most likely, no. The management now is to hide as much as possible from the user that search even exists. >> I feel that I really, really need a vector database, vector search. Hello everyone, with you is another episode of the podcast Podlodka. And today we will talk about a very interesting and fascinating topic called vector databases. It might seem that someone who is already in the know will think, "Oh, again episodes about AI," but no, we will discuss vector databases as an entity as a whole. And today we have with us Andrey Vosnetsov. He is the CTO and co-founder of Quadrant. Andrey, hello. >> Hello. Thanks for inviting me. >> Yes, and today the co-host will be the incredible, wonderful Egor Tolstoy. Egor, hello. >> Yes, hello. I missed episodes about databases very, very, very much. We had some incredibly cool episode about databases, I don't know, five years ago with Kolya Golov. Where we talked about different types of databases, their properties, and so on. And since then, I think, among such roughly adjacent episodes, we only had one about SQL, probably, and about SQL Lite. I think we had two different separate ones or just about SQL, it doesn't matter. Anyway, hello. >> Yes, and to start, Andrey, I'd like to ask for a little context. Again, when I was preparing, I managed to watch interviews with Ossetinskaya and various other things, but tell me, why are you an expert in vector databases, vector search, and so on? >> Why am I an expert? Well, it's a bit strange to talk about myself like that, but okay, I guess I should start by saying that I've interacted with search in one way or another throughout almost my entire career. My first internship was at Mail.ru, where I was involved in very low-level backend search for L.ru, that is, webscale search. Then I moved to work closer to LL, but still, this LL was in one way or another related to search, various re-ranking, and back then it was ordinary search, yes, then it gradually transformed into vector search. And probably, when you have such a golden hammer, every problem seems like a nail. And you want to solve every problem with search. Probably in this context, I reached the state of inventing my own search engine. That's probably the right way to explain why someone considers me an expert. >> Let's start, for those who are not yet in the context, by talking about what a vector database is, what vector search is, and so on. >> I'll ask an even simpler question. Maybe, what is a vector in this context at all? >> What is a vector? Yes, well, it is, yes, a good question. It can be answered with varying degrees of detail. We probably assume that our audience is developers who know more or less what neural networks are. Let's start from there. In general, a vector is a wonderful representation of any object in the form of a set of numbers. This set of numbers has a wonderful property that if two sets of numbers, two vectors are close to each other in some space, say, in Euclidean space or a bit closer to practice, in cosine space, then it means that the original objects that were converted into vectors are also close in some sense. And this property does not arise by itself; it arises because neural networks are trained in such a way as to reproduce this property. This is a rather technical explanation, but I hope someone will understand. >> Look, you were talking and comparing vectors and how it all works with how RGB colors work. I wanted to clarify here. So, in RGB, if we understand that we have these, let's call them, three coordinate axes, and we understand how to arrange them numerically everywhere. So we have some understandable frame. But how can I understand, all these examples, that there is a cat, there is a cat, and so on, there is a pancake, there is a frying pan for pancakes, and all that. How can I understand, >> what will be the dimensionality of the vector, what numbers will be there? So, if I have nothing from scratch, how can I create and comprehend something meaningful like this? >> The dimensionality of the vector is, let's say, a hyperparameter that can be chosen arbitrarily. So it is not chosen based on the task, but on how much time we have to wait for the model to train, how much time we have to wait for the search to occur. This is what the dimensionality of the vector affects. And usually, the larger it is, the better the quality, the more information we can pack into this vector. If you pay attention to the typical size of vectors, they are all powers of two. And this is not accidental; it's an optimization. All programmers, I hope, love it when all numbers are powers of two, because then they fit beautifully into memory. From the user's perspective, the dimensionality, it hardly affects anything, except for the quality and speed itself. >> And if we dig deeper, yes, then how to understand what numbers to put there, then we, roughly speaking, don't know what. This is not an analytical process; it is the process of training a neural network. That is, the neural network itself, like a black box, picks some numbers through the training process, but we ourselves probably cannot set them manually. That is, we can come up with some features. Even if we come up with features, we don't know how important they are relative to each other, for example. That is, it is impractical. It probably won't work as well as if we trained a model. That is, figuratively speaking, we give the model a fixed vector size and tell it: "Pack as much different information as possible here in such a way as to minimize the error of the training set." The training set mainly consists of, well, it's called triplets, but figuratively speaking, let's say it's pairs, pairs of objects. And the neural network sees pairs of objects. And some pairs are marked as similar. Some pairs of objects are marked as dissimilar. And the neural network tries to pick numbers in such a way that similar objects are close, and different objects are far apart. >> But vector search, it appeared even before the advent of neural networks. >> Well, it's probably hard to say what exactly appeared first. Well, yes, a vector can be created without neural networks. For example, RGB is such a good example of a vector that we can assemble manually. We can also assemble a vector from features. For example, we describe a person, for example, and as features, we write their height, their weight, their grade in university for mathematics. These will all be sets of features that can also be put into some conditional vector, right? But this vector is very unlikely to correspond well to this quality that close vectors will generate similar people. Simply because it is very difficult for us to scale these features in such a way that the vector space is well organized. That is, it can be done in principle. And, for example, the nearest neighbor search algorithm, it can work simply on manually selected features. It's just not as impressive as a neural network can be. And in general, if we return to vector databases, why they appeared in the end and how vector search in this database differs from text search, and so on. Let's say that vector search as a mass technology appeared around the time when open, accessible models began to appear that could generate these vectors. Before that, vector search, of course, also existed, but it existed as a very exclusive product for large corporations that could afford it. And primarily, large corporations could afford to train such neural networks that generate good vectors. For example, Google could afford it, Facebook could afford it, a startup with 150 people already couldn't. It either lacked expertise or didn't have enough data. And it's generally a rather risky process. You can do a lot of things wrong during training and spend a lot of time without achieving almost any result. That is, it is a non-trivial process that is difficult to formalize, which requires a lot of experimentation. However, as soon as publicly available free models appeared, which anyone could download and start using at least for General Purpose tasks, like finding similar texts, vector search immediately became much more accessible. And the question arose: how, in fact, to store these vectors? Where to keep them, how to search them quickly, how to make it convenient. Again, at the stage of large corporations, at the stage of Facebook and Google, these tasks were colored a bit differently. Convenience was secondary, scaling was primary. And practically all research, academic articles, were aimed at how to make vector search scale to billions and trillions of vectors on tens and hundreds of machines in huge clusters. That was the number one problem for vector search then, because the main consumer of vector search was corporations that needed exactly that scale. And with the advent of publicly available neural networks, this scale began to decrease. That is, it was no longer just Google that needed it, but also, say, an e-commerce store. It became accessible to them. And the question of convenience arose, one might say, how and where and with what tools to interact with all this. >> Yes, here I would like to understand how vector databases work, that is, how to store a lot of vectors simultaneously and at the same time search them quickly. Here, perhaps, we need to say our standard catchphrase, that vector databases are actually not databases, because, yes, it's a marketing term that stuck simply because, perhaps, it's easier to pronounce. And it's associated more easily, it stays in memory. But, strictly speaking, almost all vector databases that specialize in vectors are search engines. That is, their architecture is more like a search engine like Elasticsearch than a typical database like PostgreSQL. There are many reasons for this. First of all, the main thing is that vectors are not some new information that cannot be obtained from anywhere else. Vectors are always a transformation of some original object, some original text, some original dataset, which we conventionally call the source of truth, and which can always be transformed again to obtain these very vectors. >> So, excuse me, so you're saying not data, but a function, essentially, a transformation and some cache of the results of this function. Yes, yes, some cache, cache of the function's results is what we call vectors, what we put into a vector database, which is actually a search engine. Perhaps I will still use the term vector database, simply because it is easier to pronounce in Russian. But it should be kept in mind that there is a significant architectural difference between classical databases and what vector databases do. And we can talk about this separately, by the way. It's a rather interesting technical topic, in my opinion. We like it. So, having this transformed data, several things must be taken into account. Firstly, they can change. At any moment, some new version of the model might be released, and we will want to discard the old vectors and build new vectors that will be better, larger, more accurate, more efficient in some way. Who knows what? And secondly, vectors themselves are quite expensive. That is, they are often much larger than the original data. That is, if text of 50 words is approximately 100 bytes, then the average vector size with the current most popular models is 6 KB per vector. Quite a lot. That is, if you imagine a typical database table with a million rows, a table with a million rows is small by modern standards, any cloud provider can hold a million rows in a table. But as soon as a million rows are multiplied by 6 KB, we suddenly have 6 GB of data, which must also be somehow quickly accessible, meaning it cannot lie on a very slow disk. It's best if it lies in memory. That is, it's a very expensive thing in terms of machines, which, if placed next to the source of truth, will outweigh and overshadow the original data, the original use case. That is, what the database used to do very quickly and simply, just because there wasn't much data, becomes much more difficult with vectors. Therefore, for vectors, it's good to have something separate, isolated from the main workload, which can be rebuilt at any time, which can be switched from one instance to another at any time. This is a good practice. This is true, in principle, not only for vectors but also for texts and text indexes, but perhaps to a lesser extent. Simply because text indexes are simpler and cheaper in terms of machines. Vectors are heavier. >> And a vectorized, let's call it, do you understand, there's no such word. a-a-a, text and the same vector representation of an image, will the weight be the same or can it differ by orders of magnitude? >> Uh-huh. Well, it actually depends entirely on the model we use. There are models that generate large vectors for texts, and there are models that generate small vectors for images. So it's all customizable. Of interest, relevant, it's probably worth noting that there are models that are universal for images and texts. These are multimodal models, as they are called. And the whole point is that they are trained, so their training data consists of both images and texts. And these images and texts are matched with each other. Well, for example, a simple, or rather, not simple, but the most popular, most famous case is a model called CLIP, and it is trained on a dataset called Lion. What is the Lion dataset? It's a scraped internet where each found image, if it's associated with a title, with alt text, it's a training example, that as soon as you find an image on any website on the internet, it has a caption, then this is a pair of positive examples, what text should correspond to what image. And there are many such images on the internet, it's not hard to guess. The largest version of the dataset itself, I think, is 5 billion text-image pairs. And such a dataset is used to train a multimodal model that converts both text and images into vectors that are in the same space. That is, we can compare how close text is to an image and how close an image is to text, respectively. That is, we can search for images by text description and vice versa, text by image description, if one can say so. >> Can you also give a real-life example of how an image is encoded into a vector? That is, if with text I understood, again, there is this analogy with RGB, with an image, it seems a bit more complicated. Does it also use encoding of alt text, descriptions? Or some more cunning things? >> Actually, I would say that with an image it's even simpler than with text, because an image is a more or less continuous signal. That is, each pixel of an image is, figuratively speaking, a number, or even three numbers. And these three numbers, it's clear how to feed them into a neural network. That is, there are convolutional neural networks that can transform essentially any image into some internal representation of the neural network, process it further, and turn it into a vector. So, in fact, it's simpler for images. Images, probably, even started to be processed by neural networks earlier than texts. It's more complicated for texts. Texts are first converted into tokens. Tokens correspond to some basic vectors, trained in a special, unique way. And then these basic vectors, through, well, the modern architecture is transformers, are transformed into a vector through transformers. However, if we are talking about CLIP, this model that is multimodal, it does not use the image caption as input. It uses the image itself. And that's its advantage, that we can take any image and turn that any image into a vector. >> But how is it that the resulting vector, I don't know, from a picture of a dog. How is it that the vector of a dog's picture lies close to the vector of the word. Let's even complicate it with the phrase "stooped." >> Yes, yes. So, >> so we have 5 billion images from the internet. There is a very high probability that someone has posted something similar on the internet. That is, they posted a photo of a dog and wrote: "This is my dog." This serves as a positive example. That is, we tell the neural network that if it encounters such a picture and such a word, they should be together. Then the question arises: "But where are the negative examples?" That is, we don't have specially labeled pictures and words that definitely shouldn't be similar. And here, in fact, there is a whole area of research on how to get negative examples. The simplest, most intuitive one is that we just take a random picture and a random text and say: "This is not similar," most likely. Well, there's a small chance they'll be similar, but that's at the noise level. That is, 99.9% is that they will be completely unrelated pictures and texts. And then we tell the neural network, here are your positive examples, here are your negative examples. Please change the weights inside yourself in such a way that the vector we get from the text is a little closer to the dog, to the picture of the dog, and a little further from everything else that we call negative examples. And we do this millions and billions of times. If we have a dataset of 5 billion and we do several epochs, then yes, it's a very long process. It takes days, weeks on a lot of hardware, but in the end, we get a vector representation that reflects this property. >> What about examples? Well, surely out of 5 billion there is a picture of some train station somewhere, labeled as a place where it smells of dog. This is neither a negative nor a positive example. >> No, this is, yes, this is noise. That is, every dataset has some level of bad, low-quality examples. And here's the thing: the model can't remember everything. It has limited capacity, right? Therefore, it has to choose between deriving some general principles and remembering specific examples. If we do everything well, if we set up the training process in such a way that it ignores such outliers and focuses more on general principles, then our neural network will work well. This doesn't guarantee 100% quality, of course, but it's an assumption that among 5 billion, even if we have, say, 10% noise, 10% bad labels, it won't significantly affect the final quality. Here, perhaps, I would like to return a little to the level of ordinary mortals and understand, why wasn't simple search good enough, why was vector search needed? >> Well, actually, there are many examples. And the example we just discussed is one of them. That is, how to search for an image by text with ordinary search. This is actually, yes, the most interesting case. Perhaps it's not the most popular now because everyone is into LLMs and so on. But it's a very illustrative example of what can be done with vector search and what practically cannot be done with ordinary search. That is, well, there are many other examples. If we talk about ordinary text search, then, as a rule, it works well when we have large documents with a large number of words and a short query. This is the best-case scenario for text search. A large document is good because it has many words and increases the probability of correctly formulating your thought. That is, we must use exactly the synonym that is used in the document for it to match. And a short query is good because if it were long, it would be too expensive for ordinary text search. It turns into a full scan and, as a rule, it's not that effective. That is, you can conduct an experiment, try to insert 150 words into a text search query and compare how much slower it becomes because of this. And it becomes almost linearly slower. So that's another limitation of text search. Cross-language search is also an interesting example. That is, we have one example that we have trained the correspondence between images and text, but no one stops us from training the correspondence of texts in different languages to one vector, for example. And then it doesn't matter what language the document and query are in. If the neural network converts everything into such a universal vector space for meaning, not language, then the search will work without translation. The search will work, well, practically out of the box. >> But does the traditional text search work better in scenarios where the vector search works poorly or also well? Just differently? >> Absolutely, there are scenarios where traditional text search works better than vector search. For example, when we want to find documents with articles or with some digital identifiers that do not carry meaning. That is, one must understand that a vector is an entity with a limited amount of information, right? That is, we cannot pack an infinity of different words into a fixed-size vector, it cannot remember them all. And even more so, it cannot remember them if it's some unique token that may not even have appeared in the training set when we trained our model. Therefore, vector search will most likely not find such examples at all. If we have an article or some UUID, for example. For this reason, by the way, the idea of building a search by logs with vector search works very poorly. That is, if you have some service and you export logs, then most likely only ordinary text search will help you, and vector search will be very bad. Then there are some, well, special areas, knowledge domains, which are very rich in specific terms, say, medicine, or some chemistry, that is, all these substance names that are rarely encountered in ordinary life, only make sense to specialists. Most likely, in such cases, a model trained on a general corpus of texts will work quite poorly. That is, you either need to train a specialized model on a specific knowledge domain, or use standard text search. But the good news is that no one forbids doing both simultaneously. And this is, probably, the default approach to search now. You need to do both vector search and ordinary text search, and then combine the results. >> At the same time, vector search, well, it turns out, has been around for a certain number of years, so with ordinary search everything is clear, it has been modernized for decades. Yes, how do you think, if you just look at some development curve, then in vector databases, vector searches, and how all this is already quite well optimized, or is there still a lot of room for development and improvements and so on? >> Well, I would say,
The main source of optimization for vector search is, strangely enough, a compromise, right? That is, uh, actually the vector comparison operation itself is a very trivial thing, right? That is, uh, in most scenarios, dot product is used, uh, what is it called in Russian? Vector dot product. So, vector dot product is what has been optimized in computers since 1950. the most primitive thing that can be, uh, the most frequently used. And because of this, optimizing dot product specifically is, well, quite difficult. On the other hand, when we deal with vectors of quite large dimensionality, then perhaps we don't always need absolute accuracy of this dot product. Perhaps we don't even always need absolute accuracy of the search itself, right? That is, uh, will we find, uh, a vector that is absolutely the closest to ours among a billion vectors, or will we find a vector that differs by, say, one value after the decimal point of three zeros, right? >> And what does this mean in practice? How can I, uh, understand that I have been given exactly what I was looking for? And that a huge amount of resources were spent on it, or rather, an inaccurate dot product was given and I was shown something close. Will I even understand this myself on a human level? >> Most likely, no. Most likely, no. Because, uh, in principle, when there is a dataset that, uh, has millions of, uh, similar images, uh, for example, or similar texts, uh, differences of this kind are incomparably smaller than the differences that exist between different models, uh, which, in fact, produce vectors, right? That is, uh, there is a difference, but it most likely manifests itself only on benchmarks. That is, a person looking at the result will most likely not understand the difference. But we can calculate it purely mechanically. In fact, on sufficiently large volumes, even on benchmarks, it is quite problematic to calculate accuracy, because, well, to calculate accuracy, you actually need to perform a search by brute force first, right, and then using, uh, a vector database, right, let's call it that. Uh, and only then compare the result. But this brute-force search is so slow that to collect a sufficiently large sample of examples, you need to spend a day, simply because this search is so slow, on such a large volume of data. >> And how is this compromise between speed and accuracy generally sought? And how, well, I understand that there are some, again, probably tests. Uh, but at what point and in what way is it understood that this level of accuracy is already noticeable to the human eye, and it is no longer satisfied with the results of vector search. >> In fact, the most important thing for us, as those who build vector databases, is, uh, the very fact of the possibility of such a compromise, because it is, well, a non-trivial solution that, for example, in text search, I think, no one does. In text search, the result is always absolutely identical, right? That is, it is always absolutely accurate. The very fact of this compromise allows us to build such an index, right, specialized for vectors, which, uh, allows for the very possibility of an imprecise result, not an absolutely guaranteed 100% result. And within this index, there are its own parameters that can be adjusted, tweaked, uh, fine-tuned in various ways, so that in each specific application, the user can choose this compromise for themselves, right? That is, we prefer to shift this decision about accuracy versus speed to the user, because, generally speaking, we don't know who needs what level of accuracy. For example, if we are doing some kind of biometrics, then accuracy is much more important than searching for funny pictures on the internet. Right. But the very fact of such a possibility and the very fact of the existence of an index that allows for not absolutely accurate results, it allows, uh, generally speaking, to create interfaces even that also imply that the accuracy will not always be 100%. >> Are there any compromises that you have decided for the user and that's it? And no other way, I don't know, data storage, where it will be, uh, some other stories. >> Well, in fact, our open-source philosophy rather implies, uh, opening up all possible configurations to the user, right, which can be configured in various ways. On the one hand, it's good if the user knows what they are doing. On the other hand, of course, it's quite easy to step on rakes. Just make such a setting, such a configuration, which, well, will simply be unviable. This is, as it were, a design decision of what kind of product we are making. Either we are making a Cloud Native, which is assumed to be used mainly or only in the cloud as a commercial product, or we are making an open-source tool, which is available, figuratively speaking, to everyone. And how they, how users will mess around with the application of this tool. That's their responsibility, if they break something, they break it themselves, and not in the cloud. Right. But we have quite a lot of things open to the user. This is, on the one hand, a problem, and on the other hand, a bonus. >> Is it like this for everyone? It is clear that there are a huge number of tools now. How, uh, what are the differences between them, especially considering the current trend for vectors? >> No, naturally, it's different for everyone. Someone builds vector search with the aim that it must necessarily be a cloud product. And in fact, if you build it as a cloud product, then from the perspective of the cloud, it will fit into the infrastructure much more easily. Uh, internal formats can be changed, internal versions can be updated. If it doesn't break the user interface, then they most likely won't even notice it, and they have no way to control it. And if we build an open-source product, then it's a completely different story. We cannot control what the user does there. We can only assume that they at least follow the instructions for updating versions. And that's it. There are, of course, versions in between, there are versions that were cloud products and became open. There is the opposite, like us, for example, first open-source, then we move to the cloud, and there we start imposing all sorts of restrictions on top of the source. There are many different stories. It's hard to come up with. Perhaps the most interesting thing is to talk about the separation of storage and compute. This is a popular topic in databases and search engines. The main idea is that we have Amazon S3. Amazon S3 is a very cheap storage. Yes, its speed might not be as fast, but if speed is not the number one priority, then why not, right? Why not upload everything to S3 and recall processors, right, figuratively speaking, compute machines, independently of storage? So, there are solutions that rely on this architectural decision as fundamental, building their vector search based on this. This is not our story, meaning we have a different approach, but it is also a valid architectural solution that covers a certain niche. >> In the chat, a question was asked about the fact that vector features are already appearing in classic databases like Postgres, ClickHouse, and so on. >> Well, I wouldn't say that ClickHouse is a classic database, but, let's say, >> okay. With these trends in mind, how will this affect tools that are entirely focused on vector search? >> Yes? Well, as we probably discussed at the beginning, uh, vector, uh, and vector search is not a workload that combines well with classic databases that love transactional consistency, and with classic databases that love persistence of everything. This is an architectural pattern, right, that many of us have heard of. ACID, that is, this is what classic databases like Postgres are built on. Uh, this approach has both advantages, right, and fundamental limitations. It is very difficult to create a replica for Postgres, right, because it will not be completely identical. Because there must always be some main replica, and the others listen to it. It is very difficult, for example, to perform table joins if these tables are on different machines. And all these limitations, they make sense if you are working with data that requires transactional consistency. Vectors do not require transactional consistency. This is not a source of truth at all. For vectors, it is much more important that, uh, the storage we use can, for example, easily scale, because vectors are large, they rarely fit on a single machine. We want to build a distributed cluster. We want to make sure that this cluster works even if some machines in it fail. That is, uh, respectively, fault tolerance is needed. And we need eventual consistency, right? That is, we don't necessarily need our update to be applied immediately and guaranteed to all nodes. We can tolerate it if something is updated on one node earlier than on another, for example. And because of this, for vector, for vector search engines, right, the possibilities of their application are greatly expanded. Of course, if we are talking about a use case where we don't have resource constraints at all, if we are talking about a use case where everything fits well on one machine, and it fits even as long as we can keep two copies. If we want to switch from one model to another, we need to have two copies at some point. If all this fits on one machine and we are not limited by performance, then, of course, you can use any tool. But if we are talking about some kind of more or less large scale, then I think that not only for vectors, but also for text search, ordinary traditional databases will not be suitable. That is, answering another question from the chat, it is unlikely that vector databases will ever merge with classic databases and will always be a separate class of systems. >> Well, there will always be a place for, uh, vector databases built on the principle of search engines, rather than traditional SQL databases, right? That is, some use cases will naturally be covered by such hybrids, such plugins for Postgres will cover them. Naturally, this will happen. But at the same time, there will always be those scenarios, those scales, where a specialized solution is simply necessary. And how do you understand that, for example, I am developing something and I feel that I really need a vector database, vector search. What are the brightest practical application cases? >> Now, with the advent of CH GPT, with the advent of LLMs accessible to everyone, the most frequent use case for vector search is so-called RAG. Retrieval Augmented Generation. The essence is that, well, everyone knows that language models hallucinate. Everyone knows that language models do not have access to the latest information, uh, some private information on which they were not trained. Therefore, they need to expand the context of information that is relevant to the current task. And the most straightforward way to expand the context for the current task is to make a search query, find relevant documents, and put them directly into the prompt, in which we are actually asking our question. This is RAG version 1.0, right, which appeared, well, probably 2 years ago. This is now probably 80% of all use cases that, for example, we see on our platform. On the one hand, this is, of course, sad, because, well, we would like more interesting and diverse applications, we would like something more non-standard, but practice shows that this is what everyone needs most right now. One can say, of course, that vector search is not unique in this case, right? That is, we could even use ordinary search to do the same, but the question arises, how well can ordinary search do this if we are working with small documents. And we work with small documents, not even because they are small in themselves, but because we break them down into small chunks, into small fragments, so as not to clutter the context of sound models with irrelevant junk, for which we will have to pay if we add it there. And if, look, if we take and focus on the RAG case, are there any specific, I don't know, features of vector databases that specifically help tailor them for the RAG case, or not features, but, I don't know, requirements for them? That is, what makes a specific vector search engine great specifically for RAG? Uh, well, there are some functions that are more utilitarian, right? That is, for example, uh, well, from practice, it becomes known that many, uh, RAG use cases are tied to many independent subsets of documents, right? That is, what we call multi-tenancy. That is, figuratively speaking, there is some collection or, uh, website that stores documents. But each organization registered on this site has access only to its subset of documents, right? That is, the conditional Notion, right? So, Notion is a good example. Each organization has its own workspaces, and you need to search only within them. At the same time, the scale can be completely different. There are organizations that have created two pages and that's it. There are organizations with 1,000 people and millions of documents. So, all this needs to be stored efficiently, right? That is, we cannot simply create a separate instance for each such user, because it will simply be very expensive. We need to store them efficiently together, preferably in one collection, in one table, without overpaying for a global search, which, in principle, will never be needed by anyone. It needs to be noted again that to build a vector index, a lot of resources are needed. And the more documents we have, the more this index, the more expensive it is to build the index, and not linearly, right? That is, if we add 1,000 documents, it's not the same as, well, adding 500 twice, it's more expensive. Right. But with multi-tenancy, if we can somehow combine all users in one collection and at the same time disable the global index and build only a small subset for each independently, this is a very big optimization, which greatly reduces resource requirements. This is one example. Another example that I really like, it's more research-oriented, perhaps. It's not implemented anywhere yet, but experiments show that it's quite good. This is so-called relevance feedback. For those who don't know, let me explain. Relevance feedback is a very old topic in the community related to information retrieval. It has been known since 1965. The first article I found, it's already 70 years old or more. So this topic is far from new, but strangely enough, it is not popular to this day. It is weakly applied. It is very difficult to apply. Why? Because it is tied to the action of a person as an agent in this system. What is it in general? Relevance Feedback is a technique that allows, or assumes, that we will modify the search query based on user feedback, based on the results of the first query. That is, figuratively speaking, the user makes the first query to the system, gets some result, says what they like about it, what they don't like, and the system performs the search a second time, based in some way on this new knowledge obtained. And the second search result is expected to be of higher quality, more accurate. And all this, of course, is very good in theory, but users are such peculiar people that they, well, practically never leave their feedback. It is very difficult to force a user to click somewhere additionally if it is not specifically necessary for them. If someone remembers, then in the first versions of Google, for example, there were even buttons like thumbs up, thumbs down on search results. No one clicked them. It was very noisy, unhelpful, so they all left and were removed. That is, over the past 70 years, it has been practically impossible to get a user to leave their feedback. Now, with the advent of LLMs, this problem seems to be solved simply because it is much easier to get an LLM to do something than to get a user to do it, right? That is, we can ask the model if it likes the result, which result it likes more. It can express this even in some numbers. That is, you don't have to ask it in text. You can extract information from it at a lower level, but it will still be useful information that can be used for this second iteration of the search, which, in theory, should improve the result. Vector search is ideal for this, because vector search can work not with absolute values, right? That is, the limitation of this first, the first iteration from sixty-five years ago was that the user had to unequivocally say which document was relevant, which document was irrelevant to their query. And, well, as it's not hard to understand, in most cases, the first result was not relevant to anything, right? That is, there was no gradation. And this gradation, even if we could extract it from the user, applying it to keyword search is quite difficult. Vectors, however, make it quite simple, because the vectors themselves represent not binary values, right, but some weighted weighted scores. And therefore, applying such an approach to vectors is, well, an intuitively understandable task, at least for those who work with vectors. We arrive at a situation where we have an old, very well-known, but rarely used technology that becomes possible only because we have language models, we have agents that, in principle, can afford to use this technology. And this technology is something that actually requires, well, let's say, if not fundamental, then a sufficiently deep change in how we execute a search query, right? That is, we no longer have a single vector as a query, but we have a query plus its context, which, well, into which the user's feedback or the language model's feedback needs to be recorded. And the execution of this second query is something that requires changes in the search engine, starting from its interface and ending with how, in fact, this query is executed very deeply in the guts, in the data storage. Right. This is something that, perhaps, only vector databases can implement, because they, as it were, own their own code, right? They, uh, there are teams that, figuratively speaking, have the authority to change the code throughout the entire stack, from the API to where the vectors are stored. As far as I know, text engines, and it's quite difficult for them to do this, because, for example, Elasticsearch is a project that consists of two parts. There is its own Elasticsearch and there is Lucene. This is a separate library, which, well, probably overlaps with the Elasticsearch team, but it is still an independent project. To push through such a change that goes from the API to the internal guts, let's say, storage, requires a rather non-trivial interaction between the teams of different projects. Right. And for vector databases, this is something that we can do in, say, a quarter. >> By the way, I have a question for those who want to delve deeper into the guts of vector databases. It's clear that Quadrant has, uh, on GitHub, you can see the source code, but maybe there's some implementation that's more compact, simpler, smaller, from which it's easiest to start understanding how vector databases work, or do you recommend going straight to us? >> Well, in fact, uh, I probably don't recommend going straight to us. >> Well, there's Rust, obviously. Well, no, Rust is not the biggest problem. The biggest problem in our case is that we have decomposed the index quite a lot, such that, well, we have added several levels of abstraction to it, and understanding these levels of abstraction without the context of the engine as a whole is quite difficult. On the other hand, classic implementations of this vector index, by the way, it's called HNSW. Well, the one we use, uh, they are quite academic, right? That is, they are also optimized for benchmarks, they are optimized to optimize every byte. Therefore, reading this code, especially in C++, is quite difficult. Right. So I would probably find some, maybe not the most canonical implementation, but an implementation of the algorithm, and start with it. There are many libraries that simply implement this very vector index, even in Rust, which is independent of the search engine. >> You use the Hierarchical Navigable Small World algorithm. Yes, it sounds terrible in Russian. >> Well, at least it's understandable. >> Yes. >> Yes. Well, our algorithm, naturally, has gone through several stages of evolution. We have added the ability to perform filtering. We have also added these abstractions, so that it is not dependent, for example, on the vector comparison function that we use, right? That is, in our case, it has been significantly reworked. >> Right. But the basic principles, yes, are preserved. >> Yes, it's an approximate nearest neighbor search method based on graphs. And here, a question from the chat immediately comes to mind: how do vector databases differ from graph databases? >> Yes, well, they differ in everything. Well, well, here, if, if, uh, to cling to the fact that the vector index is a graph, then this graph is a purely internal data structure. Graph databases are made to store user graphs, right? That is, the user defines some graphs, and we arrange these graphs into some internal structures. Vector databases do not store a graph as such. They use a graph as an intermediate format to make it easier to search for vectors. But from an interface perspective, there is practically nothing in common between vector databases. >> And from the perspective of the tasks being solved, are they more competitors or just a general internal representation? Uh, well, there is, let's say, such a fashion or hype among some people who are involved in this task of building RAG, that it is necessary to add, uh, some graph entities, there, ontologies, to somehow build connections from documents. To be honest, I just don't believe in it. I have such an immunity to everything related to graphs, it came from university. I just don't believe in ontologies. I don't believe in any semantic web. For me, these are scary words. I might be wrong, right? That is, I admit that there is some use case for which it is useful, but I have never seen it myself. I would also like to ask about practical applications. Is there anything outside of language models, RAG, where it fits well? >> Well, the most interesting task that I personally like the most, related to vector search, is anomaly detection. We even have such a case study. Well, I'll tell you briefly. So, there is a coffee plantation. A coffee plantation is interested in understanding how good its coffee bean harvest is. And to do this, they have a special apparatus that looks like a large box with lighting and a camera looking inside it. A handful of coffee beans are thrown into it. And the task of this box is to count which of these handfuls are good beans, and which are various kinds of anomalies. That is, for example, it might be over-dried or moldy, or instead of a bean, there might be a cockroach lying there. So, there is a set of various anomalies, right, which, well, we can start classifying them, but generally speaking, they are not limited to a specific list, right? That is, there can be anything, a twig, we won't predict it in advance. And classically, this task is solved, naturally, by collecting a training dataset. The annotators sat for a couple of weeks, collected 10,000 examples. So, they labeled them: this is good, this is a twig, this is a cockroach, this is mold, this is over-dried. There are 50 different classes of anomalies. A model is trained, and it works, giving good quality, say, 98%. This is enough to roughly estimate how the harvest is better on one field and worse on another. But what to do if new anomalies appear that were not there before, right? A new bug has appeared, which was not observed before, right? It spoils our life in new ways. And in the case of classic models, the traditional path is to hire annotators again. They have to label these new anomalies again. The model has to be retrained. And we have to, well, essentially repeat this process. It might be a little easier, of course, because I don't need to
Everything to mark, but, in general, we need to go through this path. In the case of vector search, we can do approximately the same thing, only instead of training this specialized model, which distinguishes anomalies, we can train a model that generates a similarity vector. We can even train it on the same anomalies that we have already labeled, but instead of predicting a class, this model will find the closest class from those that we have stored in the database. Thus, when something new appears next time, we don't need to retrain anything. We simply add this new example of a new bug to the database, and it automatically starts recognizing it as soon as it appears. This is a cool example of how vector search works not related to text. >> A very cool example. >> Cool. What is the biggest pain when working with vector databases? Well, the main pain, naturally, comes from the fact that vectors are expensive, they take up a lot of space, they take up a lot of resources, and that's the price you have to pay to get good quality search. Naturally, all our efforts, half of the team, are directed towards lowering this price. We come up with different tricks, like how to use memory more efficiently, how to quantize vectors, so that they take up not 32 bytes, but 16 or even 1 bit. This is a process built on compromises. >> Can you give an example of the most unexpected trick you've done and that you can remember? The most unexpected, probably, is binary quantization, which, in principle, is quite trivial as a concept. What's unexpected about it is that it works, right? That is, let's say, what do we do? We have a vector. A vector is some floating-point number. Usually 32 bits per element. And we just take it and say, "If this number is greater than zero, it's one. If it's less than zero, it's zero." And thus compress the vector by 32 times. It seems, you know, that a programmer or a scientist would think that we will lose almost all accuracy. Because it's too much compression. But it turns out that for large vectors, for example, those vectors generated by models from Open AI, the losses are not as significant as we initially thought. They are, well, around 5%. A 5% loss in accuracy when compressing by 32 times is quite good. This is an unexpected result. When we measured this, we initially thought, of course, that we had something wrong with our code, something wrong with our benchmark. This shouldn't have happened. But it did. The most wonderful thing is that this compromise can, in principle, be compensated. Not at the moment when you create the index, it can be compensated at the moment when you execute the query. That is, let's say, you can request a few more results and re-rank this large top of results with really good vectors, and thus spend a little more time at the moment of query execution, but compensate for the loss of accuracy that resulted from compression. It's like something from physics, where sin X for small values equals X. And all that. >> But this doesn't work for all models. This is also an interesting result, that for some models, which may even be larger in dimensionality, this trick doesn't work. Well, when we started looking into it more, there are different explanations, that the distribution of these values is not quite normal. Well, in general, it has opened up, in fact, a whole direction of research, which is probably the most effective way to make vector search cheaper. This is quantization, and specifically binary quantization. So, you, at some point, sat down, and from your sub-project, what happened was that your own company, an open-source engine that is used in many places. Do you have a general idea of what might happen next with this whole topic in the future? Both with your product specifically, and with the entire field of vector databases in general, will it fade away because it will become something like, well, everyone will just get used to it, understand its scope of application, and that's it. Or will it develop even more? Right now, I think that, of course, nothing will fade away. Of course, it will become something like a standard, right? Just like we are now used to regular search almost everywhere. I think vector search will also be, well, almost everywhere, because it's just such a convenience, right? That is, if your search is not vector-based, then you are losing a lot of results. Well, at least in those areas where vector search makes sense, right? I'm not talking about logs, I'm not talking about some other, text representations that don't have semantics, for example, we'll leave them out. My more global vision of what will happen with vector search is that, in general, vector search is currently perceived as a smarter regular search. That is, from a practical point of view, from the point of view of developers. But in fact, vector search as such has an overlap with text search, but the functions that can be performed with it are generally not the same, right? That is, there are functions that work well in text search and don't work at all in vector search, for example, for text search it's very easy to estimate how many results matched the query. Yes, we can just count. Everyone has seen this line in Google, like "5 billion results found," right? For vector search, this simply doesn't make sense. Because in vector search, every document is similar to every query to some extent. We cannot simply cut it off with a threshold. On the other hand, for example, for vector representation, something like negative search, that is, inverted search, instead of the most similar, we look for the most dissimilar. This is something natural. This can be achieved simply by multiplying the vector by -1. For text search, this is an impossible task. It doesn't have such a data representation that would allow this. But for vector search, it's trivial, and it can be applied to the same anomaly searches, for example. For vector search, such a task is interesting, or rather, not even for vector search, but specifically for vector data representation, there are other tasks that are also interesting. For example, what to do if we have a collection of documents and we don't know anything about it, but we want to understand, somehow explore this collection of documents and extract some useful information from it? Well, the most naive approach is to build clusters, for example, for such a collection. Clusters are built quite trivially using vector representation. For example, we even have a special API function in our engine that generates a pairwise distance matrix. This is what can be directly passed to clustering algorithms to build a nice picture, right? We don't do the algorithms ourselves, we don't implement them simply because there are very many different ones, but we prepare the data for them in such a way that the algorithms themselves don't spend a lot of resources on the user's side to draw good clusters. And you can go deeper and deeper. Vector representation requires its own specialized interfaces, which are often incompatible with traditional search. That is, we cannot build clusters for traditional search. For traditional search, we cannot do diversity selection, where each object is maximally dissimilar to every other object, and so on. How to reconfigure your paradigm of thinking in such a way? It just seems to me that what still hinders the understanding and spread of vector databases, vector search, is that we are used to understanding how familiar search works, how familiar data works. And therefore, all the talk about vector representation in some incomprehensible form of things understandable to us, it flows into these everyday examples, because everyone wants to somehow impose it on their own paradigm of thinking. What can be done about this, in your opinion? Well, I think that the interfaces are to blame for this, which have trained several generations of users that search looks like this. Search looks like an input field where you need to type your keywords, press a button, and it will find results. Yes, in the best case. In the worst case, there will be a terrible panel on the side with 10,000 different checkboxes to help you choose suitable shoes in a clothing store, for example. That is, interfaces built this way are interfaces built for text search. And trying to impose vector search on existing interfaces is, well, a poor man's approach, as they say. That is, we have something, we want this something to be better. So we add vector search to it. But in reality, it seems, at least to me, that vector search can also be accessible through other forms of interface. My favorite example is, of course, TikTok, where we don't search for anything at all. We just swipe quickly or swipe slowly. And this is already enough signal to show us a funny video about a cat after, say, 10 swipes, so that we get stuck there for the whole evening, so to speak, swiping through these cats. This is an interface that allows interaction with vector search. What are other examples? Well, it's difficult, right? I assume that something might be similar to this path of exploration, but building the interface itself is a complex task, but I think it will be solved at one point or another. And it depends on us how convenient an API we will provide so that this interface can be integrated into it. So, did I understand correctly that besides TikTok, you can't even describe an example of an interface that would be suitable for vector search right now? Well, it's a difficult task, yes. I can imagine it, I can try to do something myself on the fly, but I'm not an interface specialist. Most likely, it will be bad. Most likely, no one will like it. So, in reality, probably, the direction is now towards hiding as much as possible from the user that search even exists. On the one hand, this is TikTok, on the other hand, probably, these are agents that will write a search query themselves, that will talk to the search engine themselves, do several iterations, several attempts, and only then show the result to the user. This is probably the second direction of action that is more or less clear. >> Let me ask the last classic question. Besides where to look for input, what else would you recommend to people who want to dive deeper into the field? Are there any books? Are there any must-have articles or papers that need to be read? In short, is there anything like that where we should direct our listeners? Well, it depends on what exactly you want. If you want to start applying vector search, then I would say you need to start with experiments. That is, you need to try to do something yourself following tutorials. And, at least for me, it works best. You need to first come up with a task, and then try to solve this task. And if we are talking about the internal workings of vector databases themselves, then, well, it's a difficult task. Probably, everything related to regular databases is useful here. That is, for example, when we hire people, it's important for us that a person understands how, for example, the Linux kernel works at a low level, how processor instructions work, for example, SIMD optimizations, how the GPU works, it's important. That is, we usually want to discuss very low-level things in such interviews. That is, roughly speaking, I had such a task in parallel with preparing for this episode, and I thought, what if it could be applied here? I have a large number of, well, in one area, not very structured Excel sheets with different data. >> Uh-huh. >> These are really huge massive Excel sheets. And I need to figure out how to find some patterns between these Excel sheets, where to intersect them with each other. At the same time, I don't want to read them with my eyes and do this work myself. And it seemed that one could indeed build some kind of vector based on all of this. So, if we take just such a task, how should I create a to-do list to go through this path of vectorization? >> Well, you need to answer the first question: is there any semantic component in this Excel, or is it just a table of numbers? If it's just a table of numbers, then there's unlikely to be a ready-made model for obtaining vectors that will work well with this data. >> Well, there are column headers, at least. >> Well, column headers are already better. This is at least something that vector search can handle in some form, right? So you need to identify the part that contains semantic information. If there is such a part, then you can use it, right? Then there arises a purely engineering task, how to extract it from Excel. They are probably also in different formats. >> Uh-huh. >> An engineering task arises, how to extract it, how to put it into a vector database, how to then correlate the search results with the original document. How to visualize it? Well, this is a task that a programmer can handle. You don't need to be a machine learning expert for this, you don't need to be a PhD. This is a task that has descended from the research level to the level of an ordinary programmer who hasn't done this before in the last few years. You just need to understand that not everything can be effectively converted into a vector. That is, a table of numbers is unlikely, right? For example, I once heard an idea: "What if we record a chess game as a vector?" This won't work if you don't have specialized models for chess. If there is text. If there are some images, again, not all images, right? Images with white noise are unlikely to be suitable. Images with something drawn from real life, that's another matter, right? So you need to consider the semantic component. >> Great, thank you. Yes, I'll try, and maybe it will work out. Great, Andrey, thank you so much for coming. I think we've touched on everything very well today, discussed the main nuances, that a vector database is not actually a database, but vector search. How it all works with everyday examples, where it's applicable, how it differs from regular search, what problems, pains, and the future of it all. Finally, as usual, I'd like to ask Egor, what do you love more than recording episodes about new databases? >> More than that, I only like coming up with questions for you myself. But since you asked me first, I like it more when you, dear friends, not only watch our new episodes but also rewatch the old ones. Because in addition to those that I spoiled for you at the beginning, I highly recommend listening to another episode about search, where we mainly talked about the Sphinx engine, but also touched on others. That old episode is also very cool. Well, actually, I like it when you listen to our podcast, subscribe to it wherever you can, recommend it to your friends, colleagues, mothers, fathers, brothers, children, and even force your children to listen to it in the car when you travel with them. When you join our Telegram channel and post questions for future episodes, that's what I like most. >> Yes, don't forget to give five stars on iTunes. Again, as you saw, the questions for this episode were taken partly from the chat based on the preliminary post, so yes, don't forget to join. Thank you again, Andrey, for coming to us and answering everything so well. We'll hear you next week. That's all, bye everyone. >> Thank you all. Bye everyone. >> Thank you for inviting me. Bye. M.