Transcription
And here is a question, there is what is called positive passages, these are correct contexts, and we will use them. Well, in principle, we don't need answers as such, we will discuss this, we will definitely discuss it later. Ah, actually, the first question, well, like, let's try to cope without RAGs anyway. We lived happily, something here, some new concepts, new materials. When should this be used and when should it not be used. The first question seems, well, obvious, because this is literally what you ask yourself and us, as soon as you see a new concept, a new technology. Here. And the answer here is plus or minus simple. Often we deal with domain data. We have X5, I'll remind you again, that X5 is Perekrestok, Pyaterochka is for those who didn't know. I, for example, didn't know. I was explained about it at the company when I had already gotten a job and worked there for half a year. So, in general, we have quite a lot of what is called domain knowledge, including some links to our internal portals, some processes, including not only bureaucratic processes, but also, for example, how to wash ovens. Yes, because we have our own baking in Pyaterochka and Perekrestok. Please come, eat deliciously. Here, in general. This is all domain data, and models, obviously, do not know it. They are updated with some periodicity. Oven washing instructions less often, vacation application instructions more often. And when your instructions, in principle, have a tendency to be updated, it, well, it seems to make sense, yes, to use RAGs, because, firstly, we will be training something too often in such a case, yes. And we wouldn't want to. And secondly, it was a good case, and now it will be a bad one. We will have little data, nothing will be fine-tuned, and maybe we will even make it worse, and it will even diverge and stop responding to our questions at all. So if you don't want experiments or don't have many graphics cards or, well, all of the above, then, of course, well, here they are, here are the RAGs. Their time has come, their finest hour has come. In addition, RAGs often allow us to facilitate fact-checking, where we attach links to the articles from which all this came. specific links, possibly even to text snippets. Also, well, we improve answers and reduce the probability of hallucinations. Although, of course, well, we don't get rid of them completely. However, with fine-tuning, it will be a much more frequent situation where we hear a ringing and don't know where it's from. So, in general, today during the workshop we will use Llama 2 with Langchain. Why? Because, firstly, the author was not given money for OpenAI, and secondly, because, well, anyone can do it with OpenAI, but try it with local models. This is generally the motto of all our work. And think about it, if your management allows you to upload sensitive data to OpenAI, it's probably worth thinking about. But this is also on the course, by the way. Please come, we will talk about ethics there. Also important and interesting. So, local models today will be Llama 2 with Langchain, this wonderful thing. And we can just ask a question. Today's test question is to show you. What is it like, this is about this wonderful Armenian gentleman who, in, well, somewhere in '70, created the Armenian state. And that's it, everything will revolve around the Armenian state. So, this is, in fact, what we will be dealing with. Here we have a question, here we have a context, based on which we are going to answer. As people, rational beings, and so on. We look at the question, we look at the context, and it seems we can answer something. It's not difficult to guess, in principle. And this is, in principle, a sufficiently standard story. If we have very long articles and we are going to do something with them, we can try to answer them off the cuff, without any context. This is how it is stored in my head, in the model's head. Well, for example, I rarely encounter this gentleman, and I won't answer such a question immediately. And if so, then probably incorrectly. Well, in short, as in the anecdote: "I can quickly add four-digit numbers." But that's wrong. I didn't say I could add them correctly. Well, it's about the same topic. Here are completely different years, completely different people, completely different faces. What will happen if we add a little context here? We pass it here in a similar way. We just say that here is the document. And immediately after, literally right away via Enter, the question. And here we answer. Good. Magic, it's simple enough, just like showing students a manual, letting them flip through it, and they will probably find the correct answer in the manual. But not everything is so simple. We also discussed when not to use RAGs. And this is also a big, cool, important question for business. We will discuss it closer to the end. But for now, for example, I can say for sure that if your entire knowledge base fits into the context window of your model, then you can stop, just shove all the documents into the model's context window, and we will answer. Cool. But if all the documents don't fit, if we have the entire Pyaterochka information center, which we will try to fit inside, then welcome here. Here. And we have already talked about this as if we just have some list of documents, let's pass it to the LLM, and we are cool. But if we talk a little more complicated, then we have this scheme here. It will also be duplicated in the notebook, so if you forget something, you can restore the sequence of events there. Ah, actually, here, more complex concepts appear in the spirit of encoders, document encoders, query encoders, vector search, and all this stuff we will dive into headfirst today. We will try not to drown and stay within the time limit. Ah, so, quickly about vector databases. There are many of them. They rule the roost here in the search index. This is simply, well, a cool thing if we want to store vectors somewhere and then search for these same vectors. Well, like, you won't have to encode and decode it back and forth if you can store it in encoded form from the beginning. And there are very many such vector databases. You can choose the one you like best, or the one that the security department has approved for you. But not all of them may be equally convenient. For example, PGVector, as we found out, does not support vectors of different sizes, while Qdrant does. So everything is up to you. Please search. We, in particular, use Qdrant ourselves. In many ways, this was and still is justified by the fact that there is a large community and, if necessary, it is, plus or minus, easy to maintain. How to search? Here we have such a concept as Query expansion in the presentation, but I want to go back a little here. Here we have a simple RAG. Before we added something to it and did all this, all this glitter, shine, and tuning, let's start with something basic that will become our benchmark. Ah, actually, the benchmark here is Qdrant. The very first thing we will use for simplicity is RecursiveCharacterTextSplitter. And in principle, we will work with the Langchain library. This is both a good idea and a bad idea at the same time. Why is it a good idea? Well, because it's all out of the box, and you can write one line, and it will work, and magic will happen. Why is it a bad idea? Because Langchain has terrible documentation, and it's also poorly maintained periodically. Therefore, I will probably, well, like, warn you that Langchain is mostly not a production solution for experiments, yes, for demonstrations, yes, if you are going to roll it out to production, I would think seriously. Seriously. Here. But here for clarity, we will leave it. All disclaimers are made, we can move on. Ah, actually, we first create the document structure itself. Here again, it's a Langchain document, where we create this from the database. Page content is what we will search by, what we will encode into vectors, and then perform vector search on. Metadata is everything we might need. I will immediately introduce two new concepts on which we will configure something on top. We could have done it later, but why later if we can do it earlier. So, we have both symmetric search and asymmetric search. What we are used to, what seems most understandable to us, is asymmetric search. We have a question, we search for an answer based on it. In fact, this is exactly what happens here. Page content is simply the content of the pages, it's just text. And here you can see that page content is an event about Filaret and that's it. And here is the biography. But a new concept appears in the form of symmetric search. And what is this beast at all? The fact is that sometimes we need completely unrelated concepts to answer. For example, if the text with the answer does not contain the necessary keywords, you can think of it like calling Rostelecom. Hello, my internet is broken, what should I do? It seems that the exact instruction will not contain the phrase "internet is broken, what to do." It will contain much more specific instructions with specific service names, buttons, how many times it needs to be rebooted, and who else needs to be called. And often such things are solved by what is called symmetric search. There is already a small spoiler here, it was open below. And you can see that we are searching by questions here. The question is in page content, and the context itself, which we can then use, is in the metadata. Ah, in general, just a second. In general, we can, ah, use symmetric search, using the experience of previous, ah, queries. We have already been written to about broken internet. Let's search among similar queries. Maybe we have already solved this problem somehow. This approach also helps if the questions are not related to each other in any way. You can list the entire list of questions attached to this context, and use the entire list in your search. A good example was "my career doesn't work." The answers were, the answer was "career doesn't work, what to do?" And then instructions. And this was also relevant to the question "I can't pass the course." The word "course" was not contained in the answer or in any other questions. We added it manually, and only then it worked. Therefore, this can be a bit of a workaround, but if it works, it's not stupid. In general, yes, symmetric search for your attention. And you can see how it looks here in the dataset, where, in fact, page content will be here "home," and we will search by it, and everything else is in the metadata. The same thing happens with embeddings, approximately. This is also a cool add-on on top of the retriever, on, in fact, vector search. We have the BGE M3 encoder model, which is symmetric, and you can just take it, shove it all in, encode it as is. There is also the E5 model, which in turn is asymmetric and requires different suffixes. Prefixes require different prefixes for encoding user queries and documents. This, in turn, improves search. This is written on the E5 page. All this can be read, you can browse now, but in principle, the links will be provided. Next, we will use Qdrant in our work, because, well, we use it, and this is, well, the author decided, in short. Ah, actually, for now, we will mostly play with asymmetric search, but we will also dive into asymmetric search again. We just throw in the entire list of our documents here, throw in the model that will serve us embeddings, and then it will build a vector database from this. And we can use it as a retriever. There are two options here. Firstly, you can use it as a retriever. Then we find the top one. Here is the query. And we can find, here is the relevant document here. And here we can also specify K, and then we will explicitly tell how many top documents we are looking for. As an option, you can just specify similarity search and also specify how many top documents we want to retrieve. All this is done using ordinary cosine, ordinary cosine similarity. There is nothing new under the sun here. If you are not familiar with such concepts, again, go to the course. It will be in the prerequisites block for RAG. And this is, it seems, my block, so see you there. The same can be checked, that it really works well with symmetric search. We can declare it as a retriever and find even this slightly awkward query, because, well, correct English grammar looks a bit like this. This is done on purpose. Here we find our documents. And as soon as we find a document, we have access to, respectively, not only the answer itself, but also, for example, to the context, which we can also work with. We can even, in fact, shove all this here, see that docs zero is this one single document. And see what is in Page Content. In fact, the question. We can, we can look into the metadata. This will be a regular JSON, as you wish, and get the context from here. Everything is cool, everything works. It just requires a little post-processing and dancing with a tambourine, but it works. The same also works with Qdrant search. And then we have modifications of RAG in the plan. And I suggest we pause here for a moment and move on to the promised mini Q&A session. It seems that questions are accumulating, so we can continue. Okay. Then let's move on. Actually, about modifications. I'll even collapse this here for clarity to show what awaits us. Here we have three points where we will stick our heads today. The first is Query expansion. We will look at it a little less thoroughly in the notebook, and a little more thoroughly in the presentation. This is, in fact, we are going to tinker with the user input here. And maybe it doesn't seem obvious at first, especially when clients come to you and say: "Look, we have a cool dataset here. The entire accounting department compiled it. The entire accounting department, people are surprisingly literate and educated." And then people with a significantly different level of education come to you and simply, for example, with dyslexia or other reasons, for which your query looks completely different than expected. Metrics start to drop sharply. Did they drop for us? Yes. Did we stumble? Yes. That's why we are here. In fact, the simplest thing you can think of is that we have an additional, simplest query to the LLM, where we say: "Buddy, please make something decent out of this." Correct typos, correct grammar, but try not to change the meaning. In fact, it almost managed. It almost managed here with the word "state educational institution." But it was a good attempt. We will find something here already. It's already "Armenian" with an "a" and "Filaret" with a capital letter. Almost a victory. In fact, this is a spoiler on the topic of what could go wrong here. As soon as we have new composite parts of the program, we have points of system failure. In fact, yes, we can screw up badly here, rephrase something incorrectly and get completely unexpected results. It seems that once, on the question of how to wash an oven, we rephrased it as how to sell pies. I don't remember exactly how it turned out, but perhaps the query was exceptionally awkward. In fact, here you need to be very, very careful, play with the prompt, temperature, parameters. And we have some more suggestions for you. This is, in fact, HyDE, where the idea is to expand our potential query. Well, they write us something about vacation, and we assume that we want to say here, how to apply for vacation. The problem again will be that we can invent something extra. For example, the person wanted to postpone it, not apply for it, and then problems will arise. Plus, if the knowledge is too domain-specific, we risk hallucinating quite a bit. Therefore, almost all improvements here, take them not so much to heart. Think that most likely it won't work with your data, and be surprised if it actually works. But here are small disclaimers from me every time. Another cool thing is SelfQuery. And it will definitely work well for you if you came to us from, I don't know, Cian, Spotify, anything, yes, maybe even Ozon. If they ask you for a yellow bag, you can definitely distinguish the category "yellow" and the category "bag." In fact, the meaning of SelfQuery is that we turn the user's query into a less specific query plus a JSON with tags. A JSON with tags is much easier to process than a query written in natural language. And therefore, it should work. It may not work for you if you have few filters that you can extract from the query. In fact, yes, it will work if metadata extraction is exceptionally good. And if, from the query "vacation" or "postpone vacation," you only extract, well, the category "vacation," and the word "postpone" remains. It seems we haven't gained much from this. And the last thing I'll throw in here is a fallback. This is simply that as soon as the model loses confidence in its answer, we start searching the database again, searching again. And we do this every time the model loses confidence and generates less probable logits. Well, here we risk dissolving into tears. But the topic is good if we are writing essays and really need sources. Next is the search block. And I'll interrupt for questions. Is there any point in using Faiss and Qdrant together in synergy? Will there be advantages from this? I have a slide for this. We can talk about it a little later. But as our practice has shown, it doesn't significantly increase the time, and the gain on our data was almost minimal. Next, how to assemble RAG if you are not a programmer? Are there any code services? I'm afraid to lie. I have a deformation here where I know how to program. It's hard for me to communicate with no-code services. But if I'm not mistaken, then the service or N8N, aka Nathan, can help. With query reformulation and reranker, does the delay between receiving a question and answering it increase? For example, in Perplexity, answers come out in seconds, although there is also query reformulation. How can we achieve at least comparable speed? What technologies are used for truly fast answers? Or is it all solved solely by the amount of hardware and tensor parallelism? Not only by the amount of hardware and parallelism, but also by the size of the models, the length of the prompts, and your overall approach. You can tweak a lot of things and slow down the service, or you can ask questions to a small Llama and enjoy life. With cross-encoders, it's the same, you don't have to use a giant cross-encoder. You can use a small one, or you can try other rerankers, but we'll talk about that a little later. Someone is also asking me, where to write, where do the questions come from, Katya? We take questions from two channels where the broadcast is conducted. Right now, questions are only coming from Gigaskul. Everything is forwarded to you. If something is written in the comments to ViTalentHub, we will definitely bring it. And where are the comments there? Just in the last post where the announcement was that we are starting our workshop, you can write a comment with a question. Yes, okay, thank you. In general, yes, we mentioned that we don't use Langchain. How do you implement requests to LLMs? We have an LLM set up, we make requests to it via an API, and almost everything done by Langchain was then duplicated by our mischievous cheap hands. How are keywords and user queries determined? This is a built-in feature. How to see the words by which the search is conducted? This is all done, it works on model queries. If you have ever tried to form JSON from a query, then this is it. If you have never tried, firstly, Structured Output can help you. This is like, a hint for the future. And, in fact, the idea is that we pre-determine, for example, the fields we want to see, or leave them as null if the bag color was not specified, or fill them in explicitly. For example, they write something to us. We want to form a JSON from the user's query in the format of color, brand, type, and in fact, structured output and simply queries to models capable of generating JSON as output help with all this. Therefore, you determine the keywords yourself depending on the design of your database, how it is stored, and generally what you intend to filter by. And a new wave of questions. Is there any point in rewriting the user's query to the language of the documents? Yes, definitely. Yes. This simply allows you to reduce a large, not very clear, slightly formless task to a task that you already know how to solve when your questions are literate and similar to the documents. This is a normal thing. If the documents have different writing styles, then it is enough to simply bring it to a state where it looks more or less literate or corresponds to the level of literacy and detail of the documents. How best to parse tables, formulas, and images in documents? Unfortunately, I cannot advise on images and formulas. We don't work with them yet. We currently save tables in MD format and, in fact, query them. What are we writing the API in? Python. Is there any point in speeding up using a faster language? Probably yes, but unfortunately, we are not backend developers, we write it ourselves. This does not hold us hostage. So it's planned. If Langchain can only be used for MVP, then what tools should be used for production? The orthodox Python. But you can buy your own, more detailed than in Langchain. What to do if I have many documents and cross-encoders are too slow? So, let's get to cross-encoders a little later. We started too early. If there are too many vectors and the database has become too large, vector search is greatly scattered across different sections of documents. Cluster, cluster, and search by cluster centers. Find the nearest cluster center, and then search within the cluster. Ah, actually, let me return here. I left fields for "What can go wrong?" here. And I suggest maybe checking yourself or writing in the chat, or simply answering yourself aloud, what can go wrong in general with processing an additional user query and what can go wrong. In particular, here with SelfQuery, I'll drink some water and repeat the correct answers again, and we'll move on. We have a very large number of questions coming in. I think if I throw all of them to you now, we'll get stuck here for another 10 minutes. I suggest moving on and returning to them at the end. Or in the next iteration. Especially considering that questions about cross-encoders have started. It's nice that knowledgeable people have come, but don't spoil it for others. In general, for self-checking, any processing of a user query, slightly or not slightly, adds to the processing time of their query. Firstly, and secondly, it is another point of system failure where we can stumble and, in fact, everything can go completely wrong. What is the problem with Query Expansion? The problem is that, well, we may not extract any useful filters if our query design simply doesn't allow us to extract them and we are left with an empty query. This was the case, for example, in technical support. Regarding chunking. Here, in fact, we are moving to reranking. We have already peeked in a little when we discussed how we create embeddings. I will show it here. This is a slightly outdated slide. We use something slightly different now. In particular, we tried Jina Embeddings. For now, we are working with Multilingual E5 large Instruct. In fact, different embeddings are different vector representations of the same queries. Sometimes they are more successful, sometimes less successful. It all depends on your data and what you are working with. Therefore, this is just a large space.
For maneuver. And if you don't know where to look for something that can encode your queries, then welcome to what is called MTEB. This is, uh, a benchmark, uh, leaderboard. So. Uh, here we have a leaderboard. They periodically redesign it. I am surprised every time by how it looks anew. Uh, but we can take a look. Here we have the Russian language. And, uh, it would be a huge omission if I didn't boast about who participated in the RAG benchmark. This is one of my colleagues, who will also be teaching this course in the "ProNFR" section. Roma Solomatin was involved in the RAG benchmark. And you can look at the best encoders here in the benchmark. And not only encoders, actually, everything that you might purely need in theory for various tasks. A bunch of filters. Uh, for example, here for reranking and retrieval, you can sort, uh, sort in descending order. There. So far, so good. Frea is the best, uh, encoder for retrievers according to the benchmark. But, of course, we strongly recommend checking, well, not only the top one, but try to go a little lower, maybe it will turn out a little better. Uh, and, actually, uh, here, finally, we are moving on to chunking. This is also a direct part of retrieval. This is how we divide our large texts into parts if they don't fit into the model's context window. And this, well, actually happens often. So, here is an example of chunking. This is a library with a fat hippopotamus. We offer you several chunkers here. This is, in fact, the token chunker, where we simply split by tokens. Here we have some GPT2 tokenizer. We say that, as a rule, we use the same tokenizer as the model that, uh, generates or the one that does media loss [music] as losslessly as possible. Uh, actually, you can adjust chunk size and chunk overlap – this is, uh, like here, uh, our text parts overlap with each other to make it more or less, uh, not seamless, but to avoid data loss here at the seams. The simplest thing is, we just split by tokens. We know that one word is several tokens, and therefore we can very awkwardly split part of a word there, part of a word here, resulting in something like one and a half diggers. Therefore, a tokenizer, although the simplest option, may not be the most optimal. There is also sentence chunking. And here we preserve sentence boundaries so as not to lose meaning and do not split a sentence somewhere in the middle, like a rainworm with a shovel. There is also a semantic chunker, where we not only try not to split sentences, but also try to keep semantic chunks together. The only problem is that, well, firstly, similarity needs to be defined as a concept itself, and, secondly, we still have a maximum number of tokens per chunk. And if, uh, you have a categorically close, uh, document, semantically, very, very dense, then, well, we will still split it. It still won't fit into the context window completely. C'est la vie. There. Well, and actually, the minimum number of sentences can also be set. That's not a problem. CDPM Chunk is double merging. We first group by semantic similarity, then merge similar groups. Uh, therefore, if we have, for example, a slightly ADHD-style narrative, we first talked about one thing, got distracted by another, and returned to the first one, we can merge these two parts into one chunk. Uh, this, by the way, helps, in principle, with many lyrical digressions in the text, if you have any. It had practically no significant impact on us. But I can say that, actually, both the late chunker, and the SDPM chunker, and the semantic chunker worked about the same for us. Again, a disclaimer, it might be different on your data. Just know that there is diversity and some more advanced techniques, differing minimally from each other, can still give different results on your data, different from ours. Uh, actually, the late chunker also, uh, generates embeddings for each fragment independently. Instead of generating embeddings for each fragment independently, we first encode the entire text into one vector, then split the text using various recursive rules, and extract embeddings by averaging. Uh, but it, well, it works longer. In general, you can notice, uh, during, uh, working with chunkers here, that, uh, these chunks down here, uh, practically do not change. This is due to the fact that we use documents here that are from SQuAD. That is, they are from short contexts, and for long contexts, this will be a more decisive issue. And I, uh, will launch this now for MLDRE. While we look further, I will be able to demonstrate to you how it differs on MLDRE. [music] Excuse me, I need to peek at how it looked here. These are, actually, where the texts are taken from. So. Here. Uh, we, in general, will move on. Uh, next, we will have, uh, another branching to parent document. This is when we find some small piece of, uh, a document. This again works well when we have a large article, we found some small, uh, small piece that is unequivocally relevant to us, and we can look there at what is above, what is below, uh, to capture a larger piece, uh, in order to avoid information loss. And here, regarding questions about, uh, FAISS, SVM, KNN, and all that. We tried, we took FAISS as a baseline, it worked well. We tried to take ensembles. An ensemble with SVM showed good results, but we spent a lot of time for it to work. And after many, to be honest, iterations with BM-25, we finally beat it, and now our ensemble with BM-25 works better on some projects than just raw vector search. Actually, about post-processing – this is precisely about reranking. Uh, I'll look at the questions now. The questions are not about what I've said so far, so let me throw in about cross-encoders. Ah, no, there is one about chunking. What can you say about contextual chunking based on Anthropic's paper? Did you enrich it with pieces of the original document to preserve context for stitching? Uh, we did that a long time ago, when we were just starting to work with the information center documents themselves. But this, actually, enrichment through LLM in one form or another, it just complicates and makes the database processing longer. And to be honest, well, essentially, we came to the conclusion that it is rather redundant. How to evaluate the quality of a module that modifies or clarifies the user's original question? Are there specialized metrics or frameworks for this? Good question. Uh, I, well, I have two creatures on different shoulders. I don't remember which one has an angel and which one has a demon. The angel tells me that all this can and should only be evaluated by eye. But the demon says that you can use LLM evaluation, but please, please, please, make sure that the evaluations of your LLM and the prompt with which you set up the message correlate with human evaluation. match at least, well, by 80 percent, because, well, you can set it up, but not be sure that it works correctly. There. But yes, everything related to generation, for now, we either review with our eyes or with LLM evaluation. Uh, but you have a variety of eyes, you can review it yourself, you can delegate it to the client. A question about document preprocessing. What can be done to solve the problem when there are a lot of similar documents in the condition, exactly the same pieces? Uh, if they are similar documents, similar sources, you can try something like a deduplication algorithm and put not one link in the metadata, but all links to the pieces. Then, if you are sure that you have exactly similar or very similar pieces, you can just put all the links in one basket and show one chunk. Uh, well, and on the chunk, yes, you can try to split by, uh, specific, fixed parts. Well, generally, I would rather, I don't know, I need to see the data. I can't say right now. But it's a good question, right? Deduplication algorithms plus metadata to help, in my opinion. At least, that's what I can offer off the top of my head for now. Uh, creating and curating datasets for evaluating individual components of a RAG pipeline, particularly for the retriever and query reformulation module. Are there established approaches or tools for automatic generation of such datasets, for example, using LLMs, so that they are high-quality and representative? Uh, representativeness will always be a big concern. Quality too, everything generated by the model still needs to be reviewed by eye. You can't escape that. Uh, but you can create such things, especially if you have something that is generated with zero temperature. You know that it is always generated the same way. At least you can check, uh, the system's viability and, well, poke it with a stick. If the temperature is not zero or if we want to measure quality, again, well, welcome to LLM evaluation. The devil on the shoulder, it's all that. Uh, regarding dataset representativeness. There is also a very fine line between what you consider representative and what the client considers representative, uh, that is, the one who brought you this data. If you yourself are well-versed in the domain, then, please, why not, you can look and review it yourself. If this is something you are not familiar with, if lawyers come to you and say, let's make a legal bot for legal tech, well, it will be very difficult here. This is a subject of very long disputes and very long research. I'm afraid this is something you will have to do yourself. How to compare document embeddings from different models? How to understand which model suits you better? Embeddings are about the retriever and the reranker, if you have them further. It depends on how you measure the correctness of the context. If you measure the full inclusion of a piece with a correct answer in the found context, then this is one metric. If you measure just the inclusion, well, just by the link there is some article, we pulled up some link, great, we found something, then this is, well, a little different, these are different numbers, but practically all of this will be either hit rates or, uh, precision, and you just measure where your ranking metrics will be better. The question is simply what exactly do you consider a hit? And this is a whole subject for discussion. Let me return to reranking. We are already not just one foot in, we are already two and have stomped, but for everyone, for everyone. Once again. Uh, reranking is needed to, uh, improve search quality. We initially index our documents with something quite lightweight and like, well, there's some cosine similarity, yes, it's fine. Uh, how it was encoded, so it was compared. And often we can afford to do primary processing, like extract the top 50, top-10 documents, and then re-rank them in the hope of bringing the most relevant one to the top from these ten, if for some reason it fell into the top-10, but remained at the bottom. Actually, for this, heavy, uh, including BERTs, cross-encoders are used. We, in particular, use Smarqa. We take these top, I think we re-rank the top 50 in some tasks, uh, through a cross-encoder, give new, uh, metrics, and say that this is great. There is also such an idea as reranking++. You can also add boostings to this thing. This is like in memes about "just need to add monitors." Well, a traditional data scientist's fun is to slap boostings on top of everything. It works. Actually, we calculated something here with chunks. While I was distracted, you can see what we got. Uh, we have a large context here. Well, we set a large chunk size, and did it fit completely or not? No, not only. Look, here we have sentences split separately, here an application separately, here a sentence separately. Well, that's already up to us. But the fact that the events were divided quite clearly and similarly in meaning is good. The other thing is that we sometimes lose the name Filaret, and this can be a problem. Well, this is precisely why we have to conduct all these wonderful experiments ourselves. Uh, here, for example, we decided that Semantic Chunker will remain as is. Here too, there will be only one chunk. In general, we just fit into 512. We will have to work on this more, but I will leave all this to you. Once again about reranking. Here we have a question. We can, generally speaking, for clarity, use, uh, my bad, uh, query here and use it without editing. This is always the most fun and the most realistic scenario. Uh, here we have it. We can search for this, first we will search for what we have without a cross-encoder. And we can see which documents were pulled up in the top. The most relevant will be at the top. Well, actually, it's just one here because we have Ah, let's remove Filaret. Let's say, when the Armenian state was created. That's how we live. First, we do as badly as possible, and then we show how to get out of it. Voila, we spoiled the query. Great. Here we didn't do so well. We have some kind of Urartian-Greek monarchy. Spoiled as it should be. Everything, everything is fine. And we see that here it was re-ranked, or rather, on the contrary, without re-ranking. We don't even seem to have found the necessary context. We didn't even find it. Now it will be interesting here. We can take a larger number of chunks and see what happens. Actually, you can play with all this. The idea is initially: the cross-encoder gives us scores of how similar the queries and text snippets we found are to each other. We look at them together again, but with a slightly closer look, we get new numbers by which we re-rank them. You can see the order is definitely different, but it seems we spoiled the query too much. As soon as we removed the name Filaret, it became much harder. That's how we live, that's how we work. I won't even comment on this. This is literally how we work. Let's now move on to the questions. Because there were the most questions about cross-encoders. I don't remember where I stopped. Regarding Naiton. How promising will it be to use such code solutions instead of programming everything yourself with the same FastAPI? Maybe based on your personal experience, can you tell me if there is any statistics on project requests for Naiton versus solutions via code? I can't give statistics offhand. Uh, Naiton is handled by a neighboring team, with whom we have a rivalry. Just kidding. But, honestly, because our projects practically don't overlap, uh, I can't give metrics, but even from the perspective of some experience and common sense, it seems that solutions like Naiton can be a bit unstable, unreliable, at least in the sense that here you coded everything yourself, and it, uh, doesn't change much and requires maintenance. And you know exactly what kind of maintenance it requires. And Naiton can go the wrong way and enter the wrong door at any moment. If that suits you, then okay. Again, uh, the flexibility of the system, you can code anything you want in Naiton. Well, please, be so kind, follow the current rules. Mix of semantics and keywords. Yes, I've already talked about that. Are there use cases for Colbert? No. Or at least, I don't know about them. What to do if users add many documents to the database, they start duplicating? The deduplication mechanism, there are many of them, they are very different, and you can spend, to be honest, another workshop on this. And this is a good idea, by the way. I think we were even going to discuss it. Do you use technologies that allow building a knowledge graph, using GraphRAG? Uh, we tested it, and it worked well on some data, but "well" is relative to the same numbers that we used with just the regular vanilla RAG on the scheme. Uh, we haven't found a case yet where GraphRAG would work, I don't know, in Russian, outstandingly, sorry. Multilingual language model. Yes, we haven't found such a case yet where GraphRAG would break all records and work just wonderfully. Therefore, we have stopped at, uh, vanilla RAG for now, at least because it's easier to maintain the knowledge base. How do you store test data? This, please admit honestly, if this is a question from one of my colleagues. We are currently developing a platform that would allow versioning, processing, storage, and logging of changes in spreadsheets from clients. Well, you ask, why spreadsheets? Because it's the easiest for, including, the client. If it works, please don't touch it. Uh, yes, but we are learning to process spreadsheets. We are building a whole platform, we hope it will be ready by mid-August. And you will probably even see us at some good conferences where we talk about how great we are. Do you protect against data poisoning concepts? All of this, if I understood the question correctly, remains the responsibility of the clients. A system of constant validation or testing on users already in production to correct something based on evaluations. Before release, we have a so-called golden set on which we test changes. After going into production, we get user statistics. Our clients label them. Based on the statistics, we look, this is also useful for the client if they want to update and supplement the knowledge base. Uh, and what about our time? How many more questions can I answer, because we have literally 2 minutes left, so I suggest you leave a pleasant aftertaste. So, if anything, then, uh, I think we absolutely won't have time to answer all the questions, because I see that they are being added and people are still writing. There. Uh, the topic is hot, so I suggest >> Okay. I, uh, I guess I got carried away a bit with the introduction, right, that RAG in 2025 will be of no interest. >> It's definitely interesting, because we have such a heated discussion in the comments here. People are not just asking questions, but also starting to answer each other and share experiences. Therefore, uh, it's definitely relevant and definitely topical. >> Well, write, call, we'll answer everyone. >> Uh, I think that if anything, we can still answer in the comments, yes. Uh, there can definitely be a discussion. Uh, but is there anything else on the slides that you would like to finish with? Uh, actually, I'm mostly done. The only thing is, I'll go back here to, uh, I promised to tell you when we don't need RAG, and this is something worth forwarding to your project manager and their clients. In general, if you don't have external documents on which the correctness of the answer depends, then, well, you don't need it. If the query is short and we have a difference from the wording in the documents, or if we have a difference from the wording in the documents, then we use pre-training. If we need citation, then RAG is mandatory. If the cost of error is high or domain knowledge is required, then it is also mandatory. But without RAG, we excellently answer questions about horoscopes, about where to go on vacation. And this is a whole big area for discussion. Here will be a part of my concluding remarks, where this, if anything, is a completely real prompt, part of it. We have an HR portal assistant, Sphere, Sferich. You can see Sphere in Gorky Park in Moscow. We have a whole stand there. We're showing movies. Come, enjoy. There. Well, and this is what the agent bot writes on, uh, on our Sphere portal, and it seems that mindlessly shoving such queries into RAG is, it seems we are hallucinating quite a bit if we shove general questions into RAG. Just like shoving RAG queries without applying RAG. And what to do with this, how to deal with it in the next episodes. >> Okay, then I'll take over from you. Uh, as we understood, there are very many questions. I just sent, uh, Dasha, a link to our post where we are discussing all the comments, so that she can not only take questions from Zoom, but also go in and look, and answer if necessary. Uh, there are questions about, uh, what phone to call, because Dasha, I understood, uh, made such a promise. Here, I will probably adjust and say that, uh, definitely those who join our course on LLM engineering will be able to ask questions in the chat. We are already gathering a warm group, and we maintain direct communication with all the instructors. You can ask questions. Uh, as already mentioned, the topics that we plan to cover within the course and that we will touch upon, not only RAG, but also RAG, including, will be quite a lot, but also agents, and infrastructure. You can learn about all of this in detail on, uh, our website. I hope that the colleagues from, uh, right now in the broadcast, you demonstrated, uh, the special block that we prepared. Uh, we made a separate registration for those guys who were actively with us today, asked questions, engaged in discussion in the chat, and are ready to, uh, go further with us and dedicate 252 hours. And I'm not pulling this number out of thin air, this is the real duration of our course. Uh, to dive deeply, very precisely into LLMs, because we are launching not just some basic thing, but a retraining program where you can, for almost half a year, uh, weekly complete assignments, uh, improve your skills, create your own projects for your portfolio, uh, and stay in touch with the guys who, as we understood today, work with this every day and, uh, also face quite complex topics and try to understand them. A little less than 48 hours until the end of this wonderful offer, this week we are closing registration for the course. So, you can join us, uh, understand LLMs, and do it at a pleasant price, uh, which you see on the screen. And now, I think that the guys will also, uh, demonstrate a QR code so that you can go and, uh, submit applications. And there our team, uh, of course organizers, can tell you more about how the educational process will take place, and, uh, how you can ask questions, and what projects we plan to, uh, put in our portfolio. Well, it seems, uh, I've said everything. I hope I haven't forgotten anything, about the pleasant bonus in the form of, uh, a diploma of professional retraining, I also, uh, mentioned it. Uh, if there are questions, if they remain, then I'll take literally the last ones from the chat. Uh, no, I won't take them, because they have increased there. I think we'll get stuck here otherwise. Uh, we will try to answer some of them in writing in the discussion that we have created together. Uh, well, and for those we can't answer, we will hope that we all will have a pleasant aftertaste, and we will continue to, uh, develop this in our course. Uh, that's how it is. Thank you, Dasha, huge thanks for being with us today. To be honest, I don't remember such a number of questions recently. It seemed like the topic was already known to many, and we would align the information field all together, but it turned out not to be the case. There is still a lot, a lot, a lot to dig into and grow. Uh, on this, then, we say thank you to each other for dedicating an hour and a half of your evening to the topic of RAG. And we say goodbye.