Transcription
Have you ever wondered if we can take and train a neural network, for example, by providing our own code examples, our own files, datasets, and videos, so that the neural network understands you better and works more personally for you? At this point, you might ask: "Why train a neural network at all? They are quite smart on their own and can do a lot." To which I will answer that this is indeed true. But if we imagine it conceptually, a neural network by itself is just a brain. That is, it is some kind of process tool that can understand human speech. It is trained on a general database, on books, on movies, on articles, on the internet, and it understands what humanity has described. But, most likely, it does not understand specific information about you very well. And that is precisely why a technology called RAG comes into play here. We will talk about it in detail today, and throughout this video, we will find out how RAG helps solve this problem and what it represents. We will first figure out how it works in general, why it is needed, then we will move on to more technical aspects, how it works internally. After that, I will show you an example of how to write the simplest RAG in JavaScript so that you simply understand the mechanics. After that, we will set up Cursor so that it works with local RAG and files and code examples. That is, how it can be used. And after that, at the end of the video, we will move on to more advanced mechanics. For example, we will analyze 11 RAG strategies, more advanced than just basic fundamental technologies, so that you understand which one to use in the right situations. In general, by the end of the video, you will clearly understand what RAG is, why it is needed, and how it works. And let's immediately give an example for understanding what this means. Here we have RAG, and, let's assume, a situation. We ask the neural network what the remote work policy is in our company. If we just ask this, then just imagine the face of ChatGPT, which is like: "What company is this? Why is this needed, I don't know." And it will most likely answer you some generalized nonsense, because it knows nothing about your company. But if we still want to get an answer from the neural network specifically about our case, then we can take and provide a document where our company's policy is described. We can even indicate the necessary paragraph there, where it is directly written that this is our company's remote work policy and so on. And we can tell the neural network to answer based on this. Obviously, it will read the primary source and then be able to adapt it to our output, right, that is, to answer what the policy is, what is allowed, what is not allowed, and so on. That is, it will understand some foundation. That is, we expand the functionality of the neural network by adding our own document, which the neural network reads and based on which it can provide you with information. That is, the neural network becomes just a processing tool that can work with any files and texts, and so on. In general, this is called RAG, because RAG is an abbreviation for Retrieval Augmented Generation. The meaning of this technology is that any neural network, in fact, that exists now, has a certain context window, that is, the number of tokens it can perceive. For some neural networks, it might be 128,000 tokens, for some neural networks it will be a million tokens. The point is that this is a limited amount of input that the neural network can interpret. And suppose, when you work with some large databases, for example, with company policy, then company voice, or, for example, a large number of repositories that need to be considered. If we take the example of coding, then everyone understands perfectly well that agents that can write code and so on work great. But the problem is that they don't understand the project as a whole, because to consume it entirely, a large number of tokens are needed to understand all the files, structures, and so on. That is, current neural networks cannot do this, with rare exceptions. And thus, it turns out that we can describe all this additional information in our databases, files, and so on, and then add them to the neural network. And thanks to this technology, it will have the ability to access these files without filling up its context window. Thus, it turns out that if we simply explain in the end, we have a neural network that works with a clean context window, but at the same time it clearly understands all the volume of information that we provide it. Let it be many repositories, or a huge number of PDF files of hundreds of pages. In principle, all this is feasible thanks to RAG technology. And here is a very simple example of how it works. So, here, if you look, this is the simplest diagram of the pipeline by which RAG can provide information. Here is a user, and they make a specific request to the neural network, right, in this case, it's an LLM. And suppose, we can also ask here, what is the remote work policy here, for example. Then this request goes to the LLM, that is, human speech comes here. And then the LLM starts to think according to the following block diagram. It thinks, can I answer this question independently? That is, do I have enough competence to give a generalized answer? If the answer is yes, then it answers independently, without using any databases. Well, a very simple example here could be, say, write me a function to convert kilometers to miles. It is obvious that for this function, everything has long been known. No specific data is needed. The neural network can answer independently. But in case there is something personalized, for example, we say: "Write a component in my style." The neural network is like: "Damn, what is your style?" And so on. In general, I don't understand. And therefore, the following mechanisms are activated here. So, if the answer is no, it cannot answer independently, it gets access to the retriever. This is, in fact, the first letter in RAG. This is the thing that allows interaction with an external database. For example, we have a database, it is stored in books or on the internet, but, honestly, it is not so important. We will now talk about an effective format. In fact, it's all vectors. This retriever makes a request to the database for simplicity of understanding now. Imagine that it is just a PDF of 100 pages. Just a PDF of 100 pages. And the retriever is like: "Okay, I'm going to make a request, search for what the component writing style is for this particular user." It received an answer from this database, from this PDF. And, accordingly, then it takes this answer and passes it to the LLM. That is, it says: "Ah, look, the general style of writing components is this and that." And after that, the neural network, understanding the source, generates an answer for us. That is, you see, the process is somewhat complicated, but at the same time, we get more personalized information. Now, accordingly, let's move on. So, we now need to figure out what needs to be done for RAG to work well. Look, we just discussed an example where our database was represented as a 100-page PDF. But working with such documents is very inefficient, because in the end, imagine how many resources are needed to review all 100 pages. Quite a lot, right? And if we have many such files, then, in general, the answer will be vague at best if we offer the whole file at once, because, among other things, this is processing resources. And therefore, for RAG to work effectively, we need to cut documents into certain chunks. Chunks are pieces, for example, about 500 characters, right? So, in this case, when we cut the general whole information into a small number of characters, then in this case our answer will be quite clear, right, and this is the first rule that is necessary. The second, which is no less important, is that each chunk, right, so we divided it into 500 characters, each chunk, we convert into a vector. A vector, essentially, is a data format in numbers that neural networks understand. It is more compact than human speech. And why is this done? All this is stored in special data, in special databases. This is done to enhance the search tool of RAG itself. Imagine that you, for example, in this 100-page document, search by the keyword remote work. It will surely give you exact matches, but the problem is that it won't find matches by meaning, right? So, here you can see an example to make it clearer. And if you search specifically by keywords remote work, then RAG will not find, for example, work from home. Although, in principle, it's the same thing. That is, this would be an incorrect parameter. Therefore, we want, so that we have semantic search, when the search is by meaning, even if the words themselves differ. For this, certain vectors are also needed, because they represent not the words themselves, but the meanings, right? And here you can see an example of how it works. That is, when we search for remote work, we can get close to something like, for example, home office. The words are completely different, but in meaning they are more or less suitable. And bonus is compensation, adaptation can be orientation, and so on. And the concept of vectors, if we speak very simply, is numerical data that shows the distance in meaning to certain words. This is precisely the data format that neural networks understand. That is, when we input this word, say, remote work, by vectors, it starts to determine the semantic distance to other vectors that might fall under this meaning. This is roughly how it works. So, now that we understand the principles of RAG's operation in more detail, including with vectors, we can look at this diagram, which, in principle, works similarly: there is some user, they make a request to the LLM, which is the generator. And then it works together with the retriever, which then goes to the knowledge base, for example, to the 100-page PDF. And then, you see, it breaks them into chunks, that is, into these pieces that are understandable for the LLM, and they are gradually fed into it so that it can form a request. On the right, you can see certain advanced RAG strategies, because there are different tasks, there are different criteria for accepting these tasks. But we will discuss these strategies later. You see, there are already additions here that make this process more optimized. Now I would like to go through a code example of how a simple RAG works. For this, I will use the Node.js platform and the JavaScript language. Here is a very simple code that will simply allow you to understand the principle of how it is structured internally. So. You can write it yourself if you want, the source code will be available via the link in the description. And the point here is this. First, we import ChromaDB. This is a database that works specifically with vectors. In principle, there are others, but it is one of the most popular. Next, from Chroma, we import something like the default embedding function. The embedding function is, in fact, an object. The embedder is precisely the thing that takes and allows us to translate our human words into semantic vectors. This is the mechanism we just looked at. Then we initialize Chroma itself. We add OpenAI, in fact, ChatGPT. And then what happens? We assume we are analyzing cases when we work with local data about remote work. In this case, the documents are represented as an array, but again, you understand that it can be an array, or PDFs, or videos, texts, in general, anything you want. It will just complicate the mechanics of the code itself. Here, it is important for us to understand the example itself. So, here we immediately create a new collection, CompanyPolicies, that is, company policy, and immediately add an embedder, that entity that translates into vectors. After that, we load the documents themselves, they are described here in an array. After that, we add them to the collection itself. So, there is nothing about RAG directly here yet, except that it will already be converted into vectors. This is the key point: we store it here not in text, but we store it all in vectors. And then, suppose we have a user request. We ask the LLM what our remote work policy is. That is, obviously, we are hinting that we want to get information from our file system, that is, using RAG technology. Accordingly, then we start searching for the closest chunks by meaning. We make a request to our collection with the user question. And by meaning, we search here again, because vectors are stored in the database. Accordingly, we convert them into vectors and search for these distances to the semantic load. After that, we convert directly and then assemble the prompt for the LLM. The prompt looks like this. That is, we add the information we received from our database to it, right? So, according to the diagram, remember, we received it and set this context directly in the prompt. After that, we take and insert the request here. That is, RAG itself has nothing in common with the LLM. We independently, without the LLM, get this data from the database, in vectors, in files, it's not critical. And after that, we simply form a prompt that is understandable, a context that is understandable for the LLM itself, which then answers it. After which, accordingly, we access the API of ChatGPT or another, which is not critical, to answer the person. And it will already answer in the required context with the necessary data. That is, this is a super-mega-simple implementation, which you should understand how it works through this example. Accordingly, I want to show the next example within Cursor itself, how we can set up a local RAG here that can advise us coolly. If you don't know, are not familiar with what Cursor is, how it works, and, in particular, we will talk about MCP, then on my channel there is a full course on Cursor. You can watch it, and you will understand how it works. But the point is, let's go to the settings. Here we have "And MCP". I will add a new MCP here. I already have pre-configured examples of how RAG works. I will just explain to you what it is. And the point is that this is the main tool for working with local files. Here I use the library called "MCP Local RAG" for implementation. And then I specify the base directory where my documents are stored, which are local. So, this is how this database is represented. And to demonstrate the example, I can ask, yes, let's leave it without comments for now, so that you simply understand how it works. That is, I address the Cursor agent and you see that some operations are already happening here, for example, such as "query document". That is, we request a document and it finds the file "3 context.pdf" for us, and accordingly, it explains that these are session management contexts. These are fundamental elements of context engineering, where a session represents a self-contained record of a dialogue between a user and an agent. So, it received this information from my local files. If you look at this database, here is my "dev" folder, which is indicated here, and "documents", right, I go here, and you see that I just have PDFs lying there. So, thanks to such a small setup, I was able to get access to local PDFs within Cursor. And all this works for free, great. If you look at the context itself, it's a file and a guide from Google, where it talks about context engineering. Accordingly, imagine what we can do. We can talk, for example, about how to create various agents or, for example, describe a cool architecture for a project and build our project based on it. That is, we won't have to invent some concept from scratch. No, we just put it into the database, just like any data, and then tell the neural network to work with it. So, this is the interesting scheme here. Now a little about the installation. Here, yes, you see, so everything works on Node. It's more convenient for me, since the database is frontend, specifically JavaScript. And here it assumes that we are working directly with NPX, where we get access to the "MCP local RAG" library. But for some reason, it didn't work for me, so what did I do? I copied, or rather, cloned this repository locally, added it here. After that, I did "npm install". Then in the package.json, there is a script called "build". That is, it simply builds the entire project, as a result of which we get the "dist" folder with the "index.js" file. It is precisely to this file that we specify the path. That is, this is essentially the file that will execute the actions of the library itself. That's it. After that, we add the .env file, where our local files are stored, where our database is. And that's it. Voilà, our local RAG works. You can use it perfectly. At this point, you might ask me: "Well, what is local code?" This is an example of how to use this tool to take and use your current codebase for writing code on other projects. That is, conditionally, we feed some repositories here, which are stored, in my case, in this project. You can feed a lot of your repositories here that you have written, or just some code examples, how you write components, what guides there are, what features there are, and so on, and turn it into a database. And after that, it turns out that with the help of RAG, you can not just blindly write, right, some general components, but take your specific repository and examples of your code, your architectural decisions, stylistic decisions, use of libraries, and so on, and tell the agent: "First, look with the help of RAG how it is done, and only after that write the code." You can play with it. The results are quite surprising. Now let's go back to the presentation and analyze the main mistakes that you can make when implementing a RAG system. It is clear that the first mistake, we have already discussed it, you can take and feed the entire PDF or the entire large database to your vector store. That is, it can simply get clogged up and so on. We know that it is necessary to divide it into small chunks, which range from 300 to 500 characters, which gives the most effective result. To use keyword search instead of semantic search and not use vectors. Absolutely, not to clean the text, because if the text does not contain useful information, such as, for example, HTML tags, some garbage, emojis, and so on, then, firstly, it increases the source file, and secondly, it can simply break at least. Well, and accordingly, the fourth is to make RAG without chunking, so that the work is simply more optimized. If we talk about specialized applications where RAG is really needed, then first of all, it is project documentation, when again, you don't add information about how your project is structured to the context every time, but you move it to an external database and it is stored there. That is, you access it and so on. Team knowledge base, company policy, employee onboarding works very well, which again, you store all this in isolation. QA systems, support. Very conditionally, you have a robot on your website, for example, that answers customer questions. And in order for it to understand the specifics of your project, this is precisely why a RAG system is created, so as not to spoil its context every time. Code analysis, we have just reviewed these options. Well, and auto-generation of new parts of projects in your style. I also showed an example of how you can implement this. And at this stage, when we have more or less figured out RAG and so on, we can move on to a more advanced story. We will analyze it at a high level, but so that you at least have an idea of what advanced strategies exist for implementing this technology. And what is interesting, so that you don't get bored, first, I generated this picture using a neural network. So. Here are described 11 strategies that we will now analyze. But again, to make it more fun, I generated the exact same Beginner's Guide in the style of Lord of the Rings. And it's absolutely magnificent, because here is absolutely the same information that is in the style. This is a question of what neural networks can do now, how they work. You can download the pictures in the description of this video, analyze them in more detail. And this is called Nanban Pro from Google, a cool neural network. In general, this is a separate topic, right, we will now focus on RAG. Accordingly, here are 11 strategies on how to implement for different scopes of tasks. And let's start with the first strategy to understand where to use it. And it is called re-ranking. This strategy is characterized by the fact that at the first request, we get the entire set of documents. And after that, a small ranking model takes and sorts all these documents by relevance to provide the most accurate result. If you look at the Lord of the Rings example, then, well, it turns out that it extracts all the gold nuggets and then filters them to get the most necessary ones. The second type of strategy is called RAG. It relates to Search and Retrieval Methods. And it is characterized by the fact that when making a request to the database, the model itself decides how it will be more effective to proceed. Accordingly, it is called adaptive. It chooses, decides to search the entire document, get all documents, search by meaning, and so on. That is, in general, it is fully adaptive. Well, and we rely on technology here. It accesses the retriever, which goes to the context, which goes to the LLM, and then generates the request. Well, purely for interest, the agent is indeed shown this way. That is, it decides for itself whether to read everything, search by metadata, or by search. Well, and after that, it decides how to deal with this search mechanism. The third type of strategy is called Knowledge Graph RAG. And its meaning is that here we extract not only the information itself, but also the interconnections and certain structures. The point is that this strategy is needed when, in addition to the answer itself, we also need to answer the question: what is it related to? And thus, we provide more complete information. Here it is depicted as a constellation. That is, dwarves and elves argue about who to get, Frodo or other characters. And thus, it shows how they are interconnected. I am amazed, actually, by this illustration. It helps to understand how it works much better. So, the fourth strategy is called Contextual Retrieval. And the meaning of this strategy is that before we convert our database into vectors, we add a certain context to it. In short, when reading, not only the database itself will be immediately clear, but also a brief context that describes what is happening there. Thus, the search speed will be significantly higher. Well, as tradition dictates, let's look at the analogy. Yes, you see, there is a small note that we put in the library. I think this is very, very clear. So, we are interested in the fifth strategy. It is called Query Expansion. And it relates to optimizing the request itself. Well, the point is that when we, as users, write a specific request, for example, to get the remote work policy, this strategy takes and improves the request using the LLM. For example, I recently created a GPT called "Deep Prompt Generator". The point is that you input your prompt, and it outputs an optimized prompt that literally increases the LLM's response by 3 to 10 times. That is, we optimize the prompt itself using the LLM. This is called meta-prompting. This is roughly the same story. By the way, if you need such a GPT, there will be a link in my Telegram channel. Use it for free. I have also included the meta-prompt itself there. Work with it, it's a very useful thing. So, here, as you can see, Gandalf wrote: "Find the ring." And then it turned into an amazing map, with infographics. And, accordingly, this thing can go and pass through the gates. By the way, the sixth strategy is called Multi-Query RAG. And it is characterized by the fact that instead of one request, like Query Expansion, we make several such requests at once, and then compare them for relevance. That is, if the fifth strategy gives us the most accurate result based on the request, because we optimize this thing, then in Multi-Query RAG, we optimize it, and also make several requests. Yes, it takes more resources, but it gives a much more accurate result, which is also a great strategy. Well, here, as you can see, besides one Gandalf working, we also send the Fellowship of the Ring. Also quite cool. The seventh strategy is called Context-Aware or Hybrid Chunking. And the point is that we take our 100-page document using the classic chunking model, when we cut the document, we divide it into, say, 300-500 characters. But the problem with this approach is that we don't understand if we are cutting by meaning or not. And at some point, the meaning can be simply cut off. Thus, the semantic load can be divided into different chunks, which complicates the search. In this strategy, we cut not only by the number of characters, but also by meaning. Thus, we guarantee that each chunk will contain complete information, which increases the accuracy of the answer. In the Lord of the Rings example, we see that, yes, the dwarf works more accurately with the stone, that is, he doesn't just hit it randomly, but, as we know, dwarves make good stones. In general, you get it. By analogy, there is the eighth strategy, which is called Vector Chunking. And its meaning is that we first convert all this information into vectors, and only after that, when we understand the semantic vectors, we then cut. That is, this further increases the accuracy of our analogy. And this is when the elves first understand everything, and then they divide it. Well, I didn't quite understand the analogy with the tree. I hope it is clearer to you. The ninth strategy is called Hierarchical RAG. And here the point is that to understand the entire information in our database in detail, plus to preserve the overall context, we make this request and methods of obtaining information. First, in small pieces, then in general blocks. And then we read the entire document. Accordingly, at the output, we get the ability to see exactly where and how the entire document is represented. And moreover, we understand it in detail, which also increases the accuracy. Well, yes, here, too, there is a certain hierarchy. First, we read the symbol, then we find it in the context of the entire page, and then we read the entire document before making the actual requests. The tenth strategy is called Self-Refining RAG. And the point here is that we force the LLM to check the quality of the context, the answer that we received. That is, if it considers the answer and the information received from the RAG system to be quite poor, then it makes a new request. Thus, we force the neural network and, in general, the entire system to be self-learning and self-verifying. Yes, this requires certain resources, but again, sometimes tasks arise when we need accuracy. Therefore, we consider different strategies here, right? And you see, here, accordingly, some sorcerer, well, Gandalf, obviously, talks with the Palantir. Before giving the answer itself, he checks it himself. Well, for me, this also looks logical. And, accordingly, we are left to deal with the last strategy. It is called Fine-tuned Embeddings. And its meaning is that here we work with pre-trained and fine-tuned embeddings. That is, for example, if we work in medicine, then these embeddings work in medicine. If we work in programming, then they are usually in programming. This allows for more accurate and higher-quality work with data, thereby obtaining a more accurate answer. And in our Lord of the Rings scheme, the eleventh strategy. So, 5, 6, 2, 10. I can't find it, for some reason. Well, accordingly, apparently, there are no pre-trained characters here. But anyway, this indeed relates to chunking and embedding methods, right, when we simply optimize the embedding itself, these vectors, in order to structure information more accurately. Accordingly, dear viewers, that's all. I have told you about RAG. You, hopefully, now have an understanding of what this system is, how it works, when to even write and set it up. We also had fun with various diagrams, but in my opinion, for educational materials, this is just something cosmic and super clear. So, write in the comments, whether it's clear, not clear, what else to cover. I recently wrote in my Telegram channel that we still have many terms that are important to understand in the AI world, for example, right, if you look here, it's large language models, in short, this post is directly about tokenization, embeddings, attention mechanism, in general, and so on, and so on. Accordingly, there is an idea to cover all this so that you are also in the context of what technical terms mean in the AI world, to be a good, in-demand specialist, because this is already the basis now. For this, I will need your feedback, ideally in Telegram, because there is close communication there, there is additional content, videos, posts. In general, subscribe. Well, and there will be more lessons. Bye. M.