Transcription
Good day. This video was born in agony, turned out to be quite large and complex. I decided to talk about one of the context augmentation technologies for generative AI, and it turned out that I am considering an alternative to fine-tuning a model. Using my usual ChatGPT as an example, and subjectively for myself, I figured out if such an alternative is real. And I will, of course, give my answer at the end of the video. But let's go in order. For example, why is it necessary to fine-tune such an already smart and trained LM? No matter how well-read a bot is, it usually doesn't know the topic we're interested in as well as we do. As a result, so-called artificial intelligence begins to fantasize, to hallucinate, as they say, and as a result, it lies to us very plausibly. If we ourselves are also not well-versed in the topic, then we believe this lie. How can we make our AI assistant answer questions related to our topic or even our business? Correct, upload the necessary information into it. That is, to fine-tune, to conduct fine-tuning, but fine-tuning large models is long and expensive. Constantly fine-tuning on up-to-date information is difficult because fine-tuning is a time-consuming process. And while we are fine-tuning the bot, the information may lose its relevance. And then we upload the current data so that they are always at the bot's fingertips and it can always use them for answers, reasoning, and solutions. We use RAG, Retrieval Augmented Generation, or generation with pluggable knowledge. What is RAG? How and when is it used? What is the problem with using ordinary RAG chatbots? How to use it best, and what does the language issue have to do with it? We will talk about this today. [Music] To begin with, let's recall that chatbots, also known as LLMs, also known as so-called generative artificial intelligence, answer our requests. Our request is converted into tokens and placed in the context window. Initially, it is almost empty, but as the conversation progresses, all our requests and bot responses are included in it. Immediately, the entire context window, that is, every single token from the context, is fed into the input of the language model LM. And as they pass through the layers of the neural network, the LM calculates the most suitable response tokens and their sequence. It is understood that for us, the response tokens are converted back into words. That is, once again, for the response, the chatbot uses all our requests and its previous answers so that its neural network reacts with an appropriate set of words. Usually, a conversation does not end with one request and its answer. When communicating with a chatbot, we gradually bring it up to speed on our ideas, clarify, and explain. Even seemingly unrelated questions and comments are connected by the chatbot during processing directly within the neural network. This is how the notorious context is formed. And let me repeat the key point. All context tokens are equal and enter the neural network input simultaneously. This is important, and you will understand why. But do we really need to bring the bot up to speed step by step like this? Of course, not at all. We can very simply copy and paste the text on the topic into the prompt field, where we write requests to the bot, and thus supplement the context with the necessary information. This text is limited to a few dozen pages, and it will go into the neural network as is, without any prior processing, but all of it, and this is important. However, we can attach the same text to the prompt as a file. And this will finally resemble RAG, although not fully functional in a technical sense. That is, the file or files attached to the prompt are processed entirely by the chatbot, but only a part of them is used. By the way, approximately equal in size to the text that we can paste directly into the prompt, it selects relevant parts from the files to include in the context depending on your request. What is important is that the attached information, approximately equal in volume to what we can paste directly into the prompt, also gets into the context almost entirely. But with a small amount of information, a few dozen pages, it is better to attach a file than to simply copy it into the prompt, because even a primitive RAG will filter out what is definitely unnecessary. [Music] Regardless of how RAG processes data, it ultimately gets included in the context window and goes for general processing along with requests, responses, and other context. That is, in essence, RAG is a way to include selected, in theory, the most relevant additional data in the context for generating a response. Remember, generation with pluggable knowledge? It is important to understand that RAG itself does not generate a response, but only provides additional contextual data for it. If you don't include tools for working with code and uploading special data, then the chatbot has exactly five ways to obtain data for built-in RAG. First, files attached to the prompt. Second, web search. Third, a link to a page directly in the prompt. Fourth, project files. Fifth, knowledge files of a custom GPT. [Music] Theoretically, you can attach up to 10 files of different formats to the prompt. And the size is decent: up to 2 million tokens in each file. For reference, on average, one token is 3/4 of a word in English. For Cyrillic texts, the density is lower. Tokens can be larger, but the proportion can only be determined experimentally. And although in reality you may encounter limitations on the size and number of files, theoretically 10 files of 2 million tokens each is a lot of information. This is up to 40,000 book pages, fifty or even a hundred normal thick books. A whole library simply attached to the prompt. And here comes the trap. We attach a lot of information to the prompt, but RAG selects only a couple of dozen pages from it, only a tiny part of it that it considers relevant. With tens of thousands of attached pages, RAG will select a fraction of a percent from them. Note that RAG selects information using ancient methods of semantic text analysis and simple vector calculations. Not so-called artificial intelligence. The text of the attached files is broken down into so-called chunks. Essentially, quotes of 300-500 words, and then RAG selects a couple of dozen quotes and inserts them into the context in their original, unchanged form. With a competently composed prompt, the bot will indeed be able to search and find a needle of meaning in a haystack of text, but it will be, in fact, a needle in relation to the haystack. And one can only hope that this needle is the right one, the one that is needed. It is important that files simply attached to the prompt are only for that prompt. And only what you extracted from them during the execution of that prompt to which they are attached remains in the context. It's useless to tell the bot, like: "Remember, I sent you files a couple of prompts ago?" It might nod in agreement, saying, yes, it remembers, but it will only actually remember what made it into the context. [Music] You can also instruct the bot in the prompt to find and use up-to-date information from the internet or even specify a web link to a page, article, or document. And this will also be RAG. Although it will be the most primitive RAG of all in terms of quality and quantity of information used. When searching, the bot will access a search engine, get several links, download them, clean them of obvious junk, and search the obtained text for relevant fragments, which will then be loaded into the context. And again, it will be no more than a few dozen pages. Although this is not so bad. A few links are unlikely to provide much source data. Approximately the same will happen if we specify a direct link. When working with a link, ChatGPT downloads the page and extracts the text, trying to find relevant fragments. But unlike vector databases and its own embeddings, only superficial semantic analysis and partial parsing are possible here. Within the current session, the bot does not re-access links and does not download information from the internet again. But from the first time, it loads enough into the context to tell a lot about the article. Essentially everything. And yet, if you have a link, say, to a PDF of a scientific article, it is better for you to download the PDF to your computer and attach the file to the request than to specify a link to the document. The analysis will be better. What is important here is not the volume of input data itself, but how compact and relevant it is. The less extraneous information, the higher the chance that the necessary information will fully enter the context and be taken into account by the model without truncation. The most thorough built-in RAG of ChatGPT is project files and knowledge files in the Knowledge section of custom GPTs created by paid subscribers. I have already talked about creating such a GPT. Watch it if you are interested. Approximately the same happens with these files as with those attached to the prompt. But the semantic vectors of their chunks, so-called embeddings, are stored constantly as long as the files exist. And these files can be accessed any number of times from any project chat or custom GPT chat, respectively. That is, files constantly attached to a project or custom GPT knowledge are always ready for use, similar to a small vector database. Moreover, ChatGPT accesses them with every prompt in a project or custom GPT, supplementing the context with excerpts from them that are relevant to the specific request, constantly and always. That is, the haystack of text of attached knowledge is searched every time for that very needle that is relevant to the specific request. In custom GPTs, you can even specify in the settings which knowledge has priority: that taken from the bot's main knowledge base or from files. I have done this, and it works. But in constantly attached files, the search is conducted using the same old methods of semantic text analysis and simple vector calculations. And only what is found by these ancient methods gets processed by the AI. [Music] So, in ChatGPT, as an example of a chatbot based on LLM, there are several built-in ways to update knowledge, and each is useful in its own way. If you simply need to analyze documents or documents once, for example, to find contradictions in them, it is easiest to attach files directly to the prompt. In this case, GPT will read them, extract the necessary information, and provide an answer. But you will not be able to use this file later. It is only available within a single request. When you need to find out something up-to-date, for example, breaking news, the latest scientific publications, or updated data, it is better to use the built-in internet search. GPT will perform the query itself, find suitable sources, and insert text fragments from them into the response context. If you already have a link to the desired article or page, you can simply paste it into the prompt, and ChatGPT will download the content, analyze it, and include it in the response. This works the same way as search, but without extra steps. You precisely specify what needs to be read. If you are working on a long-term project and using many related files, for example, writing a course, a business plan, or research, developing a program, it is convenient to connect all the necessary files to the project. Then JPT will remember them in all prompts within that project and use them as background knowledge. By the way, it is good practice to update and supplement project files. I do this myself constantly. Finally, if you want to create your own assistant that will use a specific knowledge base in any situation, it is better to upload the necessary materials to the knowledge base for custom GPT. Such a GPT will be able to use them in any interaction, regardless of the session or prompt. Theoretically, up to 20 files of 2 million tokens each can be connected to a project or for the needs of your own GPT assistant. Although in practice, OpenAI itself does not guarantee this, but even theoretically, this is 40 million tokens, and many sciences will fit into this volume. The famous British Encyclopedia Britannica contains 50 to 60 million tokens. Quite comparable volumes. Of course, it is very convenient to combine built-in RAG technologies, for example, within a project to conduct internet searches or analyze attached files, say, comparing them with project files, i.e., with knowledge from them. Recall, these are all built-in RAG capabilities that are very easy and convenient to use. If you understand how they work, you are essentially creating your own cloud vector database, and the chatbot uses it to answer your requests. And there are even larger subject-specific knowledge bases, and you can also connect them. There are ready-made ones with their own. There are datasets that can be converted into vector form and stored locally, loaded into one of the free engines. Programming here is your helper. If you understand even a little bit how software works, you can do it with the help of a chatbot. It will take some effort, but it can be done. I have checked, in principle. By the way, the search engines of such vector database engines are most often more sophisticated than the built-in RAG chatbot. Although they also have no relation to AI. A local database will be absolutely necessary if you do not want to expose your documents and books to the chatbot and its owners. For example, if it is medical records or court cases of real people. There is a lot of information that should not be put in the cloud. With some experience and understanding, accessible to any educated person, you can build a personal assistant that uses RAG simultaneously for Custom GPT files and several local vector stores. Talk to your chatbot, it will tell and show you everything. Now let's summarize the problem as I see it. The fact is that RAG has nothing to do with the notorious AI. It is an external procedure for it. It is obvious and undeniable that context plays a decisive role, but RAG prepares and extracts additional context using relatively primitive methods. This is a very old approach. Semantic text analysis and simple vector calculations. We do not task such a wise neural network with searching for meaning and patterns in the entire array of information that we consider relevant. We put a not-too-smart relevance filter at the input of the neural network and let it decide on what basis the answer will be obtained. This is why, among other things, there is a struggle for context size, so that more crudely filtered information gets into it, and let the already smart neural network digest it. [Music] I am not the only one who sees this problem. So far, we have been talking about the simplest, or as they say, naive RAG. It simply breaks documents into semantic pieces, finds those most similar to the request, inserts them into the context, and the response is generated based on this. All chatbots and standard libraries do this by default. However, there are several other varieties of RAG that allow improving context relevance. And I will list them for your information and add that there is active work being done to improve these algorithms. For clarity, let me repeat, RAG always stands between the user's request and the neural network input, but in some architectures, it does not just filter and transmit context. RAG acts in an agent architecture, iteratively communicating with the LLM. A smarter iterative RAG makes several passes. First, it searches for data, then the LLM determines what is missing, clarifies the question, and repeats the search. This approach is implemented through special software agents, and in a ready-made form, it is available in systems like Perplexity or WebGPT from OpenAI. Hybrid RAG combines vector search with filtering by structure and metadata. For example, we search by meaning, but only among documents created after 2020. Here, as you can see, it's the same dumb, but improved search. Modular RAG is built as a chain. Separate blocks are responsible for search, selection, ranking, and response generation. Hierarchical RAG first determines the topic or domain, for example, biology or diagnostics, and then searches within it. This is more accurate, but also not like artificial intelligence. Agent systems also use RAG as one of their tools. LLM can access memory, run external APIs, access a knowledge base, check intermediate hypotheses. Self-RAG uses LLM as a self-critic. After generating a response, the model identifies weak points, returns to the search, and supplements the information. And finally, knowledge routing systems analyze the request itself and direct it to the appropriate storage, for example, to a database of medical diagnoses or a database of medications. I have not directly encountered such an architecture, but it seems to me that it would be possible, for example, to process all additional content in parts through the LLM along with the same request, so that the neural network uses all its power to find matches and analogies. [Music] RAG is a good tool for combating the notorious chatbot hallucinations, which we have already talked so much about. Any, even the most naive RAG, fights LLM hallucinations because it adds real texts from external, implied verified sources to the context. The probability that the model will need to guess what to answer is less, the more accurate and complete the information in the files and databases. But the self-RAG mentioned above does even more in this regard. It doesn't just help the LLM, it literally forces it to check itself. First, based on the user's request, a regular RAG performs a search of external data, extracts relevant fragments, and feeds them along with the request into the LLM's context, which generates a preliminary draft response. This response is returned back to the self-RAG module, which returns it to the LLM with a demand to perform a verification of authenticity. I often do this manually, but here the process is automated. The model again receives the same set of data in the context plus its own response and, in the role of a critic, checks it. It looks for which parts of the response are confirmed by the sources and which are not. If it sees gaps, assumptions, or contradictions, it either simply regenerates the response or sends a request to RAG to supplement the context. The cycle can repeat several times until the model either receives sufficient confirmation or explicitly marks questionable statements. For example, this fact is not confirmed by the sources. It's like that very iterative method of sifting through the entire haystack to find all the needles in it. Such a wonderful thing. Generation with pluggable knowledge, RAG, makes language models significantly more useful in practice. Now we can answer the main question: when can RAG be used and when does the model need to be fine-tuned? If you are using a relatively small amount of changing information within a few hundred pages, RAG is exactly what you need. If there is an unchanging body of data of many thousands of pages on your topic, use fine-tuning. If fine-tuning is impossible or not advisable, use RAG, but create a local database on an advanced engine and use iterative RAG with self-checking. Since I am close to the medical field, I will give an example. Fine-tune the LM on medical textbooks and reference books. Create a local database of outdated medical histories with advanced RAG. And attach each current medical history to the project. This is a rough, simplified explanation within my understanding. By the way, I would be happy to discuss this topic. Express your opinions and criticize. [Music] And what does the language issue have to do with it? You might ask. And the point is that the model's training data, your requests, and RAG data must be in the same language. All effective models are trained approximately 95% in English. This applies even to various Chinese deepfakes. What to do if almost all of the model's training data is in English? So, if the context of the request, including the prompt and RAG data, consists of non-English tokens, it will find a much smaller response in the neural network's parameters, because there are incomparably fewer tokens of the same language there than English ones. Obviously, we are talking about so-called leading chatbots, truly large LLMs. And chatbots by default do not translate your requests into English. So, either translate your requests and your files, or address and provide the chatbot with information in the language most familiar to it. In short, teachers, while there is still time.