Transcription
I think this situation is familiar to each of you. When you communicate with ChatGPT, Gemini, or Claude. It understands you perfectly for the first 20 messages, and then it starts to get confused, provide absolutely contradictory data, and generally lag and freeze up, making it impossible to work with. Often people say that ChatGPT is being stupid or that there's some problem with Gemini, but they don't know why this is happening or how to actually fix it. Stanford research shows that the model gets even more confused if the main data is in the middle of the context window, and in this video, I will break down why this technically happens, and most importantly, how to fix it. My name is Roma Rai, you are on the NeuroEnlightenment channel. Let's go. I think everyone knows that every LLM model has a context window, even those who barely use them. The context window, accordingly, is how many tokens the model can perceive and remember about what was, for example, at the beginning of the dialogue. Let's say the model has 40,000 or 128,000 tokens. Gemini Pro, previous models, already had a million tokens at that time. But few people know that tokens themselves don't actually affect how well the model communicates. For example, if we look at current models, Claude will have only 200,000 tokens. ChatGPT will have around 400,000, and Gemini still has 1 million tokens. But at the same time, for example, in my work, ChatGPT can handle very long dialogues calmly, while Gemini throws out errors like these. It doesn't see the picture, for example, a real case, a screenshot that I'm attaching on the screen, it freezes, stops responding to simple messages. Here are a couple of screenshots for you too. And it's as if the context window should help and make the dialogue better. Well, actually, it's not that straightforward. In 2024, Stanford conducted a study, which resulted in this curve. And they found that in fact, the accuracy of answers is maximal when the data is at the beginning of the dialogue. Normal accuracy of answers will be when the data is at the end of the dialogue. But if your data is in the middle of the dialogue, then the accuracy of answers will drop by 30%. For example, at that time, ChatGPT 3.5 literally worked worse when data was needed where the context still remained. For example, finding information in the middle of thirty files that were sent to it. But if we removed the context altogether, and it stopped remembering anything, even then it worked better. Essentially, GPT models have a rather limited resource because it's basic math, right? At the beginning, the model will remember everything, after which the context window will fill up, and it becomes harder for it to find the necessary information. Well, let's take any person, right? If you have one A4 sheet of paper on your desk with information, you'll find it in a short period of time. If you're given a book, you'll have to try. Most likely, you'll get tired and just stop trying. Essentially, the same thing happens with LLM models, to put it in very simple terms. The so-called context rot, when the chat stops working over time, looks different for each neural network. I will tell you based on my personal experience, because I use all three, and I have something to say about all three specifically in the context of context rot. First, with Gemini, it looks extremely bad. Here's the same screenshot I showed you. Gemini, when memory completely rots, and the dialogue exceeds, that is, on average, 40 to 80 messages. It all depends on the amount of data you uploaded there. Gemini just starts to get confused. The biggest bugs with Gemini at the moment are that it stops accepting images. It doesn't see them at all. It pretends they don't exist, and it simply stops answering your questions. For example, I write to it: "What's on my plan for breakfast today?" Because I have a nutrition and training plan, etc. It's in a document, I send it to Gemini, and it, accordingly, tells me, it says: "I don't know, honestly," or something similar. With Claude, the situation is a bit different. Claude is one of the most obedient neural networks. I'll explain why. When Claude loses the context window and this context rot rots, and it rots at least three times faster than in ChatGPT and at least twice as fast as in Gemini, then Claude honestly shows you this window. It packs all the data you had in the previous context. 200,000 characters and, as it were, sends a summary to itself. Thus, for the next dialogue, you have not 200,000, for example, tokens, but 180,000. But it remembers everything in the form of a summary and continues to work without any lags or freezes. The situation is more interesting with ChatGPT. It, unlike Gemini, won't start being stupid and telling you that it doesn't know or getting confused. This almost never happens. Of course, someone in the comments will say something, but I'm talking about my experience. But ChatGPT, I think many of you are familiar with this. As you communicate with it for a very, very long time, it starts to lag terribly. It feels like something is wrong with the computer, it's just impossible. In reality, no. It's just that the context window is full, and it's time to start a new chat. So, the most interesting thing here is that although we can't do much with the context window itself, there are three standard things you need to do so that your work doesn't resemble that horror when the neural network starts to get confused. When your chat becomes too large, it's time to switch to another one. This can be quite painful. For example, you can spend a long time preparing some kind of script, after which it freezes in the middle, and you have to transfer everything to a new chat. Or, for example, I can tell you about myself. I created this service, Connectio. It's a service for business owners and HR managers that helps hire five times faster than the usual method using an AI assistant and a smart hiring funnel, because it has smart tests that you send to candidates, they complete them, and the system provides a scoring of how well this candidate fits your specific vacancy. If you have a business, you have current employees, or you are planning to hire, then try using Connectio. Three tests will be absolutely free. This is a service that I personally developed with my team. When I was developing Connectio, everything was done using no-code. I initially thought with Claude about some things that would be on the website. Well, for example, here are a couple of screenshots. And the problem was that when the dialogue ended, and I hadn't finished some function yet, for example, I need to create a CRM system now. I made half of the CRM system, and for the other half, I need to switch to a new dialogue. And here's where the difficulties arise, which is why people are willing to tolerate ChatGPT being stupid, just so it doesn't lose context. In reality, everything is done very simply. First, the simplest thing you can do when switching to a new dialogue is to copy everything from the previous one, literally upload it somewhere to Google Docs, then transfer it to any neural network, and say: "Summarize to the most important." And then use this as context for the new dialogue. Second, what you can do when switching to a new dialogue so that you don't have problems. Let me show you with an example of one of my projects. Well, for example, I have a YouTube project in Claude. Here I create scripts and everything else. And we can see an interesting thing: here we have some kind of standard master prompt recorded, which always remains. This is the first [music] part. The second part. In the context, we have files that are always needed when working, say, with YouTube. For example, here is the portrait of our target audience, how we see it. My positioning, as an author, catchy videos of different authors in the niche that have gained a lot of views, transcriptions of reference videos, transcriptions of my videos. That is, when I ask it in a new dialogue, it will already see this. In addition, if you use Claude, you have a tool called skills, which you can add. And when I ask it, like: "Do you remember what the YouTube skill did for me?" We see that it reads this skill. And this is like internal database, embedded not in the project, but in Claude's memory through settings. But we'll talk about that in a separate video sometime. In the case of ChatGPT and Gemini, so that you don't lose context when switching from one task to another and the foundation remains, just create a GPT application or a bot for yourself. This way, it will be easier for you to work on some complex project. For example, you are doing web development for some SaaS project. Then you can upload all the project documentation to the GPT application. And when you switch from a chat whose context has ended to a new chat, it will already know at least something. It will be easier for it to continue. And the last trick to switch to a new dialogue without losing the context window and so that nothing lags and it remembers everything, and it's probably one of the most important, just write to it literally, don't bother with any prompts and so on. Just write to it: "Provide a transition prompt for another dialogue." And it will transfer everything itself. That is, it will give you a prompt to start a dialogue in another window. And when you have these files that are attached to your project in the GPT application, you can have an initial instruction prompt attached, and in addition, you immediately assign it a role and explain the context with a transition prompt from another dialogue. The neural network will never lag. Well, to be more precise, there will be one time when you notice it, switch to a new dialogue, and everything will just fly. No need to suffer, no need to waste time. The second thing you definitely need to observe for the neural network to work better at the beginning of your dialogue, since we now know that it remembers best there and refers to the beginning of the data, it, well, practically forgets nothing from there. The worst thing you can do in ChatGPT is write to it: "I want to do this task. Let's think." And this is normal, actually. Asking it to interview the neural network based on the context of the task you want to do is correct. Just then, as soon as your context window ends, in another dialogue, you can switch with a master prompt with full context so that it's easier for it to latch onto and, accordingly, your work will go better. But mostly I try to do it a bit differently to make it work even better. At the beginning, I always use my enhancer before writing any request, because it helps to uncover things that I wouldn't have thought of. Let's break it down with a very simple example. I plan to write posts using ChatGPT. Create a prompt for it where I send it all my messages, it looks at them, looks at my context, how I write, my turns of phrase, remembers all the information about me, and then will be ready to write posts. I just send it the text, it writes the post. We send this to my enhancer, and as a result, we get a maximally developed prompt. This will allow any neural network not to deviate from it. You can use it for Gemini, for ChatGPT, and for Claude. In general, it doesn't matter. This enhancer, by the way, is absolutely free. I created it for myself, but I share it with each of you. Get it via the link in the description. It's absolutely free. Just like the checklist for this video will be so that you can apply all these tools immediately after watching. But here we see a rather large context initially. And along with this prompt, when I send it in a new dialogue, for example, to ChatGPT, I will also immediately upload all the necessary files, all the necessary references so that this prompt is the foundation for the future context. That is, you should understand that here, perhaps, the most important thing is not the prompt, but the context that you upload for your task. For example, if you need to do some research, upload all the files here at once, don't upload them one by one, don't upload them every hour, and so on. And the last thing to understand is that each neural network is ideally suited for its own tasks. There is simply no single neural network that you could use universally. For me personally, perhaps, Claude suits me best. And even then, I still pay for subscriptions to Perplexity, Gemini, ChatGPT, and Claude. It's just that unlike Claude, where I bought a subscription for $200, for all other neural networks I bought a standard subscription for $20. Why do I do this? Because depending on the task, and I spend half of my workday interacting with neural networks, a different neural network is best suited. Let's even consider a specific example of when and which one I would use. For example, with ChatGPT, we have three models that we can use. I never choose Auto. Well, simply because Auto, as a rule, does not choose the best model for a specific task. Here you need to understand: Instant for simple tasks that do not require a large context. For example, during the day you wanted to know some fact or calculate some example, or you just need to analyze your Excel spreadsheet, but it's only math, there's no need to think, then choose Instant. It will be enough for 30-40 messages, without any large files, but it will work faster and, accordingly, it will be easier for you to work with. The thinking model, for example, will already take your time, and it will think, even if you ask, damn, I don't know, 400 divided by 3, which is quite simple even for a human. But the thinking model can spend 10 minutes just contemplating. It will also load its context because it will remember the sources and what it did. And thus, with each dialogue, even with simple tasks, it will get loaded. Does this mean that the thinking model cannot be used? No, the thinking model is mainly used for complex tasks. It's just that there aren't that many complex tasks for the average person who uses neural networks. With Gemini, for example, I don't work on tasks with large context at all, because it loses it. That is, at some point, photos stop being uploaded to it, and so on. And most importantly, Gemini can lose context before I ask for a transition master prompt. And, accordingly, because I don't have a transition prompt, I have to copy something myself and so on. I don't like that. Therefore, I don't use Gemini for complex tasks. For me, it's just a fast neural network that I can ask something during the day. For example, about some fun fact, right? I have a configured prompt where, when I'm in a country where I don't know the language and generally don't know the cuisine, for example, I went to Turkey, there's some menu in Turkish, I'm too lazy to translate it through Google Translate. Plus, I won't understand what it is. I send it to him, he knows my taste preferences, and he will tell me what to eat as quickly as possible. Again, the context often gets filled, but it just works fast, unlike ChatGPT, where I waited 7 minutes for an answer. This is, of course, inefficient. But with Claude, I use it for large tasks, because Claude has an honest context window. That is, if the context runs out, it tells you about it. It doesn't just say it, it makes this transition prompt itself and keeps you within the same dialogue, and you can complete it until the very end. Plus, when everything is in one project in Claude, Claude's context works in such a way that it normally loads data from one dialogue to another. If it's within the same project, unlike ChatGPT and Gemini, which also have context between dialogues, but do it very poorly and often take the wrong data. So, we have studied three main things that need to be done so that your communication with neural networks doesn't seem like they are stupid, they don't know anything, they constantly lag, and that's it. Now you can work with models more honestly, understand their context window, understand when it's time to switch to a new dialogue. And in general, I hope this video was useful for you. Write in the comments, subscribe to the channel. All my projects are linked in the description. The last video is here via the hint on the screen.