Transcription
Hello community. So great that you are back. Yeah, today we have a brand new paper here on in-context learning and reinforcement learning by verifiable feedback, and we're going to merge them together.
Now, I was asked here, how do you generate here your thumbnails? Here, this is simply your my text for the thumbnail. If you're interested, welcome. So we will have a teacher model, a teacher structure. We will have your student AI structure, and we will have a brand new methodology here where we will combine different methodologies here of our AI training structures. So let's start.
I was also asked, "Hey, can you summarize artificial intelligence in just 5 minutes?" Of course, let's do this. And in the second part of this video, I'll explain to you this new methodology. We are going to have a look at in-context learning. So, if you want, this is here the machine's short-term reflex adapts to immediate stimuli here. This is the prompt without changing its underlying neural pathway, the tensor weight structure. We are working here only with activations. And then the other real important thing after supervised fine-tuning, but I merge supervised fine-tuning and reinforcement learning, is here with verifiable rewards. So this means we do have a result that we know for sure. We do have an absolute ground proof. And we only have this in mathematics or code generation when you just run it and you see, "Hey, it's working," or "It's not working at all."
If you want, this is here the formal machine education. The AI model is placed in a closed deterministic system. Mathematics, code, whatever. The ground truth with absolute the compiler passes, or it fails. And hopefully, the AI system just learns here the logic. And then we have our reinforcement learning for complex open-ended structures where there's no ground truth. If you ask, "Hey, who is the most beautiful American president of all time?" You know, you can have different opinions about this. So it depends about what is your reasoning complexity, your reasoning manifold that you apply to answer this question. If you want, this is here a socialization of AI. The neural network navigates an open-ended ambitious complexity where rewards are subjective, delayed, multi-objective. You have human preferences, you have complex strategy games, whatever.
And then, of course, we move here from a simple LLM to an agent, an AI agent. So at the core of this agent is, of course, an LLM. But the most important thing is we are now in contact with an external environment. And an agent has two things. Never forget this is here a memory, and we have a memory optimization pathways. And of course, tool call and tool use. Everything here from, uh, communication protocols here to tools, or agent-to-agent communication protocols, or commercial, uh, protocols, whatever we have. This is it. So if you want here, if you have a parallel here in neurology, this is the executive prefrontal cortex. Our agent takes the reasoning capabilities from the left and operationalizes them using physical or digital appendages, our tool use or tool call. And a hippocampus, this is our memory optimization.
Can be so easy. I sometimes, you know, what's really now fascinating is continuous machine learning. Because up until now, we just have here pre-training at the start, and then we have here the post-training. But what if we want our AI to have a continuous machine learning? And I know you might indicate, "Hey, yeah, we have a memory optimization." Yeah, but it turns out this is not sufficient. We have to be a little bit more clever to really implement this. [snorts] Because our optimal goal is an AI machine with neuroplasticity in its neural network. So the ability to learn continually without forgetting here the past knowledge. And then if we have one agent, you know, the next step is a multi-agent learning algorithm. This is completely different because, my goodness, we have now all the added complexity in new permutations. So, you know, we have to have a look at the topology here of our multi-agent configuration of our communication protocols. If you have, I don't know, a swarm intelligence, or you have an orchestrator AI, and here our agent-to-agent communications suffer from severe cascading error rates. If agent A hallucinates just a tiny little minor detail to agent B, the error compounds exponentially over all the multi-turn interactions. And, yeah, you can imagine what happens at the end.
Now, of course, this simple complexity is just what we look at at the first level of textual complexity. No, if you do only text LLMs, and of course, you might ask, "Hey, but we have to do the same. You know, for the visual." If you add the visual part, yeah, turns out we have completely different mechanisms, algorithms. Everything is completely different. Either you rely only on a textual reasoning capability, or you also want to develop a visual vision reasoning capacity. Everything will change immediately. And of course, you know, "Hey, robotics is missing here." We are multimodal actions that are now here, they feed back into the environment. If you have a robotic control, you see, and we have complete other algorithms, complete other learning algorithms. Everything is again completely different. Isn't this beautiful?
Now, what's missing out is, of course, a self-supervised pre-training that I have am not yet integrated here. Imagine you just put here a single cell in the ocean, and you just hope that you have provided this single self-cell with an intelligence that it can survive here in the ocean over the next, I don't know, 100 hours, 1,000 hours. And of course, we have to do something about multimodal scaling, time, interaction, survival processes, and everything. And this is so far in the future. We still have no idea how to handle this today. Yeah, we have alignment, safety, synthetic data generation. They are open-ended problems. But this is it. This is the first five minutes I explored here artificial intelligence. This is the state-of-the-art. Now, let's start here with the video.
Now, you know, all topics are hot topics currently in AI research. Just look here at verifiable rewards. Now, here, February 19, just yesterday, 2026, a new study here by Fudan University, Tsinghua University, here unifying gradient utilization, probability mass, and signal reliability for efficient LLM reasoning. And they developed here a new algorithm. And they now have the time, and they look at our GRO algorithm, and they tell us, you know, "What this relies on a rigid, uniform, and symmetric trust region mechanisms that are fundamentally misaligned with the complex optimization dynamics of our LLM." So now we find out what's not working. And they provide a solution. They provide here Mass Adaptive Soft Policy Optimization, MASSO, for a unified framework. Oh, let's go here. This is here latest research, yesterday, February 19, 2026. MIT and Nvidia. What are they researching? They are going here for off-policy reinforcement learning for LLMs. And they encounter also problems here. And Nvidia says, "Hey, it's not fast enough." So we propose now a variance-controlled policy optimization. You know, remember we started with our PPO by OpenAI. And now we have a general stabilization method for our PPO-style algorithms. And they have closed for minimum variance and they avoid the auxiliary value models here in our PPO. So some real brand new intelligent ideas. But this is not what I want to talk about in this video. You should maybe read here the paper to be prepared now for the video.
So, let's start. What I want to show you is here a complete new way of training an AI system. And we will have here a teacher model. We will have here a student model. We will have here cluster algorithms. And we will completely merge here our reinforcement learning here with our in-context learning. Because you remember, sometimes we have no chance at all. It's not an open-source model. So we only have context to an in-context learning. But we have to understand here about the mechanisms of ICL. So therefore, we go with an open-source model where we can analyze everything.
Now, you know, a word of warning. You know, in this video, I showed you, Google showed us here that in-context learning is inert. It is not perfectly responsive. Our AI model in-context learning has its own problems, its own faults, its own non-responsiveness of the LLM. And I explain here the latest state-of-the-art here in this particular video. And they are just yesterday, February 19, 2026. Here you see University of California, Berkeley, beautiful studies about trying to understand in-context learning in linear versus quadratic attention models, you know, on the regression task. They really want to understand what the hell is going on here with in-context learning. We still do not understand here everything that's happening. Or look at this. Here we have UC Riverside, beautiful transforming here the behavioral neuroscience discovery with in-context learning. So they go here with, um, real experiments with mice. And they say, "How can we learn here from the neuroscience discovery that we see? How can we map this to an algorithm and in-context learning for our AI with AI-enhanced tensor methodologies for our dynamic weight tensors that we have here in our transformer architecture?" There's a lot of research. These are just two papers from yesterday to show you how hot this topic is. But it's not what I want to talk to you about. I want to go with something much more fascinating. And of course, it's from Google DeepMind. They talk about improving the interactive in-context learning of an LLM from just a natural English language feedback. And this is fascinating. Of course, this was published two days before yesterday. So, okay.
But they say, "We're interested in the AI models that are self-improving. We want to have this cell that we put in the ocean, and the cell has all the genetic information in this wet biotope to grow, to multiply, to morph into other algorithms, in other forms of existence." And now they want to see this experiment here on AI self-improvement of AI systems. And they say, "By training the model to predict the teacher critique, we convert the external signal into an internal capability of the LLM, allowing here our AI model to self-correct even without a teacher." Now, this sounds like a lot of. So let me just clarify. We will go step by step, and you will see it is easy. It is a simple methodology, as I showed you in our general explanation of AI. No, in-context learning.
Now, we are here at the first one: AI self-learning. This means simply without any interactive human contribution. You don't have a human that says, "Hey, this is correct, this is incorrect," or you have a human conversation with an LLM. You really want to have an AI that is absolutely independent of anything to do with human feedback. Now, you know, our current LLMs excel at processing huge amounts of static knowledge, copy the complete internet. But they struggle to learn from real-time interaction. Somehow, our algorithms for learning are great for static knowledge, completely fail for real-time interactions. So either you go with complex prompt engineering to steer you our AI models toward desired outcomes, but you have bottlenecks, and it's not really working. So to make it worthwhile for users to teach LLMs, the model must adaptively interpret, reason, and now hold on to your socks, adjust their behavior dynamically within the context of a conversation. And let's see a conversation with a human or a conversation with another AI agent. And unlike here the standard reinforcement learning where you have a scalar reward at the end here, and you have a clear gradient for the optimization, this language feedback does not really lend itself to such direct optimization. So we, I mean, Google DeepMind has to invent a new algorithm for this.
Now, you know that in-context learning is an inference-time phenomenon, not a training-time phenomenon. So it occurs entirely with what they call fast weights. You know, this is exactly the hidden state of our transformer layer and our key-value caches of the transformer's forward pass. So we are not touching here the tensor weights here in the neural network layers themselves. And the authors demonstrate now in this new study that current models, and they go with a Gemini 2.5 Pro and a GPT-5, exhibit something they call a severe loss of neural plasticity during in-context learning. And faced with complex reasoning tasks, if a user even points out a mathematical flaw to a GPT-5 system, GPT-5 will often acknowledge the text politely, "Oh, thank you for indicating this to me," but just repeat the exact same mathematical nonsense, the error that it just made a step before. So, you correct it as a human, but the model doesn't understand that this is a correction it should integrate in its reasoning path. It's just an information, says, "Okay, if you provided me some information, nice. Let's go on with what I know." So Google tells us, "Hey, our, let's say, GPT-5 lacks the cognitive flexibility to kind of metabolize here the critique and therefore alter its reasoning trajectory."
So how we do this? And they said, "Hey, to fix here the deficiency and the fast weights in our in-context learning, and you know, between you and me, these are our activations. Now, you must permanently alter now the real weights here, our tensor weights of the networks." So now they fuse together in-context learning and reinforcement learning, but in a very special way. Because the authors use reinforcement learning as a meta-optimizer to train the network now in a very specific way, in a way to optimally learn in context. So you see, we are modifying, if you want, the tensor weight structure for the context of in-context learning. How we do this? If you use RL, we need rewards. So by rewarding now the AI model only when it successfully integrates here multi-turn feedback. Oh, yeah, we are in complex reasoning. It's not a single question or answer. We are multi-turn feedback to reach a verifiable ground truth. Our RL forces now the slow weights to sculpt the highly reactive plastic ICL algorithm within the forward part of the algorithm.
Let me give you an example. Imagine a student model generates a flawed mathematical solution, and the teacher AI provides, you know, a critique, not a solution, just a critique, you know, just a hint, says, "Hey, little buddy, you forgot to carry the one." Yeah. And this critique is embedded and stored perfectly in the key-value cache here in our activations. However, when the student generates its next token, you know, what is the intelligence? Intelligence is a simple algorithm. It computes a query, a Q. And if the slow, the real weight matrix or weight tensor structure WQ was never explicitly trained to value and to respond to a corrective feedback, either human or AI, the dot product here Q * K critique transposed will yield a low attention score. Of course, the model will literally say, "Hey, I will not pay attention here to the teacher feedback in the fast weights because I was not trained to do so." And you know, this is a pattern-matching machine AI. The intelligence is, if the pattern is there, I can do it. If you have never been trained on this pattern to pay attention to a feedback, to a critique, I [clears throat] will just ignore it. Instead, attending here heavily to its own flawed generated history of the key and thus repeat the error. And this is what the authors call here the loss of neural plasticity in AI. Such an easy example.
And now they have a solution, and they call this solution Reinforcement Learning with Language Feedback. So you see, we have one L and another L. So we have an RL to feedback. What is this? They are aggressively updating here the fast weights to change how the model routes your information internally. So the RL gradients are penalizing the network when it ignores the teacher, says, "Hey, that's not the way to go." So this forces now here in the algorithm, the real weight tensor, the weight matrix is here to realign, to modify. And after the reinforcement learning fine-tuning that is going on here in the training phase, when the model computes your Q, it is now mathematically incentivized as an optimization problem to assign now massive attention scores to the critique token in the fast weights in our activations. So what does it mean? We just have to learn this stupid AI machine how to respond here to a language feedback. And we cannot do this within-context learning alone. The activations are not enough in our architecture. Therefore, we have to go and apply here reinforcement learning fine-tuning. We really have to change here the tensor weights in a particular way so that the model suddenly learns or is enabled now to assign massive attention scores to the critique tokens coming in from the teacher. This is all there is. So if you want, in one sentence, we optimize now the static tensor fields so that the transient activations can actually do the job of in-context learning in a much better way. So we have, if you want, two interlinked objective optimizations. We start, we see that the in-context learning activations are not enough. Therefore, we have to dig deep into the black box. We have to open up the black box. We have to modify the tensor weights here with an optimization algorithm. And the objective of this optimization is that the in-context learning is going to happen in a beautiful way. A simple idea.
So Google tells us, "We stimulate now the interaction between a student and a teacher AI where the student must iteratively refine its solution by integrating now the teacher guidance." And you might say, "Yeah, familiar. This is nothing new." Well, a key insight is that generating high-quality feedback on verifiable domains like mathematics or code does not require a superior AI model, a teacher model, but rather you can do this with the same model that is your student. You just have to provide to the model that is acting as a teacher an information asymmetry. Just provide access to some ground truth that the student does not have access to. So it's rather simple. Yeah. They call this, yeah, didactic interactions. So you have a student, you have a response, you have a teacher. The teacher looks at the problem, has access to what they call privileged information, like a unit test output or a ground truth math solution. Teacher understands what the student told me is correct or incorrect. If it's incorrect, the teacher gives not the solution, but just a natural language feedback, a hint. And the student goes on. And this is an optimization circle that you know. So what we have, we have the student model is more or less identical to the teacher AI. Just the teacher has access to some privileged information. So we do have an information asymmetry in our system. We transform single-turn problems into multi-turn didactic interactions. A teacher model conditioned on privileged information like the ground truth solution provides now natural language feedback to the student model without revealing the answer, just guiding it to correct its error. How the teacher is trained and learned in a way that it provides here the best guidance is another topic, and we're not going to talk about today. Let's just assume, "Hey, it's working." And [clears throat] then you just have reinforcement learning with language feedback. The turn one teacher gives another hint. Turn two, teacher gives another hint. And, hey, suddenly the student got the right answer in turn three. So you see, just run, or maybe you hardcode here, I don't know, 10 runs or 50 runs. It's a simple methodology. But careful, what it is. It is now a training-time reinforcement learning fine-tuning that we apply because we train the student model to effectively incorporate language feedback using our reinforcement learning. The student iterates through multiple turns. If the answer is correct, a reward is granted, and the interaction ends. If it is incorrect, the teacher provides feedback. If the interaction reaches the maximum number of turns, the reward is zero. Everything that we know, it's just in another combination, in another sequence of our alignment tuning and ICL.
Now, you know, what I'm interested in? I'm interested in, is it really able this system with this algorithm to have a self-improvement AI system via world modeling? Is it really able to have an auto-deductive self-improvement? Because imagine, then I would be able as a human too to have a self-improvement here with this particular algorithm. I just have to apply it to me. No. So what is happening? There's another step that you have to carefully read the paper because almost at the end of the paper, they tell you, "And now, now we do something very special." So you have the learning from the language feedback. Beautiful. You have in-domain, and then you have general multi-turn task. Let's say you go from mathematics to coding about mathematics. Beautiful. You're normally out of domain, but you see, yes, it works fine. But then, then comes the main part. Then comes the jump to an in-context self-improvement. Now, this is different. So what we have is inference-time evaluation. So we assess, if you want to train the AI model in three settings. First, learning from the language feedback. So, meaning simply interacting with an external source of language feedback. And I'm going to show you a graph, a performance graph, how this works with a GPT-5 or a Gemini 2.5 Pro. It's disappointing. Then, general multi-turn task, like out-of-domain tasks or logic puzzles, like I do here my AI test routines. And, and this is the nice part, now the in-context self-improvement where the model plays now the role of both the student and the teacher to self-correct. But there are still something that is fascinating. Because instead of just training here the student AI to maximize the reward, they add another objective function. This student model should now, in addition to its job, predict now the teacher's next critique. You say, "Hey, listen. You see that you get feedback from the teacher AI. You have 1,000 feedbacks. You see, is there a common pattern in this feedback? Is there a common structure? How the feedback happens? What are the feedback elements? What is it indicating to to mathematical operation, to object definitions, to logical streams? What is it?" So this is a pattern, and this pattern can be learned by the student AI. So therefore, the student has a new job: predict now the teacher's next critique after it was trained on, I don't know, 10,000, 100,000 real teacher critique systems. Now, by training the student to model the feedback environment, this means a teacher's distribution, the policy pi teacher, purely from the context of its own errors, the student now, if you want, internalizes now the action and the feedback given by the teacher AI. And then at the test time, when we really run with this model, simple, the external teacher AI is removed. Because guess what? It was the original model, just with access to privileged information, like say RAG or whatever. So the student becomes now an auto-didactic AI system, alternating now in its job between generating a solution, critiquing its own solution, having learned this from the teacher patterns, and refining it, like we had here in the student-teacher cycle, yielding now, hopefully, maybe massive performance gains over standard single-turn reinforcement learning cycles.
Now, is this true? This is why I wanted to show you here how I see AI and the complexities of AI currently. In-context learning, RLVR, and the open-ended RLs. And you see, we are touching on all elements. We're also touching on continuous machine learning because we also touching on AI self-learning. So everything that is connected here to this LLM, we are addressing here in this new algorithm, in this new paper by Google DeepMind. And we are no longer training here any model in an LLM to know the answer to a mathematical problem. You are training here these low weights, the activations between you and me, to construct here, if you want, a virtual prefrontal cortex in the key-value cache itself. One that can read, hypothesize, receive the critique, understand the critique, experience maybe some algorithmic doubt, and therefore then self-correct itself. So this is now the true essence here of in-context learning, but not by its own, but in-context learning realized through reinforcement learning. So now we have a dense interaction between ICL with our activations, and we have to optimize the reinforcement learning of the tensor structure of the tensor weights to optimize for ICL. It is elegant, but jeez, you have to have this idea in the first place. Huh?
Yeah, I was talking about here the authors were giving us the information, "massive performance gain over standard single-turn reinforcement learning." And I want to see this. I want to have you the data. And I say, "Okay, this is great." And here we are. So here we have a baseline. Okay, so what we have? Number of turns, multi-turns, one to ten. And on the y-axis, we have the accuracy. 40%, 50%, 70%. If I have a baseline Gemini 2.5 Pro, this is here my performance. If I have this in the didactic interaction in the student-external teacher model, the best performance I can achieve in this configuration is here, let's say 65%. Is here the dotted line. And then Google DeepMind tells us, "If we do now this self-improvement that I just presented to you in this beautiful video." Look, this is the performance in green. After six turns, we cross over the line, and we have really a self-learning experience, a self-improving AI system. Well, just on mathematics, to be clear, it's a very narrow domain knowledge that we're working on, but it happens. It is possible. So the training the models here to learn from multi-turn didactic interaction significantly improves here their self-improvement capabilities at inference time. Yeah, you can really achieve it. If you look at turns, this is happening, which is amazing by itself.
They show us here, "What about we have here a Gemini 2.5 Flash and then we have a Gemini 2.5 Pro? Can this help us, let's say, a smaller model to get the performance of the big brother model?" So here in this dark blue, here we have here the single-turn reinforcement learning. So our classical learning experience is this blue line here. And you see this is rather a limited learning experience. Now it is below 50% success rate. But look at this new methodology. Look, we are here almost touching here the performance of Gemini 2.5 Pro. This is really a self-learning system. Can it learn? Can it add intelligence to its own knowledge? Is this really possible? Absolutely fascinating. Yeah. And they see here, "Trained through RL and multi-turn didactic interaction, Flash nearly reaches the performance of Pro on the challenging hard mathematical 2 dataset." Of course, this is a very specific benchmark. If you do this for a lot of other benchmarks, maybe your results may vary. But theoretically, there is one case where it really works. And sometimes this is all that we need.
Now, you know, I'm personally interested here in supervised fine-tuning versus reinforcement learning. Because I have multiple videos where I tell you from a mathematical point of view, supervised fine-tuning is almost the same like reinforcement learning if you choose a certain mathematical representation theory. And here they have it now: single-turn reinforcement learning and single-turn supervised fine-tuning. And this is now interesting. So we have a baseline here. This is our Gemini 2.5 Pro in red. Great. And then this is interesting. Look, if you do in blue, the single-turn supervised fine-tuning, this is the blue line here. And then you have a light gray line, the single-turn reinforcement learning. And this is almost identical, almost parallel. So this means for this particular task of mathematics here, the learning exercise between a supervised fine-tuning and a reinforcement learning yield the same result. This is fascinating. And then, yeah, really the surprise. Google has really developed here a new methodology here in this RL2F methodology for a self-improvement. Absolutely stunning these results.
But again, let's do another cross-check. We have to be sure here. What if we do not perform the weight updates? What if we do not RL in this section, but simply ask, "Can a current AI model already efficiently incorporate this feedback in context without this new methodology?" Do we have more data about this? And yes, we have it. And they have here a GPT-5 system and a Gemini 2.5 Pro. And they show us here the number of turns from one, two, three. You see, this is almost flat. This is almost horizontal. There's no growth happening. So our current models without this new methodology can somehow incorporate a little bit already the feedback in context, but in no way as efficient as this new RL2F methodology. You see here Gemini 2.5 Pro in the started green line, and GPT-5 here in this solid dark blue line. You see it's, yeah, GPT-5 is a little bit more learnable within context, but nothing in [clears throat] compared to the new methodology. And if you go here to an RSC AGI task, you see it's even flattening out faster. And this is just three turns. So there's not really this feedback in context learning happening with our current models. But if Google would implement this new methodology of self-improving, oh yeah, I'm looking forward to the next models by Google.
So there you have it. You understand now why I structured here today the overview of what is AI research today in this particular way. We talked about all of this and continuous machine learning with self-improvement. And, yeah, you noticed, "Hey, you haven't talked about AI agent at all." Yes, of course. Wait for the next video. Come on. Just one. I have here the first video that you see. Google is still working here on continuous machine learning algorithms. And we are not talking about agent optimization, single agent optimization, and not at all about multi-agent optimization. We are still stuck in the research trying to understand what the hell is happening if we really want to build a self-learning AI machine. I hope you had a little bit of fun. I hope there was some new information. Maybe you found it helpful. Why not give it a like, subscribe, become a member of my channel. Anyway, I hope to see you in my next.