Transcription
Hello community. So great that you are back. We have a brand new video, and we have a new explanation of how in-context learning really works on the tensor structure of a transformer architecture. So let us jump right into it.
Welcome to my channel Discovery. We have a look at the latest research paper, and given my latest video here of the new transformer architecture by Google. You remember we were talking about the meter controller, and I told you that a lot of it depends here on the in-context learning properties that happened during the training. Now, a lot of my viewers had a lot of questions, and I said, "Beautiful, let's jump into in-context learning." And you remember this was one of the most important papers here. This was July 2025, and we had from Google Research, of course, who else? "Learning Without Training: An AI Implicit Dynamics of In-Context Learning." And if you read this paper, you immediately understand what it is all about. The most clearest view you get if you look here at the implicit learning dynamics of ICL here in chapter 3, and they really go with you step by step, and they really show you here beautifully what they mean within what in-context learning really is.
Now, interestingly, they do this on a simplification. But as you can see in NXA, they also go to contextual blocks with skip connections. So, they do integrate now the skip connection into the contextual block. So, we already leave here the simplified version, and we already show this. I thought, "Hmm, since it is not including the nonlinearity of the learning process itself, is this really something that we should know as AI researchers about? Is this really that a knowledge that we can apply?" Now, it all changed here with this publication. This is if University of Oxford and University of South, December 12, 2025. They say, "A Simple You Know When an Artist Says a Simple Generalization of the Implicit Dynamics of In-Context Learning." And these two authors here took now the result here from July and really tried to have the calculation here done for the complete complexity of a transformer architecture, and they succeeded in doing this. So, you see here the extension to any contextual block, the extension to any block with more accurate skip connections, and then here the generalization to any block with skips and the layer normalization and implicit nonlinearities.
Now, of course, the implications are absolutely significant. Let's talk about what it means. You can have a look at the paper. You can follow the mathematics. Everything. But what, what is the deeper meaning? What does it mean for our understanding? What is this black-box AI doing? Now, you know, in the mechanism behind in-context learning, especially this learning, was not really there. We had a lot of approaches, but not really. So, we kind of lack a precise mechanical mapping how the token activations translate here to the learning process in the different layers of transformer architecture. And these two preprints provide us new with a new and beautiful mathematical unification of inference and the training process. And they have now a new insight: in-context learning is not just about retrieval here, the pre-trained knowledge, here, the intelligence of the AI. But they show us that ICL is mathematically equivalent to the fine-tuning process itself.
Let's have a closer look at this. So, they demonstrate here on both orders, in both papers here, that the transformer block processing here a specific context, let's call the context C, and a query, a human query X, behaves exactly as if it were a standard feed-forward network processing only the query X. But, and here's the big but, with its weights explicitly updated now with a new different delta W matrix. This is our delta weight tensor. And if we insert this one, we can say, "Hey, this is just a standard feed-forward network processing only on the query." So, if you want, we have two different perspectives, and where they're interlinked, they showed mathematics is the same. But my brain refused to accept this because what they say that this effectively bridges here the gap between the frozen weights of a pre-trained LLM and the dynamic adaptation required for in-context learning, and suddenly we have a bridge. How is this possible? I don't believe it.
Now, if you dive into the mathematics, you see it's not really the full complexity of a transformer block. But they built something also, Google here in July 2025, like a contextual block TWW. So, this block consists, like a normal, here we have a contextual layer. This is more or less the self-attention mechanism. And then we have our neural network layer here. This is a standard dense MLP. Beautiful. The standard deep learning view, the information processing that goes on in this transformation, that we have all this, if you want, mathematical operation in the activation space. No, a vector X passes through all the different layers of our transformer architecture, getting rotated and squashed and whatever. Now, those two preprints argue that we are looking at the wrong mathematical space. They tell us, "Hey, you should look at the weight space, even for the activation." And I thought, "Hey, no, this is not compatible." Okay, but let's go on.
So, the attention mechanism does not just produce the context vector or C, it produces also an instruction to rewrite the processing logic of the MLP. And I have to tell you, this is the first time my brain broke and said, "No, no, this is not what is standard known about the interconnect between ICL and fine-tuning here in the different layers of the transformer." How is this possible? But what is an instruction to rewrite the processing logic of an MLP without touching here the weight tensors? How should this work? And they, they really claim in-context learning is mathematically identical to fine-tuning the weights of the MLP layers. And you know my feeling was, "What is this true?" Because, you know, I work also here, I have my images that I sync. Yeah. So, we have here our grid here from our weight tensor structures from different layers. Now, we have frozen layers. Beautiful. And then we have an ICL. Then we have here an ICL stream with our context activations. And they want to tell me now that this context activations are mathematically identical to modify the frozen weight structure with a rank-one update for an implicit gradient step. Honestly, you want to tell me this is equivalent.
Okay, let's have a look at this. Now, it gets interesting. So, why I have such problems? Because imagine you have a frozen neural network. Beautiful. You change its behavior, you perform the fine-tune. No, you run here typical fine-tuning algorithms. You have your training data, and you run more or less a backpropagation. No, and you update here the physical numbers in the weight tensors here, and you have an update of this. Great. So, we really touch here the tensor weights and you really modify. And this is the beauty of fine-tuning, in contrast to ICL. Because in ICL, you feed here a prompt, here a specific context sequence into the model, and alongside your query, maybe you provide some few-shot examples here for in-context learning. And the attention mechanism looks now at this context, extracts here relevant information, and then simply builds and adds here a context vector, H context, to our query X before sending it now off here from the self-attention here to the MLP structure of the transformer. And now the MLP sees now this added component here on the vector representation, and the MLP processes the original input X, but using here modified virtual weights. So, this means that the context vector is now not only a vector representation in the, if you want, activation space, but is also here rotated into the weight space where we have modified virtual weights. And I'm still struggling to understand this idea.
So, what we have actually is a ghost weight update. And I'm really, I just want to be clear on this. So, the context prompt, our ICL, creates now a temporary virtual set of weights that exist only for that specific forward pass. And you might say it sounds crazy, but okay, let's go with this. So, this means this AI model, and let's be absolutely brutal in our formulation, this model hallucinates now in ICL a fine-tuned version of itself on the fly. Is this really the content of these two papers by Google and Oxford and Sussex and whatever? It turns out, yeah, this is exactly the point. And I have questions. And one of my questions is, how is this possible? Activation patterns do not modify weights. No, an activation pattern is something completely different. This is the vector that flows through the layers. The weight tensors. If you modify a weight tensor, how is this identical? And the answer is given by the paper. It is possible because they prove mathematically, not in a visual explanation like I try to think. They say it is possible because mathematically we can prove that there is a linearity of matrix multiplication that we can find identical mathematical terms that we can identify.
So, let's do this. It is simple. It is real simple. But it is important that we all understand it. Here in the activation space, we have the classical view. No, let W be the frozen weights here of an MLP layer in a transformer. Let X be your human query here or the embedding here of the last token. And let A be here the output of the attention layer. Attention and A. This is more or less the complete context information. Then a standard transformer. And now let's ignore the nonlinearities here for a second. I'll come back to the nonlinearities in a minute. The input to the MLP is the sum of the query and the attention output. No, via the residual stream, as I showed my you in my last video. So, if you, this is simple. So, we intuitively think now, "Okay, the input X is now shifted here by a particular vector A." Beautiful. And this is it. So, this is our activation space view. So, the term W * A is now a vector representing here the correction, or if you want, the knowledge retrieved now from the added context to our ICL prompt. Now, the authors ask more or less, "Hey, can we find now a matrix W_prime such that this correction comes now from the weights instead from the context input?" So, mathematically, it is so easy. The authors tell us, "We just want to satisfy this equation." Huh? So, we have here the, if you want, the activation space view, and then we have here the weight space view, and we just want to make that this is, can we have this is the same? Yeah, this is it. No, W * A is exactly our delta W * X. X is here the human query, if you want, and A is here the attention here of this additional context, and W is our weight, and delta W is our additional weight structure. So, to solve here for W_prime, we need a matrix that when multiplied with X produces here this particular vector W * A. Uh, distributive property. So, now they tell us mathematically, the authors tell us, "We can pretend that the correction term W * A did not come from the extra vector A, but instead came from an extra weight acting on the original vector X."
So, now we understand why they claim this because they build a mathematical framework where more or less they say, "In the end of all of our calculations, we arrive at this. We arrive here at this term and at this term from two different mathematical spaces, and we just ask, 'Hey, is it possible that our W * A becomes becomes equivalent to our delta W * X?' Can we find this?" And then this is here, if you want, a correction term that is now coming not from the content from the attention, but now we can go and operate with virtual weight structures. So, we have now this bridge suddenly connecting fine-tuning and in-context learning. Now, of course, as I showed you, you can do this here with a rank-one solution. No, if you go with an outer product, simple. No. So, mathematically, we can say adding now the attention output A to the input is identical to updating the weight matrix W by adding here a rank-one matrix delta. So, the activation pattern A from the attention layer becomes the source of the weight update delta W in ICL. So, we have now this beautiful connection between fine-tuning and ICL in a coherent mathematical theory. And now you understand why the authors can claim this identity because mathematically speaking, it is identical. But in physics, you know, whenever we see that we sort that two properties of a physical system are different, and then analyzing it mathematically, we see they are identical, we understand that we made a mistake in building this physical system because either we have neglected symmetries or we have neglected identity operation, but we built something that is not, let's call it, perfect. We built something that works, but we don't really understand why. And now, with this mathematical identity, I have a feeling that what we think is the difference between in-context learning and fine-tuning, what we coded, it turns out is indeed the identical thing, more or less. So maybe we built our transformer architecture just in a wrong way. We didn't utilize that those terms are identical.
Now, you know, we operated here with a simplified kernel representation. And the beauty of the second preprint that I showed you here is the second order here from December 12, that they took this much further, and they prove mathematically that it works here also in a realistic nonlinear deep learning scenarios. So, the second paper is just taking the results of the first paper and proving it mathematically in nonlinear deep learning. But we stay on the same complexity that we try to understand and we try to solve. So, yeah, no, the skeptics, and this is me, I have said at the beginning, no, but you have the layer norm, you have nonlinear activation, you have skip connections. No, and this is why I did not take here this publication by Google here from July 2025 really that significant because I thought, "Hey, the simple algebra doesn't hold if you input nonlinear activations." No, but this team of now proved here just two weeks ago that it holds. And this is something, okay. But they have a second insight: that the update delta W becomes specific to the token position. So, we calculate, given the nonlinearity, a delta W for each single token position. Now, this thing becomes a little bit more interesting. No. So, the weights are effectively different for every single token that is generated by our autoregressive system. The model isn't just fine-tuned once. It is now with ICL continuously refining itself at every step of the generation of the next token prediction of an autoregressive transformer architecture. Therefore, we can now stop treating here the learning, and I mean here the backpropagation, the learning process on the neural network, and the inference, the forward pass, as it would be two different things. It turns out, no. If you understand the mathematics of this, we see the forward pass is a learning step in itself, and this is mathematically equivalent here to the backpropagation. So, somehow there is a chance we can further optimize the architectures of the transformers. We can make it maybe smaller. We can make it more powerful. We have not yet optimized the architecture.
So, if I go here with my image, this is just in my head, whatever image you have. I have here, you know, my tensor structure here, my weight matrix here in, I don't know, 4,000 dimensions. And then I say, "Okay, and then this should be equivalent here to ICL that comes in here, the context, the context activation on the frozen weights." And now I have to understand that these two images are identical. So, this golden context activation are more or less just a rank-one patch that I apply here to my frozen weight tensor structure. So, ICL is mathematically proven equivalent to a rank-one update on the tensor structure. Okay, let's go with this. Okay, now I have further questions. So, you see here how I fight with these things sometimes. And say, "But wait, the activation of our neurons in the layers of the transformers is not a weight tensor update." No, an activation is something completely different. This is the vector that flows through the layers. The weight tensors. If you modify a weight tensor, how is this identical? And the answer is given by the paper. It is possible because they prove mathematically, not in a visual explanation like I try to think. They say it is possible because mathematically we can prove that there is a linearity of matrix multiplication that we can find identical mathematical terms that we can identify.
So, let's do this. It is simple. It is real simple. But it is important that we all understand it. Here in the activation space, we have the classical view. No, let W be the frozen weights here of an MLP layer in a transformer. Let X be your human query here or the embedding here of the last token. And let A be here the output of the attention layer. Attention and A. This is more or less the complete context information. Then a standard transformer. And now let's ignore the nonlinearities here for a second. I'll come back to the nonlinearities in a minute. The input to the MLP is the sum of the query and the attention output. No, via the residual stream, as I showed my you in my last video. So, if you, this is simple. So, we intuitively think now, "Okay, the input X is now shifted here by a particular vector A." Beautiful. And this is it. So, this is our activation space view. So, the term W * A is now a vector representing here the correction, or if you want, the knowledge retrieved now from the added context to our ICL prompt. Now, the authors ask more or less, "Hey, can we find now a matrix W_prime such that this correction comes now from the weights instead from the context input?" So, mathematically, it is so easy. The authors tell us, "We just want to satisfy this equation." Huh? So, we have here the, if you want, the activation space view, and then we have here the weight space view, and we just want to make that this is, can we have this is the same? Yeah, this is it. No, W * A is exactly our delta W * X. X is here the human query, if you want, and A is here the attention here of this additional context, and W is our weight, and delta W is our additional weight structure. So, to solve here for W_prime, we need a matrix that when multiplied with X produces here this particular vector W * A. Uh, distributive property. So, now they tell us mathematically, the authors tell us, "We can pretend that the correction term W * A did not come from the extra vector A, but instead came from an extra weight acting on the original vector X."
So, now we understand why they claim this because they build a mathematical framework where more or less they say, "In the end of all of our calculations, we arrive at this. We arrive here at this term and at this term from two different mathematical spaces, and we just ask, 'Hey, is it possible that our W * A becomes becomes equivalent to our delta W * X?' Can we find this?" And then this is here, if you want, a correction term that is now coming not from the content from the attention, but now we can go and operate with virtual weight structures. So, we have now this bridge suddenly connecting fine-tuning and in-context learning. Now, of course, as I showed you, you can do this here with a rank-one solution. No, if you go with an outer product, simple. No. So, mathematically, we can say adding now the attention output A to the input is identical to updating the weight matrix W by adding here a rank-one matrix delta. So, the activation pattern A from the attention layer becomes the source of the weight update delta W in ICL. So, we have now this beautiful connection between fine-tuning and ICL in a coherent mathematical theory. And now you understand why the authors can claim this identity because mathematically speaking, it is identical. But in physics, you know, whenever we see that we sort that two properties of a physical system are different, and then analyzing it mathematically, we see they are identical, we understand that we made a mistake in building this physical system because either we have neglected symmetries or we have neglected identity operation, but we built something that is not, let's call it, perfect. We built something that works, but we don't really understand why. And now, with this mathematical identity, I have a feeling that what we think is the difference between in-context learning and fine-tuning, what we coded, it turns out is indeed the identical thing, more or less. So maybe we built our transformer architecture just in a wrong way. We didn't utilize that those terms are identical.
Now, you know, we operated here with a simplified kernel representation. And the beauty of the second preprint that I showed you here is the second order here from December 12, that they took this much further, and they prove mathematically that it works here also in a realistic nonlinear deep learning scenarios. So, the second paper is just taking the results of the first paper and proving it mathematically in nonlinear deep learning. But we stay on the same complexity that we try to understand and we try to solve. So, yeah, no, the skeptics, and this is me, I have said at the beginning, no, but you have the layer norm, you have nonlinear activation, you have skip connections. No, and this is why I did not take here this publication by Google here from July 2025 really that significant because I thought, "Hey, the simple algebra doesn't hold if you input nonlinear activations." No, but this team of now proved here just two weeks ago that it holds. And this is something, okay. But they have a second insight: that the update delta W becomes specific to the token position. So, we calculate, given the nonlinearity, a delta W for each single token position. Now, this thing becomes a little bit more interesting. No. So, the weights are effectively different for every single token that is generated by our autoregressive system. The model isn't just fine-tuned once. It is now with ICL continuously refining itself at every step of the generation of the next token prediction of an autoregressive transformer architecture. Therefore, we can now stop treating here the learning, and I mean here the backpropagation, the learning process on the neural network, and the inference, the forward pass, as it would be two different things. It turns out, no. If you understand the mathematics of this, we see the forward pass is a learning step in itself, and this is mathematically equivalent here to the backpropagation. So, somehow there is a chance we can further optimize the architectures of the transformers. We can make it maybe smaller. We can make it more powerful. We have not yet optimized the architecture.
So, if I go here with my image, this is just in my head, whatever image you have. I have here, you know, my tensor structure here, my weight matrix here in, I don't know, 4,000 dimensions. And then I say, "Okay, and then this should be equivalent here to ICL that comes in here, the context, the context activation on the frozen weights." And now I have to understand that these two images are identical. So, this golden context activation are more or less just a rank-one patch that I apply here to my frozen weight tensor structure. So, ICL is mathematically proven equivalent to a rank-one update on the tensor structure. Okay, let's go with this. Okay, now I have further questions. So, you see here how I fight with these things sometimes. And say, "But wait, the activation of our neurons in the layers of the transformers is not a weight tensor update." No, an activation is something completely different. This is the vector that flows through the layers. The weight tensors. If you modify a weight tensor, how is this identical? And the answer is given by the paper. It is possible because they prove mathematically, not in a visual explanation like I try to think. They say it is possible because mathematically we can prove that there is a linearity of matrix multiplication that we can find identical mathematical terms that we can identify.
So, let's do this. It is simple. It is real simple. But it is important that we all understand it. Here in the activation space, we have the classical view. No, let W be the frozen weights here of an MLP layer in a transformer. Let X be your human query here or the embedding here of the last token. And let A be here the output of the attention layer. Attention and A. This is more or less the complete context information. Then a standard transformer. And now let's ignore the nonlinearities here for a second. I'll come back to the nonlinearities in a minute. The input to the MLP is the sum of the query and the attention output. No, via the residual stream, as I showed my you in my last video. So, if you, this is simple. So, we intuitively think now, "Okay, the input X is now shifted here by a particular vector A." Beautiful. And this is it. So, this is our activation space view. So, the term W * A is now a vector representing here the correction, or if you want, the knowledge retrieved now from the added context to our ICL prompt. Now, the authors ask more or less, "Hey, can we find now a matrix W_prime such that this correction comes now from the weights instead from the context input?" So, mathematically, it is so easy. The authors tell us, "We just want to satisfy this equation." Huh? So, we have here the, if you want, the activation space view, and then we have here the weight space view, and we just want to make that this is, can we have this is the same? Yeah, this is it. No, W * A is exactly our delta W * X. X is here the human query, if you want, and A is here the attention here of this additional context, and W is our weight, and delta W is our additional weight structure. So, to solve here for W_prime, we need a matrix that when multiplied with X produces here this particular vector W * A. Uh, distributive property. So, now they tell us mathematically, the authors tell us, "We can pretend that the correction term W * A did not come from the extra vector A, but instead came from an extra weight acting on the original vector X."
So, now we understand why they claim this because they build a mathematical framework where more or less they say, "In the end of all of our calculations, we arrive at this. We arrive here at this term and at this term from two different mathematical spaces, and we just ask, 'Hey, is it possible that our W * A becomes becomes equivalent to our delta W * X?' Can we find this?" And then this is here, if you want, a correction term that is now coming not from the content from the attention, but now we can go and operate with virtual weight structures. So, we have now this bridge suddenly connecting fine-tuning and in-context learning. Now, of course, as I showed you, you can do this here with a rank-one solution. No, if you go with an outer product, simple. No. So, mathematically, we can say adding now the attention output A to the input is identical to updating the weight matrix W by adding here a rank-one matrix delta. So, the activation pattern A from the attention layer becomes the source of the weight update delta W in ICL. So, we have now this beautiful connection between fine-tuning and ICL in a coherent mathematical theory. And now you understand why the authors can claim this identity because mathematically speaking, it is identical. But in physics, you know, whenever we see that we sort that two properties of a physical system are different, and then analyzing it mathematically, we see they are identical, we understand that we made a mistake in building this physical system because either we have neglected symmetries or we have neglected identity operation, but we built something that is not, let's call it, perfect. We built something that works, but we don't really understand why. And now, with this mathematical identity, I have a feeling that what we think is the difference between in-context learning and fine-tuning, what we coded, it turns out is indeed the identical thing, more or less. So maybe we built our transformer architecture just in a wrong way. We didn't utilize that those terms are identical.
Now, you know, we operated here with a simplified kernel representation. And the beauty of the second preprint that I showed you here is the second order here from December 12, that they took this much further, and they prove mathematically that it works here also in a realistic nonlinear deep learning scenarios. So, the second paper is just taking the results of the first paper and proving it mathematically in nonlinear deep learning. But we stay on the same complexity that we try to understand and we try to solve. So, yeah, no, the skeptics, and this is me, I have said at the beginning, no, but you have the layer norm, you have nonlinear activation, you have skip connections. No, and this is why I did not take here this publication by Google here from July 2025 really that significant because I thought, "Hey, the simple algebra doesn't hold if you input nonlinear activations." No, but this team of now proved here just two weeks ago that it holds. And this is something, okay. But they have a second insight: that the update delta W becomes specific to the token position. So, we calculate, given the nonlinearity, a delta W for each single token position. Now, this thing becomes a little bit more interesting. No. So, the weights are effectively different for every single token that is generated by our autoregressive system. The model isn't just fine-tuned once. It is now with ICL continuously refining itself at every step of the generation of the next token prediction of an autoregressive transformer architecture. Therefore, we can now stop treating here the learning, and I mean here the backpropagation, the learning process on the neural network, and the inference, the forward pass, as it would be two different things. It turns out, no. If you understand the mathematics of this, we see the forward pass is a learning step in itself, and this is mathematically equivalent here to the backpropagation. So, somehow there is a chance we can further optimize the architectures of the transformers. We can make it maybe smaller. We can make it more powerful. We have not yet optimized the architecture.
So, if I go here with my image, this is just in my head, whatever image you have. I have here, you know, my tensor structure here, my weight matrix here in, I don't know, 4,000 dimensions. And then I say, "Okay, and then this should be equivalent here to ICL that comes in here, the context, the context activation on the frozen weights." And now I have to understand that these two images are identical. So, this golden context activation are more or less just a rank-one patch that I apply here to my frozen weight tensor structure. So, ICL is mathematically proven equivalent to a rank-one update on the tensor structure. Okay, let's go with this. Okay, now I have further questions. So, you see here how I fight with these things sometimes. And say, "But wait, the activation of our neurons in the layers of the transformers is not a weight tensor update." No, an activation is something completely different. This is the vector that flows through the layers. The weight tensors. If you modify a weight tensor, how is this identical? And the answer is given by the paper. It is possible because they prove mathematically, not in a visual explanation like I try to think. They say it is possible because mathematically we can prove that there is a linearity of matrix multiplication that we can find identical mathematical terms that we can identify.
So, let's do this. It is simple. It is real simple. But it is important that we all understand it. Here in the activation space, we have the classical view. No, let W be the frozen weights here of an MLP layer in a transformer. Let X be your human query here or the embedding here of the last token. And let A be here the output of the attention layer. Attention and A. This is more or less the complete context information. Then a standard transformer. And now let's ignore the nonlinearities here for a second. I'll come back to the nonlinearities in a minute. The input to the MLP is the sum of the query and the attention output. No, via the residual stream, as I showed my you in my last video. So, if you, this is simple. So, we intuitively think now, "Okay, the input X is now shifted here by a particular vector A." Beautiful. And this is it. So, this is our activation space view. So, the term W * A is now a vector representing here the correction, or if you want, the knowledge retrieved now from the added context to our ICL prompt. Now, the authors ask more or less, "Hey, can we find now a matrix W_prime such that this correction comes now from the weights instead from the context input?" So, mathematically, it is so easy. The authors tell us, "We just want to satisfy this equation." Huh? So, we have here the, if you want, the activation space view, and then we have here the weight space view, and we just want to make that this is, can we have this is the same? Yeah, this is it. No, W * A is exactly our delta W * X. X is here the human query, if you want, and A is here the attention here of this additional context, and W is our weight, and delta W is our additional weight structure. So, to solve here for W_prime, we need a matrix that when multiplied with X produces here this particular vector W * A. Uh, distributive property. So, now they tell us mathematically, the authors tell us, "We can pretend that the correction term W * A did not come from the extra vector A, but instead came from an extra weight acting on the original vector X."
So, now we understand why they claim this because they build a mathematical framework where more or less they say, "In the end of all of our calculations, we arrive at this. We arrive here at this term and at this term from two different mathematical spaces, and we just ask, 'Hey, is it possible that our W * A becomes becomes equivalent to our delta W * X?' Can we find this?" And then this is here, if you want, a correction term that is now coming not from the content from the attention, but now we can go and operate with virtual weight structures. So, we have now this bridge suddenly connecting fine-tuning and in-context learning. Now, of course, as I showed you, you can do this here with a rank-one solution. No, if you go with an outer product, simple. No. So, mathematically, we can say adding now the attention output A to the input is identical to updating the weight matrix W by adding here a rank-one matrix delta. So, the activation pattern A from the attention layer becomes the source of the weight update delta W in ICL. So, we have now this beautiful connection between fine-tuning and ICL in a coherent mathematical theory. And now you understand why the authors can claim this identity because mathematically speaking, it is identical. But in physics, you know, whenever we see that we sort that two properties of a physical system are different, and then analyzing it mathematically, we see they are identical, we understand that we made a mistake in building this physical system because either we have neglected symmetries or we have neglected identity operation, but we built something that is not, let's call it, perfect. We built something that works, but we don't really understand why. And now, with this mathematical identity, I have a feeling that what we think is the difference between in-context learning and fine-tuning, what we coded, it turns out is indeed the identical thing, more or less. So maybe we built our transformer architecture just in a wrong way. We didn't utilize that those terms are identical.
Now, you know, we operated here with a simplified kernel representation. And the beauty of the second preprint that I showed you here is the second order here from December 12, that they took this much further, and they prove mathematically that it works here also in a realistic nonlinear deep learning scenarios. So, the second paper is just taking the results of the first paper and proving it mathematically in nonlinear deep learning. But we stay on the same complexity that we try to understand and we try to solve. So, yeah, no, the skeptics, and this is me, I have said at the beginning, no, but you have the layer norm, you have nonlinear activation, you have skip connections. No, and this is why I did not take here this publication by Google here from July 2025 really that significant because I thought, "Hey, the simple algebra doesn't hold if you input nonlinear activations." No, but this team of now proved here just two weeks ago that it holds. And this is something, okay. But they have a second insight: that the update delta W becomes specific to the token position. So, we calculate, given the nonlinearity, a delta W for each single token position. Now, this thing becomes a little bit more interesting. No. So, the weights are effectively different for every single token that is generated by our autoregressive system. The model isn't just fine-tuned once. It is now with ICL continuously refining itself at every step of the generation of the next token prediction of an autoregressive transformer architecture. Therefore, we can now stop treating here the learning, and I mean here the backpropagation, the learning process on the neural network, and the inference, the forward pass, as it would be two different things. It turns out, no. If you understand the mathematics of this, we see the forward pass is a learning step in itself, and this is mathematically equivalent here to the backpropagation. So, somehow there is a chance we can further optimize the architectures of the transformers. We can make it maybe smaller. We can make it more powerful. We have not yet optimized the architecture.
So, if I go here with my image, this is just in my head, whatever image you have. I have here, you know, my tensor structure here, my weight matrix here in, I don't know, 4,000 dimensions. And then I say, "Okay, and then this should be equivalent here to ICL that comes in here, the context, the context activation on the frozen weights." And now I have to understand that these two images are identical. So, this golden context activation are more or less just a rank-one patch that I apply here to my frozen weight tensor structure. So, ICL is mathematically proven equivalent to a rank-one update on the tensor structure. Okay, let's go with this. Okay, now I have further questions. So, you see here how I fight with these things sometimes. And say, "But wait, the activation of our neurons in the layers of the transformers is not a weight tensor update." No, an activation is something completely different. This is the vector that flows through the layers. The weight tensors. If you modify a weight tensor, how is this identical? And the answer is given by the paper. It is possible because they prove mathematically, not in a visual explanation like I try to think. They say it is possible because mathematically we can prove that there is a linearity of matrix multiplication that we can find identical mathematical terms that we can identify.
So, let's do this. It is simple. It is real simple. But it is important that we all understand it. Here in the activation space, we have the classical view. No, let W be the frozen weights here of an MLP layer in a transformer. Let X be your human query here or the embedding here of the last token. And let A be here the output of the attention layer. Attention and A. This is more or less the complete context information. Then a standard transformer. And now let's ignore the nonlinearities here for a second. I'll come back to the nonlinearities in a minute. The input to the MLP is the sum of the query and the attention output. No, via the residual stream, as I showed my you in my last video. So, if you, this is simple. So, we intuitively think now, "Okay, the input X is now shifted here by a particular vector A." Beautiful. And this is it. So, this is our activation space view. So, the term W * A is now a vector representing here the correction, or if you want, the knowledge retrieved now from the added context to our ICL prompt. Now, the authors ask more or less, "Hey, can we find now a matrix W_prime such that this correction comes now from the weights instead from the context input?" So, mathematically, it is so easy. The authors tell us, "We just want to satisfy this equation." Huh? So, we have here the, if you want, the activation space view, and then we have here the weight space view, and we just want to make that this is, can we have this is the same? Yeah, this is it. No, W * A is exactly our delta W * X. X is here the human query, if you want, and A is here the attention here of this additional context, and W is our weight, and delta W is our additional weight structure. So, to solve here for W_prime, we need a matrix that when multiplied with X produces here this particular vector W * A. Uh, distributive property. So, now they tell us mathematically, the authors tell us, "We can pretend that the correction term W * A did not come from the extra vector A, but instead came from an extra weight acting on the original vector X."
So, now we understand why they claim this because they build a mathematical framework where more or less they say, "In the end of all of our calculations, we arrive at this. We arrive here at this term and at this term from two different mathematical spaces, and we just ask, 'Hey, is it possible that our W * A becomes becomes equivalent to our delta W * X?' Can we find this?" And then this is here, if you want, a correction term that is now coming not from the content from the attention, but now we can go and operate with virtual weight structures. So, we have now this bridge suddenly connecting fine-tuning and in-context learning. Now, of course, as I showed you, you can do this here with a rank-one solution. No, if you go with an outer product, simple. No. So, mathematically, we can say adding now the attention output A to the input is identical to updating the weight matrix W by adding here a rank-one matrix delta. So, the activation pattern A from the attention layer becomes the source of the weight update delta W in ICL. So, we have now this beautiful connection between fine-tuning and ICL in a coherent mathematical theory. And now you understand why the authors can claim this identity because mathematically speaking, it is identical. But in physics, you know, whenever we see that we sort that two properties of a physical system are different, and then analyzing it mathematically, we see they are identical, we understand that we made a mistake in building this physical system because either we have neglected symmetries or we have neglected identity operation, but we built something that is not, let's call it, perfect. We built something that works, but we don't really understand why. And now, with this mathematical identity, I have a feeling that what we think is the difference between in-context learning and fine-tuning, what we coded, it turns out is indeed the identical thing, more or less. So maybe we built our transformer architecture just in a wrong way. We didn't utilize that those terms are identical.
Now, you know, we operated here with a simplified kernel representation. And the beauty of the second preprint that I showed you here is the second order here from December 12, that they took this much further, and they prove mathematically that it works here also in a realistic nonlinear deep learning scenarios. So, the second paper is just taking the results of the first paper and proving it mathematically in nonlinear deep learning. But we stay on the same complexity that we try to understand and we try to solve. So, yeah, no, the skeptics, and this is me, I have said at the beginning, no, but you have the layer norm, you have nonlinear activation, you have skip connections. No, and this is why I did not take here this publication by Google here from July 2025 really that significant because I thought, "Hey, the simple algebra doesn't hold if you input nonlinear activations." No, but this team of now proved here just two weeks ago that it holds. And this is something, okay. But they have a second insight: that the update delta W becomes specific to the token position. So, we calculate, given the nonlinearity, a delta W for each single token position. Now, this thing becomes a little bit more interesting. No. So, the weights are effectively different for every single token that is generated by our autoregressive system. The model isn't just fine-tuned once. It is now with ICL continuously refining itself at every step of the generation of the next token prediction of an autoregressive transformer architecture. Therefore, we can now stop treating here the learning, and I mean here the backpropagation, the learning process on the neural network, and the inference, the forward pass, as it would be two different things. It turns out, no. If you understand the mathematics of this, we see the forward pass is a learning step in itself, and this is mathematically equivalent here to the backpropagation. So, somehow there is a chance we can further optimize the architectures of the transformers. We can make it maybe smaller. We can make it more powerful. We have not yet optimized the architecture.
So, if I go here with my image, this is just in my head, whatever image you have. I have here, you know, my tensor structure here, my weight matrix here in, I don't know, 4,000 dimensions. And then I say, "Okay, and then this should be equivalent here to ICL that comes in here, the context, the context activation on the frozen weights." And now I have to understand that these two images are identical. So, this golden context activation are more or less just a rank-one patch that I apply here to my frozen weight tensor structure. So, ICL is mathematically proven equivalent to a rank-one update on the tensor structure. Okay, let's go with this. Okay, now I have further questions. So, you see here how I fight with these things sometimes. And say, "But wait, the activation of our neurons in the layers of the transformers is not a weight tensor update." No, an activation is something completely different. This is the vector that flows through the layers. The weight tensors. If you modify a weight tensor, how is this identical? And the answer is given by the paper. It is possible because they prove mathematically, not in a visual explanation like I try to think. They say it is possible because mathematically we can prove that there is a linearity of matrix multiplication that we can find identical mathematical terms that we can identify.
So, let's do this. It is simple. It is real simple. But it is important that we all understand it. Here in the activation space, we have the classical view. No, let W be the frozen weights here of an MLP layer in a transformer. Let X be your human query here or the embedding here of the last token. And let A be here the output of the attention layer. Attention and A. This is more or less the complete context information. Then a standard transformer. And now let's ignore the nonlinearities here for a second. I'll come back to the nonlinearities in a minute. The input to the MLP is the sum of the query and the attention output. No, via the residual stream, as I showed my you in my last video. So, if you, this is simple. So, we intuitively think now, "Okay, the input X is now shifted here by a particular vector A." Beautiful. And this is it. So, this is our activation space view. So, the term W * A is now a vector representing here the correction, or if you want, the knowledge retrieved now from the added context to our ICL prompt. Now, the authors ask more or less, "Hey, can we find now a matrix W_prime such that this correction comes now from the weights instead from the context input?" So, mathematically, it is so easy. The authors tell us, "We just want to satisfy this equation." Huh? So, we have here the, if you want, the activation space view, and then we have here the weight space view, and we just want to make that this is, can we have this is the same? Yeah, this is it. No, W * A is exactly our delta W * X. X is here the human query, if you want, and A is here the attention here of this additional context, and W is our weight, and delta W is our additional weight structure. So, to solve here for W_prime, we need a matrix that when multiplied with X produces here this particular vector W * A. Uh, distributive property. So, now they tell us mathematically, the authors tell us, "We can pretend that the correction term W * A did not come from the extra vector A, but instead came from an extra weight acting on the original vector X."
So, now we understand why they claim this because they build a mathematical framework where more or less they say, "In the end of all of our calculations, we arrive at this. We arrive here at this term and at this term from two different mathematical spaces, and we just ask, 'Hey, is it possible that our W * A becomes becomes equivalent to our delta W * X?' Can we find this?" And then this is here, if you want, a correction term that is now coming not from the content from the attention, but now we can go and operate with virtual weight structures. So, we have now this bridge suddenly connecting fine-tuning and in-context learning. Now, of course, as I showed you, you can do this here with a rank-one solution. No, if you go with an outer product, simple. No. So, mathematically, we can say adding now the attention output A to the input is identical to updating the weight matrix W by adding here a rank-one matrix delta. So, the activation pattern A from the attention layer becomes the source of the weight update delta W in ICL. So, we have now this beautiful connection between fine-tuning and ICL in a coherent mathematical theory. And now you understand why the authors can claim this identity because mathematically speaking, it is identical. But in physics, you know, whenever we see that we sort that two properties of a physical system are different, and then analyzing it mathematically, we see they are identical, we understand that we made a mistake in building this physical system because either we have neglected symmetries or we have neglected identity operation, but we built something that is not, let's call it, perfect. We built something that works, but we don't really understand why. And now, with this mathematical identity, I have a feeling that what we think is the difference between in-context learning and fine-tuning, what we coded, it turns out is indeed the identical thing, more or less. So maybe we built our transformer architecture just in a wrong way. We didn't utilize that those terms are identical.
Now, you know, we operated here with a simplified kernel representation. And the beauty of the second preprint that I showed you here is the second order here from December 12, that they took this much further, and they prove mathematically that it works here also in a realistic nonlinear deep learning scenarios. So, the second paper is just taking the results of the first paper and proving it mathematically in nonlinear deep learning. But we stay on the same complexity that we try to understand and we try to solve. So, yeah, no, the skeptics, and this is me, I have said at the beginning, no, but you have the layer norm, you have nonlinear activation, you have skip connections. No, and this is why I did not take here this publication by Google here from July 2025 really that significant because I thought, "Hey, the simple algebra doesn't hold if you input nonlinear activations." No, but this team of now proved here just two weeks ago that it holds. And this is something, okay. But they have a second insight: that the update delta W becomes specific to the token position. So, we calculate, given the nonlinearity, a delta W for each single token position. Now, this thing becomes a little bit more interesting. No. So, the weights are effectively different for every single token that is generated by our autoregressive system. The model isn't just fine-tuned once. It is now with ICL continuously refining itself at every step of the generation of the next token prediction of an autoregressive transformer architecture. Therefore, we can now stop treating here the learning, and I mean here the backpropagation, the learning process on the neural network, and the inference, the forward pass, as it would be two different things. It turns out, no. If you understand the mathematics of this, we see the forward pass is a learning step in itself, and this is mathematically equivalent here to the backpropagation. So, somehow there is a chance we can further optimize the architectures of the transformers. We can make it maybe smaller. We can make it more powerful. We have not yet optimized the architecture.
So, if I go here with my image, this is just in my head, whatever image you have. I have here, you know, my tensor structure here, my weight matrix here in, I don't know, 4,000 dimensions. And then I say, "Okay, and then this should be equivalent here to ICL that comes in here, the context, the context activation on the frozen weights." And now I have to understand that these two images are identical. So, this golden context activation are more or less just a rank-one patch that I apply here to my frozen weight tensor structure. So, ICL is mathematically proven equivalent to a rank-one update on the tensor structure. Okay, let's go with this. Okay, now I have further questions. So, you see here how I fight with these things sometimes. And say, "But wait, the activation of our neurons in the layers of the transformers is not a weight tensor update." No, an activation is something completely different. This is the vector that flows through the layers. The weight tensors. If you modify a weight tensor, how is this identical? And the answer is given by the paper. It is possible because they prove mathematically, not in a visual explanation like I try to think. They say it is possible because mathematically we can prove that there is a linearity of matrix multiplication that we can find identical mathematical terms that we can identify.
So, let's do this. It is simple. It is real simple. But it is important that we all understand it. Here in the activation space, we have the classical view. No, let W be the frozen weights here of an MLP layer in a transformer. Let X be your human query here or the embedding here of the last token. And let A be here the output of the attention layer. Attention and A. This is more or less the complete context information. Then a standard transformer. And now let's ignore the nonlinearities here for a second. I'll come back to the nonlinearities in a minute. The input to the MLP is the sum of the query and the attention output. No, via the residual stream, as I showed my you in my last video. So, if you, this is simple. So, we intuitively think now, "Okay, the input X is now shifted here by a particular vector A." Beautiful. And this is it. So, this is our activation space view. So, the term W * A is now a vector representing here the correction, or if you want, the knowledge retrieved now from the added context to our ICL prompt. Now, the authors ask more or less, "Hey, can we find now a matrix W_prime such that this correction comes now from the weights instead from the context input?" So, mathematically, it is so easy. The authors tell us, "We just want to satisfy this equation." Huh? So, we have here the, if you want, the activation space view, and then we have here the weight space view, and we just want to make that this is, can we have this is the same? Yeah, this is it. No, W * A is exactly our delta W * X. X is here the human query, if you want, and A is here the attention here of this additional context, and W is our weight, and delta W is our additional weight structure. So, to solve here for W_prime, we need a matrix that when multiplied with X produces here this particular vector W * A. Uh, distributive property. So, now they tell us mathematically, the authors tell us, "We can pretend that the correction term W * A did not come from the extra vector A, but instead came from an extra weight acting on the original vector X."
So, now we understand why they claim this because they build a mathematical framework where more or less they say, "In the end of all of our calculations, we arrive at this. We arrive here at this term and at this term from two different mathematical spaces, and we just ask, 'Hey, is it possible that our W * A becomes becomes equivalent to our delta W * X?' Can we find this?" And then this is here, if you want, a correction term that is now coming not from the content from the attention, but now we can go and operate with virtual weight structures. So, we have now this bridge suddenly connecting fine-tuning and in-context learning. Now, of course, as I showed you, you can do this here with a rank-one solution. No, if you go with an outer product, simple. No. So, mathematically, we can say adding now the attention output A to the input is identical to updating the weight matrix W by adding here a rank-one matrix delta. So, the activation pattern A from the attention layer becomes the source of the weight update delta W in ICL. So, we have now this beautiful connection between fine-tuning and ICL in a coherent mathematical theory. And now you understand why the authors can claim this identity because mathematically speaking, it is identical. But in physics, you know, whenever we see that we sort that two properties of a physical system are different, and then analyzing it mathematically, we see they are identical, we understand that we made a mistake in building this physical system because either we have neglected symmetries or we have neglected identity operation, but we built something that is not, let's call it, perfect. We built something that works, but we don't really understand why. And now, with this mathematical identity, I have a feeling that what we think is the difference between in-context learning and fine-tuning, what we coded, it turns out is indeed the identical thing, more or less. So maybe we built our transformer architecture just in a wrong way. We didn't utilize that those terms are identical.
Now, you know, we operated here with a simplified kernel representation. And the beauty of the second preprint that I showed you here is the second order here from December 12, that they took this much further, and they prove mathematically that it works here also in a realistic nonlinear deep learning scenarios. So, the second paper is just taking the results of the first paper and proving it mathematically in nonlinear deep learning. But we stay on the same complexity that we try to understand and we try to solve. So, yeah, no, the skeptics, and this is me, I have said at the beginning, no, but you have the layer norm, you have nonlinear activation, you have skip connections. No, and this is why I did not take here this publication by Google here from July 2025 really that significant because I thought, "Hey, the simple algebra doesn't hold if you input nonlinear activations." No, but this team of now proved here just two weeks ago that it holds. And this is something, okay. But they have a second insight: that the update delta W becomes specific to the token position. So, we calculate, given the nonlinearity, a delta W for each single token position. Now, this thing becomes a little bit more interesting. No. So, the weights are effectively different for every single token that is generated by our autoregressive system. The model isn't just fine-tuned once. It is now with ICL continuously refining itself at every step of the generation of the next token prediction of an autoregressive transformer architecture. Therefore, we can now stop treating here the learning, and I mean here the backpropagation, the learning process on the neural network, and the inference, the forward pass, as it would be two different things. It turns out, no. If you understand the mathematics of this, we see the forward pass is a learning step in itself, and this is mathematically equivalent here to the backpropagation. So, somehow there is a chance we can further optimize the architectures of the transformers. We can make it maybe smaller. We can make it more powerful. We have not yet optimized the architecture.
So, if I go here with my image, this is just in my head, whatever image you have. I have here, you know, my tensor structure here, my weight matrix here in, I don't know, 4,000 dimensions. And then I say, "Okay, and then this should be equivalent here to ICL that comes in here, the context, the context activation on the frozen weights." And now I have to understand that these two images are identical. So, this golden context activation are more or less just a rank-one patch that I apply here to my frozen weight tensor structure. So, ICL is mathematically proven equivalent to a rank-one update on the tensor structure. Okay, let's go with this. Okay, now I have further questions. So, you see here how I fight with these things sometimes. And say, "But wait, the activation of our neurons in the layers of the transformers is not a weight tensor update." No, an activation is something completely different. This is the vector that flows through the layers. The weight tensors. If you modify a weight tensor, how is this identical? And the answer is given by the paper. It is possible because they prove mathematically, not in a visual explanation like I try to think. They say it is possible because mathematically we can prove that there is a linearity of matrix multiplication that we can find identical mathematical terms that we can identify.
So, let's do this. It is simple. It is real simple. But it is important that we all understand it. Here in the activation space, we have the classical view. No, let W be the frozen weights here of an MLP layer in a transformer. Let X be your human query here or the embedding here of the last token. And let A be here the output of the attention layer. Attention and A. This is more or less the complete context information. Then a standard transformer. And now let's ignore the nonlinearities here for a second. I'll come back to the nonlinearities in a minute. The input to the MLP is the sum of the query and the attention output. No, via the residual stream, as I showed my you in my last video. So, if you, this is simple. So, we intuitively think now, "Okay, the input X is now shifted here by a particular vector A." Beautiful. And this is it. So, this is our activation space view. So, the term W * A is now a vector representing here the correction, or if you want, the knowledge retrieved now from the added context to our ICL prompt. Now, the authors ask more or less, "Hey, can we find now a matrix W_prime such that this correction comes now from the weights instead from the context input?" So, mathematically, it is so easy. The authors tell us, "We just want to satisfy this equation." Huh? So, we have here the, if you want, the activation space view, and then we have here the weight space view, and we just want to make that this is, can we have this is the same? Yeah, this is it. No, W * A is exactly our delta W * X. X is here the human query, if you want, and A is here the attention here of this additional context, and W is our weight, and delta W is our additional weight structure. So, to solve here for W_prime, we need a matrix that when multiplied with X produces here this particular vector W * A. Uh, distributive property. So, now they tell us mathematically, the authors tell us, "We can pretend that the correction term W * A did not come from the extra vector A, but instead came from an extra weight acting on the original vector X."
So, now we understand why they claim this because they build a mathematical framework where more or less they say, "In the end of all of our calculations, we arrive at this. We arrive here at this term and at this term from two different mathematical spaces, and we just ask, 'Hey, is it possible that our W * A becomes becomes equivalent to our delta W * X?' Can we find this?" And then this is here, if you want, a correction term that is now coming not from the content from the attention, but now we can go and operate with virtual weight structures. So, we have now this bridge suddenly connecting fine-tuning and in-context learning. Now, of course, as I showed you, you can do this here with a rank-one solution. No, if you go with an outer product, simple. No. So, mathematically, we can say adding now the attention output A to the input is identical to updating the weight matrix W by adding here a rank-one matrix delta. So, the activation pattern A from the attention layer becomes the source of the weight update delta W in ICL. So, we have now this beautiful connection between fine-tuning and ICL in a coherent mathematical theory. And now you understand why the authors can claim this identity because mathematically speaking, it is identical. But in physics, you know, whenever we see that we sort that two properties of a physical system are different, and then analyzing it mathematically, we see they are identical, we understand that we made a mistake in building this physical system because either we have neglected symmetries or we have neglected identity operation, but we built something that is not, let's call it, perfect. We built something that works, but we don't really understand why. And now, with this mathematical identity, I have a feeling that what we think is the difference between in-context learning and fine-tuning, what we coded, it turns out is indeed the identical thing, more or less. So maybe we built our transformer architecture just in a wrong way. We didn't utilize that those terms are identical.
Now, you know, we operated here with a simplified kernel representation. And the beauty of the second preprint that I showed you here is the second order here from December 12, that they took this much further, and they prove mathematically that it works here also in a realistic nonlinear deep learning scenarios. So, the second paper is just taking the results of the first paper and proving it mathematically in nonlinear deep learning. But we stay on the same complexity that we try to understand and we try to solve. So, yeah, no, the skeptics, and this is me, I have said at the beginning, no, but you have the layer norm, you have nonlinear activation, you have skip connections. No, and this is why I did not take here this publication by Google here from July 2025 really that significant because I thought, "Hey, the simple algebra doesn't hold if you input nonlinear activations." No, but this team of now proved here just two weeks ago that it holds. And this is something, okay. But they have a second insight: that the update delta W becomes specific to the token position. So, we calculate, given the nonlinearity, a delta W for each single token position. Now, this thing becomes a little bit more interesting. No. So, the weights are effectively different for every single token that is generated by our autoregressive system. The model isn't just fine-tuned once. It is now with ICL continuously refining itself at every step of the generation of the next token prediction of an autoregressive transformer architecture. Therefore, we can now stop treating here the learning, and I mean here the backpropagation, the learning process on the neural network, and the inference, the forward pass, as it would be two different things. It turns out, no. If you understand the mathematics of this, we see the forward pass is a learning step in itself, and this is mathematically equivalent here to the backpropagation. So, somehow there is a chance we can further optimize the architectures of the transformers. We can make it maybe smaller. We can make it more powerful. We have not yet optimized the architecture.
So, if I go here with my image, this is just in my head, whatever image you have. I have here, you know, my tensor structure here, my weight matrix here in, I don't know, 4,000 dimensions. And then I say, "Okay, and then this should be equivalent here to ICL that comes in here, the context, the context activation on the frozen weights." And now I have to understand that these two images are identical. So, this golden context activation are more or less just a rank-one patch that I apply here to my frozen weight tensor structure. So, ICL is mathematically proven equivalent to a rank-one update on the tensor structure. Okay, let's go with this. Okay, now I have further questions. So, you see here how I fight with these things sometimes. And say, "But wait, the activation of our neurons in the layers of the transformers is not a weight tensor update." No, an activation is something completely different. This is the vector that flows through the layers. The weight tensors. If you modify a weight tensor, how is this identical? And the answer is given by the paper. It is possible because they prove mathematically, not in a visual explanation like I try to think. They say it is possible because mathematically we can prove that there is a linearity of matrix multiplication that we can find identical mathematical terms that we can identify.
So, let's do this. It is simple. It is real simple. But it is important that we all understand it. Here in the activation space, we have the classical view. No, let W be the frozen weights here of an MLP layer in a transformer. Let X be your human query here or the embedding here of the last token. And let A be here the output of the attention layer. Attention and A. This is more or less the complete context information. Then a standard transformer. And now let's ignore the nonlinearities here for a second. I'll come back to the nonlinearities in a minute. The input to the MLP is the sum of the query and the attention output. No, via the residual stream, as I showed my you in my last video. So, if you, this is simple. So, we intuitively think now, "Okay, the input X is now shifted here by a particular vector A." Beautiful. And this is it. So, this is our activation space view. So, the term W * A is now a vector representing here the correction, or if you want, the knowledge retrieved now from the added context to our ICL prompt. Now, the authors ask more or less, "Hey, can we find now a matrix W_prime such that this correction comes now from the weights instead from the context input?" So, mathematically, it is so easy. The authors tell us, "We just want to satisfy this equation." Huh? So, we have here the, if you want, the activation space view, and then we have here the weight space view, and we just want to make that this is, can we have this is the same? Yeah, this is it. No, W * A is exactly our delta W * X. X is here the human query, if you want, and A is here the attention here of this additional context, and W is our weight, and delta W is our additional weight structure. So, to solve here for W_prime, we need a matrix that when multiplied with X produces here this particular vector W * A. Uh, distributive property. So, now they tell us mathematically, the authors tell us, "We can pretend that the correction term W * A did not come from the extra vector A, but instead came from an extra weight acting on the original vector X."
So, now we understand why they claim this because they build a mathematical framework where more or less they say, "In the end of all of our calculations, we arrive at this. We arrive here at this term and at this term from two different mathematical spaces, and we just ask, 'Hey, is it possible that our W * A becomes becomes equivalent to our delta W * X?' Can we find this?" And then this is here, if you want, a correction term that is now coming not from the content from the attention, but now we can go and operate with virtual weight structures. So, we have now this bridge suddenly connecting fine-tuning and in-context learning. Now, of course, as I showed you, you can do this here with a rank-one solution. No, if you go with an outer product, simple. No. So, mathematically, we can say adding now the attention output A to the input is identical to updating the weight matrix W by adding here a rank-one matrix delta. So, the activation pattern A from the attention layer becomes the source of the weight update delta W in ICL. So, we have now this beautiful connection between fine-tuning and ICL in a coherent mathematical theory. And now you understand why the authors can claim this identity because mathematically speaking, it is identical. But in physics, you know, whenever we see that we sort that two properties of a physical system are different, and then analyzing it mathematically, we see they are identical, we understand that we made a mistake in building this physical system because either we have neglected symmetries or we have neglected identity operation, but we built something that is not, let's call it, perfect. We built something that works, but we don't really understand why. And now, with this mathematical identity, I have a feeling that what we think is the difference between in-context learning and fine-tuning, what we coded, it turns out is indeed the identical thing, more or less. So maybe we built our transformer architecture just in a wrong way. We didn't utilize that those terms are identical.
Now, you know, we operated here with a simplified kernel representation. And the beauty of the second preprint that I showed you here is the second order here from December 12, that they took this much further, and they prove mathematically that it works here also in a realistic nonlinear deep learning scenarios. So, the second paper is just taking the results of the first paper and proving it mathematically in nonlinear deep learning. But we stay on the same complexity that we try to understand and we try to solve. So, yeah, no, the skeptics, and this is me, I have said at the beginning, no, but you have the layer norm, you have nonlinear activation, you have skip connections. No, and this is why I did not take here this publication by Google here from July 2025 really that significant because I thought, "Hey, the simple algebra doesn't hold if you input nonlinear activations." No, but this team of now proved here just two weeks ago that it holds. And this is something, okay. But they have a second insight: that the update delta W becomes specific to the token position. So, we calculate, given the nonlinearity, a delta W for each single token position. Now, this thing becomes a little bit more interesting. No. So, the weights are effectively different for every single token that is generated by our autoregressive system. The model isn't just fine-tuned once. It is now with ICL continuously refining itself at every step of the generation of the next token prediction of an autoregressive transformer architecture. Therefore, we can now stop treating here the learning, and I mean here the backpropagation, the learning process on the neural network, and the inference, the forward pass, as it would be two different things. It turns out, no. If you understand the mathematics of this, we see the forward pass is a learning step in itself, and this is mathematically equivalent here to the backpropagation. So, somehow there is a chance we can further optimize the architectures of the transformers. We can make it maybe smaller. We can make it more powerful. We have not yet optimized the architecture.
So, if I go here with my image, this is just in my head, whatever image you have. I have here, you know, my tensor structure here, my weight matrix here in, I don't know, 4,000 dimensions. And then I say, "Okay, and then this should be equivalent here to ICL that comes in here, the context, the context activation on the frozen weights." And now I have to understand that these two images are identical. So, this golden context activation are more or less just a rank-one patch that I apply here to my frozen weight tensor structure. So, ICL is mathematically proven equivalent to a rank-one update on the tensor structure. Okay, let's go with this. Okay, now I have further questions. So, you see here how I fight with these things sometimes. And say, "But wait, the activation of our neurons in the layers of the transformers is not a weight tensor update." No, an activation is something completely different. This is the vector that flows through the layers. The weight tensors. If you modify a weight tensor, how is this identical? And the answer is given by the paper. It is possible because they prove mathematically, not in a visual explanation like I try to think. They say it is possible because mathematically we can prove that there is a linearity of matrix multiplication that we can find identical mathematical terms that we can identify.
So, let's do this. It is simple. It is real simple. But it is important that we all understand it. Here in the activation space, we have the classical view. No, let W be the frozen weights here of an MLP layer in a transformer. Let X be your human query here or the embedding here of the last token. And let A be here the output of the attention layer. Attention and A. This is more or less the complete context information. Then a standard transformer. And now let's ignore the nonlinearities here for a second. I'll come back to the nonlinearities in a minute. The input to the MLP is the sum of the query and the attention output. No, via the residual stream, as I showed my you in my last video. So, if you, this is simple. So, we intuitively think now, "Okay, the input X is now shifted here by a particular vector A." Beautiful. And this is it. So, this is our activation space view. So, the term W * A is now a vector representing here the correction, or if you want, the knowledge retrieved now from the added context to our ICL prompt. Now, the authors ask more or less, "Hey, can we find now a matrix W_prime such that this correction comes now from the weights instead from the context input?" So, mathematically, it is so easy. The authors tell us, "We just want to satisfy this equation." Huh? So, we have here the, if you want, the activation space view, and then we have here the weight space view, and we just want to make that this is, can we have this is the same? Yeah, this is it. No, W * A is exactly our delta W * X. X is here the human query, if you want, and A is here the attention here of this additional context, and W is our weight, and delta W is our additional weight structure. So, to solve here for W_prime, we need a matrix that when multiplied with X produces here this particular vector W * A. Uh, distributive property. So, now they tell us mathematically, the authors tell us, "We can pretend that the correction term W * A did not come from the extra vector A, but instead came from an extra weight acting on the original vector X."
So, now we understand why they claim this because they build a mathematical framework where more or less they say, "In the end of all of our calculations, we arrive at this. We arrive here at this term and at this term from two different mathematical spaces, and we just ask, 'Hey, is it possible that our W * A becomes becomes equivalent to our delta W * X?' Can we find this?" And then this is here, if you want, a correction term that is now coming not from the content from the attention, but now we can go and operate with virtual weight structures. So, we have now this bridge suddenly connecting fine-tuning and in-context learning. Now, of course, as I showed you, you can do this here with a rank-one solution. No, if you go with an outer product, simple. No. So, mathematically, we can say adding now the attention output A to the input is identical to updating the weight matrix W by adding here a rank-one matrix delta. So, the activation pattern A from the attention layer becomes the source of the weight update delta W in ICL. So, we have now this beautiful connection between fine-tuning and ICL in a coherent mathematical theory. And now you understand why the authors can claim this identity because mathematically speaking, it is identical. But in physics, you know, whenever we see that we sort that two properties of a physical system are different, and then analyzing it mathematically, we see they are identical, we understand that we made a mistake in building this physical system because either we have neglected symmetries or we have neglected identity operation, but we built something that is not, let's call it, perfect. We built something that works, but we don't really understand why. And now, with this mathematical identity, I have a feeling that what we think is the difference between in-context learning and fine-tuning, what we coded, it turns out is indeed the identical thing, more or less. So maybe we built our transformer architecture just in a wrong way. We didn't utilize that those terms are identical.
Now, you know, we operated here with a simplified kernel representation. And the beauty of the second preprint that I showed you here is the second order here from December 12, that they took this much further, and they prove mathematically that it works here also in a realistic nonlinear deep learning scenarios. So, the second paper is just taking the results of the first paper and proving it mathematically in nonlinear deep learning. But we stay on the same complexity that we try to understand and we try to solve. So, yeah, no, the skeptics, and this is me, I have said at the beginning, no, but you have the layer norm, you have nonlinear activation, you have skip connections. No, and this is why I did not take here this publication by Google here from July 2025 really that significant because I thought, "Hey, the simple algebra doesn't hold if you input nonlinear activations." No, but this team of now proved here just two weeks ago that it holds. And this is something, okay. But they have a second insight: that the update delta W becomes specific to the token position. So, we calculate, given the nonlinearity, a delta W for each single token position. Now, this thing becomes a little bit more interesting. No. So, the weights are effectively different for every single token that is generated by our autoregressive system. The model isn't just fine-tuned once. It is now with ICL continuously refining itself at every step of the generation of the next token prediction of an autoregressive transformer architecture. Therefore, we can now stop treating here the learning, and I mean here the backpropagation, the learning process on the neural network, and the inference, the forward pass, as it would be two different things. It turns out, no. If you understand the mathematics of this, we see the forward pass is a learning step in itself, and this is mathematically equivalent here to the backpropagation. So, somehow there is a chance we can further optimize the architectures of the transformers. We can make it maybe smaller. We can make it more powerful. We have not yet optimized the architecture.
So, if I go here with my image, this is just in my head, whatever image you have. I have here, you know, my tensor structure here, my weight matrix here in, I don't know, 4,000 dimensions. And then I say, "Okay, and then this should be equivalent here to ICL that comes in here, the context, the context activation on the frozen weights." And now I have to understand that these two images are identical. So, this golden context activation are more or less just a rank-one patch that I apply here to my frozen weight tensor structure. So, ICL is mathematically proven equivalent to a rank-one update on the tensor structure. Okay, let's go with this. Okay, now I have further questions. So, you see here how I fight with these things sometimes. And say, "But wait, the activation of our neurons in the layers of the transformers is not a weight tensor update." No, an activation is something completely different. This is the vector that flows through the layers. The weight tensors. If you modify a weight tensor, how is this identical? And the answer is given by the paper. It is possible because they prove mathematically, not in a visual explanation like I try to think. They say it is possible because mathematically we can prove that there is a linearity of matrix multiplication that we can find identical mathematical terms that we can identify.
So, let's do this. It is simple. It is real simple. But it is important that we all understand it. Here in the activation space, we have the classical view. No, let W be the frozen weights here of an MLP layer in a transformer. Let X be your human query here or the embedding here of the last token. And let A be here the output of the attention layer. Attention and A. This is more or less the complete context information. Then a standard transformer. And now let's ignore the nonlinearities here for a second. I'll come back to the nonlinearities in a minute. The input to the MLP is the sum of the query and the attention output. No, via the residual stream, as I showed my you in my last video. So, if you, this is simple. So, we intuitively think now, "Okay, the input X is now shifted here by a particular vector A." Beautiful. And this is it. So, this is our activation space view. So, the term W * A is now a vector representing here the correction, or if you want, the knowledge retrieved now from the added context to our ICL prompt. Now, the authors ask more or less, "Hey, can we find now a matrix W_prime such that this correction comes now from the weights instead from the context input?" So, mathematically, it is so easy. The authors tell us, "We just want to satisfy this equation." Huh? So, we have here the, if you want, the activation space view, and then we have here the weight space view, and we just want to make that this is, can we have this is the same? Yeah, this is it. No, W * A is exactly our delta W * X. X is here the human query, if you want, and A is here the attention here of this additional context, and W is our weight, and delta W is our additional weight structure. So, to solve here for W_prime, we need a matrix that when multiplied with X produces here this particular vector W * A. Uh, distributive property. So, now they tell us mathematically, the authors tell us, "We can pretend that the correction term W * A did not come from the extra vector A, but instead came from an extra weight acting on the original vector X."
So, now we understand why they claim this because they build a mathematical framework where more or less they say, "In the end of all of our calculations, we arrive at this. We arrive here at this term and at this term from two different mathematical spaces, and we just ask, 'Hey, is it possible that our W * A becomes becomes equivalent to our delta W * X?' Can we find this?" And then this is here, if you want, a correction term that is now coming not from the content from the attention, but now we can go and operate with virtual weight structures. So, we have now this bridge suddenly connecting fine-tuning and in-context learning. Now, of course, as I showed you, you can do this here with a rank-one solution. No, if you go with an outer product, simple. No. So, mathematically, we can say adding now the attention output A to the input is identical to updating the weight matrix W by adding here a rank-one matrix delta. So, the activation pattern A from the attention layer becomes the source of the weight update delta W in ICL. So, we have now this beautiful connection between fine-tuning and ICL in a coherent mathematical theory. And now you understand why the authors can claim this identity because mathematically speaking, it is identical. But in physics, you know, whenever we see that we sort that two properties of a physical system are different, and then analyzing it mathematically, we see they are identical, we understand that we made a mistake in building this physical system because either we have neglected symmetries or we have neglected identity operation, but we built something that is not, let's call it, perfect. We built something that works, but we don't really understand why. And now, with this mathematical identity, I have a feeling that what we think is the difference between in-context learning and fine-tuning, what we coded, it turns out is indeed the identical thing, more or less. So maybe we built our transformer architecture just in a wrong way. We didn't utilize that those terms are identical.
Now, you know, we operated here with a simplified kernel representation. And the beauty of the second preprint that I showed you here is the second order here from December 12, that they took this much further, and they prove mathematically that it works here also in a realistic nonlinear deep learning scenarios. So, the second paper is just taking the results of the first paper and proving it mathematically in nonlinear deep learning. But we stay on the same complexity that we try to understand and we try to solve. So, yeah, no, the skeptics, and this is me, I have said at the beginning, no, but you have the layer norm, you have nonlinear activation, you have skip connections. No, and this is why I did not take here this publication by Google here from July 2025 really that significant because I thought, "Hey, the simple algebra doesn't hold if you input nonlinear activations." No, but this team of now proved here just two weeks ago that it holds. And this is something, okay. But they have a second insight: that the update delta W becomes specific to the token position. So, we calculate, given the nonlinearity, a delta W for each single token position. Now, this thing becomes a little bit more interesting. No. So, the weights are effectively different for every single token that is generated by our autoregressive system. The model isn't just fine-tuned once. It is now with ICL continuously refining itself at every step of the generation of the next token prediction of an autoregressive transformer architecture. Therefore, we can now stop treating here the learning, and I mean here the backpropagation, the learning process on the neural network, and the inference, the forward pass, as it would be two different things. It turns out, no. If you understand the mathematics of this, we see the forward pass is a learning step in itself, and this is mathematically equivalent here to the backpropagation. So, somehow there is a chance we can further optimize the architectures of the transformers. We can make it maybe smaller. We can make it more powerful. We have not yet optimized the architecture.
So, if I go here with my image, this is just in my head, whatever image you have. I have here, you know, my tensor structure here, my weight matrix here in, I don't know, 4,000 dimensions. And then I say, "Okay, and then this should be equivalent here to ICL that comes in here, the context, the context activation on the frozen weights." And now I have to understand that these two images are identical. So, this golden context activation are more or less just a rank-one patch that I apply here to my frozen weight tensor structure. So, ICL is mathematically proven equivalent to a rank-one update on the tensor structure. Okay, let's go with this. Okay, now I have further questions. So, you see here how I fight with these things sometimes. And say, "But wait, the activation of our neurons in the layers of the transformers is not a weight tensor update." No, an activation is something completely different. This is the vector that flows through the layers. The weight tensors. If you modify a weight tensor, how is this identical? And the answer is given by the paper. It is possible because they prove mathematically, not in a visual explanation like I try to think. They say it is possible because mathematically we can prove that there is a linearity of matrix multiplication that we can find identical mathematical terms that we can identify.
So, let's do this. It is simple. It is real simple. But it is important that we all understand it. Here in the activation space, we have the classical view. No, let W be the frozen weights here of an MLP layer in a transformer. Let X be your human query here or the embedding here of the last token. And let A be here the output of the attention layer. Attention and A. This is more or less the complete context information. Then a standard transformer. And now let's ignore the nonlinearities here for a second. I'll come back to the nonlinearities in a minute. The input to the MLP is the sum of the query and the attention output. No, via the residual stream, as I showed my you in my last video. So, if you, this is simple. So, we intuitively think now, "Okay, the input X is now shifted here by a particular vector A." Beautiful. And this is it. So, this is our activation space view. So, the term W * A is now a vector representing here the correction, or if you want, the knowledge retrieved now from the added context to our ICL prompt. Now, the authors ask more or less, "Hey, can we find now a matrix W_prime such that this correction comes now from the weights instead from the context input?" So, mathematically, it is so easy. The authors tell us, "We just want to satisfy this equation." Huh? So, we have here the, if you want, the activation space view, and then we have here the weight space view, and we just want to make that this is, can we have this is the same? Yeah, this is it. No, W * A is exactly our delta W * X. X is here the human query, if you want, and A is here the attention here of this additional context, and W is our weight, and delta W is our additional weight structure. So, to solve here for W_prime, we need a matrix that when multiplied with X produces here this particular vector W * A. Uh, distributive property. So, now they tell us mathematically, the authors tell us, "We can pretend that the correction term W * A did not come from the extra vector A, but instead came from an extra weight acting on the original vector X."
So, now we understand why they claim this because they build a mathematical framework where more or less they say, "In the end of all of our calculations, we arrive at this. We arrive here at this term and at this term from two different mathematical spaces, and we just ask, 'Hey, is it possible that our W * A becomes becomes equivalent to our delta W * X?' Can we find this?" And then this is here, if you want, a correction term that is now coming not from the content from the attention, but now we can go and operate with virtual weight structures. So, we have now this bridge suddenly connecting fine-tuning and in-context learning. Now, of course, as I showed you, you can do this here with a rank-one solution. No, if you go with an outer product, simple. No. So, mathematically, we can say adding now the attention output A to the input is identical to updating the weight matrix W by adding here a rank-one matrix delta. So, the activation pattern A from the attention layer becomes the source of the weight update delta W in ICL. So, we have now this beautiful connection between fine-tuning and ICL in a coherent mathematical theory. And now you understand why the authors can claim this identity because mathematically speaking, it is identical. But in physics, you know, whenever we see that we sort that two properties of a physical system are different, and then analyzing it mathematically, we see they are identical, we understand that we made a mistake in building this physical system because either we have neglected symmetries or we have neglected identity operation, but we built something that is not, let's call it, perfect. We built something that works, but we don't really understand why. And now, with this mathematical identity, I have a feeling that what we think is the difference between in-context learning and fine-tuning, what we coded, it turns out is indeed the identical thing, more or less. So maybe we built our transformer architecture just in a wrong way. We didn't utilize that those terms are identical.
Now, you know, we operated here with a simplified kernel representation. And the beauty of the second preprint that I showed you here is the second order here from December 12, that they took this much further, and they prove mathematically that it works here also in a realistic nonlinear deep learning scenarios. So, the second paper is just taking the results of the first paper and proving it mathematically in nonlinear deep learning. But we stay on the same complexity that we try to understand and we try to solve. So, yeah, no, the skeptics, and this is me, I have said at the beginning, no, but you have the layer norm, you have nonlinear activation, you have skip connections. No, and this is why I did not take here this publication by Google here from July 2025 really that significant because I thought, "Hey, the simple algebra doesn't hold if you input nonlinear activations." No, but this team of now proved here just two weeks ago that it holds. And this is something, okay. But they have a second insight: that the update delta W becomes specific to the token position. So, we calculate, given the nonlinearity, a delta W for each single token position. Now, this thing becomes a little bit more interesting. No. So, the weights are effectively different for every single token that is generated by our autoregressive system. The model isn't just fine-tuned once. It is now with ICL continuously refining itself at every step of the generation of the next token prediction of an autoregressive transformer architecture. Therefore, we can now stop treating here the learning, and I mean here the backpropagation, the learning process on the neural network, and the inference, the forward pass, as it would be two different things. It turns out, no. If you understand the mathematics of this, we see the forward pass is a learning step in itself, and this is mathematically equivalent here to the backpropagation. So, somehow there is a chance we can further optimize the architectures of the transformers. We can make it maybe smaller. We can make it more powerful. We have not yet optimized the architecture.
So, if I go here with my image, this is just in my head, whatever image you have. I have here, you know, my tensor structure here, my weight matrix here in, I don't know, 4,000 dimensions. And then I say, "Okay, and then this should be equivalent here to ICL that comes in here, the context, the context activation on the frozen weights." And now I have to understand that these two images are identical. So, this golden context activation are more or less just a rank-one patch that I apply here to my frozen weight tensor structure. So, ICL is mathematically proven equivalent to a rank-one update on the tensor structure. Okay, let's go with this. Okay, now I have further questions. So, you see here how I fight with these things sometimes. And say, "But wait, the activation of our neurons in the layers of the transformers is not a weight tensor update." No, an activation is something completely different. This is the vector that flows through the layers. The weight tensors. If you modify a weight tensor, how is this identical? And the answer is given by the paper. It is possible because they prove mathematically, not in a visual explanation like I try to think. They say it is possible because mathematically we can prove that there is a linearity of matrix multiplication that we can find identical mathematical terms that we can identify.
So, let's do this. It is simple. It is real simple. But it is important that we all understand it. Here in the activation space, we have the classical view. No, let W be the frozen weights here of an MLP layer in a transformer. Let X be your human query here or the embedding here of the last token. And let A be here the output of the attention layer. Attention and A. This is more or less the complete context information. Then a standard transformer. And now let's ignore the nonlinearities here for a second. I'll come back to the nonlinearities in a minute. The input to the MLP is the sum of the query and the attention output. No, via the residual stream, as I showed my you in my last video. So, if you, this is simple. So, we intuitively think now, "Okay, the input X is now shifted here by a particular vector A." Beautiful. And this is it. So, this is our activation space view. So, the term W * A is now a vector representing here the correction, or if you want, the knowledge retrieved now from the added context to our ICL prompt. Now, the authors ask more or less, "Hey, can we find now a matrix W_prime such that this correction comes now from the weights instead from the context input?" So, mathematically, it is so easy. The authors tell us, "We just want to satisfy this equation." Huh? So, we have here the, if you want, the activation space view, and then we have here the weight space view, and we just want to make that this is, can we have this is the same? Yeah, this is it. No, W * A is exactly our delta W * X. X is here the human query, if you want, and A is here the attention here of this additional context, and W is our weight, and delta W is our additional weight structure. So, to solve here for W_prime, we need a matrix that when multiplied with X produces here this particular vector W * A. Uh, distributive property. So, now they tell us mathematically, the authors tell us, "We can pretend that the correction term W * A did not come from the extra vector A, but instead came from an extra weight acting on the original vector X."
So, now we understand why they claim this because they build a mathematical framework where more or less they say, "In the end of all of our calculations, we arrive at this. We arrive here at this term and at this term from two different mathematical spaces, and we just ask, 'Hey, is it possible that our W * A becomes becomes equivalent to our delta W * X?' Can we find this?" And then this is here, if you want, a correction term that is now coming not from the content from the attention, but now we can go and operate with virtual weight structures. So, we have now this bridge suddenly connecting fine-tuning and in-context learning. Now, of course, as I showed you, you can do this here with a rank-one solution. No, if you go with an outer product, simple. No. So, mathematically, we can say adding now the attention output A to the input is identical to updating the weight matrix W by adding here a rank-one matrix delta. So, the activation pattern A from the attention layer becomes the source of the weight update delta W in ICL. So, we have now this beautiful connection between fine-tuning and ICL in a coherent mathematical theory. And now you understand why the authors can claim this identity because mathematically speaking, it is identical. But in physics, you know, whenever we see that we sort that two properties of a physical system are different, and then analyzing it mathematically, we see they are identical, we understand that we made a mistake in building this physical system because either we have neglected symmetries or we have neglected identity operation, but we built something that is not, let's call it, perfect. We built something that works, but we don't really understand why. And now, with this mathematical identity, I have a feeling that what we think is the difference between in-context learning and fine-tuning, what we coded, it turns out is indeed the identical thing, more or less. So maybe we built our transformer architecture just in a wrong way. We didn't utilize that those terms are identical.
Now, you know, we operated here with a simplified kernel representation. And the beauty of the second preprint that I showed you here is the second order here from December 12, that they took this much further, and they prove mathematically that it works here also in a realistic nonlinear deep learning scenarios. So, the second paper is just taking the results of the first paper and proving it mathematically in nonlinear deep learning. But we stay on the same complexity that we try to understand and we try to solve. So, yeah, no, the skeptics, and this is me, I have said at the beginning, no, but you have the layer norm, you have nonlinear activation, you have skip connections. No, and this is why I did not take here this publication by Google here from July 2025 really that significant because I thought, "Hey, the simple algebra doesn't hold if you input nonlinear activations." No, but this team of now proved here just two weeks ago that it holds. And this is something, okay. But they have a second insight: that the update delta W becomes specific to the token position. So, we calculate, given the nonlinearity, a delta W for each single token position. Now, this thing becomes a little bit more interesting. No. So, the weights are effectively different for every single token that is generated by our autoregressive system. The model isn't just fine-tuned once. It is now with ICL continuously refining itself at every step of the generation of the next token prediction of an autoregressive transformer architecture. Therefore, we can now stop treating here the learning, and I mean here the backpropagation, the learning process on the neural network, and the inference, the forward pass, as it would be two different things. It turns out, no. If you understand the mathematics of this, we see the forward pass is a learning step in itself, and this is mathematically equivalent here to the backpropagation. So, somehow there is a chance we can further optimize the architectures of the transformers. We can make it maybe smaller. We can make it more powerful. We have not yet optimized the architecture.
So, if I go here with my image, this is just in my head, whatever image you have. I have here, you know, my tensor structure here, my weight matrix here in, I don't know, 4,000 dimensions. And then I say, "Okay, and then this should be equivalent here to ICL that comes in here, the context, the context activation on the frozen weights." And now I have to understand that these two images are identical. So, this golden context activation are more or less just a rank-one patch that I apply here to my frozen weight tensor structure. So, ICL is mathematically proven equivalent to a rank-one update on the tensor structure. Okay, let's go with this. Okay, now I have further questions. So, you see here how I fight with these things sometimes. And say, "But wait, the activation of our neurons in the layers of the transformers is not a weight tensor update." No, an activation is something completely different. This is the vector that flows through the layers. The weight tensors. If you modify a weight tensor, how is this identical? And the answer is given by the paper. It is possible because they prove mathematically, not in a visual explanation like I try to think. They say it is possible because mathematically we can prove that there is a linearity of matrix multiplication that we can find identical mathematical terms that we can identify.
So, let's do this. It is simple. It is real simple. But it is important that we all understand it. Here in the activation space, we have the classical view. No, let W be the frozen weights here of an MLP layer in a transformer. Let X be your human query here or the embedding here of the last token. And let A be here the output of the attention layer. Attention and A. This is more or less the complete context information. Then a standard transformer. And now let's ignore the nonlinearities here for a second. I'll come back to the nonlinearities in a minute. The input to the MLP is the sum of the query and the attention output. No, via the residual stream, as I showed my you in my last video. So, if you, this is simple. So, we intuitively think now, "Okay, the input X is now shifted here by a particular vector A." Beautiful. And this is it. So, this is our activation space view. So, the term W * A is now a vector representing here the correction, or if you want, the knowledge retrieved now from the added context to our ICL prompt. Now, the authors ask more or less, "Hey, can we find now a matrix W_prime such that this correction comes now from the weights instead from the context input?" So, mathematically, it is so easy. The authors tell us, "We just want to satisfy this equation." Huh? So, we have here the, if you want, the activation space view, and then we have here the weight space view, and we just want to make that this is, can we have this is the same? Yeah, this is it. No, W * A is exactly our delta W * X. X is here the human query, if you want, and A is here the attention here of this additional context, and W is our weight, and delta W is our additional weight structure. So, to solve here for W_prime, we need a matrix that when multiplied with X produces here this particular vector W * A. Uh, distributive property. So, now they tell us mathematically, the authors tell us, "We can pretend that the correction term W * A did not come from the extra vector A, but instead came from an extra weight acting on the original vector X."
So, now we understand why they claim this because they build a mathematical framework where more or less they say, "In the end of all of our calculations, we arrive at this. We arrive here at this term and at this term from two different mathematical spaces, and we just ask, 'Hey, is it possible that our W * A becomes becomes equivalent to our delta W * X?' Can we find this?" And then this is here, if you want, a correction term that is now coming not from the content from the attention, but now we can go and operate with virtual weight structures. So, we have now this bridge suddenly connecting fine-tuning and in-context learning. Now, of course, as I showed you, you can do this here with a rank-one solution. No, if you go with an outer product, simple. No. So, mathematically, we can say adding now the attention output A to the input is identical to updating the weight matrix W by adding here a rank-one matrix delta. So, the activation pattern A from the attention layer becomes the source of the weight update delta W in ICL. So, we have now this beautiful connection between fine-tuning and ICL in a coherent mathematical theory. And now you understand why the authors can claim this identity because mathematically speaking, it is identical. But in physics, you know, whenever we see that we sort that two properties of a physical system are different, and then analyzing it mathematically, we see they are identical, we understand that we made a mistake in building this physical system because either we have neglected symmetries or we have neglected identity operation, but we built something that is not, let's call it, perfect. We built something that works, but we don't really understand why. And now, with this mathematical identity, I have a feeling that what we think is the difference between in-context learning and fine-tuning, what we coded, it turns out is indeed the identical thing, more or less. So maybe we built our transformer architecture just in a wrong way. We didn't utilize that those terms are identical.
Now, you know, we operated here with a simplified kernel representation. And the beauty of the second preprint that I showed you here is the second order here from December 12, that they took this much further, and they prove mathematically that it works here also in a realistic nonlinear deep learning scenarios. So, the second paper is just taking the results of the first paper and proving it mathematically in nonlinear deep learning. But we stay on the same complexity that we try to understand and we try to solve. So, yeah, no, the skeptics, and this is me, I have said at the beginning, no, but you have the layer norm, you have nonlinear activation, you have skip connections. No, and this is why I did not take here this publication by Google here from July 2025 really that significant because I thought, "Hey, the simple algebra doesn't hold if you input nonlinear activations." No, but this team of now proved here just two weeks ago that it holds. And this is something, okay. But they have a second insight: that the update delta W becomes specific to the token position. So, we calculate, given the nonlinearity, a delta W for each single token position. Now, this thing becomes a little bit more interesting. No. So, the weights are effectively different for every single token that is generated by our autoregressive system. The model isn't just fine-tuned once. It is now with ICL continuously refining itself at every step of the generation of the next token prediction of an autoregressive transformer architecture. Therefore, we can now stop treating here the learning, and I mean here the backpropagation, the learning process on the neural network, and the inference, the forward pass, as it would be two different things. It turns out, no. If you understand the mathematics of this, we see the forward pass is a learning step in itself, and this is mathematically equivalent here to the backpropagation. So, somehow there is a chance we can further optimize the architectures of the transformers. We can make it maybe smaller. We can make it more powerful. We have not yet optimized the architecture.
So, if I go here with my image, this is just in my head, whatever image you have. I have here, you know, my tensor structure here, my weight matrix here in, I don't know, 4,000 dimensions. And then I say, "Okay, and then this should be equivalent here to ICL that comes in here, the context, the context activation on the frozen weights." And now I have to understand that these two images are identical. So, this golden context activation are more or less just a rank-one patch that I apply here to my frozen weight tensor structure. So, ICL is mathematically proven equivalent to a rank-one update on the tensor structure. Okay, let's go with this. Okay, now I have further questions. So, you see here how I fight with these things sometimes. And say, "But wait, the activation of our neurons in the layers of the transformers is not a weight tensor update." No, an activation is something completely different. This is the vector that flows through the layers. The weight tensors. If you modify a weight tensor, how is this identical? And the answer is given by the paper. It is possible because they prove mathematically, not in a visual explanation like I try to think. They say it is possible because mathematically we can prove that there is a linearity of matrix multiplication that we can find identical mathematical terms that we can identify.
So, let's do this. It is simple. It is real simple. But it is important that we all understand it. Here in the activation space, we have the classical view. No, let W be the frozen weights here of an MLP layer in a transformer. Let X be your human query here or the embedding here of the last token. And let A be here the output of the attention layer. Attention and A. This is more or less the complete context information. Then a standard transformer. And now let's ignore the nonlinearities here for a second. I'll come back to the nonlinearities in a minute. The input to the MLP is the sum of the query and the attention output. No, via the residual stream, as I showed my you in my last video. So, if you, this is simple. So, we intuitively think now, "Okay, the input X is now shifted here by a particular vector A." Beautiful. And this is it. So, this is our activation space view. So, the term W * A is now a vector representing here the correction, or if you want, the knowledge retrieved now from the added context to our ICL prompt. Now, the authors ask more or less, "Hey, can we find now a matrix W_prime such that this correction comes now from the weights instead from the context input?" So, mathematically, it is so easy. The authors tell us, "We just want to satisfy this equation." Huh? So, we have here the, if you want, the activation space view, and then we have here the weight space view, and we just want to make that this is, can we have this is the same? Yeah, this is it. No, W * A is exactly our delta W * X. X is here the human query, if you want, and A is here the attention here of this additional context, and W is our weight, and delta W is our additional weight structure. So, to solve here for W_prime, we need a matrix that when multiplied with X produces here this particular vector W * A. Uh, distributive property. So, now they tell us mathematically, the authors tell us, "We can pretend that the correction term W * A did not come from the extra vector A, but instead came from an extra weight acting on the original vector X."
So, now we understand why they claim this because they build a mathematical framework where more or less they say, "In the end of all of our calculations, we arrive at this. We arrive here at this term and at this term from two different mathematical spaces, and we just ask, 'Hey, is it possible that our W * A becomes becomes equivalent to our delta W * X?' Can we find this?" And then this is here, if you want, a correction term that is now coming not from the content from the attention, but now we can go and operate with virtual weight structures. So, we have now this bridge suddenly connecting fine-tuning and in-context learning. Now, of course, as I showed you, you can do this here with a rank-one solution. No, if you go with an outer product, simple. No. So, mathematically, we can say adding now the attention output A to the input is identical to updating the weight matrix W by adding here a rank-one matrix delta. So, the activation pattern A from the attention layer becomes the source of the weight update delta W in ICL. So, we have now this beautiful connection between fine-tuning and ICL in a coherent mathematical theory. And now you understand why the authors can claim this identity because mathematically speaking, it is identical. But in physics, you know, whenever we see that we sort that two properties of a physical system are different, and then analyzing it mathematically, we see they are identical, we understand that we made a mistake in building this physical system because either we have neglected symmetries or we have neglected identity operation, but we built something that is not, let's call it, perfect. We built something that works, but we don't really understand why. And now, with this mathematical identity, I have a feeling that what we think is the difference between in-context learning and fine-tuning, what we coded, it turns out is indeed the identical thing, more or less. So maybe we built our transformer architecture just in a wrong way. We didn't utilize that those terms are identical.
Now, you know, we operated here with a simplified kernel representation. And the beauty of the second preprint that I showed you here is the second order here from December 12, that they took this much further, and they prove mathematically that it works here also in a realistic nonlinear deep learning scenarios. So, the second paper is just taking the results of the first paper and proving it mathematically in nonlinear deep learning. But we stay on the same complexity that we try to understand and we try to solve. So, yeah, no, the skeptics, and this is me, I have said at the beginning, no, but you have the layer norm, you have nonlinear activation, you have skip connections. No, and this is why I did not take here this publication by Google here from July 2025 really that significant because I thought, "Hey, the simple algebra doesn't hold if you input nonlinear activations." No, but this team of now proved here just two weeks ago that it holds. And this is something, okay. But they have a second insight: that the update delta W becomes specific to the token position. So, we calculate, given the nonlinearity, a delta W for each single token position. Now, this thing becomes a little bit more interesting. No. So, the weights are effectively different for every single token that is generated by our autoregressive system. The model isn't just fine-tuned once. It is now with ICL continuously refining itself at every step of the generation of the next token prediction of an autoregressive transformer architecture. Therefore, we can now stop treating here the learning, and I mean here the backpropagation, the learning process on the neural network, and the inference, the forward pass, as it would be two different things. It turns out, no. If you understand the mathematics of this, we see the forward pass is a learning step in itself, and this is mathematically equivalent here to the backpropagation. So, somehow there is a chance we can further optimize the architectures of the transformers. We can make it maybe smaller. We can make it more powerful. We have not yet optimized the architecture.
So, if I go here with my image, this is just in my head, whatever image you have. I have here, you know, my tensor structure here, my weight matrix here in, I don't know, 4,000 dimensions. And then I say, "Okay, and then this should be equivalent here to ICL that comes in here, the context, the context activation on the frozen weights." And now I have to understand that these two images are identical. So, this golden context activation are more or less just a rank-one patch that I apply here to my frozen weight tensor structure. So, ICL is mathematically proven equivalent to a rank-one update on the tensor structure. Okay, let's go with this. Okay, now I have further questions. So, you see here how I fight with these things sometimes. And say, "But wait, the activation of our neurons in the layers of the transformers is not a weight tensor update." No, an activation is something completely different. This is the vector that flows through the layers. The weight tensors. If you modify a weight tensor, how is this identical? And the answer is given by the paper. It is possible because they prove mathematically, not in a visual explanation like I try to think. They say it is possible because mathematically we can prove that there is a linearity of matrix multiplication that we can find identical mathematical terms that we can identify.
So, let's do this. It is simple. It is real simple. But it is important that we all understand it. Here in the activation space, we have the classical view. No, let W be the frozen weights here of an MLP layer in a transformer. Let X be your human query here or the embedding here of the last token. And let A be here the output of the attention layer. Attention and A. This is more or less the complete context information. Then a standard transformer. And now let's ignore the nonlinearities here for a second. I'll come back to the nonlinearities in a minute. The input to the MLP is the sum of the query and the attention output. No, via the residual stream, as I showed my you in my last video. So, if you, this is simple. So, we intuitively think now, "Okay, the input X is now shifted here by a particular vector A." Beautiful. And this is it. So, this is our activation space view. So, the term W * A is now a vector representing here the correction, or if you want, the knowledge retrieved now from the added context to our ICL prompt. Now, the authors ask more or less, "Hey, can we find now a matrix W_prime such that this correction comes now from the weights instead from the context input?" So, mathematically, it is so easy. The authors tell us, "We just want to satisfy this equation." Huh? So, we have here the, if you want, the activation space view, and then we have here the weight space view, and we just want to make that this is, can we have this is the same? Yeah, this is it. No, W * A is exactly our delta W * X. X is here the human query, if you want, and A is here the attention here of this additional context, and W is our weight, and delta W is our additional weight structure. So, to solve here for W_prime, we need a matrix that when multiplied with X produces here this particular vector W * A. Uh, distributive property. So, now they tell us mathematically, the authors tell us, "We can pretend that the correction term W * A did not come from the extra vector A, but instead came from an extra weight acting on the original vector X."
So, now we understand why they claim this because they build a mathematical framework where more or less they say, "In the end of all of our calculations, we arrive at this. We arrive here at this term and at this term from two different mathematical spaces, and we just ask, 'Hey, is it possible that our W * A becomes becomes equivalent to our delta W * X?' Can we find this?" And then this is here, if you want, a correction term that is now coming not from the content from the attention, but now we can go and operate with virtual weight structures. So, we have now this bridge suddenly connecting fine-tuning and in-context learning. Now, of course, as I showed you, you can do this here with a rank-one solution. No, if you go with an outer product, simple. No. So, mathematically, we can say adding now the attention output A to the input is identical to updating the weight matrix W by adding here a rank-one matrix delta. So, the activation pattern A from the attention layer becomes the source of the weight update delta W in ICL. So, we have now this beautiful connection between fine-tuning and ICL in a coherent mathematical theory. And now you understand why the authors can claim this identity because mathematically speaking, it is identical. But in physics, you know, whenever we see that we sort that two properties of a physical system are different, and then analyzing it mathematically, we see they are identical, we understand that we made a mistake in building this physical system because either we have neglected symmetries or we have neglected identity operation, but we built something that is not, let's call it, perfect. We built something that works, but we don't really understand why. And now, with this mathematical identity, I have a feeling that what we think is the difference between in-context learning and fine-tuning, what we coded, it turns out is indeed the identical thing, more or less. So maybe we built our transformer architecture just in a wrong way. We didn't utilize that those terms are identical.
Now, you know, we operated here with a simplified kernel representation. And the beauty of the second preprint that I showed you here is the second order here from December 12, that they took this much further, and they prove mathematically that it works here also in a realistic nonlinear deep learning scenarios. So, the second paper is just taking the results of the first paper and proving it mathematically in nonlinear deep learning. But we stay on the same complexity that we try to understand and we try to solve. So, yeah, no, the skeptics, and this is me, I have said at the beginning, no, but you have the layer norm, you have nonlinear activation, you have skip connections. No, and this is why I did not take here this publication by Google here from July 2025 really that significant because I thought, "Hey, the simple algebra doesn't hold if you input nonlinear activations." No, but this team of now proved here just two weeks ago that it holds. And this is something, okay. But they have a second insight: that the update delta W becomes specific to the token position. So, we calculate, given the nonlinearity, a delta W for each single token position. Now, this thing becomes a little bit more interesting. No. So, the weights are effectively different for every single token that is generated by our autoregressive system. The model isn't just fine-tuned once. It is now with ICL continuously refining itself at every step of the generation of the next token prediction of an autoregressive transformer architecture. Therefore, we can now stop treating here the learning, and I mean here the backpropagation, the learning process on the neural network, and the inference, the forward pass, as it would be two different things. It turns out, no. If you understand the mathematics of this, we see the forward pass is a learning step in itself, and this is mathematically equivalent here to the backpropagation. So, somehow there is a chance we can further optimize the architectures of the transformers. We can make it maybe smaller. We can make it more powerful. We have not yet optimized the architecture.
So, if I go here with my image, this is just in my head, whatever image you have. I have here, you know, my tensor structure here, my weight matrix here in, I don't know, 4,000 dimensions. And then I say, "Okay, and then this should be equivalent here to ICL that comes in here, the context, the context activation on the frozen weights." And now I have to understand that these two images are identical. So, this golden context activation are more or less just a rank-one patch that I apply here to my frozen weight tensor structure. So, ICL is mathematically proven equivalent to a rank-one update on the tensor structure. Okay, let's go with this. Okay, now I have further questions. So, you see here how I fight with these things sometimes. And say, "But wait, the activation of our neurons in the layers of the transformers is not a weight tensor update." No, an activation is something completely different. This is the vector that flows through the layers. The weight tensors. If you modify a weight tensor, how is this identical? And the answer is given by the paper. It is possible because they prove mathematically, not in a visual explanation like I try to think. They say it is possible because mathematically we can prove that there is a linearity of matrix multiplication that we can find identical mathematical terms that we can identify.
So, let's do this. It is simple. It is real simple. But it is important that we all understand it. Here in the activation space, we have the classical view. No, let W be the frozen weights here of an MLP layer in a transformer. Let X be your human query here or the embedding here of the last token. And let A be here the output of the attention layer. Attention and A. This is more or less the complete context information. Then a standard transformer. And now let's ignore the nonlinearities here for a second. I'll come back to the nonlinearities in a minute. The input to the MLP is the sum of the query and the attention output. No, via the residual stream, as I showed my you in my last video. So, if you, this is simple. So, we intuitively think now, "Okay, the input X is now shifted here by a particular vector A." Beautiful. And this is it. So, this is our activation space view. So, the term W * A is now a vector representing here the correction, or if you want, the knowledge retrieved now from the added context to our ICL prompt. Now, the authors ask more or less, "Hey, can we find now a matrix W_prime such that this correction comes now from the weights instead from the context input?" So, mathematically, it is so easy. The authors tell us, "We just want to satisfy this equation." Huh? So, we have here the, if you want, the activation space view, and then we have here the weight space view, and we just want to make that this is, can we have this is the same? Yeah, this is it. No, W * A is exactly our delta W * X. X is here the human query, if you want, and A is here the attention here of this additional context, and W is our weight, and delta W is our additional weight structure. So, to solve here for W_prime, we need a matrix that when multiplied with X produces here this particular vector W * A. Uh, distributive property. So, now they tell us mathematically, the authors tell us, "We can pretend that the correction term W * A did not come from the extra vector A, but instead came from an extra weight acting on the original vector X."
So, now we understand why they claim this because they build a mathematical framework where more or less they say, "In the end of all of our calculations, we arrive at this. We arrive here at this term and at this term from two different mathematical spaces, and we just ask, 'Hey, is it possible that our W * A becomes becomes equivalent to our delta W * X?' Can we find this?" And then this is here, if you want, a correction term that is now coming not from the content from the attention, but now we can go and operate with virtual weight structures. So, we have now this bridge suddenly connecting fine-tuning and in-context learning. Now, of course, as I showed you, you can do this here with a rank-one solution. No, if you go with an outer product, simple. No. So, mathematically, we can say adding now the attention output A to the input is identical to updating the weight matrix W by adding here a rank-one matrix delta. So, the activation pattern A from the attention layer becomes the source of the weight update delta W in ICL. So, we have now this beautiful connection between fine-tuning and ICL in a coherent mathematical theory. And now you understand why the authors can claim this identity because mathematically speaking, it is identical. But in physics, you know, whenever we see that we sort that two properties of a physical system are different, and then analyzing it mathematically, we see they are identical, we understand that we made a mistake in building this physical system because either we have neglected symmetries or we have neglected identity operation, but we built something that is not, let's call it, perfect. We built something that works, but we don't really understand why. And now, with this mathematical identity, I have a feeling that what we think is the difference between in-context learning and fine-tuning, what we coded, it turns out is indeed the identical thing, more or less. So maybe we built our transformer architecture just in a wrong way. We didn't utilize that those terms are identical.
Now, you know, we operated here with a simplified kernel representation. And the beauty of the second preprint that I showed you here is the second order here from December 12, that they took this much further, and they prove mathematically that it works here also in a realistic nonlinear deep learning scenarios. So, the second paper is just taking the results of the first paper and proving it mathematically in nonlinear deep learning. But we stay on the same complexity that we try to understand and we try to solve. So, yeah, no, the skeptics, and this is me, I have said at the beginning, no, but you have the layer norm, you have nonlinear activation, you have skip connections. No, and this is why I did not take here this publication by Google here from July 2025 really that significant because I thought, "Hey, the simple algebra doesn't hold if you input nonlinear activations." No, but this team of now proved here just two weeks ago that it holds. And this is something, okay. But they have a second insight: that the update delta W becomes specific to the token position. So, we calculate, given the nonlinearity, a delta W for each single token position. Now, this thing becomes a little bit more interesting. No. So, the weights are effectively different for every single token that is generated by our autoregressive system. The model isn't just fine-tuned once. It is now with ICL continuously refining itself at every step of the generation of the next token prediction of an autoregressive transformer architecture. Therefore, we can now stop treating here the learning, and I mean here the backpropagation, the learning process on the neural network, and the inference, the forward pass, as it would be two different things. It turns out, no. If you understand the mathematics of this, we see the forward pass is a learning step in itself, and this is mathematically equivalent here to the backpropagation. So, somehow there is a chance we can further optimize the architectures of the transformers. We can make it maybe smaller. We can make it more powerful. We have not yet optimized the architecture.
So, if I go here with my image, this is just in my head, whatever image you have. I have here, you know, my tensor structure here, my weight matrix here in, I don't know, 4,000 dimensions. And then I say, "Okay, and then this should be equivalent here to ICL that comes in here, the context, the context activation on the frozen weights." And now I have to understand that these two images are identical. So, this golden context activation are more or less just a rank-one patch that I apply here to my frozen weight tensor structure. So, ICL is mathematically proven equivalent to a rank-one update on the tensor structure. Okay, let's go with this. Okay, now I have further questions. So, you see here how I fight with these things sometimes. And say, "But wait, the activation of our neurons in the layers of the transformers is not a weight tensor update." No, an activation is something completely different. This is the vector that flows through the layers. The weight tensors. If you modify a weight tensor, how is this identical? And the answer is given by the paper. It is possible because they prove mathematically, not in a visual explanation like I try to think. They say it is possible because mathematically we can prove that there is a linearity of matrix multiplication that we can find identical mathematical terms that we can identify.
So, let's do this. It is simple. It is real simple. But it is important that we all understand it. Here in the activation space, we have the classical view. No, let W be the frozen weights here of an MLP layer in a transformer. Let X be your human query here or the embedding here of the last token. And let A be here the output of the attention layer. Attention and A. This is more or less the complete context information. Then a standard transformer. And now let's ignore the nonlinearities here for a second. I'll come back to the nonlinearities in a minute. The input to the MLP is the sum of the query and the attention output. No, via the residual stream, as I showed my you in my last video. So, if you, this is simple. So, we intuitively think now, "Okay, the input X is now shifted here by a particular vector A." Beautiful. And this is it. So, this is our activation space view. So, the term W * A is now a vector representing here the correction, or if you want, the knowledge retrieved now from the added context to our ICL prompt. Now, the authors ask more or less, "Hey, can we find now a matrix W_prime such that this correction comes now from the weights instead from the context input?" So, mathematically, it is so easy. The authors tell us, "We just want to satisfy this equation." Huh? So, we have here the, if you want, the activation space view, and then we have here the weight space view, and we just want to make that this is, can we have this is the same? Yeah, this is it. No, W * A is exactly our delta W * X. X is here the human query, if you want, and A is here the attention here of this additional context, and W is our weight, and delta W is our additional weight structure. So, to solve here for W_prime, we need a matrix that when multiplied with X produces here this particular vector W * A. Uh, distributive property. So, now they tell us mathematically, the authors tell us, "We can pretend that the correction term W * A did not come from the extra vector A, but instead came from an extra weight acting on the original vector X."
So, now we understand why they claim this because they build a mathematical framework where more or less they say, "In the end of all of our calculations, we arrive at this. We arrive here at this term and at this term from two different mathematical spaces, and we just ask, 'Hey, is it possible that our W * A becomes becomes equivalent to our delta W * X?' Can we find this?" And then this is here, if you want, a correction term that is now coming not from the content from the attention, but now we can go and operate with virtual weight structures. So, we have now this bridge suddenly connecting fine-tuning and in-context learning. Now, of course, as I showed you, you can do this here with a rank-one solution. No, if you go with an outer product, simple. No. So, mathematically, we can say adding now the attention output A to the input is identical to updating the weight matrix W by adding here a rank-one matrix delta. So, the activation pattern A from the attention layer becomes the source of the weight update delta W in ICL. So, we have now this beautiful connection between fine-tuning and ICL in a coherent mathematical theory. And now you understand why the authors can claim this identity because mathematically speaking, it is identical. But in physics, you know, whenever we see that we sort that two properties of a physical system are different, and then analyzing it mathematically, we see they are identical, we understand that we made a mistake in building this physical system because either we have neglected symmetries or we have neglected identity operation, but we built something that is not, let's call it, perfect. We built something that works, but we don't really understand why. And now, with this mathematical identity, I have a feeling that what we think is the difference between in-context learning and fine-tuning, what we coded, it turns out is indeed the identical thing, more or less. So maybe we built our transformer architecture just in a wrong way. We didn't utilize that those terms are identical.
Now, you know, we operated here with a simplified kernel representation. And the beauty of the second preprint that I showed you here is the second order here from December 12, that they took this much further, and they prove mathematically that it works here also in a realistic nonlinear deep learning scenarios. So, the second paper is just taking the results of the first paper and proving it mathematically in nonlinear deep learning. But we stay on the same complexity that we try to understand and we try to solve. So, yeah, no, the skeptics, and this is me, I have said at the beginning, no, but you have the layer norm, you have nonlinear activation, you have skip connections. No, and this is why I did not take here this publication by Google here from July 2025 really that significant because I thought, "Hey, the simple algebra doesn't hold if you input nonlinear activations." No, but this team of now proved here just two weeks ago that it holds. And this is something, okay. But they have a second insight: that the update delta W becomes specific to the token position. So, we calculate, given the nonlinearity, a delta W for each single token position. Now, this thing becomes a little bit more interesting. No. So, the weights are effectively different for every single token that is generated by our autoregressive system. The model isn't just fine-tuned once. It is now with ICL continuously refining itself at every step of the generation of the next token prediction of an autoregressive transformer architecture. Therefore, we can now stop treating here the learning, and I mean here the backpropagation, the learning process on the neural network, and the inference, the forward pass, as it would be two different things. It turns out, no. If you understand the mathematics of this, we see the forward pass is a learning step in itself, and this is mathematically equivalent here to the backpropagation. So, somehow there is a chance we can further optimize the architectures of the transformers. We can make it maybe smaller. We can make it more powerful. We have not yet optimized the architecture.
So, if I go here with my image, this is just in my head, whatever image you have. I have here, you know, my tensor structure here, my weight matrix here in, I don't know, 4,000 dimensions. And then I say, "Okay, and then this should be equivalent here to ICL that comes in here, the context, the context activation on the frozen weights." And now I have to understand that these two images are identical. So, this golden context activation are more or less just a rank-one patch that I apply here to my frozen weight tensor structure. So, ICL is mathematically proven equivalent to a rank-one update on the tensor structure. Okay, let's go with this. Okay, now I have further questions. So, you see here how I fight with these things sometimes. And say, "But wait, the activation of our neurons in the layers of the transformers is not a weight tensor update." No, an activation is something completely different. This is the vector that flows through the layers. The weight tensors. If you modify a weight tensor, how is this identical? And the answer is given by the paper. It is possible because they prove mathematically, not in a visual explanation like I try to think. They say it is possible because mathematically we can prove that there is a linearity of matrix multiplication that we can find identical mathematical terms that we can identify.
So, let's do this. It is simple. It is real simple. But it is important that we all understand it. Here in the activation space, we have the classical view. No, let W be the frozen weights here of an MLP layer in a transformer. Let X be your human query here or the embedding here of the last token. And let A be here the output of the attention layer. Attention and A. This is more or less the complete context information. Then a standard transformer. And now let's ignore the nonlinearities here for a second. I'll come back to the nonlinearities in a minute. The input to the MLP is the sum of the query and the attention output. No, via the residual stream, as I showed my you in my last video. So, if you, this is simple. So, we intuitively think now, "Okay, the input X is now shifted here by a particular vector A." Beautiful. And this is it. So, this is our activation space view. So, the term W * A is now a vector representing here the correction, or if you want, the knowledge retrieved now from the added context to our ICL prompt. Now, the authors ask more or less, "Hey, can we find now a matrix W_prime such that this correction comes now from the weights instead from the context input?" So, mathematically, it is so easy. The authors tell us, "We just want to satisfy this equation." Huh? So, we have here the, if you want, the activation space view, and then we have here the weight space view, and we just want to make that this is, can we have this is the same? Yeah, this is it. No, W * A is exactly our delta W * X. X is here the human query, if you want, and A is here the attention here of this additional context, and W is our weight, and delta W is our additional weight structure. So, to solve here for W_prime, we need a matrix that when multiplied with X produces here this particular vector W * A. Uh, distributive property. So, now they tell us mathematically, the authors tell us, "We can pretend that the correction term W * A did not come from the extra vector A, but instead came from an extra weight acting on the original vector X."
So, now we understand why they claim this because they build a mathematical framework where more or less they say, "In the end of all of our calculations, we arrive at this. We arrive here at this term and at this term from two different mathematical spaces, and we just ask, 'Hey, is it possible that our W * A becomes becomes equivalent to our delta W * X?' Can we find this?" And then this is here, if you want, a correction term that is now coming not from the content from the attention, but now we can go and operate with virtual weight structures. So, we have now this bridge suddenly connecting fine-tuning and in-context learning. Now, of course, as I showed you, you can do this here with a rank-one solution. No, if you go with an outer product, simple. No. So, mathematically, we can say adding now the attention output A to the input is identical to updating the weight matrix W by adding here a rank-one matrix delta. So, the activation pattern A from the attention layer becomes the source of the weight update delta W in ICL. So, we have now this beautiful connection between fine-tuning and ICL in a coherent mathematical theory. And now you understand why the authors can claim this identity because mathematically speaking, it is identical. But in physics, you know, whenever we see that we sort that two properties of a physical system are different, and then analyzing it mathematically, we see they are identical, we understand that we made a mistake in building this physical system because either we have neglected symmetries or we have neglected identity operation, but we built something that is not, let's call it, perfect. We built something that works, but we don't really understand why. And now, with this mathematical identity, I have a feeling that what we think is the difference between in-context learning and fine-tuning, what we coded, it turns out is indeed the identical thing, more or less. So maybe we built our transformer architecture just in a wrong way. We didn't utilize that those terms are identical.
Now, you know, we operated here with a simplified kernel representation. And the beauty of the second preprint that I showed you here is the second order here from December 12, that they took this much further, and they prove mathematically that it works here also in a realistic nonlinear deep learning scenarios. So, the second paper is just taking the results of the first paper and proving it mathematically in nonlinear deep learning. But we stay on the same complexity that we try to understand and we try to solve. So, yeah, no, the skeptics, and this is me, I have said at the beginning, no, but you have the layer norm, you have nonlinear activation, you have skip connections. No, and this is why I did not take here this publication by Google here from July 2025 really that significant because I thought, "Hey, the simple algebra doesn't hold if you input nonlinear activations." No, but this team of now proved here just two weeks ago that it holds. And this is something, okay. But they have a second insight: that the update delta W becomes specific to the token position. So, we calculate, given the nonlinearity, a delta W for each single token position. Now, this thing becomes a little bit more interesting. No. So, the weights are effectively different for every single token that is generated by our autoregressive system. The model isn't just fine-tuned once. It is now with ICL continuously refining itself at every step of the generation of the next token prediction of an autoregressive transformer architecture. Therefore, we can now stop treating here the learning, and I mean here the backpropagation, the learning process on the neural network, and the inference, the forward pass, as it would be two different things. It turns out, no. If you understand the mathematics of this, we see the forward pass is a learning step in itself, and this is mathematically equivalent here to the backpropagation. So, somehow there is a chance we can further optimize the architectures of the transformers. We can make it maybe smaller. We can make it more powerful. We have not yet optimized the architecture.
So, if I go here with my image, this is just in my head, whatever image you have. I have here, you know, my tensor structure here, my weight matrix here in, I don't know, 4,000 dimensions. And then I say, "Okay, and then this should be equivalent here to ICL that comes in here, the context, the context activation on the frozen weights." And now I have to understand that these two images are identical. So, this golden context activation are more or less just a rank-one patch that I apply here to my frozen weight tensor structure. So, ICL is mathematically proven equivalent to a rank-one update on the tensor structure. Okay, let's go with this. Okay, now I have further questions. So, you see here how I fight with these things sometimes. And say, "But wait, the activation of our neurons in the layers of the transformers is not a weight tensor update." No, an activation is something completely different. This is the vector that flows through the layers. The weight tensors. If you modify a weight tensor, how is this identical? And the answer is given by the paper. It is possible because they prove mathematically, not in a visual explanation like I try to think. They say it is possible because mathematically we can prove that there is a linearity of matrix multiplication that we can find identical mathematical terms that we can identify.
So, let's do this. It is simple. It is real simple. But it is important that we all understand it. Here in the activation space, we have the classical view. No, let W be the frozen weights here of an MLP layer in a transformer. Let X be your human query here or the embedding here of the last token. And let A be here the output of the attention layer. Attention and A. This is more or less the complete context information. Then a standard transformer. And now let's ignore the nonlinearities here for a second. I'll come back to the nonlinearities in a minute. The input to the MLP is the sum of the query and the attention output. No, via the residual stream, as I showed my you in my last video. So, if you, this is simple. So, we intuitively think now, "Okay, the input X is now shifted here by a particular vector A." Beautiful. And this is it. So, this is our activation space view. So, the term W * A is now a vector representing here the correction, or if you want, the knowledge retrieved now from the added context to our ICL prompt. Now, the authors ask more or less, "Hey, can we find now a matrix W_prime such that this correction comes now from the weights instead from the context input?" So, mathematically, it is so easy. The authors tell us, "We just want to satisfy this equation." Huh? So, we have here the, if you want, the activation space view, and then we have here the weight space view, and we just want to make that this is, can we have this is the same? Yeah, this is it. No, W * A is exactly our delta W * X. X is here the human query, if you want, and A is here the attention here of this additional context, and W is our weight, and delta W is our additional weight structure. So, to solve here for W_prime, we need a matrix that when multiplied with X produces here this particular vector W * A. Uh, distributive property. So, now they tell us mathematically, the authors tell us, "We can pretend that the correction term W * A did not come from the extra vector A, but instead came from an extra weight acting on the original vector X."
So, now we understand why they claim this because they build a mathematical framework where more or less they say, "In the end of all of our calculations, we arrive at this. We arrive here at this term and at this term from two different mathematical spaces, and we just ask, 'Hey, is it possible that our W * A becomes becomes equivalent to our delta W * X?' Can we find this?" And then this is here, if you want, a correction term that is now coming not from the content from the attention, but now we can go and operate with virtual weight structures. So, we have now this bridge suddenly connecting fine-tuning and in-context learning. Now, of course, as I showed you, you can do this here with a rank-one solution. No, if you go with an outer product, simple. No. So, mathematically, we can say adding now the attention output A to the input is identical to updating the weight matrix W by adding here a rank-one matrix delta. So, the activation pattern A from the attention layer becomes the source of the weight update delta W in ICL. So, we have now this beautiful connection between fine-tuning and ICL in a coherent mathematical theory. And now you understand why the authors can claim this identity because mathematically speaking, it is identical. But in physics, you know, whenever we see that we sort that two properties of a physical system are different, and then analyzing it mathematically, we see they are identical, we understand that we made a mistake in building this physical system because either we have neglected symmetries or we have neglected identity operation, but we built something that is not, let's call it, perfect. We built something that works, but we don't really understand why. And now, with this mathematical identity, I have a feeling that what we think is the difference between in-context learning and fine-tuning, what we coded, it turns out is indeed the identical thing, more or less. So maybe we built our transformer architecture just in a wrong way. We didn't utilize that those terms are identical.
Now, you know, we operated here with a simplified kernel representation. And the beauty of the second preprint that I showed you here is the second order here from December 12, that they took this much further, and they prove mathematically that it works here also in a realistic nonlinear deep learning scenarios. So, the second paper is just taking the results of the first paper and proving it mathematically in nonlinear deep learning. But we stay on the same complexity that we try to understand and we try to solve. So, yeah, no, the skeptics, and this is me, I have said at the beginning, no, but you have the layer norm, you have nonlinear activation, you have skip connections. No, and this is why I did not take here this publication by Google here from July 2025 really that significant because I thought, "Hey, the simple algebra doesn't hold if you input nonlinear activations." No, but this team of now proved here just two weeks ago that it holds. And this is something, okay. But they have a second insight: that the update delta W becomes specific to the token position. So, we calculate, given the nonlinearity, a delta W for each single token position. Now, this thing becomes a little bit more interesting. No. So, the weights are effectively different for every single token that is generated by our autoregressive system. The model isn't just fine-tuned once. It is now with ICL continuously refining itself at every step of the generation of the next token prediction of an autoregressive transformer architecture. Therefore, we can now stop treating here the learning, and I mean here the backpropagation, the learning process on the neural network, and the inference, the forward pass, as it would be two different things. It turns out, no. If you understand the mathematics of this, we see the forward pass is a learning step in itself, and this is mathematically equivalent here to the backpropagation. So, somehow there is a chance we can further optimize the architectures of the transformers. We can make it maybe smaller. We can make it more powerful. We have not yet optimized the architecture.
So, if I go here with my image, this is just in my head, whatever image you have. I have here, you know, my tensor structure here, my weight matrix here in, I don't know, 4,000 dimensions. And then I say, "Okay, and then this should be equivalent here to ICL that comes in here, the context, the context activation on the frozen weights." And now I have to understand that these two images are identical. So, this golden context activation are more or less just a rank-one patch that I apply here to my frozen weight tensor structure. So, ICL is mathematically proven equivalent to a rank-one update on the tensor structure. Okay, let's go with this. Okay, now I have further questions. So, you see here how I fight with these things sometimes. And say, "But wait, the activation of our neurons in the layers of the transformers is not a weight tensor update." No, an activation is something completely different. This is the vector that flows through the layers. The weight tensors. If you modify a weight tensor, how is this identical? And the answer is given by the paper. It is possible because they prove mathematically, not in a visual explanation like I try to think. They say it is possible because mathematically we can prove that there is a linearity of matrix multiplication that we can find identical mathematical terms that we can identify.
So, let's do this. It is simple. It is real simple. But it is important that we all understand it. Here in the activation space, we have the classical view. No, let W be the frozen weights here of an MLP layer in a transformer. Let X be your human query here or the embedding here of the last token. And let A be here the output of the attention layer. Attention and A. This is more or less the complete context information. Then a standard transformer. And now let's ignore the nonlinearities here for a second. I'll come back to the nonlinearities in a minute. The input to the MLP is the sum of the query and the attention output. No, via the residual stream, as I showed my you in my last video. So, if you, this is simple. So, we intuitively think now, "Okay, the input X is now shifted here by a particular vector A." Beautiful. And this is it. So, this is our activation space view. So, the term W * A is now a vector representing here the correction, or if you want, the knowledge retrieved now from the added context to our ICL prompt. Now, the authors ask more or less, "Hey, can we find now a matrix W_prime such that this correction comes now from the weights instead from the context input?" So, mathematically, it is so easy. The authors tell us, "We just want to satisfy this equation." Huh? So, we have here the, if you want, the activation space view, and then we have here the weight space view, and we just want to make that this is, can we have this is the same? Yeah, this is it. No, W * A is exactly our delta W * X. X is here the human query, if you want, and A is here the attention here of this additional context, and W is our weight, and delta W is our additional weight structure. So, to solve here for W_prime, we need a matrix that when multiplied with X produces here this particular vector W * A. Uh, distributive property. So, now they tell us mathematically, the authors tell us, "We can pretend that the correction term W * A did not come from the extra vector A, but instead came from an extra weight acting on the original vector X."
So, now we understand why they claim this because they build a mathematical framework where more or less they say, "In the end of all of our calculations, we arrive at this. We arrive here at this term and at this term from two different mathematical spaces, and we just ask, 'Hey, is it possible that our W * A becomes becomes equivalent to our delta W * X?' Can we find this?" And then this is here, if you want, a correction term that is now coming not from the content from the attention, but now we can go and operate with virtual weight structures. So, we have now this bridge suddenly connecting fine-tuning and in-context learning. Now, of course, as I showed you, you can do this here with a rank-one solution. No, if you go with an outer product, simple. No. So, mathematically, we can say adding now the attention output A to the input is identical to updating the weight matrix W by adding here a rank-one matrix delta. So, the activation pattern A from the attention layer becomes the source of the weight update delta W in ICL. So, we have now this beautiful connection between fine-tuning and ICL in a coherent mathematical theory. And now you understand why the authors can claim this identity because mathematically speaking, it is identical. But in physics, you know, whenever we see that we sort that two properties of a physical system are different, and then analyzing it mathematically, we see they are identical, we understand that we made a mistake in building this physical system because either we have neglected symmetries or we have neglected identity operation, but we built something that is not, let's call it, perfect. We built something that works, but we don't really understand why. And now, with this mathematical identity, I have a feeling that what we think is the difference between in-context learning and fine-tuning, what we coded, it turns out is indeed the identical thing, more or less. So maybe we built our transformer architecture just in a wrong way. We didn't utilize that those terms are identical.
Now, you know, we operated here with a simplified kernel representation. And the beauty of the second preprint that I showed you here is the second order here from December 12, that they took this much further, and they prove mathematically that it works here also in a realistic nonlinear deep learning scenarios. So, the second paper is just taking the results of the first paper and proving it mathematically in nonlinear deep learning. But we stay on the same complexity that we try to understand and we try to solve. So, yeah, no, the skeptics, and this is me, I have said at the beginning, no, but you have the layer norm, you have nonlinear activation, you have skip connections. No, and this is why I did not take here this publication by Google here from July 2025 really that significant because I thought, "Hey, the simple algebra doesn't hold if you input nonlinear activations." No, but this team of now proved here just two weeks ago that it holds. And this is something, okay. But they have a second insight: that the update delta W becomes specific to the token position. So, we calculate, given the nonlinearity, a delta W for each single token position. Now, this thing becomes a little bit more interesting. No. So, the weights are effectively different for every single token that is generated by our autoregressive system. The model isn't just fine-tuned once. It is now with ICL continuously refining itself at every step of the generation of the next token prediction of an autoregressive transformer architecture. Therefore, we can now stop treating here the learning, and I mean here the backpropagation, the learning process on the neural network, and the inference, the forward pass, as it would be two different things. It turns out, no. If you understand the mathematics of this, we see the forward pass is a learning step in itself, and this is mathematically equivalent here to the backpropagation. So, somehow there is a chance we can further optimize the architectures of the transformers. We can make it maybe smaller. We can make it more powerful. We have not yet optimized the architecture.
So, if I go here with my image, this is just in my head, whatever image you have. I have here, you know, my tensor structure here, my weight matrix here in, I don't know, 4,000 dimensions. And then I say, "Okay, and then this should be equivalent here to ICL that comes in here, the context, the context activation on the frozen weights." And now I have to understand that these two images are identical. So, this golden context activation are more or less just a rank-one patch that I apply here to my frozen weight tensor structure. So, ICL is mathematically proven equivalent to a rank-one update on the tensor structure. Okay, let's go with this. Okay, now I have further questions. So, you see here how I fight with these things sometimes. And say, "But wait, the activation of our neurons in the layers of the transformers is not a weight tensor update." No, an activation is something completely different. This is the vector that flows through the layers. The weight tensors. If you modify a weight tensor, how is this identical? And the answer is given by the paper. It is possible because they prove mathematically, not in a visual explanation like I try to think. They say it is possible because mathematically we can prove that there is a linearity of matrix multiplication that we can find identical mathematical terms that we can identify.
So, let's do this. It is simple. It is real simple. But it is important that we all understand it. Here in the activation space, we have the classical view. No, let W be the frozen weights here of an MLP layer in a transformer. Let X be your human query here or the embedding here of the last token. And let A be here the output of the attention layer. Attention and A. This is more or less the complete context information. Then a standard transformer. And now let's ignore the nonlinearities here for a second. I'll come back to the nonlinearities in a minute. The input to the MLP is the sum of the query and the attention output. No, via the residual stream, as I showed my you in my last video. So, if you, this is simple. So, we intuitively think now, "Okay, the input X is now shifted here by a particular vector A." Beautiful. And this is it. So, this is our activation space view. So, the term W * A is now a vector representing here the correction, or if you want, the knowledge retrieved now from the added context to our ICL prompt. Now, the authors ask more or less, "Hey, can we find now a matrix W_prime such that this correction comes now from the weights instead from the context input?" So, mathematically, it is so easy. The authors tell us, "We just want to satisfy this equation." Huh? So, we have here the, if you want, the activation space view, and then we have here the weight space view, and we just want to make that this is, can we have this is the same? Yeah, this is it. No, W * A is exactly our delta W * X. X is here the human query, if you want, and A is here the attention here of this additional context, and W is our weight, and delta W is our additional weight structure. So, to solve here for W_prime, we need a matrix that when multiplied with X produces here this particular vector W * A. Uh, distributive property. So, now they tell us mathematically, the authors tell us, "We can pretend that the correction term W * A did not come from the extra vector A, but instead came from an extra weight acting on the original vector X."
So, now we understand why they claim this because they build a mathematical framework where more or less they say, "In the end of all of our calculations, we arrive at this. We arrive here at this term and at this term from two different mathematical spaces, and we just ask, 'Hey, is it possible that our W * A becomes becomes equivalent to our delta W * X?' Can we find this?" And then this is here, if you want, a correction term that is now coming not from the content from the attention, but now we can go and operate with virtual weight structures. So, we have now this bridge suddenly connecting fine-tuning and in-context learning. Now, of course, as I showed you, you can do this here with a rank-one solution. No, if you go with an outer product, simple. No. So, mathematically, we can say adding now the attention output A to the input is identical to updating the weight matrix W by adding here a rank-one matrix delta. So, the activation pattern A from the attention layer becomes the source of the weight update delta W in ICL. So, we have now this beautiful connection between fine-tuning and ICL in a coherent mathematical theory. And now you understand why the authors can claim this identity because mathematically speaking, it is identical. But in physics, you know, whenever we see that we sort that two properties of a physical system are different, and then analyzing it mathematically, we see they are identical, we understand that we made a mistake in building this physical system because either we have neglected symmetries or we have neglected identity operation, but we built something that is not, let's call it, perfect. We built something that works, but we don't really understand why. And now, with this mathematical identity, I have a feeling that what we think is the difference between in-context learning and fine-tuning, what we coded, it turns out is indeed the identical thing, more or less. So maybe we built our transformer architecture just in a wrong way. We didn't utilize that those terms are identical.
Now, you know, we operated here with a simplified kernel representation. And the beauty of the second preprint that I showed you here is the second order here from December 12, that they took this much further, and they prove mathematically that it works here also in a realistic nonlinear deep learning scenarios. So, the second paper is just taking the results of the first paper and proving it mathematically in nonlinear deep learning. But we stay on the same complexity that we try to understand and we try to solve. So, yeah, no, the skeptics, and this is me, I have said at the beginning, no, but you have the layer norm, you have nonlinear activation, you have skip connections. No, and this is why I did not take here this publication by Google here from July 2025 really that significant because I thought, "Hey, the simple algebra doesn't hold if you input nonlinear activations." No, but this team of now proved here just two weeks ago that it holds. And this is something, okay. But they have a second insight: that the update delta W becomes specific to the token position. So, we calculate, given the nonlinearity, a delta W for each single token position. Now, this thing becomes a little bit more interesting. No. So, the weights are effectively different for every single token that is generated by our autoregressive system. The model isn't just fine-tuned once. It is now with ICL continuously refining itself at every step of the generation of the next token prediction of an autoregressive transformer architecture. Therefore, we can now stop treating here the learning, and I mean here the backpropagation, the learning process on the neural network, and the inference, the forward pass, as it would be two different things. It turns out, no. If you understand the mathematics of this, we see the forward pass is a learning step in itself, and this is mathematically equivalent here to the backpropagation. So, somehow there is a chance we can further optimize the architectures of the transformers. We can make it maybe smaller. We can make it more powerful. We have not yet optimized the architecture.
So, if I go here with my image, this is just in my head, whatever image you have. I have here, you know, my tensor structure here, my weight matrix here in, I don't know, 4,000 dimensions. And then I say, "Okay, and then this should be equivalent here to ICL that comes in here, the context, the context activation on the frozen weights." And now I have to understand that these two images are identical. So, this golden context activation are more or less just a rank-one patch that I apply here to my frozen weight tensor structure. So, ICL is mathematically proven equivalent to a rank-one update on the tensor structure. Okay, let's go with this. Okay, now I have further questions. So, you see here how I fight with these things sometimes. And say, "But wait, the activation of our neurons in the layers of the transformers is not a weight tensor update." No, an activation is something completely different. This is the vector that flows through the layers. The weight tensors. If you modify a weight tensor, how is this identical? And the answer is given by the paper. It is possible because they prove mathematically, not in a visual explanation like I try to think. They say it is possible because mathematically we can prove that there is a linearity of matrix multiplication that we can find identical mathematical terms that we can identify.
So, let's do this. It is simple. It is real simple. But it is important that we all understand it. Here in the activation space, we have the classical view. No, let W be the frozen weights here of an MLP layer in a transformer. Let X be your human query here or the embedding here of the last token. And let A be here the output of the attention layer. Attention and A. This is more or less the complete context information. Then a standard transformer. And now let's ignore the nonlinearities here for a second. I'll come back to the nonlinearities in a minute. The input to the MLP is the sum of the query and the attention output. No, via the residual stream, as I showed my you in my last video. So, if you, this is simple. So, we intuitively think now, "Okay, the input X is now shifted here by a particular vector A." Beautiful. And this is it. So, this is our activation space view. So, the term W * A is now a vector representing here the correction, or if you want, the knowledge retrieved now from the added context to our ICL prompt. Now, the authors ask more or less, "Hey, can we find now a matrix W_prime such that this correction comes now from the weights instead from the context input?" So, mathematically, it is so easy. The authors tell us, "We just want to satisfy this equation." Huh? So, we have here the, if you want, the activation space view, and then we have here the weight space view, and we just want to make that this is, can we have this is the same? Yeah, this is it. No, W * A is exactly our delta W * X. X is here the human query, if you want, and A is here the attention here of this additional context, and W is our weight, and delta W is our additional weight structure. So, to solve here for W_prime, we need a matrix that when multiplied with X produces here this particular vector W * A. Uh, distributive property. So, now they tell us mathematically, the authors tell us, "We can pretend that the correction term W * A did not come from the extra vector A, but instead came from an extra weight acting on the original vector X."
So, now we understand why they claim this because they build a mathematical framework where more or less they say, "In the end of all of our calculations, we arrive at this. We arrive here at this term and at this term from two different mathematical spaces, and we just ask, 'Hey, is it possible that our W * A becomes becomes equivalent to our delta W * X?' Can we find this?" And then this is here, if you want, a correction term that is now coming not from the content from the attention, but now we can go and operate with virtual weight structures. So, we have now this bridge suddenly connecting fine-tuning and in-context learning. Now, of course, as I showed you, you can do this here with a rank-one solution. No, if you go with an outer product, simple. No. So, mathematically, we can say adding now the attention output A to the input is identical to updating the weight matrix W by adding here a rank-one matrix delta. So, the activation pattern A from the attention layer becomes the source of the weight update delta W in ICL. So, we have now this beautiful connection between fine-tuning and ICL in a coherent mathematical theory. And now you understand why the authors can claim this identity because mathematically speaking, it is identical. But in physics, you know, whenever we see that we sort that two properties of a physical system are different, and then analyzing it mathematically, we see they are identical, we understand that we made a mistake in building this physical system because either we have neglected symmetries or we have neglected identity operation, but we built something that is not, let's call it, perfect. We built something that works, but we don't really understand why. And now, with this mathematical identity, I have a feeling that what we think is the difference between in-context learning and fine-tuning, what we coded, it turns out is indeed the identical thing, more or less. So maybe we built our transformer architecture just in a wrong way. We didn't utilize that those terms are identical.
Now, you know, we operated here with a simplified kernel representation. And the beauty of the second preprint that I showed you here is the second order here from December 12, that they took this much further, and they prove mathematically that it works here also in a realistic nonlinear deep learning scenarios. So, the second paper is just taking the results of the first paper and proving it mathematically in nonlinear deep learning. But we stay on the same complexity that we try to understand and we try to solve. So, yeah, no, the skeptics, and this is me, I have said at the beginning, no, but you have the layer norm, you have nonlinear activation, you have skip connections. No, and this is why I did not take here this publication by Google here from July 2025 really that significant because I thought, "Hey, the simple algebra doesn't hold if you input nonlinear activations." No, but this team of now proved here just two weeks ago that it holds. And this is something, okay. But they have a second insight: that the update delta W becomes specific to the token position. So, we calculate, given the nonlinearity, a delta W for each single token position. Now, this thing becomes a little bit more interesting. No. So, the weights are effectively different for every single token that is generated by our autoregressive system. The model isn't just fine-tuned once. It is now with ICL continuously refining itself at every step of the generation of the next token prediction of an autoregressive transformer architecture. Therefore, we can now stop treating here the learning, and I mean here the backpropagation, the learning process on the neural network, and the inference, the forward pass, as it would be two different things. It turns out, no. If you understand the mathematics of this, we see the forward pass is a learning step in itself, and this is mathematically equivalent here to the backpropagation. So, somehow there is a chance we can further optimize the architectures of the transformers. We can make it maybe smaller. We can make it more powerful. We have not yet optimized the architecture.
So, if I go here with my image, this is just in my head, whatever image you have. I have here, you know, my tensor structure here, my weight matrix here in, I don't know, 4,000 dimensions. And then I say, "Okay, and then this should be equivalent here to ICL that comes in here, the context, the context activation on the frozen weights." And now I have to understand that these two images are identical. So, this golden context activation are more or less just a rank-one patch that I apply here to my frozen weight tensor structure. So, ICL is mathematically proven equivalent to a rank-one update on the tensor structure. Okay, let's go with this. Okay, now I have further questions. So, you see here how I fight with these things sometimes. And say, "But wait, the activation of our neurons in the layers of the transformers is not a weight tensor update." No, an activation is something completely different. This is the vector that flows through the layers. The weight tensors. If you modify a weight tensor, how is this identical? And the answer is given by the paper. It is possible because they prove mathematically, not in a visual explanation like I try to think. They say it is possible because mathematically we can prove that there is a linearity of matrix multiplication that we can find identical mathematical terms that we can identify.
So, let's do this. It is simple. It is real simple. But it is important that we all understand it. Here in the activation space, we have the classical view. No, let W be the frozen weights here of an MLP layer in a transformer. Let X be your human query here or the embedding here of the last token. And let A be here the output of the attention layer. Attention and A. This is more or less the complete context information. Then a standard transformer. And now let's ignore the nonlinearities here for a second. I'll come back to the nonlinearities in a minute. The input to the MLP is the sum of the query and the attention output. No, via the residual stream, as I showed my you in my last video. So, if you, this is simple. So, we intuitively think now, "Okay, the input X is now shifted here by a particular vector A." Beautiful. And this is it. So, this is our activation space view. So, the term W * A is now a vector representing here the correction, or if you want, the knowledge retrieved now from the added context to our ICL prompt. Now, the authors ask more or less, "Hey, can we find now a matrix W_prime such that this correction comes now from the weights instead from the context input?" So, mathematically, it is so easy. The authors tell us, "We just want to satisfy this equation." Huh? So, we have here the, if you want, the activation space view, and then we have here the weight space view, and we just want to make that this is, can we have this is the same? Yeah, this is it. No, W * A is exactly our delta W * X. X is here the human query, if you want, and A is here the attention here of this additional context, and W is our weight, and delta W is our additional weight structure. So, to solve here for W_prime, we need a matrix that when multiplied with X produces here this particular vector W * A. Uh, distributive property. So, now they tell us mathematically, the authors tell us, "We can pretend that the correction term W * A did not come from the extra vector A, but instead came from an extra weight acting on the original vector X."
So, now we understand why they claim this because they build a mathematical framework where more or less they say, "In the end of all of our calculations, we arrive at this. We arrive here at this term and at this term from two different mathematical spaces, and we just ask, 'Hey, is it possible that our W * A becomes becomes equivalent to our delta W * X?' Can we find this?" And then this is here, if you want, a correction term that is now coming not from the content from the attention, but now we can go and operate with virtual weight structures. So, we have now this bridge suddenly connecting fine-tuning and in-context learning. Now, of course, as I showed you, you can do this here with a rank-one solution. No, if you go with an outer product, simple. No. So, mathematically, we can say adding now the attention output A to the input is identical to updating the weight matrix W by adding here a rank-one matrix delta. So, the activation pattern A from the attention layer becomes the source of the weight update delta W in ICL. So, we have now this beautiful connection between fine-tuning and ICL in a coherent mathematical theory. And now you understand why the authors can claim this identity because mathematically speaking, it is identical. But in physics, you know, whenever we see that we sort that two properties of a physical system are different, and then analyzing it mathematically, we see they are identical, we understand that we made a mistake in building this physical system because either we have neglected symmetries or we have neglected identity operation, but we built something that is not, let's call it, perfect. We built something that works, but we don't really understand why. And now, with this mathematical identity, I have a feeling that what we think is the difference between in-context learning and fine-tuning, what we coded, it turns out is indeed the identical thing, more or less. So maybe we built our transformer architecture just in a wrong way. We didn't utilize that those terms are identical.
Now, you know, we operated here with a simplified kernel representation. And the beauty of the second preprint that I showed you here is the second order here from December 12, that they took this much further, and they prove mathematically that it works here also in a realistic nonlinear deep learning scenarios. So, the second paper is just taking the results of the first paper and proving it mathematically in nonlinear deep learning. But we stay on the same complexity that we try to understand and we try to solve. So, yeah, no, the skeptics, and this is me, I have said at the beginning, no, but you have the layer norm, you have nonlinear activation, you have skip connections. No, and this is why I did not take here this publication by Google here from July 2025 really that significant because I thought, "Hey, the simple algebra doesn't hold if you input nonlinear activations." No, but this team of now proved here just two weeks ago that it holds. And this is something, okay. But they have a second insight: that the update delta W becomes specific to the token position. So, we calculate, given the nonlinearity, a delta W for each single token position. Now, this thing becomes a little bit more interesting. No. So, the weights are effectively different for every single token that is generated by our autoregressive system. The model isn't just fine-tuned once. It is now with ICL continuously refining itself at every step of the generation of the next token prediction of an autoregressive transformer architecture. Therefore, we can now stop treating here the learning, and I mean here the backpropagation, the learning process on the neural network, and the inference, the forward pass, as it would be two different things. It turns out, no. If you understand the mathematics of this, we see the forward pass is a learning step in itself, and this is mathematically equivalent here to the backpropagation. So, somehow there is a chance we can further optimize the architectures of the transformers. We can make it maybe smaller. We can make it more powerful. We have not yet optimized the architecture.
So, if I go here with my image, this is just in my head, whatever image you have. I have here, you know, my tensor structure here, my weight matrix here in, I don't know, 4,000 dimensions. And then I say, "Okay, and then this should be equivalent here to ICL that comes in here, the context, the context activation on the frozen weights." And now I have to understand that these two images are identical. So, this golden context activation are more or less just a rank-one patch that I apply here to my frozen weight tensor structure. So, ICL is mathematically proven equivalent to a rank-one update on the tensor structure. Okay, let's go with this. Okay, now I have further questions. So, you see here how I fight with these things sometimes. And say, "But wait, the activation of our neurons in the layers of the transformers is not a weight tensor update." No, an activation is something completely different. This is the vector that flows through the layers. The weight tensors. If you modify a weight tensor, how is this identical? And the answer is given by the paper. It is possible because they prove mathematically, not in a visual explanation like I try to think. They say it is possible because mathematically we can prove that there is a linearity of matrix multiplication that we can find identical mathematical terms that we can identify.
So, let's do this. It is simple. It is real simple. But it is important that we all understand it. Here in the activation space, we have the classical view. No, let W be the frozen weights here of an MLP layer in a transformer. Let X be your human query here or the embedding here of the last token. And let A be here the output of the attention layer. Attention and A. This is more or less the complete context information. Then a standard transformer. And now let's ignore the nonlinearities here for a second. I'll come back to the nonlinearities in a minute. The input to the MLP is the sum of the query and the attention output. No, via the residual stream, as I showed my you in my last video. So, if you, this is simple. So, we intuitively think now, "Okay, the input X is now shifted here by a particular vector A." Beautiful. And this is it. So, this is our activation space view. So, the term W * A is now a vector representing here the correction, or if you want, the knowledge retrieved now from the added context to our ICL prompt. Now, the authors ask more or less, "Hey, can we find now a matrix W_prime such that this correction comes now from the weights instead from the context input?" So, mathematically, it is so easy. The authors tell us, "We just want to satisfy this equation." Huh? So, we have here the, if you want, the activation space view, and then we have here the weight space view, and we just want to make that this is, can we have this is the same? Yeah, this is it. No, W * A is exactly our delta W * X. X is here the human query, if you want, and A is here the attention here of this additional context, and W is our weight, and delta W is our additional weight structure. So, to solve here for W_prime, we need a matrix that when multiplied with X produces here this particular vector W * A. Uh, distributive property. So, now they tell us mathematically, the authors tell us, "We can pretend that the correction term W * A did not come from the extra vector A, but instead came from an extra weight acting on the original vector X."
So, now we understand why they claim this because they build a mathematical framework where more or less they say, "In the end of all of our calculations, we arrive at this. We arrive here at this term and at this term from two different mathematical spaces, and we just ask, 'Hey, is it possible that our W * A becomes becomes equivalent to our delta W * X?' Can we find this?" And then this is here, if you want, a correction term that is now coming not from the content from the attention, but now we can go and operate with virtual weight structures. So, we have now this bridge suddenly connecting fine-tuning and in-context learning. Now, of course, as I showed you, you can do this here with a rank-one solution. No, if you go with an outer product, simple. No. So, mathematically, we can say adding now the attention output A to the input is identical to updating the weight matrix W by adding here a rank-one matrix delta. So, the activation pattern A from the attention layer becomes the source of the weight update delta W in ICL. So, we have now this beautiful connection between fine-tuning and ICL in a coherent mathematical theory. And now you understand why the authors can claim this identity because mathematically speaking, it is identical. But in physics, you know, whenever we see that we sort that two properties of a physical system are different, and then analyzing it mathematically, we see they are identical, we understand that we made a mistake in building this physical system because either we have neglected symmetries or we have neglected identity operation, but we built something that is not, let's call it, perfect. We built something that works, but we don't really understand why. And now, with this mathematical identity, I have a feeling that what we think is the difference between in-context learning and fine-tuning, what we coded, it turns out is indeed the identical thing, more or less. So maybe we built our transformer architecture just in a wrong way. We didn't utilize that those terms are identical.
Now, you know, we operated here with a simplified kernel representation. And the beauty of the second preprint that I showed you here is the second order here from December 12, that they took this much further, and they prove mathematically that it works here also in a realistic nonlinear deep learning scenarios. So, the second paper is just taking the results of the first paper and proving it mathematically in nonlinear deep learning. But we stay on the same complexity that we try to understand and we try to solve. So, yeah, no, the skeptics, and this is me, I have said at the beginning, no, but you have the layer norm, you have nonlinear activation, you have skip connections. No, and this is why I did not take here this publication by Google here from July 2025 really that significant because I thought, "Hey, the simple algebra doesn't hold if you input nonlinear activations." No, but this team of now proved here just two weeks ago that it holds. And this is something, okay. But they have a second insight: that the update delta W becomes specific to the token position. So, we calculate, given the nonlinearity, a delta W for each single token position. Now, this thing becomes a little bit more interesting. No. So, the weights are effectively different for every single token that is generated by our autoregressive system. The model isn't just fine-tuned once. It is now with ICL continuously refining itself at every step of the generation of the next token prediction of an autoregressive transformer architecture. Therefore, we can now stop treating here the learning, and I mean here the backpropagation, the learning process on the neural network, and the inference, the forward pass, as it would be two different things. It turns out, no. If you understand the mathematics of this, we see the forward pass is a learning step in itself, and this is mathematically equivalent here to the backpropagation. So, somehow there is a chance we can further optimize the architectures of the transformers. We can make it maybe smaller. We can make it more powerful. We have not yet optimized the architecture.
So, if I go here with my image, this is just in my head, whatever image you have. I have here, you know, my tensor structure here, my weight matrix here in, I don't know, 4,000 dimensions. And then I say, "Okay, and then this should be equivalent here to ICL that comes in here, the context, the context activation on the frozen weights." And now I have to understand that these two images are identical. So, this golden context activation are more or less just a rank-one patch that I apply here to my frozen weight tensor structure. So, ICL is mathematically proven equivalent to a rank-one update on the tensor structure. Okay, let's go with this. Okay, now I have further questions. So, you see here how I fight with these things sometimes. And say, "But wait, the activation of our neurons in the layers of the transformers is not a weight tensor update." No, an activation is something completely different. This is the vector that flows through the layers. The weight tensors. If you modify a weight tensor, how is this identical? And the answer is given by the paper. It is possible because they prove mathematically, not in a visual explanation like I try to think. They say it is possible because mathematically we can prove that there is a linearity of matrix multiplication that we can find identical mathematical terms that we can identify.
So, let's do this. It is simple. It is real simple. But it is important that we all understand it. Here in the activation space, we have the classical view. No, let W be the frozen weights here of an MLP layer in a transformer. Let X be your human query here or the embedding here of the last token. And let A be here the output of the attention layer. Attention and A. This is more or less the complete context information. Then a standard transformer. And now let's ignore the nonlinearities here for a second. I'll come back to the nonlinearities in a minute. The input to the MLP is the sum of the query and the attention output. No, via the residual stream, as I showed my you in my last video. So, if you, this is simple. So, we intuitively think now, "Okay, the input X is now shifted here by a particular vector A." Beautiful. And this is it. So, this is our activation space view. So, the term W * A is now a vector representing here the correction, or if you want, the knowledge retrieved now from the added context to our ICL prompt. Now, the authors ask more or less, "Hey, can we find now a matrix W_prime such that this correction comes now from the weights instead from the context input?" So, mathematically, it is so easy. The authors tell us, "We just want to satisfy this equation." Huh? So, we have here the, if you want, the activation space view, and then we have here the weight space view, and we just want to make that this is, can we have this is the same? Yeah, this is it. No, W * A is exactly our delta W * X. X is here the human query, if you want, and A is here the attention here of this additional context, and W is our weight, and delta W is our additional weight structure. So, to solve here for W_prime, we need a matrix that when multiplied with X produces here this particular vector W * A. Uh, distributive property. So, now they tell us mathematically, the authors tell us, "We can pretend that the correction term W * A did not come from the extra vector A, but instead came from an extra weight acting on the original vector X."
So, now we understand why they claim this because they build a mathematical framework where more or less they say, "In the end of all of our calculations, we arrive at this. We arrive here at this term and at this term from two different mathematical spaces, and we just ask, 'Hey, is it possible that our W * A becomes becomes equivalent to our delta W * X?' Can we find this?" And then this is here, if you want, a correction term that is now coming not from the content from the attention, but now we can go and operate with virtual weight structures. So, we have now this bridge suddenly connecting fine-tuning and in-context learning. Now, of course, as I showed you, you can do this here with a rank-one solution. No, if you go with an outer product, simple. No. So, mathematically, we can say adding now the attention output A to the input is identical to updating the weight matrix W by adding here a rank-one matrix delta. So, the activation pattern A from the attention layer becomes the source of the weight update delta W in ICL. So, we have now this beautiful connection between fine-tuning and ICL in a coherent mathematical theory. And now you understand why the authors can claim this identity because mathematically speaking, it is identical. But in physics, you know, whenever we see that we sort that two properties of a physical system are different, and then analyzing it mathematically, we see they are identical, we understand that we made a mistake in building this physical system because either we have neglected symmetries or we have neglected identity operation, but we built something that is not, let's call it, perfect. We built something that works, but we don't really understand why. And now, with this mathematical identity, I have a feeling that what we think is the difference between in-context learning and fine-tuning, what we coded, it turns out is indeed the identical thing, more or less. So maybe we built our transformer architecture just in a wrong way. We didn't utilize that those terms are identical.
Now, you know, we operated here with a simplified kernel representation. And the beauty of the second preprint that I showed you here is the second order here from December 12, that they took this much further, and they prove mathematically that it works here also in a realistic nonlinear deep learning scenarios. So, the second paper is just taking the results of the first paper and proving it mathematically in nonlinear deep learning. But we stay on the same complexity that we try to understand and we try to solve. So, yeah, no, the skeptics, and this is me, I have said at the beginning, no, but you have the layer norm, you have nonlinear activation, you have skip connections. No, and this is why I did not take here this publication by Google here from July 2025 really that significant because I thought, "Hey, the simple algebra doesn't hold if you input nonlinear activations." No, but this team of now proved here just two weeks ago that it holds. And this is something, okay. But they have a second insight: that the update delta W becomes specific to the token position. So, we calculate, given the nonlinearity, a delta W for each single token position. Now, this thing becomes a little bit more interesting. No. So, the weights are effectively different for every single token that is generated by our autoregressive system. The model isn't just fine-tuned once. It is now with ICL continuously refining itself at every step of the generation of the next token prediction of an autoregressive transformer architecture. Therefore, we can now stop treating here the learning, and I mean here the backpropagation, the learning process on the neural network, and the inference, the forward pass, as it would be two different things. It turns out, no. If you understand the mathematics of this, we see the forward pass is a learning step in itself, and this is mathematically equivalent here to the backpropagation. So, somehow there is a chance we can further optimize the architectures of the transformers. We can make it maybe smaller. We can make it more powerful. We have not yet optimized the architecture.
So, if I go here with my image, this is just in my head, whatever image you have. I have here, you know, my tensor structure here, my weight matrix here in, I don't know, 4,000 dimensions. And then I say, "Okay, and then this should be equivalent here to ICL that comes in here, the context, the context activation on the frozen weights." And now I have to understand that these two images are identical. So, this golden context activation are more or less just a rank-one patch that I apply here to my frozen weight tensor structure. So, ICL is mathematically proven equivalent to a rank-one update on the tensor structure. Okay, let's go with this. Okay, now I have further questions. So, you see here how I fight with these things sometimes. And say, "But wait, the activation of our neurons in the layers of the transformers is not a weight tensor update." No, an activation is something completely different. This is the vector that flows through the layers. The weight tensors. If you modify a weight tensor, how is this identical? And the answer is given by the paper. It is possible because they prove mathematically, not in a visual explanation like I try to think. They say it is possible because mathematically we can prove that there is a linearity of matrix multiplication that we can find identical mathematical terms that we can identify.
So, let's do this. It is simple. It is real simple. But it is important that we all understand it. Here in the activation space, we have the classical view. No, let W be the frozen weights here of an MLP layer in a transformer. Let X be your human query here or the embedding here of the last token. And let A be here the output of the attention layer. Attention and A. This is more or less the complete context information. Then a standard transformer. And now let's ignore the nonlinearities here for a second. I'll come back to the nonlinearities in a minute. The input to the MLP is the sum of the query and the attention output. No, via the residual stream, as I showed my you in my last video. So, if you, this is simple. So, we intuitively think now, "Okay, the input X is now shifted here by a particular vector A." Beautiful. And this is it. So, this is our activation space view. So, the term W * A is now a vector representing here the correction, or if you want, the knowledge retrieved now from the added context to our ICL prompt. Now, the authors ask more or less, "Hey, can we find now a matrix W_prime such that this correction comes now from the weights instead from the context input?" So, mathematically, it is so easy. The authors tell us, "We just want to satisfy this equation." Huh? So, we have here the, if you want, the activation space view, and then we have here the weight space view, and we just want to make that this is, can we have this is the same? Yeah, this is it. No, W * A is exactly our delta W * X. X is here the human query, if you want, and A is here the attention here of this additional context, and W is our weight, and delta W is our additional weight structure. So, to solve here for W_prime, we need a matrix that when multiplied with X produces here this particular vector W * A. Uh, distributive property. So, now they tell us mathematically, the authors tell us, "We can pretend that the correction term W * A did not come from the extra vector A, but instead came from an extra weight acting on the original vector X."
So, now we understand why they claim this because they build a mathematical framework where more or less they say, "In the end of all of our calculations, we arrive at this. We arrive here at this term and at this term from two different mathematical spaces, and we just ask, 'Hey, is it possible that our W * A becomes becomes equivalent to our delta W * X?' Can we find this?" And then this is here, if you want, a correction term that is now coming not from the content from the attention, but now we can go and operate with virtual weight structures. So, we have now this bridge suddenly connecting fine-tuning and in-context learning. Now, of course, as I showed you, you can do this here with a rank-one solution. No, if you go with an outer product, simple. No. So, mathematically, we can say adding now the attention output A to the input is identical to updating the weight matrix W by adding here a rank-one matrix delta. So, the activation pattern A from the attention layer becomes the source of the weight update delta W in ICL. So, we have now this beautiful connection between fine-tuning and ICL in a coherent mathematical theory. And now you understand why the authors can claim this identity because mathematically speaking, it is identical. But in physics, you know, whenever we see that we sort that two properties of a physical system are different, and then analyzing it mathematically, we see they are identical, we understand that we made a mistake in building this physical system because either we have neglected symmetries or we have neglected identity operation, but we built something that is not, let's call it, perfect. We built something that works, but we don't really understand why. And now, with this mathematical identity, I have a feeling that what we think is the difference between in-context learning and fine-tuning, what we coded, it turns out is indeed the identical thing, more or less. So maybe we built our transformer architecture just in a wrong way. We didn't utilize that those terms are identical.
Now, you know, we operated here with a simplified kernel representation. And the beauty of the second preprint that I showed you here is the second order here from December 12, that they took this much further, and they prove mathematically that it works here also in a realistic nonlinear deep learning scenarios. So, the second paper is just taking the results of the first paper and proving it mathematically in nonlinear deep learning. But we stay on the same complexity that we try to understand and we try to solve. So, yeah, no, the skeptics, and this is me, I have said at the beginning, no, but you have the layer norm, you have nonlinear activation, you have skip connections. No, and this is why I did not take here this publication by Google here from July 2025 really that significant because I thought, "Hey, the simple algebra doesn't hold if you input nonlinear activations." No, but this team of now proved here just two weeks ago that it holds. And this is something, okay. But they have a second insight: that the update delta W becomes specific to the token position. So, we calculate, given the nonlinearity, a delta W for each single token position. Now, this thing becomes a little bit more interesting. No. So, the weights are effectively different for every single token that is generated by our autoregressive system. The model isn't just fine-tuned once. It is now with ICL continuously refining itself at every step of the generation of the next token prediction of an autoregressive transformer architecture. Therefore, we can now stop treating here the learning, and I mean here the backpropagation, the learning process on the neural network, and the inference, the forward pass, as it would be two different things. It turns out, no. If you understand the mathematics of this, we see the forward pass is a learning step in itself, and this is mathematically equivalent here to the backpropagation. So, somehow there is a chance we can further optimize the architectures of the transformers. We can make it maybe smaller. We can make it more powerful. We have not yet optimized the architecture.
So, if I go here with my image, this is just in my head, whatever image you have. I have here, you know, my tensor structure here, my weight matrix here in, I don't know, 4,000 dimensions. And then I say, "Okay, and then this should be equivalent here to ICL that comes in here, the context, the context activation on the frozen weights." And now I have to understand that these two images are identical. So, this golden context activation are more or less just a rank-one patch that I apply here to my frozen weight tensor structure. So, ICL is mathematically proven equivalent to a rank-one update on the tensor structure. Okay, let's go with this. Okay, now I have further questions. So, you see here how I fight with these things sometimes. And say, "But wait, the activation of our neurons in the layers of the transformers is not a weight tensor update." No, an activation is something completely different. This is the vector that flows through the layers. The weight tensors. If you modify a weight tensor, how is this identical? And the answer is given by the paper. It is possible because they prove mathematically, not in a visual explanation like I try to think. They say it is possible because mathematically we can prove that there is a linearity of matrix multiplication that we can find identical mathematical terms that we can identify.
So, let's do this. It is simple. It is real simple. But it is important that we all understand it. Here in the activation space, we have the classical view. No, let W be the frozen weights here of an MLP layer in a transformer. Let X be your human query here or the embedding here of the last token. And let A be here the output of the attention layer. Attention and A. This is more or less the complete context information. Then a standard transformer. And now let's ignore the nonlinearities here for a second. I'll come back to the nonlinearities in a minute. The input to the MLP is the sum of the query and the attention output. No, via the residual stream, as I showed my you in my last video. So, if you, this is simple. So, we intuitively think now, "Okay, the input X is now shifted here by a particular vector A." Beautiful. And this is it. So, this is our activation space view. So, the term W * A is now a vector representing here the correction, or if you want, the knowledge retrieved now from the added context to our ICL prompt. Now, the authors ask more or less, "Hey, can we find now a matrix W_prime such that this correction comes now from the weights instead from the context input?" So, mathematically, it is so easy. The authors tell us, "We just want to satisfy this equation." Huh? So, we have here the, if you want, the activation space view, and then we have here the weight space view, and we just want to make that this is, can we have this is the same? Yeah, this is it. No, W * A is exactly our delta W * X. X is here the human query, if you want, and A is here the attention here of this additional context, and W is our weight, and delta W is our additional weight structure. So, to solve here for W_prime, we need a matrix that when multiplied with X produces here this particular vector W * A. Uh, distributive property. So, now they tell us mathematically, the authors tell us, "We can pretend that the correction term W * A did not come from the extra vector A, but instead came from an extra weight acting on the original vector X."
So, now we understand why they claim this because they build a mathematical framework where more or less they say, "In the end of all of our calculations, we arrive at this. We arrive here at this term and at this term from two different mathematical spaces, and we just ask, 'Hey, is it possible that our W * A becomes becomes equivalent to our delta W * X?' Can we find this?" And then this is here, if you want, a correction term that is now coming not from the content from the attention, but now we can go and operate with virtual weight structures. So, we have now this bridge suddenly connecting fine-tuning and in-context learning. Now, of course, as I showed you, you can do this here with a rank-one solution. No, if you go with an outer product, simple. No. So, mathematically, we can say adding now the attention output A to the input is identical to updating the weight matrix W by adding here a rank-one matrix delta. So, the activation pattern A from the attention layer becomes the source of the weight update delta W in ICL. So, we have now this beautiful connection between fine-tuning and ICL in a coherent mathematical theory. And now you understand why the authors can claim this identity because mathematically speaking, it is identical. But in physics, you know, whenever we see that we sort that two properties of a physical system are different, and then analyzing it mathematically, we see they are identical, we understand that we made a mistake in building this physical system because either we have neglected symmetries or we have neglected identity operation, but we built something that is not, let's call it, perfect. We built something that works, but we don't really understand why. And now, with this mathematical identity, I have a feeling that what we think is the difference between in-context learning and fine-tuning, what we coded, it turns out is indeed the identical thing, more or less. So maybe we built our transformer architecture just in a wrong way. We didn't utilize that those terms are identical.
Now, you know, we operated here with a simplified kernel representation. And the beauty of the second preprint that I showed you here is the second order here from December 12, that they took this much further, and they prove mathematically that it works here also in a realistic nonlinear deep learning scenarios. So, the second paper is just taking the results of the first paper and proving it mathematically in nonlinear deep learning. But we stay on the same complexity that we try to understand and we try to solve. So, yeah, no, the skeptics, and this is me, I have said at the beginning, no, but you have the layer norm, you have nonlinear activation, you have skip connections. No, and this is why I did not take here this publication by Google here from July 2025 really that significant because I thought, "Hey, the simple algebra doesn't hold if you input nonlinear activations." No, but this team of now proved here just two weeks ago that it holds. And this is something, okay. But they have a second insight: that the update delta W becomes specific to the token position. So, we calculate, given the nonlinearity, a delta W for each single token position. Now, this thing becomes a little bit more interesting. No. So, the weights are effectively different for every single token that is generated by our autoregressive system. The model isn't just fine-tuned once. It is now with ICL continuously refining itself at every step of the generation of the next token prediction of an autoregressive transformer architecture. Therefore, we can now stop treating here the learning, and I mean here the backpropagation, the learning process on the neural network, and the inference, the forward pass, as it would be two different things. It turns out, no. If you understand the mathematics of this, we see the forward pass is a learning step in itself, and this is mathematically equivalent here to the backpropagation. So, somehow there is a chance we can further optimize the architectures of the transformers. We can make it maybe smaller. We can make it more powerful. We have not yet optimized the architecture.
So, if I go here with my image, this is just in my head, whatever image you have. I have here, you know, my tensor structure here, my weight matrix here in, I don't know, 4,000 dimensions. And then I say, "Okay, and then this should be equivalent here to ICL that comes in here, the context, the context activation on the frozen weights." And now I have to understand that these two images are identical. So, this golden context activation are more or less just a rank-one patch that I apply here to my frozen weight tensor structure. So, ICL is mathematically proven equivalent to a rank-one update on the tensor structure. Okay, let's go with this. Okay, now I have further questions. So, you see here how I fight with these things sometimes. And say, "But wait, the activation of our neurons in the layers of the transformers is not a weight tensor update." No, an activation is something completely different. This is the vector that flows through the layers. The weight tensors. If you modify a weight tensor, how is this identical? And the answer is given by the paper. It is possible because they prove mathematically, not in a visual explanation like I try to think. They say it is possible because mathematically we can prove that there is a linearity of matrix multiplication that we can find identical mathematical terms that we can identify.
So, let's do this. It is simple. It is real simple. But it is important that we all understand it. Here in the activation space, we have the classical view. No, let W be the frozen weights here of an MLP layer in a transformer. Let X be your human query here or the embedding here of the last token. And let A be here the output of the attention layer. Attention and A. This is more or less the complete context information. Then a standard transformer. And now let's ignore the nonlinearities here for a second. I'll come back to the nonlinearities in a minute. The input to the MLP is the sum of the query and the attention output. No, via the residual stream, as I showed my you in my last video. So, if you, this is simple. So, we intuitively think now, "Okay, the input X is now shifted here by a particular vector A." Beautiful. And this is it. So, this is our activation space view. So, the term W * A is now a vector representing here the correction, or if you want, the knowledge retrieved now from the added context to our ICL prompt. Now, the authors ask more or less, "Hey, can we find now a matrix W_prime such that this correction comes now from the weights instead from the context input?" So, mathematically, it is so easy. The authors tell us, "We just want to satisfy this equation." Huh? So, we have here the, if you want, the activation space view, and then we have here the weight space view, and we just want to make that this is, can we have this is the same? Yeah, this is it. No, W * A is exactly our delta W * X. X is here the human query, if you want, and A is here the attention here of this additional context, and W is our weight, and delta W is our additional weight structure. So, to solve here for W_prime, we need a matrix that when multiplied with X produces here this particular vector W * A. Uh, distributive property. So, now they tell us mathematically, the authors tell us, "We can pretend that the correction term W * A did not come from the extra vector A, but instead came from an extra weight acting on the original vector X."
So, now we understand why they claim this because they build a mathematical framework where more or less they say, "In the end of all of our calculations, we arrive at this. We arrive here at this term and at this term from two different mathematical spaces, and we just ask, 'Hey, is it possible that our W * A becomes becomes equivalent to our delta W * X?' Can we find this?" And then this is here, if you want, a correction term that is now coming not from the content from the attention, but now we can go and operate with virtual weight structures. So, we have now this bridge suddenly connecting fine-tuning and in-context learning. Now, of course, as I showed you, you can do this here with a rank-one solution. No, if you go with an outer product, simple. No. So, mathematically, we can say adding now the attention output A to the input is identical to updating the weight matrix W by adding here a rank-one matrix delta. So, the activation pattern A from the attention layer becomes the source of the weight update delta W in ICL. So, we have now this beautiful connection between fine-tuning and ICL in a coherent mathematical theory. And now you understand why the authors can claim this identity because mathematically speaking, it is identical. But in physics, you know, whenever we see that we sort that two properties of a physical system are different, and then analyzing it mathematically, we see they are identical, we understand that we made a mistake in building this physical system because either we have neglected symmetries or we have neglected identity operation, but we built something that is not, let's call it, perfect. We built something that works, but we don't really understand why. And now, with this mathematical identity, I have a feeling that what we think is the difference between in-context learning and fine-tuning, what we coded, it turns out is indeed the identical thing, more or less. So maybe we built our transformer architecture just in a wrong way. We didn't utilize that those terms are identical.
Now, you know, we operated here with a simplified kernel representation. And the beauty of the second preprint that I showed you here is the second order here from December 12, that they took this much further, and they prove mathematically that it works here also in a realistic nonlinear deep learning scenarios. So, the second paper is just taking the results of the first paper and proving it mathematically in nonlinear deep learning. But we stay on the same complexity that we try to understand and we try to solve. So, yeah, no, the skeptics, and this is me, I have said at the beginning, no, but you have the layer norm, you have nonlinear activation, you have skip connections. No, and this is why I did not take here this publication by Google here from July 2025 really that significant because I thought, "Hey, the simple algebra doesn't hold if you input nonlinear activations." No, but this team of now proved here just two weeks ago that it holds. And this is something, okay. But they have a second insight: that the update delta W becomes specific to the token position. So, we calculate, given the nonlinearity, a delta W for each single token position. Now, this thing becomes a little bit more interesting. No. So, the weights are effectively different for every single token that is generated by our autoregressive system. The model isn't just fine-tuned once. It is now with ICL continuously refining itself at every step of the generation of the next token prediction of an autoregressive transformer architecture. Therefore, we can now stop treating here the learning, and I mean here the backpropagation, the learning process on the neural network, and the inference, the forward pass, as it would be two different things. It turns out, no. If you understand the mathematics of this, we see the forward pass is a learning step in itself, and this is mathematically equivalent here to the backpropagation. So, somehow there is a chance we can further optimize the architectures of the transformers. We can make it maybe smaller. We can make it more powerful. We have not yet optimized the architecture.
So, if I go here with my image, this is just in my head, whatever image you have. I have here, you know, my tensor structure here, my weight matrix here in, I don't know, 4,000 dimensions. And then I say, "Okay, and then this should be equivalent here to ICL that comes in here, the context, the context activation on the frozen weights." And now I have to understand that these two images are identical. So, this golden context activation are more or less just a rank-one patch that I apply here to my frozen weight tensor structure. So, ICL is mathematically proven equivalent to a rank-one update on the tensor structure. Okay, let's go with this. Okay, now I have further questions. So, you see here how I fight with these things sometimes. And say, "But wait, the activation of our neurons in the layers of the transformers is not a weight tensor update." No, an activation is something completely different. This is the vector that flows through the layers. The weight tensors. If you modify a weight tensor, how is this identical? And the answer is given by the paper. It is possible because they prove mathematically, not in a visual explanation like I try to think. They say it is possible because mathematically we can prove that there is a linearity of matrix multiplication that we can find identical mathematical terms that we can identify.
So, let's do this. It is simple. It is real simple. But it is important that we all understand it. Here in the activation space, we have the classical view. No, let W be the frozen weights here of an MLP layer in a transformer. Let X be your human query here or the embedding here of the last token. And let A be here the output of the attention layer. Attention and A. This is more or less the complete context information. Then a standard transformer. And now let's ignore the nonlinearities here for a second. I'll come back to the nonlinearities in a minute. The input to the MLP is the sum of the query and the attention output. No, via the residual stream, as I showed my you in my last video. So, if you, this is simple. So, we intuitively think now, "Okay, the input X is now shifted here by a particular vector A." Beautiful. And this is it. So, this is our activation space view. So, the term W * A is now a vector representing here the correction, or if you want, the knowledge retrieved now from the added context to our ICL prompt. Now, the authors ask more or less, "Hey, can we find now a matrix W_prime such that this correction comes now from the weights instead from the context input?" So, mathematically, it is so easy. The authors tell us, "We just want to satisfy this equation." Huh? So, we have here the, if you want, the activation space view, and then we have here the weight space view, and we just want to make that this is, can we have this is the same? Yeah, this is it. No, W * A is exactly our delta W * X. X is here the human query, if you want, and A is here the attention here of this additional context, and W is our weight, and delta W is our additional weight structure. So, to solve here for W_prime, we need a matrix that when multiplied with X produces here this particular vector W * A. Uh, distributive property. So, now they tell us mathematically, the authors tell us, "We can pretend that the correction term W * A did not come from the extra vector A, but instead came from an extra weight acting on the original vector X."
So, now we understand why they claim this because they build a mathematical framework where more or less they say, "In the end of all of our calculations, we arrive at this. We arrive here at this term and at this term from two different mathematical spaces, and we just ask, 'Hey, is it possible that our W * A becomes becomes equivalent to our delta W * X?' Can we find this?" And then this is here, if you want, a correction term that is now coming not from the content from the attention, but now we can go and operate with virtual weight structures. So, we have now this bridge suddenly connecting fine-tuning and in-context learning. Now, of course, as I showed you, you can do this here with a rank-one solution. No, if you go with an outer product, simple. No. So, mathematically, we can say adding now the attention output A to the input is identical to updating the weight matrix W by adding here a rank-one matrix delta. So, the activation pattern A from the attention layer becomes the source of the weight update delta W in ICL. So, we have now this beautiful connection between fine-tuning and ICL in a coherent mathematical theory. And now you understand why the authors can claim this identity because mathematically speaking, it is identical. But in physics, you know, whenever we see that we sort that two properties of a physical system are different, and then analyzing it mathematically, we see they are identical, we understand that we made a mistake in building this physical system because either we have neglected symmetries or we have neglected identity operation, but we built something that is not, let's call it, perfect. We built something that works, but we don't really understand why. And now, with this mathematical identity, I have a feeling that what we think is the difference between in-context learning and fine-tuning, what we coded, it turns out is indeed the identical thing, more or less. So maybe we built our transformer architecture just in a wrong way. We didn't utilize that those terms are identical.
Now, you know, we operated here with a simplified kernel representation. And the beauty of the second preprint that I showed you here is the second order here from December 12, that they took this much further, and they prove mathematically that it works here also in a realistic nonlinear deep learning scenarios. So, the second paper is just taking the results of the first paper and proving it mathematically in nonlinear deep learning. But we stay on the same complexity that we try to understand and we try to solve. So, yeah, no, the skeptics, and this is me, I have said at the beginning, no, but you have the layer norm, you have nonlinear activation, you have skip connections. No, and this is why I did not take here this publication by Google here from July 2025 really that significant because I thought, "Hey, the simple algebra doesn't hold if you input nonlinear activations." No, but this team of now proved here just two weeks ago that it holds. And this is something, okay. But they have a second insight: that the update delta W becomes specific to the token position. So, we calculate, given the nonlinearity, a delta W for each single token position. Now, this thing becomes a little bit more interesting. No. So, the weights are effectively different for every single token that is generated by our autoregressive system. The model isn't just fine-tuned once. It is now with ICL continuously refining itself at every step of the generation of the next token prediction of an autoregressive transformer architecture. Therefore, we can now stop treating here the learning, and I mean here the backpropagation, the learning process on the neural network, and the inference, the forward pass, as it would be two different things. It turns out, no. If you understand the mathematics of this, we see the forward pass is a learning step in itself, and this is mathematically equivalent here to the backpropagation. So, somehow there is a chance we can further optimize the architectures of the transformers. We can make it maybe smaller. We can make it more powerful. We have not yet optimized the architecture.
So, if I go here with my image, this is just in my head, whatever image you have. I have here, you know, my tensor structure here, my weight matrix here in, I don't know, 4,000 dimensions. And then I say, "Okay, and then this should be equivalent here to ICL that comes in here, the context, the context activation on the frozen weights." And now I have to understand that these two images are identical. So, this golden context activation are more or less just a rank-one patch that I apply here to my frozen weight tensor structure. So, ICL is mathematically proven equivalent to a rank-one update on the tensor structure. Okay, let's go with this. Okay, now I have further questions. So, you see here how I fight with these things sometimes. And say, "But wait, the activation of our neurons in the layers of the transformers is not a weight tensor update." No, an activation is something completely different. This is the vector that flows through the layers. The weight tensors. If you modify a weight tensor, how is this identical? And the answer is given by the paper. It is possible because they prove mathematically, not in a visual explanation like I try to think. They say it is possible because mathematically we can prove that there is a linearity of matrix multiplication that we can find identical mathematical terms that we can identify.
So, let's do this. It is simple. It is real simple. But it is important that we all understand it. Here in the activation space, we have the classical view. No, let W be the frozen weights here of an MLP layer in a transformer. Let X be your human query here or the embedding here of the last token. And let A be here the output of the attention layer. Attention and A. This is more or less the complete context information. Then a standard transformer. And now let's ignore the nonlinearities here for a second. I'll come back to the nonlinearities in a minute. The input to the MLP is the sum of the query and the attention output. No, via the residual stream, as I showed my you in my last video. So, if you, this is simple. So, we intuitively think now, "Okay, the input X is now shifted here by a particular vector A." Beautiful. And this is it. So, this is our activation space view. So, the term W * A is now a vector representing here the correction, or if you want, the knowledge retrieved now from the added context to our ICL prompt. Now, the authors ask more or less, "Hey, can we find now a matrix W_prime such that this correction comes now from the weights instead from the context input?" So, mathematically, it is so easy. The authors tell us, "We just want to satisfy this equation." Huh? So, we have here the, if you want, the activation space view, and then we have here the weight space view, and we just want to make that this is, can we have this is the same? Yeah, this is it. No, W * A is exactly our delta W * X. X is here the human query, if you want, and A is here the attention here of this additional context, and W is our weight, and delta W is our additional weight structure. So, to solve here for W_prime, we need a matrix that when multiplied with X produces here this particular vector W * A. Uh, distributive property. So, now they tell us mathematically, the authors tell us, "We can pretend that the correction term W * A did not come from the extra vector A, but instead came from an extra weight acting on the original vector X."
So, now we understand why they claim this because they build a mathematical framework where more or less they say, "In the end of all of our calculations, we arrive at this. We arrive here at this term and at this term from two different mathematical spaces, and we just ask, 'Hey, is it possible that our W * A becomes becomes equivalent to our delta W * X?' Can we find this?" And then this is here, if you want, a correction term that is now coming not from the content from the attention, but now we can go and operate with virtual weight structures. So, we have now this bridge suddenly connecting fine-tuning and in-context learning. Now, of course, as I showed you, you can do this here with a rank-one solution. No, if you go with an outer product, simple. No. So, mathematically, we can say adding now the attention output A to the input is identical to updating the weight matrix W by adding here a rank-one matrix delta. So, the activation pattern A from the attention layer becomes the source of the weight update delta W in ICL. So, we have now this beautiful connection between fine-tuning and ICL in a coherent mathematical theory. And now you understand why the authors can claim this identity because mathematically speaking, it is identical. But in physics, you know, whenever we see that we sort that two properties of a physical system are different, and then analyzing it mathematically, we see they are identical, we understand that we made a mistake in building this physical system because either we have neglected symmetries or we have neglected identity operation, but we built something that is not, let's call it, perfect. We built something that works, but we don't really understand why. And now, with this mathematical identity, I have a feeling that what we think is the difference between in-context learning and fine-tuning, what we coded, it turns out is indeed the identical thing, more or less. So maybe we built our transformer architecture just in a wrong way. We didn't utilize that those terms are identical.
Now, you know, we operated here with a simplified kernel representation. And the beauty of the second preprint that I showed you here is the second order here from December 12, that they took this much further, and they prove mathematically that it works here also in a realistic nonlinear deep learning scenarios. So, the second paper is just taking the results of the first paper and proving it mathematically in nonlinear deep learning. But we stay on the same complexity that we try to understand and we try to solve. So, yeah, no, the skeptics, and this is me, I have said at the beginning, no, but you have the layer norm, you have nonlinear activation, you have skip connections. No, and this is why I did not take here this publication by Google here from July 2025 really that significant because I thought, "Hey, the simple algebra doesn't hold if you input nonlinear activations." No, but this team of now proved here just two weeks ago that it holds. And this is something, okay. But they have a second insight: that the update delta W becomes specific to the token position. So, we calculate, given the nonlinearity, a delta W for each single token position. Now, this thing becomes a little bit more interesting. No. So, the weights are effectively different for every single token that is generated by our autoregressive system. The model isn't just fine-tuned once. It is now with ICL continuously refining itself at every step of the generation of the next token prediction of an autoregressive transformer architecture. Therefore, we can now stop treating here the learning, and I mean here the backpropagation, the learning process on the neural network, and the inference, the forward pass, as it would be two different things. It turns out, no. If you understand the mathematics of this, we see the forward pass is a learning step in itself, and this is mathematically equivalent here to the backpropagation. So, somehow there is a chance we can further optimize the architectures of the transformers. We can make it maybe smaller. We can make it more powerful. We have not yet optimized the architecture.
So, if I go here with my image, this is just in my head, whatever image you have. I have here, you know, my tensor structure here, my weight matrix here in, I don't know, 4,000 dimensions. And then I say, "Okay, and then this should be equivalent here to ICL that comes in here, the context, the context activation on the frozen weights." And now I have to understand that these two images are identical. So, this golden context activation are more or less just a rank-one patch that I apply here to my frozen weight tensor structure. So, ICL is mathematically proven equivalent to a rank-one update on the tensor structure. Okay, let's go with this. Okay, now I have further questions. So, you see here how I fight with these things sometimes. And say, "But wait, the activation of our neurons in the layers of the transformers is not a weight tensor update." No, an activation is something completely different. This is the vector that flows through the layers. The weight tensors. If you modify a weight tensor, how is this identical? And the answer is given by the paper. It is possible because they prove mathematically, not in a visual explanation like I try to think. They say it is possible because mathematically we can prove that there is a linearity of matrix multiplication that we can find identical mathematical terms that we can identify.
So, let's do this. It is simple. It is real simple. But it is important that we all understand it. Here in the activation space, we have the classical view. No, let W be the frozen weights here of an MLP layer in a transformer. Let X be your human query here or the embedding here of the last token. And let A be here the output of the attention layer. Attention and A. This is more or less the complete context information. Then a standard transformer. And now let's ignore the nonlinearities here for a second. I'll come back to the nonlinearities in a minute. The input to the MLP is the sum of the query and the attention output. No, via the residual stream, as I showed my you in my last video. So, if you, this is simple. So, we intuitively think now, "Okay, the input X is now shifted here by a particular vector A." Beautiful. And this is it. So, this is our activation space view. So, the term W * A is now a vector representing here the correction, or if you want, the knowledge retrieved now from the added context to our ICL prompt. Now, the authors ask more or less, "Hey, can we find now a matrix W_prime such that this correction comes now from the weights instead from the context input?" So, mathematically, it is so easy. The authors tell us, "We just want to satisfy this equation." Huh? So, we have here the, if you want, the activation space view, and then we have here the weight space view, and we just want to make that this is, can we have this is the same? Yeah, this is it. No, W * A is exactly our delta W * X. X is here the human query, if you want, and A is here the attention here of this additional context, and W is our weight, and delta W is our additional weight structure. So, to solve here for W_prime, we need a matrix that when multiplied with X produces here this particular vector W * A. Uh, distributive property. So, now they tell us mathematically, the authors tell us, "We can pretend that the correction term W * A did not come from the extra vector A, but instead came from an extra weight acting on the original vector X."
So, now we understand why they claim this because they build a mathematical framework where more or less they say, "In the end of all of our calculations, we arrive at this. We arrive here at this term and at this term from two different mathematical spaces, and we just ask, 'Hey, is it possible that our W * A becomes becomes equivalent to our delta W * X?' Can we find this?" And then this is here, if you want, a correction term that is now coming not from the content from the attention, but now we can go and operate with virtual weight structures. So, we have now this bridge suddenly connecting fine-tuning and in-context learning. Now, of course, as I showed you, you can do this here with a rank-one solution. No, if you go with an outer product, simple. No. So, mathematically, we can say adding now the attention output A to the input is identical to updating the weight matrix W by adding here a rank-one matrix delta. So, the activation pattern A from the attention layer becomes the source of the weight update delta W in ICL. So, we have now this beautiful connection between fine-tuning and ICL in a coherent mathematical theory. And now you understand why the authors can claim this identity because mathematically speaking, it is identical. But in physics, you know, whenever we see that we sort that two properties of a physical system are different, and then analyzing it mathematically, we see they are identical, we understand that we made a mistake in building this physical system because either we have neglected symmetries or we have neglected identity operation, but we built something that is not, let's call it, perfect. We built something that works, but we don't really understand why. And now, with this mathematical identity, I have a feeling that what we think is the difference between in-context learning and fine-tuning, what we coded, it turns out is indeed the identical thing, more or less. So maybe we built our transformer architecture just in a wrong way. We didn't utilize that those terms are identical.
Now, you know, we operated here with a simplified kernel representation. And the beauty of the second preprint that I showed you here is the second order here from December 12, that they took this much further, and they prove mathematically that it works here also in a realistic nonlinear deep learning scenarios. So, the second paper is just taking the results of the first paper and proving it mathematically in nonlinear deep learning. But we stay on the same complexity that we try to understand and we try to solve. So, yeah, no, the skeptics, and this is me, I have said at the beginning, no, but you have the layer norm, you have nonlinear activation, you have skip connections. No, and this is why I did not take here this publication by Google here from July 2025 really that significant because I thought, "Hey, the simple algebra doesn't hold if you input nonlinear activations." No, but this team of now proved here just two weeks ago that it holds. And this is something, okay. But they have a second insight: that the update delta W becomes specific to the token position. So, we calculate, given the nonlinearity, a delta W for each single token position. Now, this thing becomes a little bit more interesting. No. So, the weights are effectively different for every single token that is generated by our autoregressive system. The model isn't just fine-tuned once. It is now with ICL continuously refining itself at every step of the generation of the next token prediction of an autoregressive transformer architecture. Therefore, we can now stop treating here the learning, and I mean here the backpropagation, the learning process on the neural network, and the inference, the forward pass, as it would be two different things. It turns out, no. If you understand the mathematics of this, we see the forward pass is a learning step in itself, and this is mathematically equivalent here to the backpropagation. So, somehow there is a chance we can further optimize the architectures of the transformers. We can make it maybe smaller. We can make it more powerful. We have not yet optimized the architecture.
So, if I go here with my image, this is just in my head, whatever image you have. I have here, you know, my tensor structure here, my weight matrix here in, I don't know, 4,000 dimensions. And then I say, "Okay, and then this should be equivalent here to ICL that comes in here, the context, the context activation on the frozen weights." And now I have to understand that these two images are identical. So, this golden context activation are more or less just a rank-one patch that I apply here to my frozen weight tensor structure. So, ICL is mathematically proven equivalent to a rank-one update on the tensor structure. Okay, let's go with this. Okay, now I have further questions. So, you see here how I fight with these things sometimes. And say, "But wait, the activation of our neurons in the layers of the transformers is not a weight tensor update." No, an activation is something completely different. This is the vector that flows through the layers. The weight tensors. If you modify a weight tensor, how is this identical? And the answer is given by the paper. It is possible because they prove mathematically, not in a visual explanation like I try to think. They say it is possible because mathematically we can prove that there is a linearity of matrix multiplication that we can find identical mathematical terms that we can identify.
So, let's do this. It is simple. It is real simple. But it is important that we all understand it. Here in the activation space, we have the classical view. No, let W be the frozen weights here of an MLP layer in a transformer. Let X be your human query here or the embedding here of the last token. And let A be here the output of the attention layer. Attention and A. This is more or less the complete context information. Then a standard transformer. And now let's ignore the nonlinearities here for a second. I'll come back to the nonlinearities in a minute. The input to the MLP is the sum of the query and the attention output. No, via the residual stream, as I showed my you in my last video. So, if you, this is simple. So, we intuitively think now, "Okay, the input X is now shifted here by a particular vector A." Beautiful. And this is it. So, this is our activation space view. So, the term W * A is now a vector representing here the correction, or if you want, the knowledge retrieved now from the added context to our ICL prompt. Now, the authors ask more or less, "Hey, can we find now a matrix W_prime such that this correction comes now from the weights instead from the context input?" So, mathematically, it is so easy. The authors tell us, "We just want to satisfy this equation." Huh? So, we have here the, if you want, the activation space view, and then we have here the weight space view, and we just want to make that this is, can we have this is the same? Yeah, this is it. No, W * A is exactly our delta W * X. X is here the human query, if you want, and A is here the attention here of this additional context, and W is our weight, and delta W is our additional weight structure. So, to solve here for W_prime, we need a matrix that when multiplied with X produces here this particular vector W * A. Uh, distributive property. So, now they tell us mathematically, the authors tell us, "We can pretend that the correction term W * A did not come from the extra vector A, but instead came from an extra weight acting on the original vector X."
So, now we understand why they claim this because they build a mathematical framework where more or less they say, "In the end of all of our calculations, we arrive at this. We arrive here at this term and at this term from two different mathematical spaces, and we just ask, 'Hey, is it possible that our W * A becomes becomes equivalent to our delta W * X?' Can we find this?" And then this is here, if you want, a correction term that is now coming not from the content from the attention, but now we can go and operate with virtual weight structures. So, we have now this bridge suddenly connecting fine-tuning and in-context learning. Now, of course, as I showed you, you can do this here with a rank-one solution. No, if you go with an outer product, simple. No. So, mathematically, we can say adding now the attention output A to the input is identical to updating the weight matrix W by adding here a rank-one matrix delta. So, the activation pattern A from the attention layer becomes the source of the weight update delta W in ICL. So, we have now this beautiful connection between fine-tuning and ICL in a coherent mathematical theory. And now you understand why the authors can claim this identity because mathematically speaking, it is identical. But in physics, you know, whenever we see that we sort that two properties of a physical system are different, and then analyzing it mathematically, we see they are identical, we understand that we made a mistake in building this physical system because either we have neglected symmetries or we have neglected identity operation, but we built something that is not, let's call it, perfect. We built something that works, but we don't really understand why. And now, with this mathematical identity, I have a feeling that what we think is the difference between in-context learning and fine-tuning, what we coded, it turns out is indeed the identical thing, more or less. So maybe we built our transformer architecture just in a wrong way. We didn't utilize that those terms are identical.
Now, you know, we operated here with a simplified kernel representation. And the beauty of the second preprint that I showed you here is the second order here from December 12, that they took this much further, and they prove mathematically that it works here also in a realistic nonlinear deep learning scenarios. So, the second paper is just taking the results of the first paper and proving it mathematically in nonlinear deep learning. But we stay on the same complexity that we try to understand and we try to solve. So, yeah, no, the skeptics, and this is me, I have said at the beginning, no, but you have the layer norm, you have nonlinear activation, you have skip connections. No, and this is why I did not take here this publication by Google here from July 2025 really that significant because I thought, "Hey, the simple algebra doesn't hold if you input nonlinear activations." No, but this team of now proved here just two weeks ago that it holds. And this is something, okay. But they have a second insight: that the update delta W becomes specific to the token position. So, we calculate, given the nonlinearity, a delta W for each single token position. Now, this thing becomes a little bit more interesting. No. So, the weights are effectively different for every single token that is generated by our autoregressive system. The model isn't just fine-tuned once. It is now with ICL continuously refining itself at every step of the generation of the next token prediction of an autoregressive transformer architecture. Therefore, we can now stop treating here the learning, and I mean here the backpropagation, the learning process on the neural network, and the inference, the forward pass, as it would be two different things. It turns out, no. If you understand the mathematics of this, we see the forward pass is a learning step in itself, and this is mathematically equivalent here to the backpropagation. So, somehow there is a chance we can further optimize the architectures of the transformers. We can make it maybe smaller. We can make it more powerful. We have not yet optimized the architecture.
So, if I go here with my image, this is just in my head, whatever image you have. I have here, you know, my tensor structure here, my weight matrix here in, I don't know, 4,000 dimensions. And then I say, "Okay, and then this should be equivalent here to ICL that comes in here, the context, the context activation on the frozen weights." And now I have to understand that these two images are identical. So, this golden context activation are more or less just a rank-one patch that I apply here to my frozen weight tensor structure. So, ICL is mathematically proven equivalent to a rank-one update on the tensor structure. Okay, let's go with this. Okay, now I have further questions. So, you see here how I fight with these things sometimes. And say, "But wait, the activation of our neurons in the layers of the transformers is not a weight tensor update." No, an activation is something completely different. This is the vector that flows through the layers. The weight tensors. If you modify a weight tensor, how is this identical? And the answer is given by the paper. It is possible because they prove mathematically, not in a visual explanation like I try to think. They say it is possible because mathematically we can prove that there is a linearity of matrix multiplication that we can find identical mathematical terms that we can identify.
So, let's do this. It is simple. It is real simple. But it is important that we all understand it. Here in the activation space, we have the classical view. No, let W be the frozen weights here of an MLP layer in a transformer. Let X be your human query here or the embedding here of the last token. And let A be here the output of the attention layer. Attention and A. This is more or less the complete context information. Then a standard transformer. And now let's ignore the nonlinearities here for a second. I'll come back to the nonlinearities in a minute. The input to the MLP is the sum of the query and the attention output. No, via the residual stream, as I showed my you in my last video. So, if you, this is simple. So, we intuitively think now, "Okay, the input X is now shifted here by a particular vector A." Beautiful. And this is it. So, this is our activation space view. So, the term W * A is now a vector representing here the correction, or if you want, the knowledge retrieved now from the added context to our ICL prompt. Now, the authors ask more or less, "Hey, can we find now a matrix W_prime such that this correction comes now from the weights instead from the context input?" So, mathematically, it is so easy. The authors tell us, "We just want to satisfy this equation." Huh? So, we have here the, if you want, the activation space view, and then we have here the weight space view, and we just want to make that this is, can we have this is the same? Yeah, this is it. No, W * A is exactly our delta W * X. X is here the human query, if you want, and A is here the attention here of this additional context, and W is our weight, and delta W is our additional weight structure. So, to solve here for W_prime, we need a matrix that when multiplied with X produces here this particular vector W * A. Uh, distributive property. So, now they tell us mathematically, the authors tell us, "We can pretend that the correction term W * A did not come from the extra vector A, but instead came from an extra weight acting on the original vector X."
So, now we understand why they claim this because they build a mathematical framework where more or less they say, "In the end of all of our calculations, we arrive at this. We arrive here at this term and at this term from two different mathematical spaces, and we just ask, 'Hey, is it possible that our W * A becomes becomes equivalent to our delta W * X?' Can we find this?" And then this is here, if you want, a correction term that is now coming not from the content from the attention, but now we can go and operate with virtual weight structures. So, we have now this bridge suddenly connecting fine-tuning and in-context learning. Now, of course, as I showed you, you can do this here with a rank-one solution. No, if you go with an outer product, simple. No. So, mathematically, we can say adding now the attention output A to the input is identical to updating the weight matrix W by adding here a rank-one matrix delta. So, the activation pattern A from the attention layer becomes the source of the weight update delta W in ICL. So, we have now this beautiful connection between fine-tuning and ICL in a coherent mathematical theory. And now you understand why the authors can claim this identity because mathematically speaking, it is identical. But in physics, you know, whenever we see that we sort that two properties of a physical system are different, and then analyzing it mathematically, we see they are identical, we understand that we made a mistake in building this physical system because either we have neglected symmetries or we have neglected identity operation, but we built something that is not, let's call it, perfect. We built something that works, but we don't really understand why. And now, with this mathematical identity, I have a feeling that what we think is the difference between in-context learning and fine-tuning, what we coded, it turns out is indeed the identical thing, more or less. So maybe we built our transformer architecture just in a wrong way. We didn't utilize that those terms are identical.
Now, you know, we operated here with a simplified kernel representation. And the beauty of the second preprint that I showed you here is the second order here from December 12, that they took this much further, and they prove mathematically that it works here also in a realistic nonlinear deep learning scenarios. So, the second paper is just taking the results of the first paper and proving it mathematically in nonlinear deep learning. But we stay on the same complexity that we try to understand and we try to solve. So, yeah, no, the skeptics, and this is me, I have said at the beginning, no, but you have the layer norm, you have nonlinear activation, you have skip connections. No, and this is why I did not take here this publication by Google here from July 2025 really that significant because I thought, "Hey, the simple algebra doesn't hold if you input nonlinear activations." No, but this team of now proved here just two weeks ago that it holds. And this is something, okay. But they have a second insight: that the update delta W becomes specific to the token position. So, we calculate, given the nonlinearity, a delta W for each single token position. Now, this thing becomes a little bit more interesting. No. So, the weights are effectively different for every single token that is generated by our autoregressive system. The model isn't just fine-tuned once. It is now with ICL continuously refining itself at every step of the generation of the next token prediction of an autoregressive transformer architecture. Therefore, we can now stop treating here the learning, and I mean here the backpropagation, the learning process on the neural network, and the inference, the forward pass, as it would be two different things. It turns out, no. If you understand the mathematics of this, we see the forward pass is a learning step in itself, and this is mathematically equivalent here to the backpropagation. So, somehow there is a chance we can further optimize the architectures of the transformers. We can make it maybe smaller. We can make it more powerful. We have not yet optimized the architecture.
So, if I go here with my image, this is just in my head, whatever image you have. I have here, you know, my tensor structure here, my weight matrix here in, I don't know, 4,000 dimensions. And then I say, "Okay, and then this should be equivalent here to ICL that comes in here, the context, the context activation on the frozen weights." And now I have to understand that these two images are identical. So, this golden context activation are more or less just a rank-one patch that I apply here to my frozen weight tensor structure. So, ICL is mathematically proven equivalent to a rank-one update on the tensor structure. Okay, let's go with this. Okay, now I have further questions. So, you see here how I fight with these things sometimes. And say, "But wait, the activation of our neurons in the layers of the transformers is not a weight tensor update." No, an activation is something completely different. This is the vector that flows through the layers. The weight tensors. If you modify a weight tensor, how is this identical? And the answer is given by the paper. It is possible because they prove mathematically, not in a visual explanation like I try to think. They say it is possible because mathematically we can prove that there is a linearity of matrix multiplication that we can find identical mathematical terms that we can identify.
So, let's do this. It is simple. It is real simple. But it is important that we all understand it. Here in the activation space, we have the classical view. No, let W be the frozen weights here of an MLP layer in a transformer. Let X be your human query here or the embedding here of the last token. And let A be here the output of the attention layer. Attention and A. This is more or less the complete context information. Then a standard transformer. And now let's ignore the nonlinearities here for a second. I'll come back to the nonlinearities in a minute. The input to the MLP is the sum of the query and the attention output. No, via the residual stream, as I showed my you in my last video. So, if you, this is simple. So, we intuitively think now, "Okay, the input X is now shifted here by a particular vector A." Beautiful. And this is it. So, this is our activation space view. So, the term W * A is now a vector representing here the correction, or if you want, the knowledge retrieved now from the added context to our ICL prompt. Now, the authors ask more or less, "Hey, can we find now a matrix W_prime such that this correction comes now from the weights instead from the context input?" So, mathematically, it is so easy. The authors tell us, "We just want to satisfy this equation." Huh? So, we have here the, if you want, the activation space view, and then we have here the weight space view, and we just want to make that this is, can we have this is the same? Yeah, this is it. No, W * A is exactly our delta W * X. X is here the human query, if you want, and A is here the attention here of this additional context, and W is our weight, and delta W is our additional weight structure. So, to solve here for W_prime, we need a matrix that when multiplied with X produces here this particular vector W * A. Uh, distributive property. So, now they tell us mathematically, the authors tell us, "We can pretend that the correction term W * A did not come from the extra vector A, but instead came from an extra weight acting on the original vector X."
So, now we understand why they claim this because they build a mathematical framework where more or less they say, "In the end of all of our calculations, we arrive at this. We arrive here at this term and at this term from two different mathematical spaces, and we just ask, 'Hey, is it possible that our W * A becomes becomes equivalent to our delta W * X?' Can we find this?" And then this is here, if you want, a correction term that is now coming not from the content from the attention, but now we can go and operate with virtual weight structures. So, we have now this bridge suddenly connecting fine-tuning and in-context learning. Now, of course, as I showed you, you can do this here with a rank-one solution. No, if you go with an outer product, simple. No. So, mathematically, we can say adding now the attention output A to the input is identical to updating the weight matrix W by adding here a rank-one matrix delta. So, the activation pattern A from the attention layer becomes the source of the weight update delta W in ICL. So, we have now this beautiful connection between fine-tuning and ICL in a coherent mathematical theory. And now you understand why the authors can claim this identity because mathematically speaking, it is identical. But in physics, you know, whenever we see that we sort that two properties of a physical system are different, and then analyzing it mathematically, we see they are identical, we understand that we made a mistake in building this physical system because either we have neglected symmetries or we have neglected identity operation, but we built something that is not, let's call it, perfect. We built something that works, but we don't really understand why. And now, with this mathematical identity, I have a feeling that what we think is the difference between in-context learning and fine-tuning, what we coded, it turns out is indeed the identical thing, more or less. So maybe we built our transformer architecture just in a wrong way. We didn't utilize that those terms are identical.
Now, you know, we operated here with a simplified kernel representation. And the beauty of the second preprint that I showed you here is the second order here from December 12, that they took this much further, and they prove mathematically that it works here also in a realistic nonlinear deep learning scenarios. So, the second paper is just taking the results of the first paper and proving it mathematically in nonlinear deep learning. But we stay on the same complexity that we try to understand and we try to solve. So, yeah, no, the skeptics, and this is me, I have said at the beginning, no, but you have the layer norm, you have nonlinear activation, you have skip connections. No, and this is why I did not take here this publication by Google here from July 2025 really that significant because I thought, "Hey, the simple algebra doesn't hold if you input nonlinear activations." No, but this team of now proved here just two weeks ago that it holds. And this is something, okay. But they have a second insight: that the update delta W becomes specific to the token position. So, we calculate, given the nonlinearity, a delta W for each single token position. Now, this thing becomes a little bit more interesting. No. So, the weights are effectively different for every single token that is generated by our autoregressive system. The model isn't just fine-tuned once. It is now with ICL continuously refining itself at every step of the generation of the next token prediction of an autoregressive transformer architecture. Therefore, we can now stop treating here the learning, and I mean here the backpropagation, the learning process on the neural network, and the inference, the forward pass, as it would be two different things. It turns out, no. If you understand the mathematics of this, we see the forward pass is a learning step in itself, and this is mathematically equivalent here to the backpropagation. So, somehow there is a chance we can further optimize the architectures of the transformers. We can make it maybe smaller. We can make it more powerful. We have not yet optimized the architecture.
So, if I go here with my image, this is just in my head, whatever image you have. I have here, you know, my tensor structure here, my weight matrix here in, I don't know, 4,000 dimensions. And then I say, "Okay, and then this should be equivalent here to ICL that comes in here, the context, the context activation on the frozen weights." And now I have to understand that these two images are identical. So, this golden context activation are more or less just a rank-one patch that I apply here to my frozen weight tensor structure. So, ICL is mathematically proven equivalent to a rank-one update on the tensor structure. Okay, let's go with this. Okay, now I have further questions. So, you see here how I fight with these things sometimes. And say, "But wait, the activation of our neurons in the layers of the transformers is not a weight tensor update." No, an activation is something completely different. This is the vector that flows through the layers. The weight tensors. If you modify a weight tensor, how is this identical? And the answer is given by the paper. It is possible because they prove mathematically, not in a visual explanation like I try to think. They say it is possible because mathematically we can prove that there is a linearity of matrix multiplication that we can find identical mathematical terms that we can identify.
So, let's do this. It is simple. It is real simple. But it is important that we all understand it. Here in the activation space, we have the classical view. No, let W be the frozen weights here of an MLP layer in a transformer. Let X be your human query here or the embedding here of the last token. And let A be here the output of the attention layer. Attention and A. This is more or less the complete context information. Then a standard transformer. And now let's ignore the nonlinearities here for a second. I'll come back to the nonlinearities in a minute. The input to the MLP is the sum of the query and the attention output. No, via the residual stream, as I showed my you in my last video. So, if you, this is simple. So, we intuitively think now, "Okay, the input X is now shifted here by a particular vector A." Beautiful. And this is it. So, this is our activation space view. So, the term W * A is now a vector representing here the correction, or if you want, the knowledge retrieved now from the added context to our ICL prompt. Now, the authors ask more or less, "Hey, can we find now a matrix W_prime such that this correction comes now from the weights instead from the context input?" So, mathematically, it is so easy. The authors tell us, "We just want to satisfy this equation." Huh? So, we have here the, if you want, the activation space view, and then we have here the weight space view, and we just want to make that this is, can we have this is the same? Yeah, this is it. No, W * A is exactly our delta W * X. X is here the human query, if you want, and A is here the attention here of this additional context, and W is our weight, and delta W is our additional weight structure. So, to solve here for W_prime, we need a matrix that when multiplied with X produces here this particular vector W * A. Uh, distributive property. So, now they tell us mathematically, the authors tell us, "We can pretend that the correction term W * A did not come from the extra vector A, but instead came from an extra weight acting on the original vector X."
So, now we understand why they claim this because they build a mathematical framework where more or less they say, "In the end of all of our calculations, we arrive at this. We arrive here at this term and at this term from two different mathematical spaces, and we just ask, 'Hey, is it possible that our W * A becomes becomes equivalent to our delta W * X?' Can we find this?" And then this is here, if you want, a correction term that is now coming not from the content from the attention, but now we can go and operate with virtual weight structures. So, we have now this bridge suddenly connecting fine-tuning and in-context learning. Now, of course, as I showed you, you can do this here with a rank-one solution. No, if you go with an outer product, simple. No. So, mathematically, we can say adding now the attention output A to the input is identical to updating the weight matrix W by adding here a rank-one matrix delta. So, the activation pattern A from the attention layer becomes the source of the weight update delta W in ICL. So, we have now this beautiful connection between fine-tuning and ICL in a coherent mathematical theory. And now you understand why the authors can claim this identity because mathematically speaking, it is identical. But in physics, you know, whenever we see that we sort that two properties of a physical system are different, and then analyzing it mathematically, we see they are identical, we understand that we made a mistake in building this physical system because either we have neglected symmetries or we have neglected identity operation, but we built something that is not, let's call it, perfect. We built something that works, but we don't really understand why. And now, with this mathematical identity, I have a feeling that what we think is the difference between in-context learning and fine-tuning, what we coded, it turns out is indeed the identical thing, more or less. So maybe we built our transformer architecture just in a wrong way. We didn't utilize that those terms are identical.
Now, you know, we operated here with a simplified kernel representation. And the beauty of the second preprint that I showed you here is the second order here from December 12, that they took this much further, and they prove mathematically that it works here also in a realistic nonlinear deep learning scenarios. So, the second paper is just taking the results of the first paper and proving it mathematically in nonlinear deep learning. But we stay on the same complexity that we try to understand and we try to solve. So, yeah, no, the skeptics, and this is me, I have said at the beginning, no, but you have the layer norm, you have nonlinear activation, you have skip connections. No, and this is why I did not take here this publication by Google here from July 2025 really that significant because I thought, "Hey, the simple algebra doesn't hold if you input nonlinear activations." No, but this team of now proved here just two weeks ago that it holds. And this is something, okay. But they have a second insight: that the update delta W becomes specific to the token position. So, we calculate, given the nonlinearity, a delta W for each single token position. Now, this thing becomes a little bit more interesting. No. So, the weights are effectively different for every single token that is generated by our autoregressive system. The model isn't just fine-tuned once. It is now with ICL continuously refining itself at every step of the generation of the next token prediction of an autoregressive transformer architecture. Therefore, we can now stop treating here the learning, and I mean here the backpropagation, the learning process on the neural network, and the inference, the forward pass, as it would be two different things. It turns out, no. If you understand the mathematics of this, we see the forward pass is a learning step in itself, and this is mathematically equivalent here to the backpropagation. So, somehow there is a chance we can further optimize the architectures of the transformers. We can make it maybe smaller. We can make it more powerful. We have not yet optimized the architecture.
So, if I go here with my image, this is just in my head, whatever image you have. I have here, you know, my tensor structure here, my weight matrix here in, I don't know, 4,000 dimensions. And then I say, "Okay, and then this should be equivalent here to ICL that comes in here, the context, the context activation on the frozen weights." And now I have to understand that these two images are identical. So, this golden context activation are more or less just a rank-one patch that I apply here to my frozen weight tensor structure. So, ICL is mathematically proven equivalent to a rank-one update on the tensor structure. Okay, let's go with this. Okay, now I have further questions. So, you see here how I fight with these things sometimes. And say, "But wait, the activation of our neurons in the layers of the transformers is not a weight tensor update." No, an activation is something completely different. This is the vector that flows through the layers. The weight tensors. If you modify a weight tensor, how is this identical? And the answer is given by the paper. It is possible because they prove mathematically, not in a visual explanation like I try to think. They say it is possible because mathematically we can prove that there is a linearity of matrix multiplication that we can find identical mathematical terms that we can identify.
So, let's do this. It is simple. It is real simple. But it is important that we all understand it. Here in the activation space, we have the classical view. No, let W be the frozen weights here of an MLP layer in a transformer. Let X be your human query here or the embedding here of the last token. And let A be here the output of the attention layer. Attention and A. This is more or less the complete context information. Then a standard transformer. And now let's ignore the nonlinearities here for a second. I'll come back to the nonlinearities in a minute. The input to the MLP is the sum of the query and the attention output. No, via the residual stream, as I showed my you in my last video. So, if you, this is simple. So, we intuitively think now, "Okay, the input X is now shifted here by a particular vector A." Beautiful. And this is it. So, this is our activation space view. So, the term W * A is now a vector representing here the correction, or if you want, the knowledge retrieved now from the added context to our ICL prompt. Now, the authors ask more or less, "Hey, can we find now a matrix W_prime such that this correction comes now from the weights instead from the context input?" So, mathematically, it is so easy. The authors tell us, "We just want to satisfy this equation." Huh? So, we have here the, if you want, the activation space view, and then we have here the weight space view, and we just want to make that this is, can we have this is the same? Yeah, this is it. No, W * A is exactly our delta W * X. X is here the human query, if you want, and A is here the attention here of this additional context, and W is our weight, and delta W is our additional weight structure. So, to solve here for W_prime, we need a matrix that when multiplied with X produces here this particular vector W * A. Uh, distributive property. So, now they tell us mathematically, the authors tell us, "We can pretend that the correction term W * A did not come from the extra vector A, but instead came from an extra weight acting on the original vector X."
So, now we understand why they claim this because they build a mathematical framework where more or less they say, "In the end of all of our calculations, we arrive at this. We arrive here at this term and at this term from two different mathematical spaces, and we just ask, 'Hey, is it possible that our W * A becomes becomes equivalent to our delta W * X?' Can we find this?" And then this is here, if you want, a correction term that is now coming not from the content from the attention, but now we can go and operate with virtual weight structures. So, we have now this bridge suddenly connecting fine-tuning and in-context learning. Now, of course, as I showed you, you can do this here with a rank-one solution. No, if you go with an outer product, simple. No. So, mathematically, we can say adding now the attention output A to the input is identical to updating the weight matrix W by adding here a rank-one matrix delta. So, the activation pattern A from the attention layer becomes the source of the weight update delta W in ICL. So, we have now this beautiful connection between fine-tuning and ICL in a coherent mathematical theory. And now you understand why the authors can claim this identity because mathematically speaking, it is identical. But in physics, you know, whenever we see that we sort that two properties of a physical system are different, and then analyzing it mathematically, we see they are identical, we understand that we made a mistake in building this physical system because either we have neglected symmetries or we have neglected identity operation, but we built something that is not, let's call it, perfect. We built something that works, but we don't really understand why. And now, with this mathematical identity, I have a feeling that what we think is the difference between in-context learning and fine-tuning, what we coded, it turns out is indeed the identical thing, more or less. So maybe we built our transformer architecture just in a wrong way. We didn't utilize that those terms are identical.
Now, you know, we operated here with a simplified kernel representation. And the beauty of the second preprint that I showed you here is the second order here from December 12, that they took this much further, and they prove mathematically that it works here also in a realistic nonlinear deep learning scenarios. So, the second paper is just taking the results of the first paper and proving it mathematically in nonlinear deep learning. But we stay on the same complexity that we try to understand and we try to solve. So, yeah, no, the skeptics, and this is me, I have said at the beginning, no, but you have the layer norm, you have nonlinear activation, you have skip connections. No, and this is why I did not take here this publication by Google here from July 2025 really that significant because I thought, "Hey, the simple algebra doesn't hold if you input nonlinear activations." No, but this team of now proved here just two weeks ago that it holds. And this is something, okay. But they have a second insight: that the update delta W becomes specific to the token position. So, we calculate, given the nonlinearity, a delta W for each single token position. Now, this thing becomes a little bit more interesting. No. So, the weights are effectively different for every single token that is generated by our autoregressive system. The model isn't just fine-tuned once. It is now with ICL continuously refining itself at every step of the generation of the next token prediction of an autoregressive transformer architecture. Therefore, we can now stop treating here the learning, and I mean here the backpropagation, the learning process on the neural network, and the inference, the forward pass, as it would be two different things. It turns out, no. If you understand the mathematics of this, we see the forward pass is a learning step in itself, and this is mathematically equivalent here to the backpropagation. So, somehow there is a chance we can further optimize the architectures of the transformers. We can make it maybe smaller. We can make it more powerful. We have not yet optimized the architecture.
So, if I go here with my image, this is just in my head, whatever image you have. I have here, you know, my tensor structure here, my weight matrix here in, I don't know, 4,000 dimensions. And then I say, "Okay, and then this should be equivalent here to ICL that comes in here, the context, the context activation on the frozen weights." And now I have to understand that these two images are identical. So, this golden context activation are more or less just a rank-one patch that I apply here to my frozen weight tensor structure. So, ICL is mathematically proven equivalent to a rank-one update on the tensor structure. Okay, let's go with this. Okay, now I have further questions. So, you see here how I fight with these things sometimes. And say, "But wait, the activation of our neurons in the layers of the transformers is not a weight tensor update." No, an activation is something completely different. This is the vector that flows through the layers. The weight tensors. If you modify a weight tensor, how is this identical? And the answer is given by the paper. It is possible because they prove mathematically, not in a visual explanation like I try to think. They say it is possible because mathematically we can prove that there is a linearity of matrix multiplication that we can find identical mathematical terms that we can identify.
So, let's do this. It is simple. It is real simple. But it is important that we all understand it. Here in the activation space, we have the classical view. No, let W be the frozen weights here of an MLP layer in a transformer. Let X be your human query here or the embedding here of the last token. And let A be here the output of the attention layer. Attention and A. This is more or less the complete context information. Then a standard transformer. And now let's ignore the nonlinearities here for a second. I'll come back to the nonlinearities in a minute. The input to the MLP is the sum of the query and the attention output. No, via the residual stream, as I showed my you in my last video. So, if you, this is simple. So, we intuitively think now, "Okay, the input X is now shifted here by a particular vector A." Beautiful. And this is it. So, this is our activation space view. So, the term W * A is now a vector representing here the correction, or if you want, the knowledge retrieved now from the added context to our ICL prompt. Now, the authors ask more or less, "Hey, can we find now a matrix W_prime such that this correction comes now from the weights instead from the context input?" So, mathematically, it is so easy. The authors tell us, "We just want to satisfy this equation." Huh? So, we have here the, if you want, the activation space view, and then we have here the weight space view, and we just want to make that this is, can we have this is the same? Yeah, this is it. No, W * A is exactly our delta W * X. X is here the human query, if you want, and A is here the attention here of this additional context, and W is our weight, and delta W is our additional weight structure. So, to solve here for W_prime, we need a matrix that when multiplied with X produces here this particular vector W * A. Uh, distributive property. So, now they tell us mathematically, the authors tell us, "We can pretend that the correction term W * A did not come from the extra vector A, but instead came from an extra weight acting on the original vector X."
So, now we understand why they claim this because they build a mathematical framework where more or less they say, "In the end of all of our calculations, we arrive at this. We arrive here at this term and at this term from two different mathematical spaces, and we just ask, 'Hey, is it possible that our W * A becomes becomes equivalent to our delta W * X?' Can we find this?" And then this is here, if you want, a correction term that is now coming not from the content from the attention, but now we can go and operate with virtual weight structures. So, we have now this bridge suddenly connecting fine-tuning and in-context learning. Now, of course, as I showed you, you can do this here with a rank-one solution. No, if you go with an outer product, simple. No. So, mathematically, we can say adding now the attention output A to the input is identical to updating the weight matrix W by adding here a rank-one matrix delta. So, the activation pattern A from the attention layer becomes the source of the weight update delta W in ICL. So, we have now this beautiful connection between fine-tuning and ICL in a coherent mathematical theory. And now you understand why the authors can claim this identity because mathematically speaking, it is identical. But in physics, you know, whenever we see that we sort that two properties of a physical system are different, and then analyzing it mathematically, we see they are identical, we understand that we made a mistake in building this physical system because either we have neglected symmetries or we have neglected identity operation, but we built something that is not, let's call it, perfect. We built something that works, but we don't really understand why. And now, with this mathematical identity, I have a feeling that what we think is the difference between in-context learning and fine-tuning, what we coded, it turns out is indeed the identical thing, more or less. So maybe we built our transformer architecture just in a wrong way. We didn't utilize that those terms are identical.
Now, you know, we operated here with a simplified kernel representation. And the beauty of the second preprint that I showed you here is the second order here from December 12, that they took this much further, and they prove mathematically that it works here also in a realistic nonlinear deep learning scenarios. So, the second paper is just taking the results of the first paper and proving it mathematically in nonlinear deep learning. But we stay on the same complexity that we try to understand and we try to solve. So, yeah, no, the skeptics, and this is me, I have said at the beginning, no, but you have the layer norm, you have nonlinear activation, you have skip connections. No, and this is why I did not take here this publication by Google here from July 2025 really that significant because I thought, "Hey, the simple algebra doesn't hold if you input nonlinear activations." No, but this team of now proved here just two weeks ago that it holds. And this is something, okay. But they have a second insight: that the update delta W becomes specific to the token position. So, we calculate, given the nonlinearity, a delta W for each single token position. Now, this thing becomes a little bit more interesting. No. So, the weights are effectively different for every single token that is generated by our autoregressive system. The model isn't just fine-tuned once. It is now with ICL continuously refining itself at every step of the generation of the next token prediction of an autoregressive transformer architecture. Therefore, we can now stop treating here the learning, and I mean here the backpropagation, the learning process on the neural network, and the inference, the forward pass, as it would be two different things. It turns out, no. If you understand the mathematics of this, we see the forward pass is a learning step in itself, and this is mathematically equivalent here to the backpropagation. So, somehow there is a chance we can further optimize the architectures of the transformers. We can make it maybe smaller. We can make it more powerful. We have not yet optimized the architecture.
So, if I go here with my image, this is just in my head, whatever image you have. I have here, you know, my tensor structure here, my weight matrix here in, I don't know, 4,000 dimensions. And then I say, "Okay, and then this should be equivalent here to ICL that comes in here, the context, the context activation on the frozen weights." And now I have to understand that these two images are identical. So, this golden context activation are more or less just a rank-one patch that I apply here to my frozen weight tensor structure. So, ICL is mathematically proven equivalent to a rank-one update on the tensor structure. Okay, let's go with this. Okay, now I have further questions. So, you see here how I fight with these things sometimes. And say, "But wait, the activation of our neurons in the layers of the transformers is not a weight tensor update." No, an activation is something completely different. This is the vector that flows through the layers. The weight tensors. If you modify a weight tensor, how is this identical? And the answer is given by the paper. It is possible because they prove mathematically, not in a visual explanation like I try to think. They say it is possible because mathematically we can prove that there is a linearity of matrix multiplication that we can find identical mathematical terms that we can identify.
So, let's do this. It is simple. It is real simple. But it is important that we all understand it. Here in the activation space, we have the classical view. No, let W be the frozen weights here of an MLP layer in a transformer. Let X be your human query here or the embedding here of the last token. And let A be here the output of the attention layer. Attention and A. This is more or less the complete context information. Then a standard transformer. And now let's ignore the nonlinearities here for a second. I'll come back to the nonlinearities in a minute. The input to the MLP is the sum of the query and the attention output. No, via the residual stream, as I showed my you in my last video. So, if you, this is simple. So, we intuitively think now, "Okay, the input X is now shifted here by a particular vector A." Beautiful. And this is it. So, this is our activation space view. So, the term W * A is now a vector representing here the correction, or if you want, the knowledge retrieved now from the added context to our ICL prompt. Now, the authors ask more or less, "Hey, can we find now a matrix W_prime such that this correction comes now from the weights instead from the context input?" So, mathematically, it is so easy. The authors tell us, "We just want to satisfy this equation." Huh? So, we have here the, if you want, the activation space view, and then we have here the weight space view, and we just want to make that this is, can we have this is the same? Yeah, this is it. No, W * A is exactly our delta W * X. X is here the human query, if you want, and A is here the attention here of this additional context, and W is our weight, and delta W is our additional weight structure. So, to solve here for W_prime, we need a matrix that when multiplied with X produces here this particular vector W * A. Uh, distributive property. So, now they tell us mathematically, the authors tell us, "We can pretend that the correction term W * A did not come from the extra vector A, but instead came from an extra weight acting on the original vector X."
So, now we understand why they claim this because they build a mathematical