Transcription
For all their incredible power, most artificial neural networks have a fundamental flaw. They have no concept of time.
Take this network right here. This is AlexNet. When it was unveiled in 2012, it marked a turning point in the history of AI. AlexNet is a deep neural network built for just one thing: scene recognition. You can feed it an image and it spits out a list of 1,000 probabilities telling you what it thinks is in the picture. For example, you show it this picture right here and its output neurons fire up. Most are silent, close to zero, but one neuron, number 29 in the list, lights up with a value near one. We look up class 29 and sure enough, it stands for axolotl. Impressive.
But what if we wanted to analyze a movie? The straightforward approach would be to feed in one frame at a time and look at the predictions. But this method is deeply flawed. Each analysis is completely independent of the rest. The network has no memory and no context. In fact, you could shuffle the movie's frames into a completely random order and the network wouldn't even notice. It is like an expert with an extreme case of retrograde amnesia. It can tell you what it thinks is in the image, but the moment that image vanishes, it forgets it ever existed.
This is a massive problem because it's not how our brains work at all. When we watch a movie, our perception of the current frame is profoundly shaped by the one we just saw before. We build context. We anticipate what's next. We understand the arrow of time. So how do we build a neural network that does the same thing? How do we endow a machine with memory? That is the motivation behind recurrent neural networks. Machines that bake the concept of sequence into their very fabric.
But to understand how we build time into the machine, we first must get a clear picture of the network itself. So let's get a very quick reminder on the classic neural networks. The fundamental building block of a neural network is the neuron. You can think of it as a tiny evidence-waiting machine. It receives incoming signals, multiplies each one by a corresponding weight, and sums them all up, building an internal state. Think of it as voltage building up across a cell membrane. This is where the computation lives.
However, neurons don't communicate their voltage numbers directly to their neighbors. Instead, they convert that internal state into a spike train, a sequence of distinct electrical pulses sent through the wires to other neurons. A mathematical abstraction for this is an activation function, sigma. It takes the internal state and maps it to the actual signal sent downstream. Typically, it might look like a threshold gate, sending only positive numbers through and squashing the negative values to zero.
But a neuron by itself doesn't really do much. To enable useful computations, thousands of these neurons are organized into layers. All neurons in a specific layer look at the exact same signals coming in from the layer before them, but just weight them differently. Writing out the math for every single neuron would be a nightmare of indices. This is where the beautiful shorthand of linear algebra comes in. It allows us to stop thinking about individual neurons and start thinking about the state of the layer as a whole.
Consider any pair of adjacent layers, layer L-1 and layer L. First, let's bundle the internal states of all the neurons in a layer into a single object, a vector. Think of it as a column of numbers representing the internal pressure of every neuron in that layer. The question is: given the state of layer L-1, how do we determine H sub L? Well, layer L doesn't see the raw internal states of the previous layer directly. It sees the signals generated by those states. So, first the previous layer must fire. We apply our activation function to the previous state. Then the signals travel along the connections to the next layer. Since every neuron in layer L-1 connects to every neuron in layer L, these weights form a massive grid of numbers, the weight matrix W sub L. This matrix represents the wiring diagram of a pair of layers. When we multiply this matrix by the incoming signals, we're calculating the weighted sum for every neuron in the new layer simultaneously. This gives us the new internal voltages.
So that entire web of interactions compresses into one elegant equation. We take the old internal state, convert it to the signal through sigma, run it through the wiring with a weight matrix, and that establishes the new internal state. This is the fundamental formula for a feedforward neural network. It's a static, one-way transformation of information. By stacking many of these layers together, we can build a machine that does remarkable things like mapping the pixels of an image to the label of a handwritten digit.
So, we've captured the entire logic of the feedforward network in a single elegant equation. Fire and project, fire and project, layer after layer. But notice something crucial about it. The new state depends only on the signal coming in from the layer before it. It has no knowledge of what happened 5 minutes ago. And this is exactly what we're about to change. Let's introduce time into the equation.
Think about real physical systems like a capacitor or a vibrating membrane of a drum. They don't just reset to zero instantaneously. They carry the echo of their past states. So let's rewrite our fundamental equation for the state of layer L at time T. It is now influenced by what signals the previous layer is sending right now, just like in the feedforward case. But it also senses an echo of its past self. Here, M is a general memory function that describes how states propagate in time. And depending on the choice of M, you get different species of neural networks. Let's think about what would be the most natural choice.
To clearly see things, let's change the layout. The horizontal axis here shows the progression across layers of the network, as before. But now there is a vertical axis that shows the progression of time across the elements of the sequence. On this 2D grid, each node receives two sources of information: an arrow flowing into it from the left, communicated by the previous layer, as well as an arrow flowing into it from the top, information communicated across time from its past self via the M function.
Now imagine you are a researcher inventing this for the very first time and you are pondering what the memory function should be. Here is the most natural choice. Let's take the propagation logic of horizontal arrows and make the vertical arrows have the same functional form, making the grid symmetric. After all, from feedforward networks, we know that this pattern of activation function followed by a linear projection with a set of weights—this fire and project—works pretty well. So let's have a separate set of recurrent weights so that the temporal propagation of state is a fire-and-project transformed copy. In other words, M has the exact same form as the feedforward transformation from one layer to the next. And then the actual state is just a sum of those two similar-looking terms, just with different sets of connection matrices: one for how each neuron in a layer connects to neurons in the next layer, and one for how each neuron connects to its neighbors in that same layer, communicating information across time. And this is exactly what the researchers tried initially in the '80s. This is the vanilla formulation of recurrent neural networks you'd normally find.
However, there is a major problem in practice. While vanilla RNNs can track what happened a few time steps ago, their memory horizon is severely limited. They are fundamentally incapable of learning long-range dependencies. And the reason is baked into the very operation we chose for the echo. Think about what happens to a piece of information as it travels along the vertical axis. At every single time step, it gets passed through sigma and then multiplied by W_rec. That is, it gets processed, squished, rotated, and projected. After 10 time steps, the original signal has been processed 10 times. After 100, 100 times. It's like a game of telephone, but at every step, the message isn't whispered. It's paraphrased, condensed, and reinterpreted.
In hindsight, this shouldn't surprise us. Remember, we chose this memory function by copying it from the feedforward pathway. And the feedforward pathway was designed to throw information away. That is its entire purpose: to map all possible images of cats in different poses, lighting, and on different backgrounds onto the same output. In other words, compression, not preservation. We took the operation that was deliberately built for progressively discarding variation and asked it to do the exact opposite: to preserve information faithfully across time. So no wonder that it fails.
And here lies the key insight. To store information reliably across time, we need a pathway where information can flow without being repeatedly processed, carried forwards, largely intact, with only selective, controlled modifications. In fact, the deep learning community already stumbled upon this exact insight, but in a different context. As vision networks grew, people realized that even across layers, it's useful to preserve some information unchanged. The breakthrough was the residual connection, a direct shortcut that lets a signal bypass the transformation of a layer entirely. This was the revolution that made very deep networks trainable.
Our vanilla RNNs are missing exactly this across time. Instead of a handful of processing stages horizontally, we have hundreds or thousands of time steps vertically. And we need important information to ripple through unchanged. We need a residual connection-like mechanism but for memory.
If you're curious about the people and stories behind the ideas we discussed, from the key breakthroughs in neural network design to the hardware that made it all possible, I'd highly recommend checking out the book "The Thinking Machine" on Shortform, who are kindly sponsoring today's video. Shortform offers in-depth book guides that go way beyond simple summaries. They unpack the key ideas and weave in related insights from other books and research papers, which really helps to see the big picture. Their library covers a huge range of topics from science and technology to psychology, with new guides being published every week, and subscribers actually get to vote on what books to cover next. They also have a browser extension that can generate similar in-depth guides for articles and YouTube videos you encounter online. If you want to supercharge your reading, follow the link down in the video description for a free trial and 20% off the annual membership.
So, what is the simplest echo that preserves information instead of processing it? What if, instead of the fire-and-project operation, the echo is just: keep a fraction alpha of your previous state? This alpha is a single knob that controls memory. Let's explore what happens as we turn it. When alpha equals zero, the echo vanishes. Each time step is independent. We're back to the amnesic feedforward network we started with. When alpha equals one, the state is fully preserved and new input is simply added on top. This looks exactly like the residual connection we were looking for.
So, problem solved? Well, not quite. When the residual connections are used across layers, the number of layers is fixed, say 10 or 50. The network is always the same depth. Every training example passes through the same number of additions, and the network learns to calibrate its own outputs accordingly. The architecture is built around a fixed, known amount of accumulation. Sequences don't have this luxury. A video might be a handful of frames. Or it might be the extended version of Lord of the Rings, half a million frames.
With alpha equals one, the new state equals the previous state plus new input. Unroll it, and the state is a running sum of every input ever received. After 10,000 time steps, it's a pile of 10,000 contributions stacked on top of each other. Nothing is discarded, but nothing is findable either. It's like never throwing away a single piece of mail. Technically, nothing is lost, but your desk is buried, and every single letter is equally inaccessible. This is not memory. This is hoarding.
So, the right value must be somewhere in between. Let's set alpha to be between 0 and 1. And now something interesting happens. Recent inputs remain strong, but older inputs fade exponentially. This is a leaky bucket. Information pours in and slowly drains out. And here is the satisfying twist. This turns out to be nature's favorite memory mechanism. A neuron's membrane voltage works exactly this way. Charge builds up from synaptic inputs and leaks away through ion channels in the membrane. In fact, one of the most widely used models in computational neuroscience, the leaky integrate-and-fire neuron, is precisely this equation.
But this leaky bucket has a problem of its own. Right now, alpha is a single number shared by every neuron and fixed for all time points. But say you're watching a movie. A character's name mentioned once in the opening scene needs to persist for the entire film. The exact framing of each shot is useful right now, but irrelevant a moment later. A single alpha cannot do both. High enough to retain the name, and it also retains a growing pile of stale visual details. Low enough to flush the details, and the name fades too.
What we need is for every neuron to have its own retention rate, one that changes at every time step depending on the context. The fix is to replace the scalar alpha with a vector f(t), one gate per neuron, recomputed at each time step. Notice that the memory function M now takes the input as an argument too, because what you should forget depends on what is arriving. But where does this forget gate come from? It needs to look at both what the layer is currently holding and what's coming in, and produce a number between 0 and 1 for each neuron. We already have a machine that does this: a small neural network with a sigmoid activation. When the neuron's gate is close to one, its state passes almost untouched. When it's close to zero, the old value is erased, making room for new information.
On our 2D grid, the vertical arrows now carry adaptive values, each controlled by a small side circuit that reads both the echo from above and the input from the left, and decides how much of the echo to let through. This gated retention is the core mechanism at the heart of a family of architectures known as gated RNNs. In practice, these architectures often involve additional refinements. The two most prominent members of this family are GRUs and LSTMs. They differ in their specific plumbing. The GRU pairs our forget gate with a complimentary update gate, while the LSTM separates what a neuron knows from what it's shouting to its neighbors by maintaining two state vectors instead of one. But these are engineering choices. The core mechanism in both is the one we just derived: a learned, adaptive valve on the echo. And that single idea—selective, context-dependent forgetting—is what finally gave recurrent networks the ability to learn long-range dependencies.
Looking back, here is what we have done. We started with a static, memoryless network and asked how to give it a sense of time. The answer was a single additional term: the echo. And the entire zoo of recurrent architectures turned out to be different answers to one question: What should the memory function be? A symmetric copy of the feedforward path gives you a vanilla RNN, elegant but forgetful. A fixed scalar decay gives you a leaky integrator, nature's default. But a learned, context-dependent gate gives you the GRU and LSTM networks that can finally choose what to remember and what to forget.
But we've only scratched the surface. We haven't talked about how these networks are actually trained. How do they propagate errors backwards in time? We haven't explored what recurrent networks can teach us about the brain or the fascinating field of reservoir computing, where we leverage the complexity of recurrence without training it at all. But those are stories for future videos. If you enjoyed the video, share it with your friends, subscribe to the channel if you haven't already, and press the like button. Stay tuned for more computational neuroscience and machine learning topics coming up.