Transcription
This is an activation Atlas. It gives us a glimpse into the high-dimensional embedding spaces modern AI models use to organize and make sense of the world. The first model to really see the world like this, AlexNet, was published in 2012 in an 8-page paper that shocked the computer vision community by showing that an old AI idea would work unbelievably well when scaled. The paper's second author, Ilya Sutskever, would go on to co-found OpenAI, where he and the OpenAI team would massively scale up this idea again to create ChatGPT.
This video is sponsored by KiwiCo; more on them later. If you look under the hood of ChatGPT, you won't find any obvious signs of intelligence. Instead, you'll find layer after layer of compute blocks called transformers. This is what the "T" in GPT stands for. Each transformer performs a set of fixed matrix operations on an input matrix of data and typically returns an output matrix of the same size. To figure out what it's going to say next, ChatGPT breaks apart what you ask it into words and word fragments, maps each of these to a vector, and stacks all of these vectors together into a matrix. This matrix is then passed into the first transformer block, which returns a new matrix of the same size. This operation is then repeated again and again—96 times in ChatGPT 3.5 and reportedly 120 times in ChatGPT 4.
Now here's the absurd part: with a few caveats, the next word or word fragment that ChatGPT says back to you is literally just the last column of its final output matrix, mapped from a vector back to text. To formulate a full response, this new word or word fragment is appended to the end of the original output, and this new, slightly longer text is fed back into the input of ChatGPT. This process is repeated again and again, with one new column added to the input matrix each time, until the model's output returns a special stop word fragment. And that is it. One matrix multiply after another, GPT slowly morphs the input you give it into the output it returns.
Where is the intelligence? How is it that these 100 or so blocks of "dumb" compute are able to write essays, translate language, summarize books, solve math problems, explain complex concepts, or even add the next line of this script? The answer lies in the vast amounts of data these models are trained on.
Okay, pretty good, but not quite what I wanted to say next. The AlexNet paper is significant because it marks the first time we really see layers of compute blocks like this learning to do unbelievable things—an AI tipping point towards high performance in scale and away from explainability. While ChatGPT is trained to predict the next word fragment given some text, AlexNet is trained to predict a label given an image. The input image to AlexNet is represented as a three-dimensional matrix, or tensor, of RGB intensity values, and the output is a single vector of length 1,000, where each entry corresponds to AlexNet's predicted probability that the input image belongs to one of the 1,000 classes in the ImageNet dataset—things like tabby cats, German Shepherds, hot dogs, toasters, and aircraft carriers.
Just like ChatGPT today, AlexNet was somehow magically able to map the inputs we give it into the outputs we wanted using layer after layer of compute blocks after training on a large dataset. One nice thing about vision models, however, is that it's easier to poke around under the hood and get some idea of what the model has learned. One of the first under-the-hood insights that KFY Sutskever and Hinton show in the AlexNet paper is that the model has learned some very interesting visual patterns in its first layer. The first five layers of AlexNet are all convolutional blocks, first developed in the late 1980s to classify handwritten digits, and can be understood as a special case of the transformer blocks in ChatGPT and other large language models.
In convolutional blocks, the input image tensor is transformed by sliding a much smaller tensor called a kernel of learned weight values across the image and, at each location, computing the dot product between the image and kernel. Here, it's helpful to think of the dot product as a similarity score: the more similar a given patch of the image and kernel are, the higher the resulting dot product will be. AlexNet uses 96 individual kernels in its first layer, each of dimension 11 by 11 by 3, so conveniently we can visualize them as little RGB images. These images give us a nice idea of how the first layer of AlexNet sees the image. The upper kernels in this figure show where AlexNet has clearly learned to detect edges or rapid changes from light to dark at various angles. Images with similar patterns will generate high dot products with these kernels. Below, we see where AlexNet has learned to detect blobs of various colors. These kernels are all initialized as random numbers, and the patterns we're looking at are completely learned from data.
Sliding each of our 96 kernels over the input image and computing the dot product at each location produces a new set of 96 matrices, sometimes called activation maps. Conveniently, we can view these as images as well. The activation maps show us which parts of an image, if any, match a given kernel. Well, if I hold up something visually similar to a given kernel, we see high activation in that part of the activation map. Notice that it goes away when I rotate the pattern by 90°; the image and kernel are no longer aligned. You can also see various activation maps picking up edges and other low-level features in our image.
Of course, finding edges and color blobs in images is still hugely removed from recognizing complex concepts like German Shepherds or aircraft carriers. What's astounding about deep neural networks like AlexNet and ChatGPT is that from here, all we do is repeat the same operation again, just with a different set of learned weights. For AlexNet, this means that these 96 activation maps are stacked together into a tensor that becomes the input to the exact same type of convolutional compute block—the second overall layer in the model. We can make our activations easier to see by removing the values close to zero. Unfortunately, in our second layer, we can't learn much by simply visualizing the weight values and the kernels themselves.
The first issue is that we just can't see enough colors. The depth of the kernel has to match the depth of the incoming data. In the first layer of AlexNet, the depth of the incoming data is just three because the model takes in color images with red, green, and blue color channels. However, since the first layer computes 96 separate activation maps, the computation in the second layer of AlexNet is like processing images with 96 separate color channels. The second factor that makes what's happening in the second layer of AlexNet more difficult to visualize is that the dot products are really taking weighted combinations of the computations in the first layer. We need some way to visualize how the layers are working together.
A simple way to see what's going on is to try to find parts of various images that strongly activate the outputs of the second layer. For example, this activation map appears to be putting together edge detectors to form basic corners. Remarkably, as we move deeper into AlexNet, strong activations correspond to higher and higher-level concepts. By the time we reach the fifth layer, we have activation maps that respond very strongly to faces and other high-level concepts. What's incredible here is that no one explicitly told AlexNet what a face is. All AlexNet had to learn from were the images and labels in the ImageNet dataset, which does not contain a person or a face class. AlexNet was able to learn completely on its own both that faces are important and how to recognize them.
To better understand what a given kernel in AlexNet has learned, we can also look at the examples in the training dataset that give the highest activation values for that kernel. For our face kernel, not surprisingly, we find examples that contain people. Finally, there's this really interesting technique called feature visualization, where we can generate synthetic images that are optimized to maximize a given activation. These synthetic images give us another way to see what a specific activation layer is looking for. By the time we reach the final layer of AlexNet, our image has been processed into a vector of length 4,096. The final layer performs one last matrix computation on this vector to create a final output vector of length 1,000, with one entry for each of the classes in the ImageNet dataset.
KFY Sutskever and Hinton noticed that the second-to-last layer vector demonstrated some very interesting properties. One way to think about this vector is as a point in 4,096-dimensional space. Each image we pass into the model is effectively mapped to a point in this space. All we have to do is just stop one layer early and grab this vector. Just as we can measure the distance between two points in 2D space, we can also measure the distance between points or images in this high-dimensional space. Hinton's team ran a simple experiment where they took a test image in the ImageNet dataset, computed its corresponding vector, and then searched for the other images in the ImageNet dataset that were closest, or the nearest neighbors, to the test image in this high-dimensional space. Remarkably, the nearest neighbor images showed highly similar concepts to the test images. In figure four from the AlexNet paper, we see an example where an elephant test image yields nearest neighbors that are all elephants.
What's interesting here, too, is that the pixel values themselves between these images are very different. AlexNet really has learned high-dimensional representations of data where similar concepts are physically close. This high-dimensional space is often called a latent or embedding space. In the years following the AlexNet paper, it was shown that not only distance but directionality in some of these embedding spaces is meaningful. The demos you see where faces are age or gender shifted often work by first mapping an image to a vector in an embedding space and then literally moving this point in the age or gender direction in that embedding space, and then mapping the modified vector back to an image.
Before we get into activation atlases, which give us an amazing way to visualize these embedding spaces, please take a moment to consider if this video sponsor is something that you or someone in your life would enjoy. I was genuinely really excited to work with this company; they make incredibly thoughtful educational products. By using the link in the description below, you're really helping me make more of these videos. This video sponsor is KiwiCo. They make these fun and super well-designed educational crates for kids of all ages. They have nine different monthly subscription lines to choose from, focusing on different areas of STEAM, and you can also buy individual crates, which are great for trying out KiwiCo and make amazing gifts.
Growing up, I was constantly building. Here I am building a tower outside my house to my second-story bedroom. I was obsessed with electronics and would have absolutely loved projects like this pencil sharpener from the Eureka crate line, which is focused on science and engineering. I really believe that this type of hands-on, self-driven learning is magical. When I really think about my own education, it's the times that I've been fully absorbed in projects like this that I learned the most. Now that I'm a dad, I really want my kids to have the same kind of experiences. KiwiCo really does an amazing job boxing up start-to-finish projects like this. My daughter just got the Panda crate for fine motor skills; it includes these special crayons specifically designed to help her learn different ways of grasping. You can see her here insisting that she gets to bring them in the car with us. Huge thanks to KiwiCo for sponsoring this video. Use the discount code "Welch Labs" for 50% off your first month of a subscription.
Now back to AlexNet. There's some really amazing work that combines the synthetic images that maximize a given set of activations with a two-dimensional projection, or flattening out, of the embedding space to make these incredible visualizations called activation atlases. Neighbors on the activation atlas are generally close in the embedding space and show similar concepts the model has learned. We're getting a peek into how deep neural networks organize the visual world. Looking at the synthetic images that most activate neighborhoods of neurons, we can visually walk through the embedding space of the model, seeing it make smooth visual transitions from concepts like zebras to tigers to leopards to rabbits.
Moving to the middle layers of the model, we can see less fully formed but still meaningful concepts. Moving along this path amazingly correlates with the number and size of pieces of fruit in an image. The same principle applies in large language models: words and word fragments are mapped to vectors in an embedding space where words with similar meanings are close to each other, and the directions in the embedding space are sometimes semantically meaningful. There's some incredible, very recent work from the team at Anthropic that shows how sets of activations can be mapped to concepts in language. These results can help us better understand how LLMs work and can be used to modify model behavior. After clamping a set of activations that correspond to the concept "Golden Gate Bridge" to a high value, the LLM the team was experimenting with began to identify itself as the Golden Gate Bridge.
AlexNet won the ImageNet Large Scale Visual Recognition Challenge by a wide margin in 2012, the third year the challenge was run. In prior years, the winning teams used approaches that, under the hood, looked much more like what you might expect to find in an intelligent system. The 2011 winner used a complex set of very different algorithms, starting with an algorithm called SIFT, which is composed of specialized image analysis techniques developed by experts over many years of research. AlexNet, in contrast, is an implementation of a much older AI idea: an artificial neural network where the behavior of the algorithm is almost entirely learned from data. The dot product operation between the data and a set of weights was originally proposed by McCulloch and Pitts in the 1940s as a dramatically oversimplified model of the neurons in our brain.
In the second half of each transformer block in ChatGPT and at the end of AlexNet, you'll find a multi-layer perceptron. The perceptron is a learning algorithm and physical machine from the 1950s that uses McCulloch and Pitts neurons and can learn to perform basic shape recognition tasks. Back in the 1980s, a younger Jeff Hinton and his collaborators at Carnegie Mellon showed how to train multiple layers of these perceptrons using a multivariate calculus technique called backpropagation. These models, a couple layers deep, were remarkably pretty good at driving cars. In the 1990s, Yann LeCun, now Chief AI Scientist at Meta, was able to train five-layer deep models to recognize handwritten digits.
Despite the intermittent successes of artificial neural networks over the years, this approach was hardly the accepted way to do AI right up until the publication of AlexNet. If this was obviously the way to build intelligent systems, we would have done it decades earlier. As Ian Goodfellow writes in his excellent deep learning book, at this point, deep networks were generally believed to be very difficult to train. We now know that algorithms that have existed since the 1980s work quite well, but this was not apparent circa 2006. The issue is perhaps simply that these algorithms were too computationally costly to allow much experimentation with the hardware available at the time.
The key difference in 2012 was simply the scale of data and scale of compute. The ImageNet dataset was the largest labeled dataset of its kind to date, with over 1.3 million images, and thanks to Nvidia GPUs in 2012, Hinton's team had access to roughly 10,000 times more compute power than Yann LeCun had 15 years before. LeCun's LeNet-5 model had around 60,000 learnable parameters; AlexNet increased this a thousandfold to around 60 million parameters. Today, ChatGPT has well over a trillion parameters, making it over 10,000 times larger than AlexNet. This mind-boggling scale is the hallmark of this third wave of AI we find ourselves in today, driving both their performance and the fundamental difficulty in understanding how these models are able to do what they do.
It's amazing that we can figure out that AlexNet learns representations of faces and that large language models learn representations of concepts like the Golden Gate Bridge, but there are many, many more concepts these models learn that we don't even have words for. Activation atlases are beautiful and fascinating but very low-dimensional projections of very high-dimensional spaces, where our spatial reasoning abilities often fall apart. It's notoriously difficult to predict where AI will go next. Almost no one expected the neural networks of the 80s and 90s, scaled up by three or four orders of magnitude, to yield AlexNet, and it was almost impossible to predict that a generalization of the compute blocks in AlexNet, scaled up by four orders of magnitude, would yield ChatGPT.
Maybe the next AI breakthrough is just another three to four orders of magnitude of scale away, or maybe some mostly forgotten approach to AI will resurface as AlexNet did in 2012. We'll have to wait and see. Are you mad that I called the blocks of compute "dumb"? Not at all. Describing the compute blocks as dumb highlights the impressive nature of how simple operations can combine to produce intelligent behavior. It's a great way to emphasize the power of the underlying algorithms and training data.