Transcription
The takeaway from this is that if you're going to be a startup working on kind of new computing technology, you shouldn't be aiming for a 10x. You should be aiming for a 10,000x.
Hello everyone. Uh, I'm Trevor from Extropic. Um, I'm going to tell you a little bit about what we've been doing in the last two years, which is shrouded in extreme mystery. Cool.
So, I don't know if you guys have heard, but this whole AI thing is kind of big. A lot of people talking about it, and you wouldn't believe the things they're saying. It's crazy. Yeah. No, I was looking into this recently for something else, and I found that the big tech CEOs are actually making a lot of really specific claims about what they think the near future is going to look like. And you know, you you can read through this yourself, but the kind of central pattern is they're all predicting that we're all going to be using AI basically all the time um within, you know, a few years from now, right? And that's awesome. I think I think everybody in this room wants that and thinks that would be cool. And you know, the result of that is when they say stuff like that, stock price goes up and everyone's happy, right?
But when they make these kind of claims that tease up uh physicists like me to kind of try and do a firmy estimate of what the cost would actually be of that kind of future, right? Like what would it actually take for all of us to have an AI assistant that we're using all the time every day, right? Because there's a cost to running AI models as we all know because we're here. AI uses electricity, right? And the more you use AI, the more electricity you draw. And specifically the type of models that most of us are using most of the time um actually have this really easy way to predict the uh resource utilization that's behind them. Right? So you know for transformers basically you can come up with a simple formula that tells you how much power you would need to supply a data data center or many data centers um to process a particular amount of tokens per second. Right? And basically you can come up with this kind of you know Drake equation type model that will tell you exactly how much power you would need to supply the world with the particular token throughput um for a particular size of a transformer model.
And if you plug in all of the right parameters that you would need for, you know, all of us to have kind of an agentic AI system running all the time, even if that system is only processing all of the text we interact with every day, what you'll find is that those AIs are going to be drawing about half of the total grid, right? So half of all of the power in the US would be going towards these systems. And if you extend this even just to one fps video, which is some uh what someone like Mark Zuckerberg is really betting on with his glasses, you can come up with truly insane numbers like, you know, 10x, 20x, 30x the total grid. And that's just because, you know, these models are really big and really power hungry. And if you extend this further to, you know, the the type of robotics stuff that people were talking about this morning, the numbers would go off the chart, right? So hundreds of thousands of times more power than we can supply today. And you know obviously this is all the wild extrapolation and I'm off by a lot I'm sure but this is kind of just giving an order of magnitude of what's actually possible right and so you know if you look at at um models that incorporate kind of supply side constraints in terms of power and hardware their most aggressive projections of growth in the next few years um only go up to about 10% of the total grid right so there's clearly a huge gap between expectation and reality and you know these these really high estimates of the video models, they correspond to about a 20,000 gawatt draw compared to the 500 gawatts that we use today. And again, you shouldn't you should take all of these numbers with like a whole cup of salt because I'm extrapolating wildly, but I think you get the point.
And so, if we want to actually realize this future that I think we all want, um, we're going to need a lot more resources. And our two options are basically to produce a lot more energy or to make computers a lot more efficient. Spoiler, these are both really, really hard problems.
So, in terms of more energy, basically every big tech company has started to kind of also become a energy supplier, right? So, pick a name, Microsoft, Amazon, whatever. Um, they all have projects underway where they're bringing up like gigawatt scale energy production facilities for data centers. And that's great. There there's two problems with this, you know, especially from my perspective as someone who's involved with startups. Energy tech is a really really hard path for a startup. Not something that I would ever do, although I think there's probably people here this week that are. And the other thing is that, you know, a one gigawatt power project is massive, but from the math I just showed you, even if I'm off by a factor of 10 or 100, that's not even close to enough to what we actually need, right? So, if we're going to go the power production route, this is going to be like a a massive global scale effort. you know that's going to conclude in something that looks like a Dyson sphere which I think will happen anyways but it's going to take a really long time.
The other option is making computers much more efficient. And this is a really hard technical problem which I'll get to in a second. But the other thing that makes this problem really challenging is uh it's really hard to find people that are capable of meaningfully working on this because you kind of need to be an expert in both, you know, hardware and device physics and also algorithms, right? So this this is um potentially one of the like most technically all-encompassing things you can work on today. The other thing, and this is the real kicker as a startup is that GPUs are getting better at a very predictable exponential rate. If you just plot the GPU, you know, flops per jewel as a function of time, it's uh doubling about every two years. Despite all the naysayers that thought this was going to stop, it just keeps going because engineers are resourceful, right? So, if I'm a startup that's trying to go out and build a more efficient type of computing, I'm fighting against this kind of constant incremental gain that will just kind of bull me over if I'm not fast enough, right? And so, the takeaway from this is that if you're going to be a startup working on kind of new computing technology, you shouldn't be aiming for a 10x. You should be aiming for a 10,000x, but nobody will fund that. So you see there's a lot of kind of society scale problems that make this this particular thing hard.
Now from a technical perspective, why is it hard to make a computer more efficient? This problem is actually kind of a bit of a rock in a hard place. So if you look at where energy gets spent in a computer today, it's actually mostly in charging capacitors, right? So computers are made out of wires and metal. And when the voltage in a piece of metal changes, there's a energy change associated with that. And today that energy kind of just gets dumped when you discharge a capacitor, right? And so if you want to make that energy smaller, you can either make capacitance smaller or you can make voltage smaller. You can't make capacitance smaller anymore. We've kind of made that as small as we can for various reasons. It's actually quite a complex problem. And voltage is actually constrained by some pretty fundamental uh thermodynamic considerations about how transistors work, right? So there's this kind of intrinsic noise to transistors as a scale that's called VT. It's just the the thermal voltage. And as as V starts to approach VT, you your signal will start to be consumed by this noise. Moreover, from the same physics, these this thermodynamic effect limits the off current in a transistor. Right? So when a transistor is switched off, it actually still conducts some electricity. And the amount of electricity that gets conducted is actually an exponential function of what's called the threshold voltage, which is the on voltage of the transistor. Right? So as I as I drop the threshold voltage to make this V squared term smaller, there's another uh source of energy consumption that gets exponentially larger. Right? So you're when you're designing a computer these days, you're kind of boxed in on both sides by these two conflicting constraints. This is a a pretty hard problem.
Um, a lot of ways that people try and solve this are by getting away from this one half CV squared entirely. Right? So that's that's kind of the angle of these photonic computing approaches that try to use light because light doesn't interact with anything. Um, this is also the angle for adabatic logic. They try and get rid of this charging scheme. You can go even crazier to stuff like quantum computing which is a completely different paradigm. very low voltages but you know very very hard problem and this is kind of this this is probably like the most important equation in the world right now if you want to solve this.
I think what you have to do is look back on why are things the way they are today right from both an algorithm and hardware perspective and if you really think it through you'll realize that there's no fundamental reason why anything is the way it is today it was really just a series of coincidences you know back in 2012 there was this original work where people figured out GPUs were really useful for deep neural networks And that kind of set off this cascade where AI researchers would push the frontier of algorithms and they'd want better GPUs. So then Nvidia would make better GPUs, right? And then they do even crazier stuff with those GPUs. And then you end up with some kilowatt monstrosities that data centers are full of, right? And we look at that now and we're like, well, we're so we're kind of we're so entrenched in this way of doing things that we can't even think about doing something else, right? And so we're kind of stuck. And so I think at this point tech companies are bringing up data centers right to power this feedback loop. I think it's very prudent to look back and ask is there a better way to do things that has you know nothing to do with the current paradigm.
And so that's kind of what the way we're approaching things at Extropic which is these thermodynamic limitations about transistors. We're kind of asking is there a way we can use that to our advantage right instead of having it form this kind of brick wall that blocks progress. And the way that you could consider doing that is by kind of using this intrinsic thermal noise and transistors as a computational resource instead of having it just be a burden, right? And so what that corresponds to is building a probabilistic computer. So a computer that doesn't compute deterministically, it doesn't comput using deterministic signals. It computes using probabilistic signals. And from the perspective of machine learning, this actually makes a lot of sense because all of the algorithms that we're dumping all this energy into are probabilistic, right? Like transformers or auto regressive LLM sample from distributions. It's even a bit more on the nose when you look at diffusion models because those are really like a a simulation of a noisy physical system, right? So there's a lot of connections that you can draw between physics and uh machine learning that make this kind of probabilistic hardware actually quite natural.
And so you know what we've been working on over the last few years is actually realizing kind of a minimal version of this. And what I'm showing on the screen here is kind of a hello world type uh demo of our tech. we have kind of put together this probabilistic hardware system um that performs a generative modeling task and we've compared it head-to-head against traditional GPU based algorithms like VAEEs and GANs on a simple benchmark which is this so-called fashion emnest data set right the kind of punch line is if if you want performance par between our hardware and a VAE you need to use about four times more energy than us or sorry four orders of magnitude not four times that's an important detail and you you know, this kind of improvement is at the scale that we actually need to solve this problem I pointed out, right? And Nvidia is not going to beat orders of magnitude, right? So, this this thing actually has a chance of getting off the ground. And that's really what we're going to be striving for for the next few years is uh building a actual computer that does this instead of just a simulation of one.
And so, I'm just going to talk a little bit about what what is behind that plot there. Uh, this is going to be a bit of a teaser because I don't have time. Um, but there's going to be a paper out on this sooner rather than later. And so the key thing here is merging energy based models which are a you know not so popular form of machine learning with denoising diffusion which is extremely popular. And the idea here is that it turns out it's very straightforward to build a piece of hardware that implements a probabistic piece of hardware that implements energy based models at a very low level. But energy based models are not a good model class on their own because it turns out as you try and use an EBM to fit more and more complex data sets, it becomes uh completely intractable to sample from. Right? So the amount of energy you have to expend to sample from a complex EBM grows extremely rapidly. And this is kind of why EBMs haven't taken off as generative models of their own. But if you go if you look at how denoising model works, the way it works is basically you start with your data at time zero and you gradually add noise to it until you get a pure noise, right? So you convert your image into noise and then you try and train a machine learning system to go backwards from the noise to the image, right? And it turns out, you know, if you look at the math of this, um, if you make the time step small enough in this forward process, you end up with a really simple distribution you can sample from like a gausian. Now, if you want to take bigger steps, you need to be able to sample from a richer distribution, which you can do with an EVM. And I don't have time to totally flush this out, but the idea is you can use our probabilistic hardware to approximate these large steps in the reverse process of a denoising model. And what this does is it it alleviates this kind of intractable sampling problem that EBMs normally have. Right? So by merging our hardware with these den noising models, we basically have it have a way to use our hardware efficiently to solve a problem in machine learning. And this is a bit much to explain in four minutes. So you'll have to stick with that and read the paper when it comes out.
And so the kind of key technology we had to develop to make this possible was a good source of randomness that is mass manufacturable. Right? So this see might seem a bit contradictory because I just told you that a big problem with transistors is that they're noisy. But it turns out that actually harnessing this noise and using it to do some kind of computational task is difficult and is a you know pretty tricky engineering problem. And so this is something that we have kind of cracked over the last two years. uh basically we figured out how to harness the noise and transistors to sample from computationally useful distributions and in fact we actually you know built a test chip recently and and put a lot of this theory to practice and validated it. So you know basically we can build a circuit um that's just a handful of transistors. It takes up a few square microns on a chip and that circuit acts as a programmable source of random bits. Right? So like it's a it generates ones or zeros and I can bias the probability of whether or not it's one or zero electrically. And this random number generator is is brutally efficient, right? So it's it's around 10,000 times more efficient than a pseudo random number generator use on a CPU or something, right? So really really low energy consumption. And so when you combine this really efficient RNG with the uh probabilistic hardware architecture that I barely described a minute ago, you get that plot that I showed you earlier. Again, kind of a tease. You're going to have to read the paper, but I hope that gets you excited at least.
And the plot I showed you earlier was a kind of simple data set, a bit of a hello world. And you know the reality of this is that this probabilistic thing is probably a subruine in a bigger machine learning system. It's probably not carrying the whole load by itself, right? And so this slide is a bit of a research snapshot, not really a complete result where we're we're looking into how do we combine these probabilistic computing subutines with traditional neural networks to solve more complex problems. And so you know what we're looking at here is we're using a combination of autoenccoders and GANs to embed a less trivial data set this CR10 which is about 10 years more advanced than the last one into the space of one of these probabilistic computers and kind of do this generative modeling in latent space right and so you can think about it as you know a GAN usually starts from Gaussian noise and shapes that into whatever an image instead of seeding the GAN with just Gaussian noise we seed it with the the output of this den noising process right so the noise has a lot of structure and correlation that's useful and by doing that you can save uh quite a bit of deterministic compute over a traditional solution. So you know this snapshot here shows about an order of magnitude improvement but I think this can be pushed a lot further and that's kind of what we're working on right now and that's you know mostly why I came to this conference is to try and get people excited about this and uh come on board.
So you know right now I'm looking to bring on kind of a cohort of research residents to push this hybrid machine learning angle. If you actually join, you'll have more than 15 minutes to understand what I'm talking about and and maybe you'll get a real idea of what we're doing. Extropic is a a really awesome company, very small, very elite. We have this amazing analog design team that has over 75 years of combined experience that unlocked this uh novel hardware that I was talking about. We've got some really specialized talent in transistor noise modeling. And we have some really excellent proistic machine learning researchers that kind of put all of this work together. And there's just a ton of work to do. So if if if working on this kind of pure probabilistic machine learning excites you, I encourage you to scan this QR code here and it'll take you to our website. Also exciting, over the summer, you're going to start hearing a lot more about us. And there's actually going to be kind of an early access program for our hardware and algorithms and uh this QR code will take you to kind of like a alpha signup page for that.
Cool. So, thanks for uh your attention and feel free to come talk to me later if you have questions. [Applause]