Transcription
[music] Jeff Dean, thank you for joining us here in Sunday San Diego, right in front of the Nerves conference center. You are the chief scientist at Google, Kotech lead, and you all recently made an announcement about a version of the new TPU chip.
>> Yeah.
>> Let's talk about it. The seventh generation of TPUs.
>> Yeah. Y
>> What's special about it? Uh, I mean, like every next generation of TPU, it's better than the previous one, and you know, it has, uh, quite a lot of new capabilities. Uh, it has, you know, it's connected together into these very large configurations that we call pods. Um, I think it's 9216 chips or something like that per pod. Um, and it has, you know, much higher performance, especially for lower precision floating-point formats like FP4. Um, so that's going to be really useful for training large models, uh, for inference, for a lot of things like that. So, we're pretty excited about it.
>> Nice. If zooming out, Google started building TPUs for their own internal needs. Google's pre-eminent AI applications company, AI research organization in the world, and the need for control of the full vertically integrated stack was the original motivation, as I understand it, if I've read about it. And then eventually externalizing access to those to be a global competitor in the ecosystem of accelerators, people who build and sell accelerators. And now there's a lot of people excited about the opportunity for there to be a a massive market for TPUs. How do you relate with your role at Google to the objectives of Google's internal use of TPUs versus the marketplace that you're competing in to to kind of compete and enable millions and billions of people outside of Google to get the advantages through reselling TPUs in the competitive space?
>> Yeah, I mean, uh, the origin of the TPU program was really for our own internal needs, initially focused on inference. So, even as far back as 2013, you know, we saw that, uh, kind of deep learning methods were going to be very successful, and every time we trained a slightly larger model with more data, the results got better in things like speech and vision. And, uh, I started to do some back-of-the-envelope calculations of like, what would happen if we actually wanted to serve this much better speech model that's more compute-intensive to say, 100 million users for a few minutes a day. And [snorts] the compute requirements got quite scary. Uh, we would actually need to double the number of computers Google had overall in order to just roll out this improved speech model.
>> Wow.
>> Um, if we wanted to do it on CPUs. And so that was really the the genesis of, hey, if we build specialized hardware that is tailored for these kinds of ML, uh, computations, you know, essentially dense, low-precision linear algebra, um, we could actually do be way more efficient. And that was borne out. The first TPU ended up being 30 to 70 times more energy-efficient than contemporary CPUs or GPUs and 15 to 30 times faster.
>> And that was 2015, you said?
>> Yeah. So we started, uh, the thought experiment was 2013. The chips that landed in our data center in 2015, and we wrote a paper about.
>> Pre-transformer architecture?
>> Pre-transformer. Yeah. So, uh, we actually were focused on speech recognition and kind of vision convolutional models at the time. We squeezed in a little bit of design change at the last minute into the TPUv1 to make it support LSTMs as well, uh, which were kind of in vogue at the time for language modeling, and that also enabled us to support, uh, language translation tasks. And then subsequent versions of TPUs have focused much more on much larger scale systems that are not just a single PCIe card, but are, you know, a whole machine learning supercomputer, including the latest Ironwood one.
>> Um, and, you know, every generation has been a big improvement in both energy efficiency, performance per dollar, all these things that we that we care about, um, and enable us to scale much larger training jobs, much, you know, more serving of of requests to lots of users.
>> And of course, the transformer architecture itself, born at Google, pretty similar timeline, but with the TPU invented before that, and then transformer architecture happening. Do you think there was serendipity in terms of co- uh, design between the applications of the transformer architecture as they've grown up to change the world as we know it now, and Google's access to this vertically integrated hardware stack?
>> Yeah, I mean, every every generation of TPU, we really try to take advantage of the co-design opportunities we have with, you know, having a lot of researchers thinking about where are, you know, ML computations we're going to want to run, you know, two and a half to six years from now. Going, which is the the exercise you have as a hardware designer is like trying to predict a very fast-moving field. It's not a very easy thing. But having a lot of people kind of seeing where the field is going or, you know, this kind of thing might be interesting. We're not quite sure yet, but we could put in this kind of hardware feature or this particular kind of capability, and if we did that and this turned out to be important, then we could have the hardware support there ready when that thing, you know, hopefully bears out, you know, that it is an important thing. And if it doesn't pay off, then sometimes you've just devoted maybe a small area, piece of the chip area to this thing that turned out to be less important than you thought. But you really do want to be prepared for if this thing matters a lot, your hard work can support it.
>> Yeah.
>> So it's an interesting forecasting exercise is forecasting the whole ML field and trying to guess what we want. Well, if we could let one person do it, uh, Chuck Norris of Computer Sciences would get my vote, and has has obviously enough votes that that you are doing it at Google. And with your track record at Google, there's a legacy of inventing things for Google's internal needs. Google being the world's best systems building company for the applications Google's built, which have now become many-headed map-producing Google file system, something you did inside, you co-invented or invented inside of Google. And then eventually you have been able to witness that what Google built and demonstrates to the world as value and then publishes with the TPU architecture, obviously the transformer, the ideas in the transformer are paper themselves. But now do you think that there's a tipping point with Ironwood for the rest of the world to sort of have access to the advantages that Google has had? And I would imagine if I put myself in your shoes, this experience where it's like, that was awesome, and we did it at Google, and we paved the way, and now like, holy moly, the rest of the world is also getting all the benefits. Researchers think about impact, and that feels like the moment we live for, to be able to have it. And if you're if you feel like you're at the tipping point on the TPU moment.
>> Yeah, I mean, I think obviously we've been using TPUs now for more than a decade, or about a decade, and been really happy with them. And the co-designed properties really make them sort of useful for the kinds of machine learning computations we want to run. And we've also been renting them externally through our Cloud TPU program for a number of years, and so many, many customers are using them for all kinds of things. Uh, we've built a bunch of software layers on top of TPUs that make them sort of quite convenient and easy to use. So you have, I mean, the most well-worn path for TPUs is JAX on top of Pathways, which is an internal system we've built that, uh, we're sort of working to see if cloud customers would want, uh, access to,
>> On top of XLA, which is a compiler, ML compiler with a TPU backend. And so what this, um, tends to mean, at least for Pathways, you know, all of our Gemini development and research and training large-scale training jobs run on top of that stack. And Pathways is this nice system that we we built, uh, I guess starting about seven years ago, that gives you the illusion of a single system image across, you know, thousands or tens of thousands of chips. And so you can have like a single Python process running your JAX code, and instead of it showing up as four devices where you're running on a single TPU node, it shows up as your JAX process has access to 20,000 devices. And you, it just sort of naturally works and figures out underneath the covers exactly what transfer mechanisms to use and which, you know, which network to use. It should use the within a pod, a TPU pod, it should use the high-speed interconnect, and across pod boundaries, it'll use the data center network, across metropolitan areas, it'll use long-distance links, and so on.
>> Um, so we actually run, you know, very large-scale training jobs where we have a single Python process driving multiple TPU pods in multiple cities.
>> Nice. Great. Well, maybe we can shift topics.
>> Sure.
>> You've been talking a lot, I think, lately about the state of funding for academic research.
>> What's your message?
>> Yeah, I mean, uh, actually my colleagues, Hoza and Partha Rangadath, and I, along with Magda Balazinski at the University of Washington, recently published, uh, one article in a whole special issue of the Communications of the ACM that was devoted to, you know, uh, the impact of, um, you know, academic research. And in our section, we discussed all the academic research that Google, as a company, was built on, you know, all the things that we relied on in terms of like TCP/IP and, you know, uh, you know, advanced risk processors, and, you know, the internet, and, uh, the Stanford Digital Library Project, which is what sort of, uh, provided the funding for the original version of PageRank at Stanford.
>> Oh, yeah.
>> And my colleague Dave Patterson also had an article in that issue about all the amazing things that have come out of his, um, and his Berkeley colleagues, many different five-year labs. And so it's just really important, I feel, to have, you know, a vibrant academic, uh, research, uh, ecosystem, in the US and also in the world, because that often, those early-stage creative ideas are the things that lead to major, major, uh, breakthroughs and innovations. You know, the whole of the deep learning revolution actually built on academic research from 30, 40 years ago. You know, the inventions of neural networks and backpropagation and things like that are all, you know, central to what what we're doing even today and have been really important in the world. So, you know, I I advocate that we should have a vibrant, uh, academic funding model for academic research because the returns are quite large to society.
>> Yeah. Excellent. And you and I and Dave Patterson and Joel Pin know are on the board of L Institute, which was born in part out of a paper that you and Dave and I and a bunch of seven other authors published called "Shaping AI's Impact on Billions of Lives," where we advocated for the ways that AI research might impact society in areas like civic discourse and healthcare and science and job reskilling and journalism and more policy. And then we also advocated that there, in addition to things like 10xing down on NSF-style funding, we can explore and, uh, prototype other types of funding. So L Institute raises money from successful technologists who donate to a non-L Institute, which is a nonprofit 501(c)(3), which then in turn is running a moonshot grant program specifically dedicated to funding research labs, three to five-year research labs with three to five PIs, 30 to 50 PhD students, targeting AI's impact on society in those areas I just said: scientific progress, healthcare, job reskilling, and civic discourse. And you've been an an advocate for these alternative funding models as well, in addition to the traditional ones.
>> It was a lot of fun working on that paper with you and and Dave and, uh, the many other co-authors we had. You know, I think the, um, the thing I liked about that paper is we looked at a bunch of different areas where AI would have an impact, and some of them, you know, if we get it right, will be amazingly positive impact, and other areas, you know, it's a little less clear. There might be some, uh, negative consequences of AI, and what can we do overall across all these different areas to maximize the potential upside of AI, both from a technical computer science research, ML perspective, but also in conjunction with policymakers and with, you know, people in those fields like health or education or scientists. And then also looked at the way in which we could all work together to sort of maximize those benefits and and minimize the downside.
>> And specifically with research efforts that are in the three to five-year time horizon that fit into a lab, which is in contrast to a lot of the hype we hear in AI right now that like this like pursuing AGI or super intelligence, contrasted to trying to help with medical, you know, like success with frontline healthcare, can you mitigate the the drudgery that typical doctors feel, or eliminate obstacles that radiologists might have to actually using the technology that already exists? So I think it made it feel very much much more real and specific and achievable.
>> I really like the three to five-year time horizon kind of thing with an ambitious sort of set of people around a particular kind of thing they're trying to achieve because I feel like, um, often that gets lots of different people working together with a a mix of skills in order to sort of really push forward something. Uh, and it's not so distant that it won't have impact, but it's not so short a time period that you can't conceive of doing something ambitious, right? Even in my own career, I've tended to think of like, when I start on a new project, what could we do in three to five years? And I think that's a a delightful time range to to consider.
>> Nice. Yeah. And I'm wondering if you could share maybe some of your favorites. One thing I found while working with you on that paper was always delightful is just how well connected you are to seems like dozens and dozens of bleeding-edge projects by some of the most innovative thinkers and researchers and builders in the world. You both angel invest in them, and you, you know, are generous in donating your time and energy to advise ambitious, impactful research projects that want to go make a difference, from climate to science and specific discourse in healthcare. I think healthcare is one of your passions on the program committee that we're built, we we built for the moonshot grant program, which we now have all of our applications, including Turing Award winners and Nobel laureates, and coverage from the top universities. So everything's working according to plan so far for funding some research that actually moves the needle on these areas of society. Curious with your background and with so much exposure to so many active projects, if you could just share some of your, like, one or two of your favorites.
>> Yeah, I mean, I think I am quite passionate about the application of AI to health in various ways. And I think the the moonshot, if you like, would be, how can we as society use every past decision that's been made in health to inform every future decision, right? And that's a super hard goal because there's all kinds of, uh, impediments to doing that. There's like very real privacy concerns. There's complicated regulatory requirements that differ for every jurisdiction. But I think if we kind of aspirationally try to say, what could we do so that we can learn from every past decision that's been made in a way that helps us have every clinician and every person themselves be informed and make better decisions in the future? That would be like an awesome, amazing goal. And I think, you know, a three to five-year moonshot around that might be able to make some progress to that. Probably can't get all the way there, but it would be pretty amazing even if it made it partway to that.
>> Is your sense that with the current capabilities of AI systems, the challenge for that would be more in the fitting the the adapting the existing health medical health records, legal considerations, and what the lawyers for insurance providers and the comp the hospitals themselves? That might make it all sound very hard, like more of an implementation challenge than the capabilities? Or do you think that the the capabilities have a ways to go before we would get the benefits?
>> Yeah, I mean, I think there's a bunch of interesting technical researchy questions in there, but there are a bunch of kind of grungy, how would you get the data in the right form to be able to learn from it, because it's in every different healthcare system. It's in slightly different forms and so on. You probably have to use things like privacy-preserving machine learning or federated learning or things like that. So, how would you make that work on a technical perspective?
>> Um, because you're not going to be able to move healthcare data from where it sits. Instead, you're going to need to be able to learn on the data in a privacy-preserving way in a whole bunch of different, you know, environments. So there are real technical challenges, but there's also, as you, as you say, legal and regulatory kinds of challenges as well. But, you know, I think that's part of why you want to have a whole group of people thinking about these issues with different kinds of expertise, right? Like you need some people with machine learning expertise and, you know, computer systems building expertise, as well as legal and policy and regulatory expertise.
>> Yeah, makes a lot of sense. Any other projects that come to mind as a as a favorite?
>> You know, I'm kind of enamored these days about how can we make our computing systems even more efficient than the latest cutting-edge TPUs or GPUs. I feel like there's room there for interesting and innovative approaches for, you know, much lower cost, uh, say, inference, which seems like it's going to be a a major thing in the world more than it already is. Going back to even to the original 2013 napkin sketch for why TPUs should be born in the first place.
>> Yeah.
>> I mean, if you redo that napkin sketch now, you're going to realize that we want, you know, first, much lower latency systems than we have today, uh, as well as much more throughput. And performance per watt is going to be a really important thing. So what can we do that would make way lower, uh, you know, energy systems that still provide the the same quality and performance?
>> Mhm. How do you see the relationship between all of the like the massive amount of research happening inside the Gemini team, inside DeepMind more at large, and in the now zooming out one more layer to the AI ecosystem beyond Google, the relationship for the academic research and research happening beyond Google's bounds and what happens inside Google? Traditionally, things like the Transformer paper, like MapReduce, you've had these channels of exporting innovation outside of Google. I imagine you also have you imported innovation and built on the shoulders of giants outside, and that's why like you gave examples already. Um, have you, do you see that evolving these days as Google with such a massive investment and such a leadership position in Gemini and in the hardware kind of up and down the entire stack, has it evolved and does it need to continue to evolve, especially as we as we continue to face the we're trying to innovate for funding models for the others, but it's not looking good [laughter] in my opinion, and I can't speak for you, but, uh, curious your thoughts on that that dynamic, that relationship at the bound of Google and innovation happening in traditional mechanisms?
>> Besides, I mean, I I think there's obviously continual evolution about, uh, you know, publishing models and so on, or publishing, uh, you know, characteristics. So I think in this current competitive dynamic, we tend to not publish the secret sauce inside our architecture of our Gemini model, say, but we do publish a lot of stuff in the sort of earlier stage research, uh, aspects of, you know, here are interesting new kinds of model architectures that we haven't proven out, but we've experimented with at small scale to publish them so that the rest of the ecosystem can, you know, pick up those ideas and explore them as well or build on them. And we also kind of look at the broader publishing happening in the rest of the the community and sort of look at, uh, you know, how could we adapt some of those to some of the problems we're seeing.
>> Um, and I also don't think publishing has to be a we publish it or we don't kind of thing. There's really a continuum there about when do we publish and what do we publish. So I'll give you an example in the computational photography work that Google Research has been doing for many, many years. We have awesome researchers in that field. Uh, they often, well, almost annually, come up with a really cool new thing that can go into the Pixel camera pipeline. So things like Night Sight or Astrophotography or Magic Eraser, where you can erase like that person who wandered in front of your photo that you didn't want in the photo,
>> In the first place.
>> Um, and so what we tend to do there is we put it out into the Pixel, the next Pixel N+1 phone that's coming out, and then we sort of wait a little while, and then we submit a SIGGRAPH paper about the innovations that went into that feature. So it's sort of a little bit of a delay. We take advantage of it in our products first, and then we sort of let the rest of the community know about, you know, what is happening underneath the covers, um, and they can build on it. So I think that's a pretty nice thing, and there's this nice continuum of, you know, not just the end points being being choices.
>> Can you think of any off the top of your head examples of papers or ideas in that category you just mentioned of kind of the earlier more experimental stuff where you are publishing is either here at NeurIPS or happened recently that you're finding exciting and have been engaging with the the non like outside Google? And I'd love to talk a little bit more about how it's been how challenging has it been or has it been to organize inside Google? Because it's it's an extremely impressive large organization and I can imagine internal conferences even happening that would be a little tiny any version of NeurIPS or something like that. But before you answer that one, yeah, I'm curious if you have any specific examples.
>> Yeah, I mean, just one off the top of my head. Uh, there was a paper published by some Google Research, Google researchers here,
>> Uh, on kind of a hybrid between the transformer and recurrent models that's called Titan, I think is the name. So it's sort of looking at how can you have much longer context by using a recurrence relation, uh, but using chunks of tokens rather than individual, uh, small tokens, and learning to kind of compress the the sort of very porky representation of every token into something that's a little more compact, and then have a whole sequence of those that you use recurrent steps on. So that's just a good example of, you know, that is not in our Gemini models. It, it could be in the future, but it does seem like an interesting idea to explore.
>> Uh, one more thing on the, uh, internal, every every,
>> Somebody told me a call about a conference. They're like, "Going to a Google conference." And I was like, "Oh, that sounds so cool."
>> We have a Google Research conference. It has like 6,000 attendees every year. Uh, and I know there's a sentiment if you talk to the PhD students here that the Google Research conference might have papers that feel a year ahead of the papers you're seeing at at NeurIPS, just because there there is a gap between what's happening in the open and what's happening at Google. So, um, I'm wondering besides making a conference, how have you found it to sort of be able to build an organization that is so innovative and is able to generate the the frontier, the state-of-the-art progress that that we're all
>> Yeah. I mean, I think one of the one of the reasons the internal research conference might feel a little bit like that is often, you know, for an external thing, you have to be quite far along in your research idea to get it accepted and published. And the internal conference, you know, there's a whole range of of maturity of the work. And so people are perfectly willing to have lightning sessions of like cool early-stage results that aren't really fully baked yet. And you get like 10 of those in an hour session. So I think part of that is, yes, it hasn't been published externally, but also part of it is is just trying to, you know, circulate some of the ideas that are being explored with your colleagues, and it has to be a little less fully poly.
>> Yeah. No, I'm I'm inspired by that. I feel like NeurIPS is really impressive,
>> Very large, and maybe there's room for that architecture of a conference to be exported as well in terms of innovation.
>> I mean, the workshops here feel a little bit more like that because it's earlier stage work and so on, but it's,
>> Still a fairly traditional thing of like a PDF of some paper-like artifact. And here are often these things are just talks with a few slides or not necessarily a full paper that someone had to write up.
>> Cool. Okay. Well, I think that's a wrap for us. Thanks for taking the time. Appreciate all your thoughts.
>> Thank you. Appreciate it.
>> Enjoy the rest of NeurIPS. And
>> It is beautiful here. It is.