Transcription
Today I'm getting a tour of one one of Jane Street's training data centers here in Texas from Ron Minsky who co-heads the technology group and Daniela Corvo who heads the physical engineering team. Thanks for coming out here to do this.
>> Thank you. Cool. Let's get started.
>> So here's one where we have our our training cluster of GP300s and VL72. Are you allowed to say what is currently happening on this cluster?
>> What actual training jobs are running right now? High level.
>> We do a bunch of different kinds of models. Some of the models are for training LLMs. Some of the models are for training all sorts of custom architectures that are more adapted to the trading problems and the trading data sets.
>> Earlier you were explaining to me that this was originally not a facility that was built to handle 200 kowatt racks and so you had to retrofit it to be liquid cooled and everything. You know, these cabinets, these GB300 cabinets consume at peak about 140 kW. Compare that to tra traditional air cooled, you're talking about 10 to 40 KW. So, a lot more. So, on the perimeter here, you can see some of this air cooling equipment here that would have fed a traditional air cooling. Some of it remains. We use some of it for the air aircooled load that we have. About 15% of these cabinets are aircooled. And here inside the GPU, you can see how the how the cooling kind of flows from these quick disconnects in the back, taking a fluid in, routing it to these cold plates that are sitting on top of the GPU, and then coming back out at a warmer temperature. So when you slide this sled in, it automatically connects to liquid supply and return and 54vt power. Right? So these just slide in powered and cooling all within the sled. There are still some components in here that are air cooled, but about 85 to 90% is cooled via those full plates of the heat loads.
>> To what degree have we had concerns about leaks, right? I feel like you spent, you know, decades worrying about not having water in our data centers and now we're like putting it on on purpose. Like how big of a deal is that?
>> So inside they have something there's underneath there's there's um these ropes that detect leaks. So there's a management side of the switch of this server that will send out an alert if there they sense a leak in there. Furthermore, there's leak detection underneath the floor. If it drips and falls under the floor, we'll be able to sense it there and isolate via a valve. But it is true that if something here fails, you are at risk of destroying the server. How often is there early? Not often, but this stuff is new. You know, it's yet to be seen over time how this how this works out.
So, I guess it's surprising to me that like it was not a problem to get this get a liquid cooling going in a facility that was originally built for air cooling and like lower power densities. I don't know. Why did it work?
>> That's how you define problem. I mean, it was an engineering challenge.
>> I feel like you're hearing the version of the story after all the problems have been worked out. These guys have spent a huge amount of effort figuring out like this place was built in a kind of intermediate point when we knew we had to scale up a lot but we didn't know what the shape of the coming compute was. And so I think one of the things that the guys here did really well was like take to heart the importance of optionality of like oh yeah there's a lot of different futures and we need to build some stuff that will work for multiple different ones. And I think that worked really well but required a lot of hard thinking and planning to make it land well.
You know, one thing I'll say is there's a couple of ways to do liquid cooling. So, we have fluid coming from the roof, from the chillers on the roof down here, maybe about 18° C. Um, we use the same fluid for the air cooling, so it's fungeible within the data center. We can move the fluid around.
>> Oh, that's nice.
>> Um, so what we do is we send this in this device here, make sure that every single cabinet has the right amount of flow. You don't want the ones at the beginning of the road to receive too much flow and the ones at the end to be starved. So what these devices, these valves, they control how much fluid goes to each cabinet so that they're balanced. Um,
>> How how do they measure whether they have the right amount of fluid?
>> Ultrasonic flow meter here. So ultrasonically, it's measuring the fluid and measuring how much flow in liters per minute and capping that at some rate that we predetermined based on the heat load that it's rejecting. Um, that liquid comes in, it goes to this heat exchanger here. So there's inside that heat exchu is a heat exchanger that transfers heat between this building loop building cooling loop and a what we call a technical water loop inside there which is needs to be very very clean and filtered down to 25 microns so you don't plug the coal plates on the GPUs. You want very good efficient heat transfer there. Um it's filled with a liquid uh a mix of distilled or deionized water and uh propyline glycol 25% of propyline glycol. Um that's to inhibit any bacteria or algae growth. Um and that bacteria could grow in there if you don't have the right ratios and plug the cold plates and plug the the uh the heat exchange between the GPU and the cold plate.
>> I don't love the world where we have to worry about bacteria growing in our servers.
>> So you you have leaks, you have bacteria, all these different new things to worry about. Um making sure you have proper flow between all the devices. Uh aircooled data centers, you you know, you put the cabinet in and just flood the room with air and and and transfer that heat. So,
>> Was there just an area underneath that you could have used?
>> Raised floors traditionally were used to to supply air. Uh so, there's ways you could supply air that air would come out here. A lot of the new solutions are going to overhead piping because it takes time to build these raised floors and it's it slows down projects. So, a lot of the piping systems going above overhead now for for speed of deployment. But we like this here because you see that blue uh that blue wire there. That'll sense a leak. If one of these connections is dripping, it'll touch that that rope there and send a signal to say that there's a leak. So, we're able to contain a leak under the floor and and and measure it where overhead it's kind of right into your data center. So, we have um 4,32 GPUs here in 56 racks. What we try to do on the uh on the power side is make sure we balance our power. You can't overload certain areas. So, you kind of, you can see how we have this busway here distributing power. And we're very conscious about how many racks are on each bus. Make sure you don't go over amperage and trip breakers. And you could be in the middle of a training run and overload a breaker and you'd have to go back to some bookmark.
>> What is the hourly price on a black wall rack?
>> Like, there's two ways of pricing it, right? There's how much does the hardware itself cost and the power and all of that. And then there's the opportunity cost.
>> That's right. That's right. And we actually think about opportunity costs when we think about all of this comput stuff very intensively. And like because we're in a world where compute is relatively inelastic, you end up in places where there's a real crunch even internally where it's like ah it's you know it takes time to get new compute online and available and you can get to a case where it's like people are all kind of bidding for the same compute and you're like wow it just becomes incredibly expensive because the stuff that we get out of this is super valuable to the business. And so the opportunity cost tends to dominate the hardware cost even though the hardware costs are not small.
>> Yeah. So that's the other question. If this facility is connected to the grid and you presumably had asked the grid beforehand for a certain amount of power um and now you move to much denser compute. How are you and this is still like using the grid. It's not behind the meter. How are you able to get the power in here?
So because the compute is moving ger what it what you know if we have some power allocated from the utility um we end up just using lot less space within the data hall right so you know like everything just gets smaller and you can see in this data hall it's got a lot of space uh that we don't need um you know so you're trying to respect that that power capacity you have from the utility whether it's utility or behind the meter you still have to respect that overall value but you want to ride as close to it as possible
>> That's why you can afford to put a podcast studio in this Although there are reasons to want higher density like the networking setups themselves are incredibly complicated and like I don't know one thing I always feel when I go into one of our data centers is like what a beautiful job people has done getting the wiring right that's actually quite hard and the more you have stuff spled out the harder it is to get all the wiring done and it's also worth noting like most of the wires you see here out of the cages are fiber but the stuff inside of the the fastest stuff is all uh is all copper. Light moves more slowly in fiber than electrons move in copper. So you really are at many different levels optimizing for the latency of all of these networking rates.
>> There's about 8,000 km of fiber in this deployment. And here you go. This is what happens when you go really dense. Uh but you you you do have enough power to fill everything there out.
>> So it depends how we move power around. You know, one of one of the ideas of of being flexible and funible with our power and cooling is that we overbuild our distribution. So, while we're limited on the upper end, um you're able to move power around by loading up different rows, right? So, we have these UPS's that supply power to our site, but when we distribute that power out, if we could move it around, it allows us to say, "Hey, we're going to, you know, we're going to grow some CPU here or we're going to grow some GPU here." So, you know, we have some headroom in there to do other things. and this is just opportunity areas for us to do that. Uh we want to be ready to go if the business needs some more comput. We have a place to put it. So you you do some amount of pre-building for future you know future opportunities that come up.
>> So what are these things?
>> So these are breaker panels. These these distribute out to those bus bars that you've seen. So um this is where you know power comes in and you're going to have to break that power out to to go in different paths. So there's a lot of distribution. You can see all this overhead conduit that we had to put in. This is all carrying power cables out to the data hall. So, you're you're very conscious when you're distributing power. It's it's less funible than than than cooling. Uh with the cooling, you can kind of oversize the pipes and move it around. With with power, you have breakers and and current limits. So, you're very you have to be very careful with with what you load up and where and making sure that you don't end up in a situation where you're tripping a breaker. Um we have protections in place. So, if we we trip one of those four buses, we're still good. So, there's some redundancy. Um but still it's still an interruption something we want to avoid.
>> What would cause you to trip a breaker?
>> Uh just high current. So adding too much load on uh a single bus way uh or single connection or uh pushing too far in the over subscription and saying oh well you know we think we're going to be overs subscribed by 10% and actually things kind of shoot up to 15% or 20%. Um there there are times where you do have time to respond. Um, but if you're you're too far over the limit, the breaker's going to trip and you're going to
>> And is the uh is the current pattern determined by software or by uh by hardware alone?
>> It's it's by hardware. It's controlled somewhat by by software, I think. Um, so what Nvidia is doing, they have this kind of LPS systems. It's a load management system that they're rolling out in some of their new cabinets. Really, what they want to do is keep that load profile flat. They're building in more bulk capacitance in those power shells, so capacitors in there. And they're also trying to get the software to allow the u the peak load and the average load to be much tighter. So you have this flat profile.
>> A place where software comes in is monitoring. We actually put a huge amount of effort into building our own monitoring tools so that we can in one pane of glass that they like to say like we can monitor every aspect of the system and look for problems and sometimes even drive reactions to the system where like sometimes we might want to shut off a workload if it is drawing too much power. And having like a unified system that can see all the things has been a huge kind of step up in being able to run these things in a highly reliable way.
So that that software system Ron mentioned is actually pulling information from these breakers and performing some logic behind the scenes. It's it's topology aware and what it will do like Ron said it can shut down nodes so we don't trip that
>> Because you want to run as close to the edge as you can cuz like the hardware is incredibly valuable. So you do want to overs subscribe. You do want to kind of run near that edge but you also need to be safe. And so you building these safeguards to let you pull back in a controlled way.
>> If if somebody switched one of these switches, would a some training workload stop right now?
>> Yeah, I mean this site's been been
>> This site is currently live. So yes, so here you can see a little bit of the scale of of of our liquid cooling. So you know these are we call buffer tanks. So, they're here to to help with a situation where maybe there's a interruption in power and the chillers on the roof restart and lose cooling. This is almost like a a thermal battery storing some energy for us to keep those GPUs cool uh while the chillers come back online. Um, also, as these workloads come up and down, you're going to have a kind of a movement of temperature. So, this helps dampen that that effect out. These are the traditional air cooling units that you'll see in almost every air cooling data center. Um, really just pulling that hot air back and supplying it back into the data hall. Those wheels up there are are valves. So you mentioned leaks, right? What happens if there's a leak? Well, you have places where you can isolate the system to kind of work on it to fix a leak or something. These uh orange things uh just have more chain in them. So they're not hitting you in the head now. You can reach them with a ladder uh take more chain out and turn the valve closed or open.
>> That's great. Such a simple solution. You know, it's it's interesting as these data centers and and the compute gets so dense where the place where you have the compute is getting smaller and smaller and the places where you're supporting that compute is getting larger and larger. The infrastructure, the transformers, the the chillers are getting bigger and bigger. So the sites are are very much this much compute and this much infrastructure to support that compute.
>> What did the analogous system to this 20 years back look like?
>> So yeah, it was like dramatically more primitive. Like early on we literally just had like computers like sitting out with us in the room with all the people like the one of our comput clusters we called the hive and I remember the first version of the hive was literally like six Dell boxes stacked on top of each other like at the end of you know the row. That's why they're called hive box. Yes, that's right. Yeah. like Hive was the name like a very like actually the first the first work that I did at Jane Street was doing like quantitative research on trading strate strategies and I was like oh yeah I guess we need like a cluster and like that pile of six boxes was our first cluster and the trading systems themselves we also had there and that actually was like more important to have out with us and it took us time to convince ourselves it was okay to like actually go and put it in another room in Iraq because we actually wanted the ability to make sure we could turn the damn thing off. Like if something went wrong, just the comfort of like I could unplug it was there. And you know, it took time to convince people that we had enough control over the systems and we understood enough about where things were and just would be able to find them if we had to go in there that things were cleanly labeled enough
>> That we were comfortable taking these things and moving them in the back.
>> Yeah.
>> I mean, there were ups and downs like literally at some point, you know, one of the people who was like cleaning the office like unplugged one of the trading systems in the middle of the day like as they were vacuuming. So you know in the end it is in fact better to have it all in a data center but I don't know early on it was more of a shoestring operation and we were just kind of figuring things out.
>> Uh but did did it not need to be even back then colllocated with the exchanges? So an important thing to understand about our early uh trading is like we were super not fast right like there there trading like latencies matter at lots of different orders of magnitude and like sometimes it matters whether you're responding in you know seconds or milliseconds sometimes microsconds like these days the very fastest systems we care about you you're talking about whether you can turn around a packet in under 100 nonds
>> Yeah okay I definitely want to ask you more about that when we get to the podcast yeah interesting