Transcription
Jane Street are partners of my podcast, and one of the fun ideas we had is, why don't I come visit a data center, uh, for training that you guys run? So, I just got a tour of this, uh, Texas data center from Ron Minsky, who co-heads the technology group, and Dan Ponttovo, who heads the physical engineering team. So, thank you, guys, for showing me around. It's worth, I've never been here before, so I also got, I was also getting a tour, which was great.
Previously, I was confused. Well, how can you be doing GPU things if you need to be trading on nanoseconds? And maybe you can talk through, what is the actual time horizon of the trading you guys do? Can you afford to have be running big models to, in, in the middle of making trading decisions?
I think the thing to understand here is there isn't one time horizon. There are many time horizons. Uh, there are trading systems we build and trades that we do where, in order to be competitive, you actually have to turn around a packet in, in under 100 nanoseconds. And like, that's a very different regime, right? You know, people sometimes talk about like, oh, can you guys write high-performance stuff in OCaml? It's like, we can, but like for this kind of speed, it's like it doesn't matter if you write in OCaml or Rust or C++. You can't use a CPU, right? You are going to be on an FPGA that's like, direct wire attached to the network, and you're going to be turning around the packet so fast that if you like, attached an oscilloscope to the wire on the way in and the wire on the way out, you would see the packet start to leave before it's done being consumed. So, it's like a very different, very specialized regime. But like, when you're in that time regime, you really can't do very much computation. The decisions you're making are going to be very simple. And in fact, there's this kind of whole curve of trade-offs between how smart is the decision that you're making, be it a model or some other kind of maybe even like handwritten decision-making process, and how fast the turnaround is. And like, the, the right way to build, uh, an optimal trading strategy is really to have a kind of ensemble approach where for some kinds of decisions, you're making very simple decisions very quickly. For some kind of decisions, you're operating at the scale of, you know, instead of thinking of a hundred nanoseconds, maybe like a handful of microseconds or tens of microseconds or hundreds of microseconds or milliseconds. And in some cases, there are processes where if you can get that decision turned around, you know, in an hour or that day, that's totally fine. And you're, you're kind of competitive on a time basis at each of these horizons. Uh, but you're making very different kinds of decisions at all of them.
Maybe you can't say, but what, what is it exactly these models are predicting? Like, surely it's just not the next thing in the order book, or maybe it is?
Right. So, we're definitely like dancing towards stuff that's hard to talk about. But I think the simplest and most important one that we've been thinking about, like, we think about it now, but like also 25 years ago when I started at Jane Street, when I was building like models out of linear regression, you know, and stuff like that, like a very useful kind of thing is to predict a fair value for a thing, like, what do we think this thing is worth? And that fits in, in a very kind of composable way into lots of different trading processes. That's not the only kind of thing that we use as a prediction target, but it's an important one.
It seemed like a meme I was getting for a while about like, what trading firms do is like, you, you got to get the colo and the, where the NASDAQ exchanges and it's very important that your machines are right there without thinking without thinking too much about the exact details of what we put where. Like, your inference processes might be on CPU, might be on FPGA, might be on GPU, depending on the kind of constraints of how much compute you need, how big the model is, what kind of latency turnaround do you need. And yeah, like bigger, slower things you can put farther away. It's annoying to have to put all the compute right by the exchange. And for the stuff that's like really, really fast, like being in the colo isn't enough, you care about like how long is the spool of wire that gets you there. Like, you're literally like measuring out the length of the fiber runs when you're, when again, when you're like at this very, very low nanosecond scale. Um, but in general, like the, like bigger models give you a lot more flexibility in terms of where they physically go. If we're putting GPUs in some of these collocated, uh, facilities that are next to the exchanges right now, you have to work with, with their rules. You know, who is, who is that provider? Who is, who is giving you that space? Yeah. And your power, your cooling, all those constraints now are maybe slightly tighter than if you have a facility, facility that you're designing and operating. Um, so you're now having to kind of come up with ways to, hey, maybe I could only get one GPU in a rack because it consumes so much power. So now I have to spread it all out rather than being able to do liquid cooled in one rack. So these are all, uh, things we need to keep in mind as our compute, you know, compute.
You guys recently signed a $6 billion compute deal with CoreWeave.
Mhm.
What are you going to use that for?
The rest of the AI world has scaling laws. We have scaling laws too, and there are lots of models that we want to train. I think the thing that's interesting and maybe different between us and the kind of more traditional AI labs is the amount of diversity in model architecture and the amount of experimentation that we're doing. So, a lot of the value you get from all of this is just people are like trying lots of very different new things in the model designs and giving researchers just like faster iteration time so they can discover more ideas and drive more innovation. It just turns out to be incredibly important.
In the case of these foundation labs, there's some gain from having trained just one model that does everything that is fully general rather than building a bunch of custom different models. Can you give me a sense of why there's a different trade-off at Jane Street?
For us, some of the specialization is about being adapted to consume the right kind of data, right? And there are just like many possible data sources that we might be feeding in. There are like a bunch of just differences in the data rates that we need to achieve. Like, just like another thing that like just makes us need to kind of specialize some of what we're doing is just like the overall kind of both inference and trading dynamics are made different by just like the, the bytes to flop ratio being different. We have like way more data that we are using to train the models, but the data is kind of byte for byte less informative just because financial data is very noisy.
Yeah. Um, and so the models tend to be smaller, and the data tends to be noisier, noisier, and there tends to be a lot more of it. And it's also different between different models that we build for different applications, right? As we try and figure out like, how can we leverage more of the information that we get. It's like, oh, now there's like all of the kind of decisions from like, how do we store and load data efficiently, to how do we shape the model, to how do we make the inference process, you know, have both the throughput and latency that it needs. Those, there's going to be a whole different set of trade-offs there. And so there's just like a lot of value in kind of working that out and picking the the best thing that you can do for, for different applications.
What is the, um, inference workload actually, or how does it compare to what your traditional big chatbot LLM company is doing, in broad strokes?
Latency matters more, as you might expect. Um, batching is still an issue. Like, depending on the model you're doing, you might have models or a part of models that are kind of disaggregated for different symbols that you're looking at. And so the same kind of like pulling in data from multiple sources and batching them together makes a difference. I think another thing that's interesting is just like the data rates are really high. Like the amount, the, the aggregate data rate that you get in a large LLM LLM lab from like all of the different users is also very high. But the amount of sequential data that you're going to get from any one user is not that high. Whereas when you, the data that you're pulling is the bytes that are coming out of like the NASDAQ feed, it's like, oh man, the data rate that you want that kind of sequentially consumed in one domain, kind of causally one after the other, is really high. And so again, like the dynamics change. And like, but I think a lot of the same kind of basic engineering questions are not so dissimilar, but like all the constants are twiddled to different places, and so you end up making different choices.
What, what does that mean in terms of how you, how you had to design these systems, whether it's in terms of storage or whatever else?
Yeah, there's, I think more emphasis on the performance of the data loading than you might otherwise see. I think we're doing a lot of work to build out our own kind of large-scale, uh, data storage system, our own kind of internal object store, um, where we've like used various kind of vendor products, but, um, over time, I think for some of these research-focused use cases, we kind of need to operate at a much larger scale and need also to deal with a diversity of data centers, right? And this is like less a training time, less a inference time, and more of a training time question of like, we just can't get all the compute we want all in the same place. And I don't know, I feel like in general, like an important trick in like effectively running a technical organization is feeling, is figuring out what shortcuts you can take. One shortcut that we were like privileged to be able to take for many years is we got to pretend like there was only one CPU architecture on the planet. Like, everything was for like x8664. We pretended like none of these other things existed. Uh, and that simplified a bunch of things. Uh, and we also had like one big research data center and one big storage cluster, and that also simplified a ton of things. And actually, both of those have now been unwound. Like, you just can't get the amount of power, like you cannot wire in enough thunderbolts into like the same data center to power all the things you need. You need to get the data centers built all over the place. So, there's a big disaggregation problem, and that gives you a problem like, oh, now you have to think about like your compute scheduling and your storage scheduling being intertwined one with the other, and there's a ton of data. So moving around is like non-trivial. Um, and also we had to give up on this x86 only thing because Nvidia has a bunch of cool new products that mean that you need to support ARM.
Now, zooming out, I want to ask a very naive question. There's maybe a naive view that, uh, you know, if you have AGI, it can like immediately do what Jane Street does. Give me a sense of like, why that naive view is naive.
Yeah. And I don't want to totally discount it. Like, you know, there's a world that we should take seriously where like, you know, we're going to build large language models or some other AI systems that are like strictly smarter than all humans on the planet and more capable at all cognitive tasks and like, yeah, that's going to be weird, and that's like a different, that's a, that's a different, a different state of things. Um, and in that case, yeah, you know, maybe large amounts of things that Jane Street does will be automated away, and, you know, maybe we'll all just like, you know, sit back and, you know, drink more margaritas or something. I don't know what that world looks like, but it doesn't feel like we're particularly close to that now. I think that like, in general, I think it's like easy to underestimate the richness and complexity of the work, both that like a company like Jane Street does, but really that is done in kind of any really like ambitious, high-difficulty, like company-scale task. I think trading in particular feels to me as like, kind of AGI-complete, sort of like NP-complete. It's like, meaning like that like all of the different problems of the world end up influencing what you're doing in a trading context, because at the end of the day, trading involves figuring out what things are worth, which means making predictions about the future, and lots of different things flow into that. And as various pieces of that get automated, you know, you have the usual thing of like, the other hard parts that we don't yet know how to automate well, that ends up being where the competitive edge lies. I feel like humans and like human cognition are like more valuable than ever. Like, I have never been more desperate to hire more engineers and more traders than I am today, because everything people are doing is more valuable than it was. I mean, some of this is just me being somewhat skeptical that we are quite as close to the models that are like smarter than humans at all the things as some people seem to think.
Maybe it's like physical infrastructure, like actually getting the colo. Maybe it's actually like the software infrastructure that you build. Like, give me a sense of what it is that would.
Yeah, we build like a huge variety of complicated pieces of software, have people thinking about lots of different trading problems, some of which are not very electronic at all. Like the business is just like way more diverse than I think people give it credit for. And there's an idea of like, oh yeah, it's like it must be that like simple thing where you just like, you just have smart people who like make smart decisions and write good software. And like, if we could just automate the smartness part, that would be the whole thing. And I think it's just way more complicated than that.
What, what do you mean by the non-electronic parts of trading?
I mean, there's still trading that happens via chat between people talking to each other and making decisions, and like someone like sizing up how much adverse selection they think the person on the other side of the phone represents. That's like still a real part of the business. Um, there's just like, you know, there's just different kinds of securities that have taken longer to get more automated. The bonds business, for example, is just like not nearly at the level of automation that you see in equities. Indeed, we, I think we were kind of confused about this of like, I think those of us who have been like in the business for a while. We kind of, I mean, I started a little too late to really see the kind of transition of equities becoming electronic. But I think people who are, you know, paying attention a little earlier than me were like, yeah, and I guess everything else comes next. And like, you know, what it's been like, you know, 25, 30 years, and like not everything has gone that way. The systems are still, you know, we don't have a lot of people like standing on the floor of exchanges anymore, but there's still lots of trading that is deeply intermediated by humans and human judgment.
King of which, how much are humans in the loop on between the model and the trading decision?
Many of your most profitable days happen when like weird stuff happens and there are events and the world kind of goes crazy, and like nobody knows what's going on, and like that's when it's like very hard to provide liquidity in those contexts, and so you get paid more for doing it, and there's often a lot of volume on days like that. And doing that well often involves human judgment of like thinking about like, how is today different from all of the other days? And, you know, to the degree that we can, we want to build the models that work well through phase transitions, but also we think humans work better than models do through phase transitions. And sometimes you need this kind of meta-judgment to decide what to do. And so, there's an, even for the systems that are largely automated, there are decisions to be made by the people who are watching. And we always have people who are watching, right? I think an important part of trading is paying attention to and thinking about what's happening during the trading day, even if the individual transactions are going by far too fast for a human to kind of weigh in on a kind of transaction-by-transaction basis.
Dan, what, what have been the more notable changes over the last 20 years that you've been doing in buildings like these?
Yeah, people are actually care about data centers and want to talk about it. You know, I've been working on cooling for a while, and now all of a sudden people, people talk about it and and and think it's interesting. So that's like, that's fun and exciting. And for, for folks on my team, I think they feel that way as well. There's people who have been in the data center industry for 20 years that kind of still want to do it the way they used to. And I think that's kind of falling by the wayside now. Uh, you're finding ways where people are, um, challenging previous thoughts. Hey, these, my entire data center is backed up by generators. But generators are some of the longest lead time items you can buy. So maybe we take those away and only put it for a core part of the system that needs that resiliency. That gets our GPUs on six months faster. Let's do it. So those are things that, uh, you know, maybe, maybe it's not the best engineering decision, but it's truly the best business decision. And I think it's stuff like that that has been coming up more and more often.
It feels like every year people change the, what, what is bottlenecking scaling AI compute right now as you're doing more negotiations and trying to acquire more compute. What, what is the current bottleneck and what do you expect it to be for?
Putting aside compute and memory and all that fun stuff. So generators, uh, transformers, um, some of the cooling equipment that's used now for the liquid cooling is, is in, in a lot of demand. So, um, and it changes rapidly. What I tell you today is is going definitely going to be different two weeks from now. Um, we do this thing, we work very closely with internal teams on the procurement side, uh, to to stock up on some of this stuff. Stuff that we know is fungible across all our data centers. We will warehouse and have it ready to go. Um, there's components like generators where you're not going to put a giant generator in a, in a warehouse, or, or, you know, for instance, if you're doing something behind the meter like a turbine, you're going to, you're going to have to think about those markets a little bit more. Um, where you're getting them, where you're staging them, you can't just leave them off to the side. Um, so I think the components definitely change. Those are some of the big ones. And, uh, you know, as we get to more and more density, you know, I think one hope is that the buildings get a little bit smaller and maybe and, and, and we're able to like, you know, build the buildings faster, get all that compute kind of in a nice tight bundle, and then all the infrastructure around it's got to, got to be maybe pre-built and delivered to site, right? Modular data centers or modular infrastructure is becoming more and more of a thing where these components, especially the long lead components, are being, um, designed and built offsite and shipped to site. So almost as close to plug-and-play as you can get.
Well, one of the points you made earlier is that as, uh, as the racks themselves get more, uh, dense, you know, more and more of the data center is like the infra around the actual racks, which actually is kind of similar to, um, like a, a package on like a, a chip, right? Or like a chip on a package. It's like the, the compute is a very small part of the total area of the package.
Yeah, it's, it's interesting. Thing. I mean, I, I don't, you know, um, it, it doesn't solve any problems per se. I mean, maybe it creates, it creates others. Sure. Like, you know, you get to a one megawatt rack, right? People are like, what does that even mean? One megawatt in a rack? And, you know, you know, the, the cooling, kind of the pipe is just going to get larger that you're bringing there. And, uh, the amount of power, whether it's kind of the AC power that we're using now or 800 volt DC where it's, where it's going in the future. You still have to bring all that, those components to a spot. And the thing that's like interesting from our point of view is like, you know, we could design these, these engineering things, but at the end of the day, whether it's Nvidia or an ASIC or who, they have to sell a component that can work in a data center, and they're, they're thinking very hard about what they sell, um, because you need people to use it, right? If you're, if you build a one megawatt data center, one megawatt rack, but there's no way to power and cool it, kind of useless. So, you know, we're working very closely with, with, kind of almost everyone in, in that space to think about what are the components you need to be able to support these next generations.
Because the lead times you're talking about, you know, over a year sometimes, and you're just, you're deciding on the infrastructure before you're placing an order for the chips.
So, you know, you're trying, for instance, the, you know, TPUs, they use lower temperature water and they're, they're half as dense as, as, you know, an NBL72 GP300, right? So that requires a different strategy, and, and you want to make sure you can handle those in the future.
One of the things that allows hyperscalers to commit to large amounts of compute is that they have some reserve use for excess compute that they're not using for training or inference of LLMs at a particular time. For example, like Meta, if they're not using some of the GPUs they bought, they can just say, we'll just make our Insta ad, uh, serving model slightly better for today. What is the equivalent sort of reserve use of compute for Jane Street that's just a lower bound on how much that's worth for you?
Part of what's going on is like, in many ways, we're just like very compute-constrained. There's lots of innovation and experimentation and new ideas that people have that is bounded by the amount of compute that we have. And so like, in some ways, like if we just think about like, like we do a, we try and, and do a kind of moderately rigorous job of thinking about the value of the new different runs that we can do, and the value of the runs that we're turning away is really quite high, right? So like, we're doing what we think are the most valuable things, but, you know, if, you know, if it turns out we have more compute than we need for those, there's just like a ton of other, uh, research and experimentation that we can do in that space. So like, we're, we're, we're nowhere near to like being like, oh, too much compute, like we sort of have, have the opposite problem. I think there's also really low-hanging fruit in that direction. Just like, it's valuable to retrain the models more often. There's some decay in the quality of models over time, and being able to rerun them, like that's, that's kind of has immediate and clear value to the firm. Uh, there's also some amount of bulk inference tasks that that we can do that can like fill in the gaps in the systems where there's nothing else to schedule. Um, so we don't quite have the thing that looks like the analog of like the Instagram ad serving thing, but there is just like a ton of other like kind of dark space of like things that we're not doing, but we would if we had more compute. So we're like pretty unconcerned about getting value out of these. Here, thing, there is like, there is a bunch of embedded bets, like we are like investing a lot of money in, in this stuff, and you could imagine that like things won't get better at the rate that we are thinking they will in terms of like the value of the individual models and trades that we're doing, and like, it's a competitive environment. Maybe other people will out-compete us. We're like, I think one part of remaining good is like always being nervous about other ways that competitors can like figure out doing similar things to what you're doing and reduce the value of, of that. So like, there are ways, there are ways that it might not work out, but, uh, certainly with anything like the current mix of compute jobs that we have, we're just like very far from having this problem. It's, it's interesting to, this doesn't exactly answer it, but like, you know, you could disconnect the, uh, the powering the data center from the chips and say, okay, well, you know, I, I might need to use this compute later, let me commit to the data center and the power now, but like delay the decision on the chips, which are very expensive, right? And, and just be slightly long power and data center for that, that point of time where you might need that compute, um, and then we'll build in situations where, hey, maybe we can kind of offload some of that capacity to somebody else. It's much easier for us, I, is to offload power and data center capacity than it is the chips themselves for obvious reasons. But, uh, you can, you can really bifurcate those two.
This also changes the considerations around hiring. I mean, you already have like the highest bar for hiring, but it just increases even more. If you hire one more person, that is one person who will need compute to do their experiments, and that compute is going to be traded off against somebody else who's excellent on your team who could be doing experiments themselves.
I hear what you're saying, but we don't think, oh, it would be weird to hire more researchers because then we'd have to give them more compute. It's more like the research is incredibly valuable. The researchers are incredibly valuable. This is a good argument for buying more compute. Um, and so we're like very eager to grow the amount of compute. Like these days, we are in something like the range of like tens of thousands of GPUs, and we will, in not too long, be in the range of hundreds of thousands of of GPUs. And we think it's like well justified by the business. Like, you know, it's not, it's, it's not like, you know, we're worried about like, oh, you know, can we justify it based on like the penals of the trading strategy? It's like, no, no, no, it's like these are clearly good investments. Um, so it doesn't feel like it's slowing us down on the hiring front. In some ways, the, the biggest impediment to growth is that it takes time to like, really train people and absorb them into the culture and kind of build, build them up and build the place up. Like, we want Jane Street to continue to be a great place to work. Like, I, I just don't think of the hardware thing as at all being the thing that slows us down. And, and I think the real limiting factors are finding great people and having the mentorship capacity for them.
I guess this might be a good opportunity for you guys to mention what kinds of roles you're currently hiring for.
Oh man, why don't you start in the, in the engineering space?
Yeah, I, I'll start. I mean, I think, so, we're generally just looking for really smart people, people that, that, that are interested in, in, in doing this stuff, and, and that's, you know, mechanical engineers, electrical engineers, project managers, architects, people that help design and build some of these spaces. And, you know, our, our, our remit, uh, in my team is, is really to, to find the spaces, to design them, to construct them, and then to operate them, right? So, it's full lifecycle. So, in each one of those, you kind of need people, you know, lots of engineers, lots of what we call physical engineering, which is a made-up term that, that we came up with, but, you know, mechanical engineers and structural engineers, maybe electrical engineers, those types of folks.
And, and machine learning and trading in general is really like a whole team sport, and so we want to hire people from lots of different backgrounds and with lots of different capabilities. Uh, we're certainly like very excited to hire people with kind of, you know, specific, like machine learning backgrounds of like, you know, designing architectures and building models in various cases. We both, I mentioned that we have like a bunch of like custom architectures and stuff for like our own bespoke kind of kinds of data that we need, like the data kind of characteristic of the markets. Um, we also build LLMs, and people who have experience in all sorts of part of the lifecycle of LLM training, we're interested in hiring and have been growing that area. Um, you know, we, we hire lots of like people with like generally good scientific and technical backgrounds from like math and CS and physics and engineering and stuff to be traders, and like, there's a kind of mix of skills there. Uh, but that's like an area we continue to be very excited to hire in. On the software engineering side, there's like a general software engineering role, which we're always eager to get great people for, that, you know, I think just rewards a little bit, you know, it feels a little silly to say, but just like, you know, as Dan was saying, smart, curious people with really good CS backgrounds, uh, you know, fit into that generalist role, and there's lots of different kinds of things they can end up doing. There's also a bunch of interesting specialized areas where we really are excited. Like, here's a thing that's kind of new. With all of this scale, we are much more interested in fleetwide optimization than we were in the past. Like, we, our old view about about performance optimization was that it was much more about, you know, making the things that were most speed-critical as fast as possible. And more generally, yeah, compute's kind of cheap, and like people are expensive, and we're not spending that much time optimizing our general compute. But like, man, we're doing a lot of general compute now. You know, you start investing billions of dollars in this stuff, and it just becomes more valuable there. And there are people who have experience in doing this at some of the hyperscalers, and we'd love to hire more people with that kind of background to think about the optimization problems that we're hitting, which are like related in important ways, different, but like, you know, so it's like both a related challenge and a new one. Um, we're like, we do a lot of fun, like hardware engineering stuff. We're like working on our own ASICs. People with that kind of experience is super exciting. Um, one thing that we mentioned a little earlier at lunch was like, we're starting to think about building out a formal methods team, using basically mathematical proof to make software engineering more effective.
That's like a new, very speculative area, and we're like very excited to find people there. We feel like that's a kind of a set of a whole community of people who in the past I feel like I've always had to disappoint by like, yeah, we're not interested in formal methods, but like, I think the whole AI revolution makes formal methods suddenly a much more interesting field, and so it's a place we're excited to invest in. So I don't know, and like, I don't know, project managers, people who do front-end dev, actually, like for most of Jane Street's experience, we pretended like this whole web thing had never happened, and like almost all, all of our tools were just like in the terminal, but, you know, it turns out it's useful to be able to like, draw a straight line and, you know, have a tooltip and things like that. So, we've actually invested a lot in building really good tools for doing front-end development and building tools for people, and having great front-end engineers who are both really good software engineers and have a good sense of what it means to make an application that's good for a person is really important. I say like, as, as a general meta point about all of this, I think that like, in all of the like legitimate and real excitement around AI tooling, I think people sometimes like, kind of miss out on the importance of the human element of all of this. I think that we really, we really care a ton about building tools that are good for people, and that comes, that includes the AI tooling itself, right? I think trying to drive tooling in a way that increases human understanding and agency and efficiency is like, that's the core thing. We are limited more than anything else by the amazing people who work there, and like being able to find more of the right people and grow the organization so that we can get more done. Uh, and so we have a very kind of human-oriented way that we think about the systems that we build. Um, it's been really cool to have you guys, um, make these fun puzzles and challenges. I think in general you do that, but also you, um, you've been, uh, you guys have made a couple for the listeners of the podcast in particular. And I think people who are listening to this might find it interesting to check those out. Um, uh, including one, by the way, which not only was nobody who submitted to the competition able to solve, but Jane Street itself cannot solve, which, um, uh, which involves finding back doors to various LLMs that have a trigger phrase baked into them. Anyways, I mentioned this because, um, uh, to the extent people are interested in learning more, I think these are the kinds of fun puzzles that might give some indication of what work is like and, um, uh, why things like fun place.
Yeah, puzzles are a deeply embedded part of the culture. So, it's kind of great to use them as a way to reach out to people as well.
Yeah. Yeah. Um, I guess the plug here in this case is janestreet.com/doresh. Uh, so that people can learn more about the open worlds and about all these puzzles.
Yep. Awesome. Cool. Thanks for doing this, guys.
Thank you very much.
Our pleasure.