Transcription
Hello everyone. Today we will talk about how to build clusters for AI and workloads without going crazy, and along the way, save your time, the company's money, and your nerves while you build systems. Today I will talk about my pain, my failures, and my suffering. I hope this will help you in the future. Who am I? My name is Danila. Don't be alarmed that the presentation slides are in English. Initially, this presentation was shown at the Google campus. My name is Danila. I am a co-founder of a startup parser, where we tried to reinvent data collection from the internet using AI, and also a Lead Drops engineer at Intel, where I am involved in developing various strategic initiatives related to AI. I've hit enough bumps, doing AI solutions or products. And I hope that this experience will be useful to you today. What will the presentation be about today? First, we will talk about how to create an AI startup, because an AI startup is currently an extremely hyped topic. Zuckerberg is trying to hire the best minds with millions of billions of options in this field. So, obviously, you need an AI startup. Second, how to build a very expensive AI startup. If you have infinite money, for example, the money of Mark Zuckerberg himself. Third, what are the most common mistakes I have encountered and seen during the construction of AI systems? And fourth, how to avoid these mistakes or minimize their impact. Let's start. To understand the main problems related to AI workloads, it's probably easiest to compare them with traditional workloads, which I think many listeners have worked with. In traditional workloads, we usually write some code, it's quite deterministic, and it performs some function. In the case of AI, we usually use probabilistic models, which may have some issues with output uniformity. Of course, it depends on how you have configured them, but this is a large class of tasks in any business related to AI. Second, the amount of traffic in traditional workloads usually depends on the number of users. In the case of AI workloads, right from the start, you have a very large workload, because inferencing with modern models, even relatively small ones, is generally a computationally demanding task. And even to run some small models with 7 billion parameters, without quantization, you need at least 40-90 GB of RAM. Third. Usually, in traditional workloads, there isn't much parallelism; we mainly achieve scaling through some infrastructural tricks. In the best case, the code works asynchronously. That's already a big win. Not in all projects. In the case of AI workloads, we often process many tasks in parallel; these are GPU-centric tasks. There's a lot of work involved in how to parallelize this correctly, how to load graphics cards correctly, and not pay a lot of money for it. Fourth, as I mentioned regarding scaling containers in the cloud, or even on on-premise servers with traditional workloads, everything is quite easy. Everyone knows how to use a load balancer; these are quite simple concepts. In the case of AI, there is a huge problem with how to correctly process and distribute data to different users based on contexts, so that the model doesn't hallucinate, so that it starts up quickly, so that some training jobs run quickly and are well-parallelized. This is all a very big problem. I think this slide was convincing enough to make it clear that an AI startup can be the best opportunity to demonstrate your engineering skills. Therefore, I propose we do a thought experiment where we create our own AI startup during today's lecture. Imagine you have, for example, company X, and you want to create some AI product, something related to AI. Then, of course, it turns out that X is some startup of Elon Musk, which was bought by another startup of Elon Musk. And therefore, we will choose another name, let it be A, another AI company. Before starting an AI company, you need to understand very clearly why we are using AI at all. And I think the most sensible approach is to categorize AI companies into two types, or rather, AI solutions, not companies, into two types. The first type is AI as a feature; this is the most common scenario. As examples, you can think of some additional Gmail features, recommendations in Spotify, or web scraping in Parserr. These are moments where we apply artificial intelligence to solve some tasks approximately, a bit stochastically, but very elegantly and creatively. There is a second type of company; they are the most hyped. This is AI as a product. It's very easy to distinguish: if you remove artificial intelligence from these companies' businesses, these companies cease to exist. Examples include all companies that make foundational models in the first place, and products around them in the second place, such as Mistral, ChatGPT, Runway. The same OpenAI, their most popular product is obviously ChatGPT, but in fact, they trade access to the model, and this is a large part of their revenue. What do you think, if you are creating an AI startup for the first time, which of these two tempting paths is better to take? Write in the chat. The profitable one, right? The first question was correct; the first answer from Evgeny is correct from a business perspective, and Alexander's answer is correct from the perspective of the question asked. Yes, it's easiest if you want to do a lot of AI and not go directly into AI as a product. And it's a very powerful tool to implement interesting features with it and solve previously unsolved problems. But starting to build foundational models without many years of experience, at least in training these models, well, I wouldn't recommend it. Alternatively, the problem of lack of experience can be solved with a large amount of venture capital. But if you have a large amount of venture capital, then you probably don't need this presentation. If we are doing AI as a feature, of all the things we mentioned above, that is, all these four points, we are effectively left with two. And since in the case of AI as a feature, we usually don't do training. Either if we do, we build some relatively small models, our AI usually performs some isolated feature, and we will spend less resources, both human, strategic, and financial, on training and inference. It's enough to have some algorithms or even an LLM through a model to solve some problems. Therefore, we are effectively left with two problems that we cannot get rid of. This is a large stream of data that is constantly moving somewhere, that is coming in, that needs to be processed somehow, that comes in as input. And a probabilistic result, because from the perspective of the product and from the perspective of the user, it would be desirable that if the same set of data is given to the model or to a feature or to some summary, it would be good to always get the same result, or regenerate it only upon request. If the result is not idempotent, it will rather annoy the user. To consider how to best deal with these problems, let's imagine that our startup, which we have imagined, has decided to do summarization as a feature. And for this, the startup itself has decided to focus on Healthcare, and in particular, telemedicine. Therefore, allow me to introduce Jack, a friendly LLM bot that helps heal people through a screen. What if we think about it, it's a very common use case for language models now, judging by what I see from other people. How can we build it most cheaply, quickly, and without additional difficulties? First, I would recommend, if you are doing this for the first time, and even not for the first time, to see if there are any templates and best practices. The industry is changing very rapidly now, and there are already some products, companies, libraries, and frameworks that are trying to standardize in some way. And here we are not talking about API standardization for communicating with LLMs. We are talking specifically about the approach, that is, how to build an agent system, a RAG system, or some digital assistant, or some application that recognizes images. There are several platforms that I know of, such as LangChain, which I have actually worked on, it's an Open Platform for Enterprise. LangChain. I have some skepticism about all these solutions as a professional, so I would be careful with it. After all, most solutions in the industry are currently raw in terms of implementation, but in terms of looking at the idea or approach, I strongly recommend turning to these solutions. In LangChain, there are a lot of ready-made architectures that you can use. And this is still a big pain for people, because I often go to Reddit and see people seriously writing: "How do I build a RAG system?" I'm a prompt engineer, I have no idea how a RAG system works. For such cases, templates are ideal. Next point. It's worth planning in advance the tools you use to choose something extremely simple, but at the same time very powerful and with a future outlook. What are the recommendations here? First, take the simplest. So that the level of intuitive use is at the level of a stick, so that it's clear what it is and how to use it without instructions. And this basic level of understanding should immediately be noted by everyone. Often, it depends on your background, because I, as someone who mainly deals with infrastructure, really like to use Kubernetes, because I use it quite well. But if you, for example, are a person with a software background or more what I've encountered, a prompt engineer, then you will most likely use some Docker runner in the cloud. Which is actually also not bad, because you can migrate from it, and you don't have to think that it won't work and immediately create problems for yourself in the future, but use something extremely simple. Second, it should have a future outlook. And third, I highly recommend being careful with vendor lock-in. A good example here is the use of various APIs. Currently, there are a large number of models, but the main vendors are obviously OpenAI, Anthropic, and Google with their Gemini family. And regarding the use of APIs, OpenAI or Anthropic are great. Most frameworks support them separately. However, if you decide to use Gemini, then Gemini has separate frameworks. And if you have already started using Gemini, then it is very difficult in terms of communication with models and prompts to switch to something else. And since there is still such a race going on, it is actually very difficult not to stumble upon this, so you need to be careful with this. The same applies to issues related to infrastructure. That is, I am a big proponent of Kubernetes, due to my career experience, because I have seen many times where companies, due to vendor lock-in, could not leave the cloud or migrate somewhere, or paid colossal amounts for telemetry services. And this gradually, of course, sinks the company. For a startup, this is critical at all, because you may have an excellent startup. A cool idea. You will find market fit. After all the trial versions of the products run out, your Excel spreadsheet simply won't add up, because before that, the cost of using some service was zero, and now it's several thousand dollars, and you simply don't fit into the budget. Third, try to deliver the product as quickly as possible. This is super important here. If this sounds like common sense, the first part of this presentation, but I still stumble upon it. As an engineer, I always want to do everything beautifully, accurately, so that everything scales immediately and works coolly. But when you are building your own company or startup, especially in a field like AI, you need to deliver the product to the user as soon as possible, because otherwise, while you are working on a good prototype, it will turn out that you spend 3 months, and the user absolutely doesn't need it. Therefore, for this, we can not hesitate to use vendor lock solutions if we understand that they will indeed accelerate this prototyping. And here, in the case of creating prototypes or some experiments, which is more specific for more mature startups, you can use more specific clouds for some research, such as Oracle Cloud, for example. Here I am not advocating for the use of Oracle Cloud. It's just that smaller players, who have fewer offerings and less infrastructure, or some Denver, they simply offer lower prices. And you can use the same Nvidia graphics cards, or even some special ASICs, but essentially at a huge discount. And this greatly helps save on experiments and research, because you know that this feature or this model might never reach production. In this regard, I have a story about how the hare beat the tortoise. This is a story from my own experience, where we built an application for a very long time, rolled out a new agent feature, and wanted everything to scale well, so that we could handle any workload at all. And of course, I wanted to do everything perfectly. The partner, on the contrary, said: "Faster, faster, faster!" As a result, what turned out when the feature was finally ready? First, there was a bottleneck that was seriously underestimated: data processing, and the data, when sent in large chunks, created a bottleneck, and the entire system hung because no one expected that users would behave like this. Second, as a result, the infrastructure had to be redone. And if we had tried to deliver the prototype directly from the start, everything would have been okay. And we would have received these insights about the product much, much earlier. Accordingly, we can move on to a more hardcore part of the story, in which we successfully launched our medical assistant. And we decided to go further. We would like to create a model that perceives itself not as a doctor, but as a robot nanny from Smeshariki, who wants to cure a person in all possible ways. And here I must introduce you to Pablo, a model that is absolutely obsessed with your health. We are introducing her so that I can tell you how AI model training clusters are built and what the problems are. What do you think, if you decided to build infrastructure for model training, where would you start? What would you think about first? Write your versions in the chat. Yes, but we are now building infrastructure for training a language model. We want to create a model that knows everything about human health. So, where do we start designing the cluster, infrastructure, solution, all of this? Let's assume we have requirements. People in the chat are writing "Ajax." Apparently, this is infrastructure as code. Well, not quite. We have requirements and everything else. We need to have exclusively engineering requirements. Architecture of what? Architecture is already close. Architecture of what? Data storage is a good question, but the answer is not super obvious. And actually, both the model size and the dataset for training, since we are training the model, we don't actually need a dataset for RAG. Data storage is indeed a big problem for training. The load is specific to inference. That is, in the case of training, we control the load. Simply, the larger our model and the smaller our infrastructure, the longer, in fact, we will train this model. Actually, as my experience and the experience of my colleagues shows, what you should start with is network architecture, because often when you build anything in data centers or even use clouds for training, which is, of course, terribly expensive, the network issue is critical for training our model in the cloud with a slightly easier network. But the idea, in general, is the same: the most important thing we need to avoid is to solve the problem of oversubscription. Because if we have an oversubscription problem, that is, when some graphics cards are idle, and some are conversely, well, not graphics cards, but chips are idle, and some are conversely not fully loaded, this is a big problem during training, because we are essentially burning money. How can this be solved? First, use RDMA fabrics, as they do at Meta, essentially, these are network solution designs with throughput that directly read and transmit signals to the device memory, which solves many problems. Second, use third-tier or second-tier designs so that all nodes in the cluster are available and all nodes where training load can be sent. Third, more specifically, rather, if you are building in a data center, you should use all sorts of trendy things from Nvidia, like NVLink or NVSwitch, because essentially you can combine several cloud clusters this way. This often works out of the box. And thanks to this, you can distribute the load more effectively. And fourth, also specific to data centers, is the use of large switches, really huge switches. A switch, if someone doesn't know, is, roughly speaking, a network card. To connect as many servers as possible into one network and avoid problems. The new network architecture looked something like this. This picture actually shows a three-tier network architecture. Second, as correctly written in the comments, is the storage architecture or storage location, because AI training is quite input-intensive, and therefore, essentially, the main task is to create the correct storage. Why is it important to create the correct storage this way? Because we need to try to fill all our GPUs in the cluster and those that are available as quickly and efficiently as possible. The only way to do this is to design the storage correctly. Therefore, here you should use scalable file systems like Ceph or some more specific solutions, because, as far as I know, Meta has its own solution that allows for quite efficient unloading of graphics cards. Second, use NVMe with high throughput and a large amount of memory, because this greatly helps optimize data chunks stored in this storage. Third, regarding storage, you need to simplify the storage architecture as much as possible. Ideally, the minimum number of additional layers of SSD and memory should be used so that graphics cards can take data directly. And fourth, local cache. It greatly speeds up training; it is often underestimated. And fifth, asynchronous checkpoints, which allow, in case of problems during training, to save progress to disk. And since this is a distributed system, this is a non-trivial task that should be considered from the very beginning. Third, how to choose GPUs correctly. Here, actually, many think that the biggest problem is network and storage. In the case of GPUs, everything is very easy. The balance is roughly that one CPU core per 1 GPU, and the amount of RAM in the system should correspond to the amount of RAM in the GPU. Second, you need to choose the appropriate GPU. In principle, you should familiarize yourself with the specifications of the architecture and cores and understand what is better for you to use. For example, some solutions, I believe, Google's diffusion models train quite well on TPUs. Nvidia, actually, as a secret, is good for general corpus infrastructure. That is, if you don't fully understand what exactly you need to train, Nvidia GPUs are super great. For others, actually, there are many more specific chips that are better suited for training language models, for training models related to reasoning, and so on. Third, the idea is to group GPUs. Fourth, you need to make sure that, again, this is related to storage, that our RAM and our storage are not a bottleneck, and that it has enough throughput to transfer data to the graphics card. Fourth, actually, how to manage all of this. Here, no matter how people argue on forums, it seems nothing better than Kubernetes has been invented. Actually, I haven't seen other production solutions except for Flow, which is built on Kubernetes, to build ML pipelines and train some solutions. For this, you also need to use special schedulers. That is, often it's not ordinary Kubernetes, but some Slurm plugin in Kubernetes or LSF to manage batch jobs on GPUs. Third, is the use of frameworks for distributed computing. And fourth, is Intel PyTorch Lightning. Someone turned on their microphone. And fifth, the use of gang scheduling. In principle, these are special techniques with a very simple concept. In a classic cluster, we don't think much about where our loads are distributed. In the case of AI clusters, it is very important that some data processing tasks are performed, for example, on the same server or in the same location. And therefore, it is very important to configure various additional settings in your cluster. I don't think I should go into specifics here about what needs to be adjusted. Fifth, one of the most underestimated things is properly configured monitoring and observability, because the training process, in principle, corresponds to ordinary production, meaning something can suddenly break, fly off, data might not arrive. And therefore, monitoring and observability are one of the most underestimated issues. What is important here? GPU telemetry, which should be tracked. Metrics around network and storage. Metrics around training that are non-obvious for many engineers, to track progress by epochs. And fourth, various alerts. For example, if your GPUs are idle or some tasks are not reaching them, if you get an alert in time, it will save a huge amount of money. And fifth, optimization based on all this data. In my experience, as a rule, if your GPUs are idle in the cluster for more than 5% in total, you should work on optimizing the delivery of workloads for training, because 5% on a large cluster is a huge amount of money wasted. Regarding clouds, everything is easy here. Second, data locality. That is, you need to try to make data accessible absolutely everywhere. Third, find a way to manage this whole zoo simultaneously. And fourth, comply with regulations. Fifth, actually, it's the same as the second point, but the idea is that some parts of the infrastructure should simply be made available everywhere. What are the main problems in cluster design, even though many things, even those we are discussing now, may seem obvious? First. It's the complete disregard for the fact that data is constantly moving somewhere, coming from somewhere, new variables appear. Second, incorrect use and underestimation of how important it is to correctly distribute the load to GPUs. Third, storage problems. And fourth, mixing different types of loads where they shouldn't be. For example, where a small task occupies a large graphics card and a large task does not fit into this large graphics card, simply because you haven't written one line of code. What are the conclusions from all of this? First, correctly define the tasks you intend to solve. Second, use the right tools during training. And third, change and adapt your approaches. Now, we will communicate with you. Before I answer your questions, subscribe to my social media, especially my Telegram channel. I actively post about AI chips there every day. So, let's start, probably, with the questions that Daria sent me. How to estimate the minimum resources for a case? Let's launch a load test. There will be no load. And target sizing. Here it's important to distinguish what kind of product you have, because you need to compare training and inference. Training, for training, this question is incorrect. Probably, inference was meant here. In the case of inference, the problem arises that if you don't use a ready-made model available through an API, you already have a very high resource consumption. And through an API, everything seems very clear. Why not go crazy? Is it that difficult? There have been cases. I haven't actually heard of cases, but many people who do AI, perhaps I do too. But there are big problems, especially in issues related to such utilitarian activities. Like creating a cluster itself, network, storage, and so on. It's all quite confusing and quite stressful, I think. Tell me, have you built infrastructure for inference and for training, for both? How often do GPUs fail from experience? This is a good question. I probably don't have metrics. But I would say that for training, honestly, I have never been involved in procurement, so I can't say. Complexity, where there are nuances. We talked about nuances today. Have you worked with agents for transforming product migration from legacy platforms to more modern ones? For example, migrating a Cbase product from Solaris to Ubuntu using an AI solution. Can you recommend any AI tool for migration? I think this is not very related to the topic of today's lecture, but for tools, I can only recommend engineers. Let's look at the questions from the chat. Are there any normal metrics that can show the utilization of RAM, VRAM, and NVMe IO, specifically utilization in percentage? Generally, it depends on the cluster, but from a system design perspective, it always seems more logical to measure latency directly, because percentage utilization, well, we want it to be 100%, because we are trying to use all our equipment to the maximum to avoid waste. But in fact, I would ask you to turn off your microphone, because there are some unclear sounds. But in fact, it is most logical to measure latency, for example, in how data is delivered, how much time it takes for task scheduling. This, by the way, is also a very cool metric. So, it depends, as we discussed, the system is quite extensive and depends on the level at which we are trying to solve the problem. Basically, from what I've encountered, the biggest problem is task scheduling, because a task hangs and it might not be scheduled for an hour or two, or a bit less. No one will notice this. Some amount of workload might be idle because of this. And there are many things related to optimization. And then we dig deeper. That is, how quickly our data is loaded into the graphics card, how often our checkpoints fail, how much this hardware is utilized. I hope I answered Evgeny's question. How often do GPUs break if used at 100%? There is a big factor here depending on what GPUs we use, but there are usually chip degradation metrics, as far as I remember. And depending on this, you just order. Often, it's ordinary managers who order new GPUs. But, I would say, depending on the intensity of training, because essentially the main wear they experience is temperature. And as a result, there are many operational features, mainly handled by technicians and their managers. Listen, can I ask a question by voice? Yes, please. I am, of course, far from designing data centers, but I am interested. Let's say we've estimated how much data we need to store. To whom, actually, do we need to go to have it physically implemented? Who will do it, remove everything? What do you mean? So, there are several options. If you are a company that does independent training and has specific tasks, then, well, you likely have engineers and technicians who can do this and physically assemble the cluster. In the case of clouds, you mostly need qualified platform engineers to do this. Okay. Let's wrap up then. Thank you very much, Danila, for the report. Thank you to everyone who came and asked questions. Come to our events, subscribe to our channel, and to Danila's social media channels. And then, until next time. Everyone.