Transcription
Success Success Colleagues, good day. How do you hear? Over, pluses. Excellent. Thank you very much. So, you see the presentation, and we will now go through it with you. So, today we have an open lesson on the course "Me and the Architect". A very important profession right now. My name is Lebedev Mikhail, and I will tell you about it now. We will get acquainted and go through this description. So, the recording is in progress. You can hear me. So, my name is Lebedev Mikhail. I work in management. And, uh, my life background consists of the fact that I was a system administrator for a long time, then I developed in various directions: Windows, Linux, uh, network equipment, telephony, and so on. And this allowed me to move to the position of a system architect. Further, I continued to develop in the same direction. Uh, so, I realized that I needed project management knowledge, I studied PMBOK, technical writing was necessary for me to write documentation, uh, development skills were needed for me to write some mini-services. That's it. And, uh, so, since 2018, I moved into Data Science. So, my transition actually lasted about 3 years. While I was a system architect, I studied machine learning, took courses, and solved some problems. I participated in competitions, and as a result, I managed to seamlessly transition to the position of a data scientist. And since 2022, I have been teaching. I am an associate professor at the Higher School of Economics, I teach at Otpus. If you want to chat with me, here is my Telegram on the screen. So, please write your questions in the chat. I will answer them as much as possible, so as not to get sidetracked, well, maybe I won't answer them immediately. So, the webinar recording will be sent, you can watch this recording on VK later. So, today we will discuss the design of architectures for artificial intelligence services, get acquainted with available tools, uh, so, architecture on the example of LLM architecture. We will look, chat about the course, and then reflect on our meeting. So, if possible, please tell me about yourself, so that I can perhaps adjust or answer some of your specific questions about your path during the meeting. In fact, I was a system architect, became a data scientist, and today, in this course and in this lecture, we combine these two aspects. So, well, let's talk, before talking about artificial intelligence architecture, artificial intelligence services, let's first talk about ordinary architecture, then talk about artificial intelligence, then combine it into artificial intelligence architecture. So, well, a brief excursion. So, this is what physical servers look like. And they are servers. And, uh, since ancient times, there has been some duplication in them. In the upper picture, you see four, uh, hard drives installed. So, in the lower server, we have two power supplies. Well, if you haven't seen what they look like in racks yet. These are rack servers. Accordingly, our data centers must be built in such a way that two power inputs are provided. two inputs, uh, power, internet, telephony, and so on. Well, so that we have redundancy, well, say, the format of our data center. Well, in general, what else do we see here? That, accordingly, if we lose power or one of the power supplies fails, we simply pull it out while it's running and put it back in, and everything works. Well, this is how we ensure fault tolerance at the physical level, because today we will also talk about this. About fault tolerance, disaster tolerance, and so on. So, here you see a motherboard inside. Here, here are slots for two processors, uh, two, so to speak, two slots for RAM and so on. There are also connectors for RAID servers and so on. So, all this is done so that we do not lose the information that is stored on our disks, on our servers, on our service. So, please note, we had four disks in the upper server. Well, let's consider a simple example. Here are disks 014. These are four physical disks in our server. So, how is data arranged? We duplicate our data. Here is file A1, or a block of files A1, it is duplicated on two disks. Also, files A2 are also duplicated. Accordingly, what do we understand from this? That if disk A1 fails, if disk one fails, then we always have a copy of our data on the second disk. Well, and accordingly, they are combined into a RAID, into a mirror. Everything together is combined into a RAID, or into RAID 10. Thus, we achieve fault tolerance at the physical level if the hard drives fail. There is a very good, well, an old option - this is RAID 5 with Hotspare. It allows us to always have the ability to remove one of the four disks, or if it breaks, we physically remove it from the server and then insert a new one, initialize it, and the data will be spread across all four disks again and will not be lost. That is, the server will not need to be turned off. This is, uh, one of these types of disk aggregation to avoid data loss. Well, it is also visible that the data is distributed in some way. You see, the disks are not fully occupied so that there is always an opportunity to redistribute data between them. Now, a small excursion into building high availability. So, let's imagine that here we have a server or a workstation, and a star denotes, uh, our, well, let's say it's an Excel file or a 1C file with our salary. That is, what is very important to us, we need to save it. So, well, we understand that if, uh, during salary payment, this server fails, then we will not receive our salary. Therefore, what do we do? As IT specialists, we make a duplication, we install another server, and we make some replication of our data. That is, we rewrite this file with some frequency. So, this is how information is duplicated. There are pros and cons. Uh, so, well, firstly, it's cheap, secondly, you still need to make some copies so that they are consistent. Uh, if changes are made, then you need to understand where the correct data is now, on which server, and so on. In general, this is the first basic case. So, now, uh, how to say it? Another server appears, called a Witness server. This is a witness server, so to speak. It is needed to determine, during, for example, a reboot, power outage, and it is necessary to determine which of the servers is currently the main one. So, if, for example, the first server was the main one, and the second was the backup, then after rebooting, the first one, for example, takes a long time to recover, the second one starts up, and it can consider itself the main one, and then we will have data duplication. That is, some service, users, for example, the same thin client connects to 1C, for example, they will go to the second server and data duplication will begin. Therefore, there is a witness server, which is asked during server startup who is the main one. And thus, we always know who is the main one. That is, they will not boot up until they wait for the boot of this witness server. Well, and here we have the concept of high availability. This is a quick restart of the service on another server in case of failure. That is, we can always, uh, reboot one of the servers, for example, if we need to do some update or it breaks, then we always have a second backup. Now, this copying of files back and forth is not very convenient for us, and we can get some network storage, purchase some disk shelf, and move our files there. And our servers will connect to this disk shelf. The files will be physically protected. So, now we have two servers. We can easily switch our services between them. For example, if we have fault tolerance configured, then instant switching will allow us, in general, to ensure that the service is available. But note, there is still some redundancy here. That is, we are spending three times more money than in the first option. Now, in fact, this was not the best situation. We need a more fault-tolerant situation. For example, what could be the situation? If we have server one, and we are now rebooting it to do some update on it, then we will have two servers running for some, possibly extended, period of time. These are two and three. And it can happen that server 2 fails. Well, that's possible. Then you will only have server one under number three left, which will bear the entire load, but all services will work, well, perhaps in a slightly degraded mode or something like that, well, in any case, this will give us the opportunity to fix the second server and finally reboot the first one so that they are always in a fault-tolerant configuration. And the last principle of fault tolerance is disaster tolerance. This is when we distribute our services in different cities or in different data centers so that if, for example, in some city or part of the city, the internet cable is dug up, or even two of them, then the entire data center will not be available. Our service, for example, Client Bank will not be available. This is a problem. Therefore, we, uh, I will try to detach now. Our service will continue to work. And this is important if we are building such a high-availability infrastructure. So, well, what do we see here? That architects must calculate the number of these servers, their parameters, make some specific reserve, no more, no less, so as not to waste the budget. And the necessary RPO and RTO parameters, which are written here, target recovery time and target recovery point, so that they meet the requirements that were presented to us. That is, RTO is the maximum time during which the system can be unavailable. That is, well, roughly speaking, we can wait 3 minutes for the client bank to not work. Well, some minor failure, perhaps clients won't notice. So, and the target recovery point is, for example, all transactions that clients made during these 3 minutes, they should not be lost. That is, we must ensure that there is a minimal period of data loss. Or, if the service is non-critical, then we can restore it in other ways, then RTO RPO can be a week, a day, an hour, and so on. So, and now let's move a little from, well, we have considered the physical paradigm, uh, so, the division of resources. And now let's look at it at the logical level. Well, earlier, uh, at the very beginning, we had some infrastructure. In this case, it's a server on which we installed an operating system. Well, let's say, let it be Windows, Linux, and so on. And it had some application. In this case, AP1 is, well, for simplicity, let's take some 1C, where our salary is calculated. Further, since one disk does not use the resources of the entire server, we would like to reuse them to save money, so we can run several applications on one operating system: 1C, mail system, domain, and so on. And problems arise. If it is necessary to reboot, for example, the mail domain, it may, well, after reinstallation of an update or for some reasons, it may force the entire server to reboot along with the operating system, and then the other two servers that were running those two services will also reboot. This is not very good, because these services may be needed, they are not needed to be interrupted. Therefore, on the one hand, we save resources, we distribute one-third of all our resources to three services, but there are problems with rebooting. Then the following option was invented. This is what we had. We move here. So, hypervisors were invented, that is, some program that acts as an intermediary between hardware and operating systems. And the operating system that runs on these hypervisors understands, it, well, it thinks that it has physical resources underneath it. Accordingly, in the hypervisor settings, we allocate the required amount of these virtual resources. This will be CPU, processor, memory, hard disk, and so on, and managed. Well, and for example, these hypervisors are VMware, Hyper-V, Citrix, and KVM. Well, uh, good products, in general, that allow you to move virtual machines on the fly, reboot, create, and so on. And it is convenient to allocate resources. At the same time, under this infrastructure, there can be several servers, and the hypervisor manages several servers at once. And it even manages servers in different cities. And we can move virtual machines between these cities, well, roughly speaking, on the fly, well, in certain ways. So, well, in general, we have talked, and here it also matters that we have licenses, we can save on licenses by sharing resources in this way. And plus, if we need to reboot some application now, then it can reboot along with the guest operating system, but at the same time, all other services will not be affected. Well, and accordingly, we reduce the cost of ownership of this service. And the next one, I noted for myself, is virtualization for adults. So, there is containerization, when we install a certain operating system on some infrastructure, and on it we install, say, Docker or Kubernetes, that is, some layer for containerization. And now each service runs inside, also, well, I call it a virtual machine, because I am closer to a system architect, but these are pods, these are containers that are created on the fly, in RAM with minimal, uh, writing to the hard disk. Thus, they work very quickly. They load quickly, are easily saved, changes are made. In general, it is a very flexible system, in fact. So, but a little simpler than, uh, what is done by hypervisors in paid versions. Uh, so, colleagues, if you have questions, please write, and I will move on for now. Now I would like to move on, since we can logically move from physical infrastructure, from virtualization infrastructure to servers, to the concept of Hadoop. This is, well, a whole ecosystem, it also has hard drives, processors, servers. But what is also good about this ecosystem is that it can be very convenient, scalable. That is, we can easily create new, add new cheap servers. There is also some orchestrator that manages all this. And, well, now all machine learning, all large farms use Hadoop to store their data, perform some calculations, and train their models. Well, we will not go into what software products are here now. We will only touch a little on the MapReduce technology, which also has significance in our fault tolerance paradigm. So, Hadoop has a distributed file system called HDFS, which is very similar to the infrastructure I just described to you. That is, there are some hard drives. Data nodes are some server, it has certain resources, and there are certain programs that allow tasks to be parallelized. This happens, uh, well, I'll explain now. So, this is roughly what a Hadoop server looks like. It also has everything duplicated. Uh, for example, data nodes, this is, uh, this HDFS system of hard drives, a file system where our files are located. That is, you can add these data nodes. So, new files will be written to these nodes, replicated, as I showed you in RAID 5 Hotspare. That is, the data always has at least three replications, that is, our files are located on three different servers, data nodes. If two nodes fail, then we always have a backup copy on the third. So, we have objects for Hadoop, Spark Master, Spark Workers, which also balance the load, and additional servers to manage, in general, this whole zoo. So, and what is worth emphasizing in this paradigm is MapReduce. This is when we, uh, MapReduce technology, when we have some data, well, a large file, for example, 1x1 GB of some text files, numerical, and we need to find, I don't know, the maximum number among these, say, 5 billion, I don't know, for example, lines, then, uh, what can we do? We can split, uh, using map technology, split this file, say, a terabyte, for example, into pieces. Perhaps these pieces are already on our data nodes. That is, we have 100 servers. On each of these 100 servers, for example, I don't know, 100 MB of these file pieces are located. That is, roughly speaking, we don't even need to write our file separately to these servers. It is possibly already conveniently laid out there, so that using the Reduce technology on each of these 100 servers, in these 100 MB, we will find the maximum value, and then using the final reduce, we will find the maximum of those maximums that we found before. That is, in general, our paradigm, we have split the file, or it was already split into a large number of pieces. Each server separately, with its own resources, is engaged in the task of finding the necessary data. Then everything is aggregated using MapReduce technology, and the final result is given. Thus, such a cheap, scalable parallelization system. And this is what is actively used in banks and everywhere in companies. Well, uh, I also wanted to show you a rating of databases, which was relevant, well, as of June 2025, when I took this screenshot, because it is no longer available now, without access technologies. Oracle was in first place, then PostgreSQL, and so on. And, for example, Redis is a very fast, good, simple database, which has fallen a bit from sixth place. So, well, this is the distribution of databases, which now, uh, its rating, in general, which is better or worse. Now, let's talk a little more about infrastructure. While we have the concept of large services or large, uh, these LLM systems, language models. So, but, as we have already understood, splitting our large service gives us some advantages. Thus, we can use resources more optimally, make reboots more optimally. Therefore, microservice architecture is also popular now. And it is also easy to scale. So, uh, well, if you look at this diagram, we have some kind of monolithic system where we have, for example, five services in one, in one, uh, well, in one process. And there are microservices, which are separated, and we can put them on different servers, not necessarily a large amount of this monolith, which is heavy, which, perhaps, works well, but consumes more resources. We can use microservices to use our resources more optimally. Well, and there are some pros and cons, that yes, there is simplicity, and flexibility, and high fault tolerance, but there are also problems with testing, and sometimes even resource consumption increases, and complex communication. You need to understand well where which services are installed. So, now let's move a little closer to the topic. This is what kind of architectures exist that we need to calculate and understand. For example, we come to some job, and the Rexis system works there. Rexis is a recommendation system. It is quite complex. It also consists of services, microservices, that there is some model, some request comes to it, then there are certain indices, pre-calculated, certain filters, that is, Python libraries, scripts that are involved in searching, filtering, scoring, and sorting for this scoring system. That is, our task, for example, is to ensure that we select the necessary infrastructure to implement this service. And a slightly more complex system here talks about, uh, these GPUs, video cards, and systems for parallel computation of neural networks. So, here is an example that, uh, that these are quite complex systems now, which we will have to calculate and calculate their characteristics, depending on the GPU, on some additional improvements, which change quite often. That is, uh, this area we are talking about now, the course we are looking at our lecture for, it changes quite rapidly, it is very flexible, and you need to be on trend to, uh, perhaps, reconfigure the infrastructure quickly for these changing services. Now let's move a little to the practice of what kind of artificial intelligence services we will have to work with, for which we are preparing the infrastructure. So, well, text processing is developing the most now, but I would like you to better understand the theory, what you will have to deal with, this is the concept of embeddings. That is, when we talk about text processing, all models, even those that process images or sound, they all work with numerical data. That is, we must somehow convert images, sounds, texts into numbers. Here is an example of image processing. This is a convolutional network, to which we feed an image. Here it is shown on the left. It goes through some path. There are convolutional layers, max-pooling layers, but not so many unique layers here. Convolution, pooling, dense, and dropout. Well, in fact, four, four types of networks. Well, simply, if you are not familiar, I will quickly explain that when we feed an image, for example, of size 224x224, even if it's just one channel of grayscale, we feed it to our neural network, which has a certain number of inputs. A neural network is a network consisting of a large number of neurons in a specific sequence. It has input layers, hidden layers, output layers. So, in particular, the input layer is this first layer. The output layer is where the word "target" is written. All other layers are hidden layers. And these layers, these neurons, react to our image in a certain way. And if during training they memorize in the weights of these neurons, they memorize, in general, some real numerical values, thanks to which, in fact, if we look at this image, we take a two-dimensional image and stretch it, if you notice, it stretches into a dimension, well, in this case, 1000 classes. That is, any image of size 224x224, well, let's say with three channels too, it eventually converges to one numerical vector. A numerical vector is, well, it's actually a tensor, because now everything is called tensors. This is, if we talk about Python, it's a list of, uh, some real values. In the end, using a specific layer called Softmax, we get a probability of some class from these thousand real values. That is, we have 1000 classes. We know that in the first place is the class "car", in the second place is the class "elephant", in the third place is the class "house". And thus, our network works in such a way that by feeding an image of a certain class, we ultimately get the largest number in a specific place under a specific index in this last vector, tensor, or embedding of dimension 1000, but it will sum up to one, but it can be, for example, the largest number will be 0.1. It's just that all other numbers are also less than 0.1. But in total, they will give a thousand ones. So, at the index where we have the largest number, this means that this is the class we have, uh, at the input. So, well, this correspondence is fully recorded during network training. That is, again, in the zeroth place, under index one, we have a car, under index two, a TV, then an elephant, and so on. Thus, we always know how we get the class from the image. So, once again, I would like to stop here precisely on this figure, that by passing
Another kind of image through any image through this network, we end up with 1,000, uh, vectors of length 1,000 values. Well, then they will turn into probabilities, but for now, they are just some real numbers. That is, for example, -15, 0.15, and so on. That is, some numbers, thousands of values. Well, and, uh, so, for example, we want to solve a problem, uh, we need to create a service that will allow our employees to pass, uh, into the enterprise, without showing, uh, their pass, but by photograph, well, in the sense, by image. Then, in principle, in order to train such a network, we would need to ask for a thousand photos of each employee, where they are in a good mood, in a bad mood, with different hairstyles, with mustaches, without mustaches, with glasses, without, in order to, in general, train our network, tanned, and so on, or not. Uh, but, uh, this problem can be solved more simply, if we take a pre-trained network, and there are already quite a lot of pre-trained networks on the internet, uh, which have, however, a specific input. That is, we will have to, well, crop the input image, that is, our camera will have to output this dimension 224x24 to our network, of our two people, oh, of our person who wants to enter our company. And by running their image through this network, we get some kind of vector of length 1,000. So, if we record, once we have run this person through, taken this vector, or at least 10 photos, just taken some average value, then, uh, each person in our office will have completely different vectors. Uh, well, somewhere slightly similar, but they will be different. They will just have a length of 1,000. And if an employee comes, for example, shaves off his mustache, then, uh, we take a photo of the person without a mustache, this man, run it through our network with our service, get this vector of length 1,000, and compare this vector with the vectors we have in the database. So, even if a person shaves off his mustache or puts on glasses or changes his hairstyle, the vector of this person in the database and the one, well, respectively, of the same person who came to work, will still be closer by cosine distance. That is, if we take, uh, cosine distance, it will be less, in general, well, some threshold can also be chosen so that a stranger does not enter, because he will not resemble any of our employees. In general, in this way, we, uh, change the representation of our image, sound, or text from one space to a space of dimension one, for example, to 1,000. That is, this is what I fundamentally wanted to explain at this point. Uh, well, this is because, uh, our neural networks work with these embeddings, and with which we create the architecture. Next, I wanted to tell you about LLMs, that, uh, since, uh, LLMs do not work with text, then texts are divided into tokens. A token is, uh, not what, well, although, yes, also, what we possibly pay for. Initially, a token was, for example, a sentence divided by spaces. It was a word. Well, when commas, punctuation marks, and so on were removed, some unnecessary stop words, then a token was just a word. Uh, but now, please note, this paradigm has been abandoned, and now a token can be a part of a word. Here is shown, uh, the segmentation of the sentence "Residents of Ufa. In the evening of March 20, a rare phenomenon was observed." See, with a period, that is, this is from an open tokenizer, that is, a device or algorithm that divides into tokens. So, what are tokens? It, in general, divides into pieces and assigns a unique number to each piece. That is, a letter has this unique value. And then our LLM network will work with this value. Well, in general, this is the network released about a year ago by OpenAI, called GPT. For, please note, for the English language, it does not divide our words so much. Well, in English, you see, it contains a whole word. Well, because in English there are fewer, uh, fewer unique words. Well, and, uh, it is more preferable. That network was apparently trained on large texts. Well, and Russian also works. In general, here I wanted to show you that we convert text into numbers in the following way. We divide into some tokens. This is a trainable system. That is, why it was divided this way or not. This was all during the training of the network. It could have chosen to combine "red" or "oye", for example. We don't play with this parameter, just the number of unique words. Well, here it is visible, for example, that the maximum value is approximately around 130,000. Well, that's right. Uh, in this GPT system, there are about 130,000 unique values. There are hieroglyphs, numbers, and letters. Notice, you see, numbers have a small value because they occur more often, because, uh, why is the period at the end? Well, okay. So, well, in general, we talked, uh, that we turn a token into a number, uh, and also we can turn a sentence into an embedding, uh, what I showed you here, that is, into some vector, and our network will work with this vector. So, well, here is another type of encoding. Uh, well, in principle, this is what we just discussed, this is when we encode some tokens with numbers, and then feed these numbers, uh, into our network. But not only numbers, but also conditional embeddings. In this case, the dimension is two. Uh, that is, this is this vector, as I told you, 1,000, about 1,000 such values, only in this direction. Here, in this case, the dimension is two. That is, some numbers, they are also adjusted during training. What exactly a token will have what vector. Because, in principle, uh, tokens that are closer in meaning should also have semantically close cosine distances and these vectors. But this, how to say, needs to be checked, in general, inside the network. Well, and then, uh, we turn these words into these tokens. Then, using neural networks, uh, we, as it were, see, we pull from two-dimensional sequences into one-dimensional. And here at the end we have some class, right, in the case, for example, this is a positive class, that is, this sentence, this tweet, for example, is positive, well, kind, and this is a negative, uh, tweet. Well, networks allow this. So, what I showed you, the network is currently relatively simple. Even more complex networks have appeared, these are so-called RNNs. These are recurrent neural networks. What is interesting about them? That when we feed something into them, they produce a signal at the output, but at the same time, automatically, for example, when we want to translate the word "I love you", then, uh, when we feed "I", uh, "I" turns into a vector, well, we replace it with some vector, with some values, for example, as we replaced "I" here with this value and fed it into our neuron, then when we feed the next word "writing" or "love", then at the output of our neuron, because it has feedback, a superposition of these two words is formed. And thus, at the output, it has, uh, a superposition of the two words that entered it. Thus, it remembers the previous word and the current word. And thus, this allows, well, to remember sentences and better process, uh, sequences. There are problems with this, that it cannot be parallelized, that the gradient vanishes during training. Well. But these problems are circumvented in various ways. Well, in general, uh, this is a simple network, when we just feed a signal, loop it, and it remembers the sentence. There is the LSTM network - this is so-called long short-term memory, that is, uh, long short-term memory. What is interesting here? Here also, these cubes are trainable weights. That is, some numbers are selected here, for example, so that here you see, the forget gate. That is, when the signal passes here, there is a multiplier. If this seventh output is zero, then we have zero here, and, uh, the signal is multiplied by zero and disappears. Thus, for example, a period arrives, that is, the previous sentence has ended. That's it, we no longer need to accumulate words in long-term memory. Then we simply, uh, forget the previous sentence with the help of this gate, it's over, and we start accumulating new sentences. Well, and here the input influence is processed, well, and also regulated, how much data we should pass to the next state. GRU is slightly simplified. And, uh, the progenitor of LLMs, which have now become popular, are transformers. This is the very first transformer. Here it is, uh, small, but it's just for general development. So, here it is shown that NX means that this block is called an encoder block, it serves to create these embeddings. That is, if we, uh, quickly return to this picture, then, roughly speaking, this embedding is this part, which creates this thousand-dimensional vector of length 1,000. And the second block, the decoder, takes only this thousand-dimensional vector as input, and it must restore the sentence, for example, a translation. That is, we encoded the sentence "I love you". We got some vector of length 1,000. We feed it into the decoder. And from this, uh, from this embedding, we get the translation "I love you". And it, accordingly, goes through this entire chain. So, this NX is like this block, it can appear several times. That is, this encoder, this block, in this case, for this lecture, it appears six times, the decoder appears the same number of times. Such a system. And within this system, all the knowledge that we embed in it is stored in certain weight schemes, weight matrices. And this is what allows us to make the amazing generations that we are currently observing. But this is the first basic scheme, with which everything began. And, well, if we look at it more closely, then this, you see, this encoder, that piece, it consists of several networks. And at the same time, there is parallelization of requests. And this is very good, because, well, it trains faster. That is, we just talked about how for these recurrent networks, which have feedback, they remember sentences, that they, uh, well, do predictions and translations well, but, uh, but they are long to train, and they, uh, until the entire word or sentence passes through this recurrent network, we do not get an answer. Here, accordingly, we can parallelize, uh, feed several words at once. Well. And at the same time, uh, and at the same time, this system sees all the words simultaneously. It also has so-called attention, that is, it understands which words are more important, which are less important, and therefore also remembers this with certain weight schemes and knows which word, for example, to generate correctly next, so that the translation is consistent. These are general words about transformers. And I think there is no slide, right? Ah, so, and now these were old simple schemes, and now we have complex schemes. This is one of the simple schemes that belongs to the complex ones. Note, you see, it says here that there are several blocks from one to D. Mm. Dit bl. So, here it is drawn. Mm. Dбlк. Here it is. That is, inside, here are these, and inside one complex neural network, there are many blocks consisting of other neural networks, which are trained and allow for such text generation. Well, this is also for general development. So, you will hear everywhere that there are matrices like K value query, in which, in fact, the entire, well, the entire library of knowledge that we put into this network during training is contained. That is, here, I had an analogy that it's like a hamster with nuts. That is, the more nuts - these are knowledge from different areas - the more nuts are put inside our network, our hamster, the better it will remember and the more data it will have inside it, from which it can generate something. That is, all the magic is precisely in these K vol query matrices where, simultaneously with attention, simultaneously with positional vectors, the model remembers the data that entered it. Well, here is a small screenshot of how networks develop very quickly. So, uh, GPT and BERT were the main, uh, main, how to say, networks. BERT created more. It is an encoder, it creates embeddings, GPT is for generation. It is a decoder. Well, and T5 is also a joint architecture. Well, and there are many of them, so, uh, note, they have appeared. And, well, our task is to learn to prepare for this architecture, so that these services work, because they are not yet sufficiently, uh, flexible, they require a lot of resources. So, here I have, uh, a link, not a link, but a QR code to a Jupyter notebook, which allows, which will show you, uh, how artificial intelligence services look like. So, uh, I don't know if you are familiar with Google Colab. This is an infrastructure that Google provides for free. So, if you have an account, you go to colabarch.google.com. Uh, go to this link, and you will have this library, this Jupyter notebook, where you can execute code. So, the most important thing here, which I will show now, is the runtime type changer. These are the resources available to you. Either it's a CPU, that is, a virtual machine will be raised for you at this moment, configured, uh, where you will perform some algorithms on the CPU, or on a GPU. A GPU is, uh, video cards that allow you to process neural networks well. Well. And there are also TPUs and other types of cards, well, so to speak, paid and free, which allow us to run our neural networks. So, we have chosen GPU and click connect. What is happening now? It is allocating resources, as we have already seen in Docker, in Kubernetes, uh, before, that a pod or virtual machine is created, in which we can work. It is already configured, it already has certain libraries installed. Well, let's look at the resources. So, what resources are available to us. You see, we have 12 GB of RAM, uh, GPU, 15 GB of RAM on the video card. Well, and there is also 112 GB of hard disk. Well. And I will not show this notebook for long. The point is that we will be able to install additional libraries, which will allow us to download artificial intelligence services locally. In this case, we will now download locally to Google's servers, and they will work there, for example, recognizing images, creating voice, recognizing, and so on. But if you do this at home, run this code, then you will only need to adjust the infrastructure a little bit so that everything works, and the libraries do not conflict with each other, then you will download a basic ChatGPT locally and be able to run artificial intelligence services locally. So, there is a Transformers library. It was updated today, I think. And from it, certain functions are loaded. And what? Let's just see how it works. So, here we have installed the necessary artificial intelligence libraries. Here we have, uh, loaded these artificial intelligence libraries. And here we have simply downloaded some picture from the internet using this URL. Just the first one that was on the internet, it's a car. And, uh, we can use a service called computer vision and classify the image. We want to understand what is depicted in this picture. So, we run this code. So, how many lines are there? 2, 4, 6, 8, 10 lines are needed for us to first download the picture. This is where we download the picture. So, here we connect to a site called Hugging Face and download a Google model that allows image recognition. It will be downloaded locally. It weighs 347 MB. And so, if it's too small for you, tell me, I will increase the size. So. Uh, after that, we send our image to a specific function, the feature extractor, which we just loaded, then we apply all this to this model. Well, that is, what is a feature extractor? This is when we send the image, which is two-dimensional, to this feature extractor, what I showed you, run it through this whole network, well, of a certain dimension, and it gives us this vector, say, a thousand-dimensional one. We send it to the model. The model makes a prediction. Next, we, well, slightly differently, but for basic understanding, it's quite enough. In general, we get a vector at the output, with which we apply the Argmax method, and, uh, at the output, we get what I told you, a vector of a thousand values, where the largest value's index is our class. In general, this is done with this line. And it shows which class. So, Racer, Racer Car, and Racing Car. That is, it shows that it is a racing car. So, we downloaded a file of 347 MB using these lines. We can pass a URL or just a picture from our hard drive to this code, and it will recognize what is drawn in this picture. Let's go to this link, so you say, "Where did I get this code? How, where did I write it? Is it difficult, not difficult?" So, we have landed on a site called Hugging Face. And it has models here. And here in models, you can see the domains where you can download models. For example, audio to text, text to text, or, well, I don't know, what is this about, visual question answering. Visual question answering. So, what else do we have? Today we will see depth estimation in an image. Image to video. You feed images into this code, and it makes a video for you. Text to image. Also today, by the way, this will also be text to 3D, and so on. Text classification, you see, this relates to text in red. Computer vision, this relates to images. Well. So, you need to go to the link. Here we have object detection - this is what we were looking at with the car. And here there are a large number of 100 or more pages of certain models of people who uploaded them, well, trained them, retrained them, added something of their own. Here you can see how often they are liked, how often they are downloaded, and use this model. Well, roughly speaking, go into it. And here is the description. And you can even go, you see, to an open Colab. Immediately there will be code on how to use it. But we will use the link that was there before. We just went to Google Colab, we went to this model, it's called this. Well, we chose it on the previous screen. And scroll down to "How to use". How to use? And here we simply, you see, copy the code from here, paste it there. And we have our own local service that allows us to perform transformations. We substitute our URL, and that's it. And it works. We looked. Often there is a question, what is this model? That is, this model that we downloaded, what does it represent? In fact, it's, uh, well, the model consists of two files. The first file is in JSON format. This is like the skeleton or shell of our neural network, uh, which, well, should be created. That is, on the slide that I showed you, there were these layers, when we, uh, compressed from a dimension of 224 into a vector of length 1,000. So, how many layers were there, how many neurons, this is all written in this JSON file. Accordingly, the infrastructure, when loading, knows that if there are so many layers, everything is written in this file, and it will be able to restore it. That is, as it were, the edges are in one small JSON file. Well, roughly speaking, this is the JSON, uh, of these edges. And in the model itself, 347 MB, these are the real numbers, what I also showed you in the presentation, values like -1, -0.15, and so on. That is, there is a very large number of neurons. That is, if we say that our neural network contains 100 million neurons, then, roughly speaking, there are 100 million numerical weights. values are recorded there. These weights were trained by someone, and, uh, we can reuse them or fine-tune them, or somehow reuse them. So, let's go further. Well, let's load a picture of cats, apply our model. You see, it's no longer necessary, and it's not necessary to load it, because it's already loaded into RAM. Applied. Well, I did it in vain. In general, it determined that these are not cats, in fact, well, the model doesn't determine them very well. It determined a control panel. Well, let's look at a giraffe now. We take a giraffe, download its picture, send it to our model. Well, however, you see, it identified it as a zebra. Well, because the model is small, old, well, it doesn't identify well. So, well, let's try again. Object detection. What, what does the next model allow? From Facebook. So, it allows, uh, the entire code is reusable. You can go to the link to see it. Just copy it. What does it do? It downloads a model of 167 MB. And we get, what did we get? It identified a control panel with 99% probability in a location. And here are the coordinates of this location. So, with these coordinates, we can draw a so-called bounding box, that is, a box here. And, uh, and accordingly, highlight where it found these, uh, these control panels. This is a chair it found, and a cat it found. Well, a couch, this chair is visible, you see? Uh, well, this is the starting coordinate, and this is the second coordinate diagonally. From these two coordinates, we can always restore a rectangle. Well, if you are interested, then look at the code at the link, run it, and reuse it. Well, and in the end, uh, for example, you see, a service that we downloaded allows you to remove the entire background, and we get what is in the foreground. Well, and a mask. Let's go further. And we looked at, uh, we looked at what kind of fault-tolerant infrastructure there is, and we looked at Docker, Hadoop, uh, and so on. Now we have looked at AI services a bit, which we will need to cover with infrastructure. And what is most important now is, uh, LLM networks. So, a little ahead. For some reason, I have a bug here. Well, okay. So, a bug is an advanced LLM network that allows, uh, well, an LLM network is what I showed you, a transformer, what was the encoder, decoder, that hamster with nuts. So, uh, so that it finds our answers even better, then, uh, there are such things as RAG, which stands for Retrieval Augmented Generation. In general, it allows us to generate our queries even better. If we look at it simply, before, when we made queries in a search engine, we were given something, we asked, and we were given 10 or 15 links, and we had to find the right link, the right information on that link ourselves. This is the old internet, as they say. So, and we complained that it was not found immediately. You have to scroll through several pages until you find what you want.
But now, uh, a new internet has come to replace it, when new search engines, when we are forced, uh, to write questions well. That is, here is the new question-answering system, what we now call a prompt, it means that when asking a question, we must also add some additional contextual information, so that the model can better, uh, generate an answer for us. That is, in fact, we are now teaching people search engines to formulate questions more correctly, so that they can find answers for us better. This is an interesting, uh, observation I would like to make. Well, and the simplest example of this RAG is that we have a network. LM network - this is what we talked about, uh, a neural network. So, we make some request. A request is textual information. Find me something or write me something. What happens under digit one? We turn our request into vectors. That is, what we talked about, a vector of length, say, 1,000 or 512. We, uh, turn the request into a certain vector. That is, we do this embedding. Accordingly, for each individual different request, we will have our own unique embedding, this vector. And thus, um, for example, for football texts, uh, the embeddings will be of a certain type, as we said, a man with a mustache or without a mustache, meaning very close in distance. It works exactly the same way in LM and in RAG. That is, somewhere we have texts that we, uh, trained on, we represent them in the form of vectors. And we don't remember the texts themselves, but we remember this vector. That is, a large piece of text, I don't know, War and Peace, we send it into our network. At the output, we get some vector, from which, theoretically, War and Peace can be restored back using a decoder. So. And, uh, if, for example, we take some, uh, sentence from, uh, this text "War and Peace", then we can find a correspondence with football, with vectors related to football, compare with cosine distance or with vectors related to War and Peace. Accordingly, War and Peace will be closer in cosine distance. So, our answer will be searched there. Well, we, accordingly, create a vector and look for cosine similarity with the data that we have already digitized. What is chunking? This is when our vector, for example, is very large. Well, War and Peace, in fact, cannot be compressed into a single thousand-dimensional vector. It would be too crude. Therefore, we break it into chunks, that is, into pieces, say, by, I don't know, by 100, for example, let's imagine by 100 tokens and, uh, send them in portions like this. In general, what do we do? Using cosine similarity, we determine, uh, compare with vectors, get the top k closest in cosine distance, and feed them along with the request, uh, into the user's request. That is, a person asked, uh, who wrote War and Peace? We, uh, sent this question to our database. There we found some data about War and Peace, about Tolstoy. Then we put it all together and sent it to the LLM along with the question, who wrote War and Peace. And then it will be able to answer us better, because it will understand what it is about. Well, and accordingly, it will give us an answer. This is how the network works. And for, uh, LLMs, which are now good at generating text, we need to create an architecture for them. And what are the tricks here, what these services consist of some parts, for example, Vector DB databases, FIS, and so on. We also have a problem. If many people connect to our LLM simultaneously, it thinks for a long time, you probably also noticed, it can think for a minute, or 20 seconds. In general, we need to ensure that there is some load balancing for our users. So, initially, maybe 10 users can use it, and then maybe the whole company, 10,000, and so on, and more users. Accordingly, we must somehow do load balancing. We will look at this now. So, also, well, we can use engines for this, uh, vector databases are necessary, because we store data in these vectors, so that we can quickly find these vectors, their correspondence, indices, and so on. Well, a cache is needed, what I told you about Redis, it's also a very fast database for local calculations. And for analyzing how our network works, whether it has freezes or not, Prometheus and Grafana are used for tracking metrics and so on. And Kubernetes, so that we can quickly create new instances if, uh, many people start using our service simultaneously. Now, let's look at a few basic principles, uh, of creating architectures for artificial intelligence services. First, scalability is needed, well, here, in particular, horizontal and vertical using Kubernetes, separation into components, asynchronous processing, which relates to scalability, fault tolerance, which we have covered, replication, backup nodes, traffic redirection, Kubernetes - these are the infrastructure components that will allow us to achieve fault tolerance. We need to increase performance. Well, of course, it is quite expensive physically to buy farms of video cards, so there are some additional, uh, software-level options for quantization or model compression, storage of private requests, optimization, load distribution, modularity. Well, separation into microservices. This is exactly what we talked about, microservices fit well here too. And we need to learn to calculate all this so that this system works, works fast, works well, and is fault-tolerant. Well, and security, where would we be without it? So that the question asked by the user is answered to that user, and not to another user or not to everyone. This also needs to be considered. Feedback, model updates, logging, analysis, also specific services are needed to build our architecture. Well, and there are also cloud platforms, containerization, orchestration, and so on. But if we are talking about large firms, then, accordingly, we will need to learn to deploy locally. This is in the firm where we will get a job. So, there are still a few slides left. So, how do video cards physically look? Well, you are probably already familiar with them. These are the gaming video cards from which everything started, but, uh, they are being improved, so special motherboards, special power supplies, cooling, and so on are needed for them. Accordingly, this is the physics, which is quite expensive, or very expensive, which we need to be able to service now. Well, here are just pictures of what it looks like. There is Tesla, a farm, and so on. Here you see some kind of disk shelf, I just noticed. And now there is hardware, and there is software. In this case, when we talked about data, prompts, embeddings, these were some numerical vectors. So, when we have a large model trained by serious guys, Google and others, they use a large number of digits after the decimal point to encode these numbers into embeddings, because, uh, the fewer, for example, digits after the decimal point we have, the less precise our model is, it works worse. But, accordingly, the more of these values there are, the more they need to be stored in memory, stored on a hard drive, so the sizes increase sharply. Therefore, our and researchers' options are, for example, to download someone else's model, which has a large number of decimal places, a large dimensionality, and do some kind of truncation or rounding. And, accordingly, the dimensionality will be smaller, it will work a little worse, but, in principle, it can be reused. This is important for calculating RAM. Now, this is the only formula in this lecture, let's look at it. So, for example, we can download some model from Hugging Face, where we went today, a model already known to you like Llama or Deepseek, directly from there. It will be local on your machine and will generate some answers in the same way. And this quantization is precisely the number, the dimensionality of real numbers, the number of decimal places, and, uh, the weights, which we have a lot of, they occupy a certain amount of memory. That is, each of these real numbers in RAM occupies a certain amount of space. So, accordingly, we must, how many parameters, how many neurons we have in the network, so many, uh, multiply by the dimensionality of one of these real numbers. That is, if we have a sixteen-bit, uh, not generally, FP16 dimensionality, then on the next picture we will see how it looks, then 2 bytes per parameter will be used. That is, if we have, for example, 7B, 7 billion, that is, 7 billion neurons in this network, and for each, uh, for each neuron we need 2 bytes to write this number of such dimensionality, uh, 16, then we need 14 GB of RAM for our network to work. But this is not all, a lot of additional memory is also needed, for example, for activations, several gigabytes may also be needed. And from colleagues, I took this table, where, for example, we take some network with 32 billion parameters. If its dimensionality is 32, then 128 GB of RAM is needed. If we reduce the dimensionality by half, then less RAM is needed, and so on. That is, down to 16, we can deploy it on our local machine, but then it will be a network that will hallucinate a lot, because everything has been cut off from it, all its variability. So, I have already shown you this slide. These are the resources available to us in Google Colab for free. Well, the link is to the code that I showed you. So, questions. And there's still a little bit left, well, a couple more options. Load balancing can be done through this option described. Or, again, colleagues, uh, suggest using DNS load balancing, BGP, HAProxy, RabbitMQ, and so on. And all this, this is the infrastructure that supports, that provides load distribution. And the network itself also goes further with its parameters. Therefore, the picture becomes even more complex. This is what we need to learn in the course, so that we can support and create our infrastructure. Now, I would also like to tell you a little bit, literally, I will take a little of your time, about the directions of machine learning. That is, for example, if you decide to move from the system architect role to the field of machine learning. Well, not completely, but, for example, so that you can create infrastructure for these services. So, there are several directions. Well, in this case, four are shown here. The first is data engineers - these are people who are involved in data delivery. That is, they work with tables, logs, parsing of sites, and so on. And they put all this into a table and then send it. They can be used in artificial intelligence libraries, in machine learning models, for predictions. Next, what we are closer to - these are ML engineers, software engineers - these are people who understand hardware, operating systems, Linux, virtualization, and so on. That is, those who create infrastructure for machine learning services. There are also business analysts - these are people who have a good understanding of data. That is, they may not know anything about machine learning, about hardware, but they know well. For example, if it's about accounting, then debit and credit, they know everything very well. There are also individual researchers who, um, are only involved in these high-level models, build some hypotheses. So. But in reality, there are quite a lot of professions, and those who do fraud, recommendation systems, text processing. These are all, as it were, separate specialties. So. And let's just look again. I barely, in fact, found a vacancy today that is relevant to our lesson - this is the design of corporate AI system architecture. And what needs to be done here? See, configure and deploy server racks, which we studied today, connect local and cloud for corporate tasks. Development and assistants. Well, this is essentially what I showed you in Google. There is also what can be attributed to this. Technical specifications, testing, handover to the team. And what is required? Experience in design, implementation, experience working with LLMs. Pay attention, with databases, integration with other services, so that we can transfer data to the LLM and get the necessary answers, understand these specific libraries and technologies related to LLMs, such as LangChain, LlamaIndex, and so on, experience working with GPUs, and experience in building AI systems and NLP pipelines. That is, to be an architect, you still need to understand quite a lot about what these new types of neural networks, LLMs, are, how they work, and so on. But on the other hand, there are very few such vacancies. That is, the competition, I would say, is minimal, because this essentially combines two professions: data scientist and architect. So. And as a result of the course, you will be able to get, this is to create your own project, that is, to set up some LLM architecture locally and present it further. To include it in your resume, to update it, to improve your position. It is optimal to post the code on GitHub so that people can see what you have done. As an option, write an article on Habr, because there is only one article on architecture, conditionally, like this, plus or minus, which can be cited, in general, there is not much data. Well, and some real tasks that are currently important in practice are data labeling. That is, now, in order to use these LLMs, for example, in a bank, for example, or in some company, it works with its internal documents. These documents may have some closed structure, and sending this data outside for some servers abroad to process the data is not an option. Therefore, it is necessary, there is an important and in-demand task now - this is, data preparation within companies, data upload, LLM upload within companies that will process this data. And, well, this is a sought-after profession, in general. Accordingly, data parsing is also important. Well, and again about data labeling. That they are in certain formats. Not only everything is in text, but it can also be logs, it can be Word, PDF that need to be recognized. So, this also needs to be learned. This is very useful and important now. Well, and training current existing LLMs, or training them from scratch on your own data is also quite in demand now. So. If possible, please evaluate my presentation via the link. It is indicated at the very beginning of our chat. Next, and literally 2 more minutes about the course, what you can see via the link, what it represents, what knowledge you will be given. So, what is it? And strategic foundation, and architecture planning, quality, security, infrastructure, patterns, and you will do a project. Well, what I told you, if you, uh, well, I will answer questions now. So, the course program, the teachers for this course, but everything will be clearer later. Well, what's good about it and how it differs is that these are live webinars, you can always see recordings, homework is checked, feedback is given quite closely with colleagues, they communicate in chats. So, my hours are beeping, buzzing, because there are also student conversations in the chats. And you will be able to do a project. A project is your own service, with which you can sell yourself to transition to a new profession or earn money. So. And updates, yes, we update courses quite often based on your feedback, which you could also give me. So, via the link in the QR code, you can see more about this course, that it starts on December 24th, lasts for 4 months. So. Well, and, let's summarize, so as not to rewatch. We talked about the basics of architecture, about fault tolerance DR Full Tolerance. We looked at how the cloud is structured, talked about microservices, talked about LLMs, looked at how texts are processed, what tokens, vectorizers, embeddings are, what LLMs and RAGs are, we also touched upon. The lecture was quite intensive today. We looked at the properties of artificial intelligence architecture, looked at LLM infrastructure, some load balancing, talked about professions, about demand. Well, and the next open lesson you can see on December 15th about system design. There are some discounts here. Ah, it doesn't show what was written here. Well, this promotion didn't work, so. Two courses 20%, three courses 30%, some very New Year's discounts, you can read in the OTUS news channel and come to study with us. So, colleagues, questions. So, Docker is not a hypervisor, but, uh, in my understanding, so, yes, thank you for the question, that Docker is, well, a special kind of hypervisor, so, okay, I won't argue about that. So, the dimensionality of the embedding is one, and its length is 1,000. Yes, thank you. This results in a tensor of dimensionality 1 by 1,000. So. Well, and we figured out the VRAM. That's all, colleagues, thank you very much. I wish you good luck. Goodbye.