Transcription
Kafka is a technology that is used absolutely everywhere today. It is listed in the requirements for all job openings. It is used in all projects from Big Tech to startups. And the ability to work with it is key for any modern developer. And therefore, in this video, with magnificent animations, we will break down all the key features of Kafka. And I guarantee that by the end of this video, you will have a simply wonderful understanding of what it is and why it is needed at all.
Hello, my name is Vlad. I work as a backend developer in one of the best companies in the world and live in Amsterdam. On my channel, I talk about how you can become a super-powerful programmer in the shortest possible time. And in this video, with super-clear diagrams and extremely simple words, we will break down all the key aspects of Kafka. We will talk about what a message broker is, what problems it solves, what Kafka is specifically, what topics are in Kafka, what partitions are, and why Kafka is so popular and used in companies like Netflix, Google, Yandex, and Ozone. Yes, everywhere. And I guarantee that by the end of this video, you will simply be able to consciously start using Kafka in any of your projects, whether for study or at work. Let's go.
And, of course, to understand Kafka well, you need to figure out where it originally came from and what problems it, in general, solves. How does any modern web application typically look? You have users, you have some frontends, all sorts of buttons, forms, input fields, and so on. This can be a web frontend, that is, what you see in the browser, a JavaScript application, or it can be a mobile frontend – a mobile application. And these frontends communicate over the internet with what is called a backend. This is a separate program, running on some server, which contains all its logic, all the logic of your application, and this program possibly also communicates with some database. This is how a typical modern web application looks.
And historically, it happened that developers, when creating backends for their applications, the most high-load programs, where all user requests from the frontend fly, where all operations are performed, everything happens together, all features are located there. Such applications are usually written, for example, in Java and Spring. So, historically, programmers created them in the form of so-called monoliths. This is when all the code of your application is concentrated in one single project. That is, it is literally a single codebase, a single Git repository. All features are located there. If you launch your application, you launch all these features on one server. That is, it is a whole indivisible application, which, to launch, you need to put on one computer entirely and turn it on. That is, all features are interconnected, and they provide the functionality of your application. But because they are very closely linked to each other, the larger your application becomes, the less convenient it is to develop, the more expensive it is to maintain. And therefore, at some point, people decided that it was time to move to the concept of microservices. Which, in general, they did. They said: "Here we have a monolith, it consists of some different features that, as it happened, interact with each other due to a single project. What if we move each of the separate features into its own separate project, all also written, for example, in Java?" Yes? But it will be a project completely isolated from all other features. It will run on its own separate server, yes, so not connected to any others, and perform only strictly defined functions. Thus, development becomes easier. It is cheaper to launch and scale such applications, that is, to handle large loads, and much, much more.
If you want to learn more about microservices, then I have a very, very detailed video on the channel, also with animations, "Microservices in 1 Hour." It covers all the basics, all the problems of monoliths, and all the dividends that microservices give to developers. Why everyone wants them today, be sure to watch it.
So, these separate applications, which are called microservices, now interact not through code, not through method calls, as before in a monolith, that is, literally code called code, right, but over the internet, because they are different programs running on different computers. And to transfer data to each other and, as it were, to assemble the entire system of your application, they need to use, well, some kind of communication channel. The most popular communication channel today is, in general, the HTTP protocol, which we use to transfer data over the internet. Data is literally text in a specific format. Accordingly, we transfer it over the internet from one program to another, get some data back from that program in response, and do something with it.
But it happened that this data transfer over the internet, using HTTP protocols, has a number of problems. And therefore, in the modern world, when people develop a microservice architecture, they very often use not only simple data transfer over the internet but also message brokers, such as, for example, Kafka, which we will talk about today. But before we get to what Kafka specifically does here on this diagram, we will first understand why this internet interaction between microservices has problems and what exactly they are, because Kafka appeared as a solution to these problems. And for a complete understanding of when and how to use it, you need to understand what problems it solves. Only then can you understand whether I need to add it to my project or if, in principle, everything is fine. And this is the skill of a true developer.
A quick pause to thank you for watching this video. It would be great if you liked it and subscribed to the channel. It really helps.
So, what, in general, are the problems with the HTTP protocol and data transfer over the internet? We said that we can simply send data, some text, from one program to another, running on different computers, in general, yes, and get some data back in response when that program, to which we sent it, performs some operations with what we sent it. But not everything in the world is so ideal. The thing is that there is nothing in the world that works 100%. And it happened that all networks over which you transfer data, particularly the internet, are unreliable. And data, when you transfer it, in general, over the internet, can disappear in the middle, literally between two programs, I don't know, a failure can occur, data may not be delivered completely with some probability, and you have no control over it. It is impossible to guarantee that data will always be delivered accurately with every request from one point to another.
Moreover, you can successfully deliver data, yes, from one program to another, but when you receive a response from that program back, this is also data transfer over the internet, but only in the opposite direction. And the same situation arises there, meaning you might even send a request but not receive a response to it. That is, networks in the real world are unreliable. And, accordingly, when I develop my program, if I send requests to another program, I always, as a developer, have to think about it. What happens if the internet connection breaks in the middle? Will I have to resend the data or make sure that I wait for a response for a certain time, then try to send the request again, and so on, and so on. This imposes additional obligations on the developer. And besides, doing this, well, it's not always easy, depending on what specific operations and data you are sending.
Moreover, when we use HTTP protocols and send data over the internet in this way, it turns out that the program that is the sender of this data, after sending it to another program, must wait for a response from that other program. It cannot simply send data and go about its business, because it is obliged to receive a response to ensure that the data was accurately delivered over the internet, because if there is no response, then either it was lost in the middle during sending, or during receiving the response, it was lost, yes, and we cannot allow that. And therefore, we honestly must wait in the sender program. This is not always convenient if the operation that the recipient program will perform with our data is very, very long. And then it turns out that the sender program has sent the data, in general, and stands, waits, and does nothing while the other program processes this operation. Yes. And we are, in general, in a situation where our program starts to work slower because of this, simply.
You might tell me, "Well, Vlad, I can send data asynchronously, yes, process it in the recipient program in another thread, and, in general, the sender will do something else while waiting, and then, if a response comes, then, in general, we will consume it, and we will implement the logic for additional attempts if the data does not arrive through some timeout. We can take care of all this, yes. But there is another interesting thing here. When you send data from one program to another, and the second program, let's assume, receives it and starts some long operation to process this data, where is this operation performed, in general? It is performed in the memory of the computer on which the recipient program is running. What if you send data about some user payment in your application over the internet? Another program receives it. Let's assume everything is fine with the network. And it starts processing this payment, in general, yes. And let's say this operation is quite long. And it turns out that in the middle of this operation, the computer on which the recipient program is running, in general, suddenly explodes, burns out, is unplugged, it simply fails. It turns out that the program that is currently processing is simply turned off. And, in general, where was the data that it was processing stored? It was stored in its RAM. If I now restart this program after fixing this failure, then it turns out that the data it was processing has simply disappeared. It turns out that on the sender's side, I either have to wait a very, very long time and constantly try to get this data and send this data while the server that failed is being restored, or I have completely lost the data. And the server can be down for a very long time, in general. Yes. And this is the main problem. In the modern world, I cannot afford to lose data just because my computer broke down and the RAM was cleared after restarting it, or it became unusable and was thrown away. If, for example, this is very important data, such as payment data, user registration data in super-important institutions, it can be anything, something that our application must guarantee will definitely happen, and suddenly it turns out that we have lost the data. But our users will be extremely unhappy, we will be quickly kicked out of business, and there will be nothing to develop, because no one will use an application that loses their money data, in general. Well, imagine you made a transfer to someone in a banking application, and this transfer was lost in the middle because a server somewhere turned off. Well, you will definitely switch to another application, especially if it's a substantial amount. And, in general, this is one of the most important problems that have to be dealt with.
Kafka is excellent, but the most important thing is to learn to use it in your projects, to understand when exactly to add it to your architecture, what tasks it solves, which feature will work more efficiently using Kafka. At the link in the description, I have added a huge guide, a cheat sheet for Kafka, where we break down all the key concepts. There are a huge number of diagrams. I explain why Kafka is so fast. You can dive into it even deeper. We also discuss a lot of typical interview questions about Kafka, so you can prepare for them even better. You can get this guide completely for free. So, follow the link in the description and prepare for interviews at the best companies in the world.
Moreover, when we are developing, and let's say we have application one that sends data, and there is application two that receives this data over the internet. And then we say: "Well, actually, we have a couple more applications, microservices, where we want to send the same data from application one to them as well." And what should we do then, as developers? We need to go into the code of application one and add the logic right there. In addition to sending data over the internet to application one, you also need to send it to application two, to application three, and so on, and so on. Yes, and the more such applications appear that are interested in the exact same data that our first application sends, the more additional code I will have to write in the sender application, yes, because it needs to know where it is sending this data. And thus, the code of this application, the more connections it has, the more interested parties, a, services in its data, although it is the same data, the more recipients there are, the more code I have to write. And any change to your application is a potential bug. Moreover, data is also sent over the internet here, which means that for each application separately, the same problems that we discussed above apply. And thus, your system, besides being not reliable in this case, is also not scalable, because tomorrow another service appears that needs to receive the same data for its, say, operations, and you need to change the Sender microservice as well, and then another one appears. In general, systems evolve, and you constantly have to update this, but it's not very efficient, it's error-prone, it's slow, in general, it's not very good. And people, naturally, started to fight this in every possible way. And it is precisely to solve these problems that special programs called message brokers appeared. And Kafka is one of these programs.
Well, let's break down in super simple words what this message broker is at all. Imagine the same situation. We have a microservice that sends some data, and we have a microservice that needs to receive this data and do something with it. Taking into account all the problems with data transfer simply via the HTTP protocol, people thought that there is a high probability that some error will occur, in general, during the transfer of this data or the data will be lost. And therefore, it would be great if, when we transfer data from one point to another, there was some intermediate link in the middle that could store the data we are transferring, so that if some problem occurs during their transfer, they would still remain in this place where they were stored, and could then be retrieved again by the recipient. And thus, we could hold them in the middle in case something goes wrong during transfer or on the other side.
A message broker is simply a special program that runs on a regular computer, which knows how to receive some data that you send to it. Usually, it's text in some format, yes? You can write data into this program from one side, it will store it internally, and on the other side, another program is connected to it, which is the recipient of this data. And it constantly polls this message broker program to see if there is any new data for it that it needs to process. And if such data suddenly appears in the message broker, someone sent it there, then the message broker tells this program: "Here, please, a message for you. Take it and go process it."
And in a situation where we have a program that sends data to a message broker, such a program that sends, publishes, produces messages for the message broker, is called, in this terminology, in general, a producer, yes? Producer is an English word that means manufacturer, that is, a producer of data, a producer of messages that are transmitted through a message broker. And the recipient on the other side in such an interaction is called a consumer. Consumer is an English word that means consumer. That is, it is a program that consumes, takes messages from this message broker, which is essentially a separate intermediate program.
And this situation, when we introduce this intermediate link into this interaction in the form of a message broker, which, in general, stores the data in the middle that we transfer from one point to another, yes, gives us a huge number of advantages compared to the situation we had before with regular HTTP interaction. Let's look at the advantages we have gained.
Firstly, we can now truly process this data that we are transferring asynchronously. That is, what happens? Here we have a posts microservice on the left, which sends some data. But now it doesn't communicate directly with the notifications microservice, but with the help of a message broker. That is, it put the data into the message broker, and the notifications microservice already took it from there. And what happens? That the posts microservice, when it put the data into the message broker, in general, the message broker guarantees that it has definitely caught this data, stored it, and now it will not be lost anywhere. No problem can arise. This is a guarantee that the message broker program provides. Accordingly, in this situation, the posts microservice can send this data. When it, in general, ended up in the message broker, it says: "Well, I am generally no longer interested in when and who will process this data. I know that I sent it. I know that the message broker guarantees that this data will be delivered. I don't know exactly when, but I know that it will definitely be delivered. This is what a message broker does for you. And as a result, the posts microservice, having sent this data, can immediately go about its other business. It doesn't need to wait for a response from the notifications microservice that the data was successfully received, the operations were performed, yes, because it no longer needs to ensure that everything was successfully delivered. The message broker guarantees this. It says: "I'll take care of everything." And the posts microservice: "Great, here's the data, take it, and I'm off to do my own thing." And as a result, the posts microservice, that is, the producer, any producer in this architecture, starts to work much more efficiently. That is, it simply does not wait for a response from other consumers of its data. And the message broker, having received this data from the producer, stores it to guarantee that it will not be lost anywhere. And the notifications microservice, that is, the consumer in this architecture, takes it at some point when it is ready to process it. And let's assume it's a very long operation, but it takes this data, starts, in general, processing it. And regardless of how much time it takes, the posts microservice, as it didn't worry about it, so it doesn't worry. It doesn't wait at all for the Notifications microservice to process it, because it is no longer directly connected to the Notifications microservice, yes, and, accordingly, can simply go about its business.
But you might tell me, Vlad, what will happen with that problem when the notifications microservice, in general, can simply fail in the middle of executing this long operation? What if its server, yes, burns out at that moment? And here the advantage of a message broker reaches a new level. If my notifications microservice burns out, yes, and fails for some time, and all data from its memory is erased, yes, then the message that it has already taken from the broker, it did not manage to fully process, everything failed in the middle, but still, we said that the message broker stores all the data that comes into it. Thus, when we restart the notifications microservice on the same computer or install a new, not broken computer, and launch it, then, accordingly, the notifications microservice will be able to come back to the message broker and ask: "Are there any unprocessed messages for me?" And it will tell him: "Yes, here, this is the very message that you already tried to process, but you didn't succeed. Try again, I have stored it for you." And thus, you do not lose messages, you are simply sure that they are stored somewhere, something failed, but after some time, when the components of my system return to work, they will be able to try to process this data again, even if they managed to lose it from their own memory. The message broker has stored it for you, and you can try to process it again. And this applies to all messages that you send to it. And this is a magnificent advantage. Your system becomes much more reliable when there is a message broker that holds all the data that you transfer between different microservices. And you are sure that they will definitely, one day, you don't know when, yes, how long your microservice will be down, when it will be ready to consume this data. But you definitely know that one day it will process this data, it will not be lost anywhere.
But moreover, imagine the same situation when we publish some message in the posts microservice. Now the notifications microservice takes it, and then several more microservices appear that also want to consume the same message. Exactly the same, the same data, the same piece of information. Previously, we were forced to modify the posts microservice and say that it should send this piece of information to all interested consumers. But now, since this piece of information is stored within the message broker, any interested microservice can now simply connect to this message broker and say: "I need the same messages that your notifications microservice consumes." And it tells him: "Yes, I have everything saved. It's all in me. Take it. Just take it and process it further." And thus, any new microservice that you add to your system and that is interested in the same data does not affect the posts microservice, that is, the producer, in any way. The producer doesn't even know how many consumers of its data exist. It's not interested. It knows that there is a message broker. It pushes data into it. And who will consume it? It doesn't matter at all. And there can be a hundred of them. And they can all consume a copy of the same data for different needs. And thus, you ensure that your system scales flexibly in terms of functionality. When you need to consume the same data, you set up a new microservice, connect it to the existing message broker that manages these messages, and start consuming them. And even from a convenient point for you, which is possible in Kafka, but that's a separate topic. As a result, we don't change the posts microservice at all, which means there can be no bugs there. And it simply continues to work as it did before. This is simply a magnificent advantage in terms of speeding up and reducing the cost of your development and improving the quality of your system. And that's why message brokers are super popular.
But you might also tell me: "Well, Vlad, in such an architecture, we have a message broker, a separate program running on a separate computer, and you say that it stores the transferred data. In general, there. But what happens if this computer burns out? What happens if the computer with the message broker burns out? Well, its hard drive or RAM burns out. All the data, it turns out, disappears. The computer simply broke. What to do about it? We will definitely talk about this a little later. We will solve this problem. Stay with us and be sure to watch the video further.
But as for message brokers, there are different ones. Message broker is a collective name for different programs that provide these services for transferring data from one point to another and storing it. There is, for example, ActiveMQ, a rather old message broker. There is RabbitMQ. A very popular solution in the modern world. Redis can be used as a message broker. Perhaps you didn't know about this, but it can be done. There are special tools for this. But the most popular in modern realities, which is used in the vast majority of projects, is, of course, Kafka. It is a separate type of message broker, a separate program that is called that, but it is a message broker. Nevertheless, it has several important features that significantly distinguish it from other message brokers. And we will talk about them now.
So, how does Kafka differ from all other message brokers and why is it so popular and everyone wants it? Look at the diagram. Here we have Kafka, this message broker. We publish data to it from the posts microservice, and the notifications microservice consumes it. But, of course, the nature of the messages that we publish to Kafka can be different depending on who publishes them and when. Yes, Kafka is one, but there can be many types of messages. Well, for example, I can publish a message from the posts microservice that a post has been released and the notifications microservice needs to notify everyone who is subscribed to the author of this post. And also, for example, if I process likes for these posts in the posts microservice, I
I can send the author notifications about likes, about comments, and all of this would be different messages, containing different data. Well, for example, a notification about a like is 100% different from a notification about a comment. Yes, it's trivial, because a comment contains some text. Even in the current diagram, it looks as if I am just shoving all messages, regardless of what data they contain, into this Kafka blindly. And on the other side, the notification microservice takes them out and has to understand what has actually arrived, to parse this data, to comprehend it. This message relates to comments, this message relates to likes, this one to posts. Well, the code of each recipient or consumer of these messages would then be very complex. We would have to write this logic of calculation in each of the recipients, even if you consume the same data, what specific message has arrived, is it about comments or about likes. Well, that would be, well, unimaginably complex. And therefore, in Kafka, there is the concept of topics. What is a topic in Kafka? In fact, Kafka is not just a dumb pipe into which you shove any data from one side and receive it on the other side. Kafka actually represents a set of so-called topics. What is a topic? Topic is an English word that translates as "theme." Essentially, when you work with Kafka, within this program, you can create as many different topics, i.e., message themes, as you like. A topic is effectively a channel that you, as a developer, create for messages of a specific type. Thus, by creating a topic, you say: "Into this channel, into this topic, only messages of a specific theme will be published from one side. For example, you can create a topic for post notifications. You can create a separate topic for notifications about the creation of groups in your social network. You can create a separate topic for notifications about comments, about likes, about some reactions, about recommendations, about anything, about messages, right? And thus, depending on the content of the message, producers who publish this message to Kafka will shove it not just into Kafka, but into some specific topic, i.e., into some separate channel dedicated specifically to messages of this type. And then, on the other side, the recipient of such messages will know exactly that if I received this message from the groups topic, for example, that some group was created in my social network, then I know for sure that this message relates to group creation. I don't need to parse it, I don't need to understand what it represents myself on the consumer side. I know that if I consumed it through the groups channel, then, in general, only messages about groups can be there. So, I know immediately how to work with them. Yes, they have a specific format, they definitely relate to this subject area. And thus, your recipients do not get confused in the huge stream of messages, right, when they, in general, consume them. They know that they are consuming them from different channels. It's like on TV, there are also channels. On one channel, only news is shown. And you know that if you switch there, there will only be news. On another channel, only sports are shown, and it's called sports. You have clear expectations. If I want to watch football, I switch to the sports channel. It's the same in Kafka. If I want to receive notifications about group creation, I connect to a specific topic for group creation. If I want to receive information about, say, likes, comments, I connect to the topic that contains this information. Yes, and thus I know exactly what to expect. Of course, all these topics exist within the framework of one program, one Kafka. You connect as a Kafka administrator and can create as many topics yourself as you need to properly organize interaction in your system. And moreover, if you look at this diagram, different microservices can act as producers to different topics. That is, globally they all write to Kafka, but they can write to different topics that exist within this Kafka, depending on the type of messages they publish, right? And moreover, each microservice you have can simultaneously be a producer to some one topic, right, to which it writes data to Kafka. And simultaneously, it can also be a consumer, i.e., a data consumer from another topic. Thus, you can build much more complex systems that consume some data, transform it somehow, publish the transformed data to another topic, and so on, and so on. All this interaction can be in any direction. And you can build it as you wish precisely thanks to topics, because a topic is the same as a channel. They are absolute synonyms. And Kafka allows you to thus properly organize the redirection of your data, different messages to all recipients who are specifically interested, right? And, accordingly, when you scale your system, you bring in a new microservice and say: "My microservice is very interested in notifications about someone subscribing to an author of some posts, and you just find the corresponding topic." Here it is, the subscription topic, for example, it's called. Connect this microservice to this topic and receive only messages specifically related to subscriptions. Because on the other side, there are producers who specifically publish such messages to this topic. Not to some other one, but they have code written that says: "Publish specifically to this topic." But, of course, you, as a developer, must ensure that you put certain data into a certain topic. Which topics and how many to create is also up to you. You, as a developer, must think about what types of messages exist in my application, what I want to publish here, what to this topic, and so on. This is about organizing the interaction of your system, and you, as an engineer, must deal with these problems and think about them. But topics give you a huge amount of flexibility in this regard. A curious viewer, in general, will tell me: "Vlad, you said you have a producer. It's a program running on one computer that sends data to a message broker, which is another program running on another computer. And before that, you said that, in general, when we transfer data from one program, from one computer to another computer, we transfer it over the internet. So, if here you have a producer on one computer, and Kafka on another, then aren't you transferring data from producers to Kafka over the internet? Over the internet. If networks are not reliable, as you say, then can't the same problem arise that we had with ordinary HTTP interaction, and you would be absolutely right. The exact same problem can arise here, because it's just data transfer over the internet. But then, what is the advantage of using Kafka, in general, in this case, over ordinary internet transfer? The fact is that Kafka, as a program, has certain guarantees that it gives to its users. If you use Kafka, you can configure it in a certain way so that it guarantees a certain format of message delivery. For example, when launching Kafka, you can say that it should guarantee that messages that you publish to it will be delivered to the consumer, i.e., the recipient of these messages, at most once. Note, this is a very important condition. At most once. This means that a message that you publish to it with such Kafka settings will be delivered to the other side either exactly once, or less, i.e., zero times. So, this implies that a message may not be delivered, it may be lost. And such a guarantee, which is specified in Kafka settings, you can change it. We will talk about what else there is. Such a guarantee is called "at most once," which translates from English as "at most once." Why does this happen? Precisely because of the problem we described above. If, with such a guarantee, with such Kafka settings, I publish a message from the posts microservice to Kafka itself, and in the middle of transferring this message over the internet, some error occurs, right, and the message is not delivered, then the posts microservice will proceed, in general, with its own business, because it sent everything to Kafka and, in general, can forget about it. And Kafka will not inform it that the message was not delivered. With such settings, it will work exactly this way. Accordingly, in some situations, such messages can simply be lost during internet transfer. The same can happen on the other side, when consuming these messages from Kafka, because they are also transferred over the internet. If a message, in general, arrives at Kafka, but then, for example, the Notification microservice tries to retrieve it from there, and it, in general, disappears, and Kafka does not save it internally, with such "at most once" settings, it guarantees delivery, then it will also be lost. You might ask, why does such an "at most once" delivery guarantee even exist? That is, we are saying that with certain Kafka settings, a message that we transfer may not be delivered, it may be lost. We just discussed that this is a big problem. Kafka tries to give its users, in general, flexibility. Sometimes there are messages that you transfer through Kafka, and if they are lost, nothing bad will happen. For example, when you transfer some analytics data, for example, how many times a user clicks on certain buttons. Well, like, if we lose one click, well, it probably won't significantly affect all our, uh, overall metrics, or when we transfer some specific logs that are not very important, but it's useful to have them to extract some patterns of interaction, and so on, and so on. That is, if I lose some message with some probability, obviously, it won't happen constantly, it can just happen with some probability, then nothing bad will happen from its loss. But if I specifically say in Kafka that deliver all messages to me at most once, then in fact, Kafka will work faster with such a configuration, because it simply doesn't need to do internal saving of all this data it receives. It simply doesn't need to make additional attempts to deliver these messages. And thus, when a message successfully travels through this delivery path, it, in general, travels very quickly, because there are no overheads for its, say, saving and additional attempts. If it gets lost, then it gets lost, right. This is some kind of exchange, what we pay for faster message delivery. It just happens that there are situations when you have a lot, a lot of data that can be lost, but because there is a lot of it, you write a lot of it to Kafka. And if you said, "We need to guarantee that all of them are definitely delivered, we need to make sure they are saved," and so on, and so on, then it simply works slower. And if you understand that you have some part of the data that you are okay with losing, you can specify in Kafka that for this data, let the delivery format not be guaranteed, but we will deliver them at most once. That is, we will definitely deliver at least once, but we can also lose zero times delivery to the other side. This is possible. But Kafka is not valued for this. In fact, the "at most once" delivery guarantee is provided by, for example, Redis, and it will be very, very fast in this regard. As a message broker, everything will fly through it like a bullet. And if you want, for example, an "at most once" guarantee, then think about Redis for processing some data that can be lost. Redis loses data when used as a message broker. But Kafka is not valued for this guarantee, of course. Kafka is valued for the guarantee called "at least once," i.e., at least once. Kafka, with certain settings, can guarantee that any message will be delivered to the recipient, and this is very important, at least once. Not exactly once, but at least once. This means that it will definitely be delivered once, but it might be more. The same message. How is this done and what are its consequences? We have the same diagram, but let's assume we have configured Kafka for "at least once" delivery guarantee. This is a very important guarantee. Most people use it. In this situation, when we transfer data over the internet from the producer to Kafka, instead of just sending it, in general, and forgetting about it, the producer will wait for a response from Kafka for some time, the so-called acknowledgment or ack. This is an English word, which means confirmation. The posts microservice. The producer, sending data, will wait for confirmation from Kafka that Kafka has received it. As soon as the posts microservice receives this confirmation from it, it continues its work and is sure that the data has been written to Kafka. And Kafka sends it confirmation only when it has received the data and saved it internally, not just in RAM. Yes, it will also put it on the hard drive, and only then will it send back confirmation that the data has been received. If the producer does not receive this acknowledgment, it will automatically try to send this data again to, in general, Kafka, it will try to send it there until it finally receives this acknowledgment, i.e., confirmation that they have been successfully delivered, and only after that will it proceed to all its other operations. And in this situation, you see that we guarantee that the data will definitely be delivered to Kafka. And moreover, they will be saved on the hard drive. Kafka stores everything on the hard drive. And this guarantees that even if, for example, the notification microservice now consumes this data and it gets lost somewhere in the middle of receiving it, it can ask for it again, because it is stored on the hard drive and has not gone anywhere. Kafka can give it to him again. That is, the data will definitely not be lost with such a configuration, but this configuration has one very important drawback. That it says that this delivery is at least once, "at least once." That is, what situation can arise: I publish data from my producer to Kafka. And, let's assume, they successfully reach Kafka over the internet in this situation. It saves them internally, everything is successfully written to the hard drive, and should now send back confirmation to the producer that they are, in general, written. While sending this confirmation back, an error occurs during data transmission over the internet. That is, the producer does not receive confirmation due to unreliable networks, although the data has been successfully written. And what does it do in this situation? It tries again, because it is waiting for confirmation. It definitely needs to know that the data has been written. And it sends a copy of the same data again to Kafka. Absolutely the same data, it just tries to send it again. And Kafka receives it, writes it to its disk, and sends, for example, confirmation back to the producer, which now reaches. It receives it. Okay, I successfully sent the data. I can go about my business. But what happened at this moment? The message that the producer was sending to Kafka was duplicated because the first confirmation was lost. Now there are two identical messages in Kafka, simply duplicates, and both will be consumed on the other side. Kafka does not delete duplicates under any circumstances. It cannot do anything like that. It receives messages, delivers them to the other side. And that's it. This is precisely the "at least once" guarantee, meaning one message can be duplicated in Kafka two or more times and then be consumed by the recipient one or more times. Yes. And accordingly, on the recipient's side, you need to ensure that you correctly process duplicates if Kafka is configured for the "at least once" guarantee. Nevertheless, an important guarantee here is that the message will never be lost. No matter what happens, if you have the "at least once" guarantee, Kafka will deliver this message to the other side. The only question is how many times, exactly once or more. Accordingly, since we have such a guarantee, "at least once" on the consumer side, i.e., on the recipient's side of these messages, knowing that Kafka is configured for delivery at least once, we must always assume the probability that this guarantee will work in a negative way. As developers, we must always expect the worst. And we must ensure that our message consumer has a special filter that somehow eliminates duplicates of these messages. And this is very important, because there are different situations. There are messages, the duplication of which will not lead to any consequences. But if, for example, you are working with some super important messages, for example, money transfers, right, and you send someone some amount, this message, let's say, passes through Kafka somehow, for example, it is duplicated, and it turns out that you send money to the person not once, but twice, and money is debited from your account twice. So, indeed, in such situations, events can develop quite sadly. And therefore, consumers must always ensure that they know exactly how to handle duplicates. Sometimes you can just ignore them. But if you, thinking about this, realize that you cannot ignore potential duplication, then you need to make sure that your consumer is idempotent. This is a very important word that often appears in interviews. What does it mean? Your consumer is idempotent in a situation where it has logic that allows it to work with duplicate messages as if it were always a single message. That's all there is to idempotent consumers and idempotency. It's the ability of your service to process duplicated data as if it arrived there once. That is, no consequences when duplication does not affect the operation of your application. This is called that your consumer is idempotent. Of course, there are a huge number of ways to guarantee idempotency. Idempotency guarantees can be achieved in completely different ways. Sometimes your application is naturally idempotent because duplicates do not affect the logic you have written there in any way. And sometimes you notice that there are problems, and you need to add some filters. You need to check in the database if we have already received such a message, processed something, and somehow simply cancel them, ignore them, and so on, and so on. That is, you always have to think about this. How to do it depends on each specific situation. You have to think about it. How to ensure that your consumer is idempotent? But when you work with the "at least once" guarantee in Kafka, right, and duplicates are possible on the other side, you always have to think about it. You always have to sit down and see if my consumer is idempotent and should it be. If so, how to ensure this idempotency? This is the developer's task. There are different ways. Well, now for the most interesting part. You might have asked before, why is it that if our Kafka is running on a separate server, then if this server turns into a pumpkin, in general, then we won't lose messages? This seems illogical, right? Well, a computer just explodes. Well, of course, we will lose everything that was saved on it. How does Kafka guarantee 100% that all messages will be delivered if they are written to Kafka? In fact, the situation here is quite simple. No one ever runs any programs on a single server. If you launch your Kafka, you set it up, literally launch this program on one computer and place another computer next to it, on which you launch another Kafka program with the same configuration as the first, with the same topics. And these two programs are connected to each other, they know about each other. These are internal Kafka settings. You can do this. A group of such servers on which the same Kafka is running, which forms the system, is called a cluster. Just a smart word to denote a group of servers on which the same program is running, which coexists in the system. And these separate Kafkas on different servers know about each other. And moreover, when you publish data, say, to Kafka for you from the producer, these data are actually sent to both one server and the other, and saved on both servers. And thus, if tomorrow one of the servers burns out, right, breaks, then there is always the second one, which also saved this data and through which all communication goes. This is a basic principle of replication. Not only Kafka works this way. All your microservices should always be replicated. All your databases, because all computers have a probability of breaking, and running any program in production on only one makes no sense, because you can lose everything. Even on two, it makes no sense. You should have at least three machines in each Kafka cluster. Then the probability that all of them will break is quite low. But usually there are even more machines. But you might say to me here: "Vlad, well, we launched two machines, right, with Kafka. It turns out we write a message. On the other side, in general, the consumer picks up these messages." But it turns out that the data is copied to two servers. We read only from one server, well, so as not to duplicate the message, right? And we will read from the second one. Only if the first one fails, we will switch to it and ask from it. But it still collects copies of this data, still saves them there. And so that, if anything, we can switch to it. Well, it looks like we are not using the full potential of the second server. That is, it is simply on standby, as they say, to back us up against data loss. Well, it's not very efficient. We would like to include this second server in the work somehow. And here begins the main advantage of Kafka. why Kafka is used in huge companies, why it can process countless amounts of data, why everyone wants it in all vacancies, everywhere, why every project uses it. This is partitions and scaling of Kafka. Well, we said that Kafka has topics, right? A topic is a special channel for transmitting information of a specific type. Specific messages go into one topic, i.e., messages of one theme, messages of another theme go into another topic, but within one Kafka. And we said that a topic is, well, something like a channel. Just a channel for transmitting information of a specific type. But in reality, in Kafka, each topic is also divided internally into so-called partitions. Partition, if translated into Russian, is, well, something like a part. A part, some part of this topic. That is, essentially, you have one channel where messages of a specific type go, but within this channel, there are even smaller channels where this message can be published. When a message arrives at a topic, then within this topic there can be one or more partitions, thus subdivisions within this topic. All these partitions contain data of the same type. That is, if we have, for example, a post topic, then all partitions of this topic, which relate to posts, can contain only post data, because they belong to this topic. But the message itself, for example, when a new post is released and the posts microservice publishes this information to the posts topic in Kafka, this message will actually go into a single one. For example, we have five partitions, and this message will go into only one of them. It has, as it were, entered the topic entirely, but only into one of these partition sub-channels within this topic. And, accordingly, on the other side, it will be retrieved from this partition. And thus, all messages that we publish to this topic will be more or less evenly distributed across all available partitions, i.e., subdivisions of this topic, as it were, sub-channels, and move through them to the other side. And on the other side, the consumer will, in general, retrieve them from all available partitions, without giving priority to anyone, but simply as if the data stream into the topic is broken down into several smaller streams to, as it were, divide these data into some additional channels, although this is still within the same topic, and the data there is obviously of the same type, it's just parallelized, as it were, across different channels. You will tell me, why, Vlad, this complication? We previously published all data of the same type into one topic. Why can't there be just one channel for transmitting all this data, and on the other side we would parse it? A valid remark. But look what partitions give you. In fact, if you have five parallel, as it were, sub-channels in one topic, then it turns out that you can publish data in parallel into five channels and in parallel retrieve them on the other side from five channels. If you have exactly one channel, then data is sequentially filled into it, like a queue. That is, you must also retrieve them sequentially from it, on the one hand, right? And if you have five parallel channels for
If you have one topic, then you can publish to five parallel channels simultaneously, and consume from five parallel channels simultaneously. That is, this, in principle, allows you to increase the throughput of each specific topic, because in different streams you can consume these messages from different partitions and not wait for their sequential processing. If you had one single channel, they would simply line up in a queue there. Thus, you simply consume a large array of information faster. An important note here is that if you have multiple partitions, then, obviously, the order of messages following each other is violated. This is an important nuance about Kafka. If you have a topic and it has several partitions, yes, that is, subdivisions of messages within this topic into parallel streams, then, of course, if you published one message to one partition and the next message to another partition, then these messages can be consumed in any order. For example, I will consume from the second partition before from the first. And thus, the order of messages through Kafka may not be the same as the order of publishing these messages. It is very important to remember this. Nevertheless, Kafka has the ability to guarantee the order of succession. It is obvious that a partition is an indivisible element, an indivisible sequence of messages that go through it, yes, and there, well, a real queue happens within the partition. Nothing can be consumed there before something else. And, accordingly, if messages fall into the same partition, then they are, obviously, consumed sequentially, as from a regular queue. First come, first served. At the Kafka level, you can guarantee that certain messages always end up in the same partition in your topic to ensure that they are consumed sequentially. This can be done if you add additional information to your message in the form of a so-called key. And this is just another field with data that you attach to your message that you send to Kafka. Kafka guarantees that all messages that have the same value in the key field will be directed to the same partition in this topic. And thus they will line up one after another and be consumed sequentially on the other side. But if it doesn't matter to you whether messages are consumed sequentially or in parallel, and your logic works correctly regardless of this, you don't have to worry about it. Just dump all messages without keys into the topic. They will be randomly, evenly distributed across partitions and consumed in parallel on the other side. This is very convenient for increasing throughput, but moreover, it provides a powerful tool for distributing this data across different servers and thus scaling Kafka to a practically infinite stream of data. How does it work? If we said that we have one topic for posts, all information about posts is published there, and it has five partitions, that is, five sub-channels through which these messages diverge, then we can take, we had an idle Kafka server that we set up for replication, yes, so as not to lose data. We can say that let three out of five partitions of this topic remain on the first server, and the remaining two we will move to the second server. And then it turns out that the data, when sent to Kafka, will not only be evenly distributed across partitions, but also go to different physical servers, which increases performance, because each server has limited resources: RAM, processor, and hard drive. Yes, and if the data stream is huge and each partition, in general, works at the limit of its throughput, by moving these partitions to separate servers, we can guarantee that our topic itself is distributed across multiple physical machines. And each machine, since it has more resources to process one partition, processes data faster, and our Kafka becomes less sluggish. Literally, communication happens more efficiently because there are more resources. The throughput of each machine becomes much higher. And as you can see, however many partitions you have in each topic, that's how many machines you can distribute your entire topic across. That is, you send conceptually, you think about it. I send my post message to the topic, to Kafka, but in reality, this message will end up in some random, more or less partition of this topic on some random machine where this partition is located, yes, and then it will be consumed from there by a consumer. This is very convenient. In this configuration, we can put five machines, yes, and each partition will have its own machine. So, it will work more efficiently, because each partition will have its own resources on that server. Well, you might say: "Vlad, well, okay, if each message goes to a separate partition and only to it, yes, it doesn't go to any other of the five available, to some random one only there, then it turns out that if I distributed the data across two servers like this and one of these servers breaks, then I, for example, lose two partitions of my topic." And it turns out that, first, I lost data again. Secondly, my throughput for the entire topic has also decreased, because I had five partitions. Now I only have three partitions, where messages are lined up sequentially and not processed in parallel. But it looks like complete nonsense. We have again encountered the same problem of data loss. Yes, but here too, Kafka developers have figured out how to solve this problem. They also thought about what would happen if one of the servers just turned into a pumpkin. Well, for goodness sake, all the data from these partitions will be lost. The topic will remain with some partitions, but these will disappear. Well, how do we deal with data loss on some server? We just copy this data. Accordingly, when you distribute your topic across different servers thanks to partitions, that is, you distribute partitions across different servers, then, in fact, besides moving these partitions to a separate machine, you also leave a copy of this partition on another machine. Thus, it turns out that when we publish a message to some partition on the main machine responsible for processing this partition, we also publish a copy of this message to a copy of this partition on another server. For example, I send a message to partition 3, which is running on the first server, but on the second server there is a copy of Partition 3 as a backup, and we also publish this message there. The consumer takes this message only from the main copy, from the main server. And that copy of the partition is simply needed so as not to lose this data in a situation where one of the servers fails, yes? That is, now, if my first server turns into a pumpkin, then my second server understands this, because these two servers are connected, and activates these copies of partitions that were on the first server as primary, and starts continuing to serve all consumers from them. And since copies of these messages that also fell into the main partitions were published to them, they contain exactly the same data, they are also saved in the same way, and we have lost nothing. Everything is copied reliably. Moreover, Kafka ensures that when it returns a confirmation to the producer that it has received the data and everything has been saved successfully, it only does so when both the primary partition has been updated and the copy of this partition has also received your data. Moreover, there are not just two copies of these partitions. There are usually many more, at least three on different servers. Accordingly, you copy data more times. The more copies of servers and partitions you have, the slower the information transfer speed, of course, because a copy of the data needs to be saved to each computer. But nevertheless, you have high guarantees of delivery and message preservation. Your system is much more reliable. And this is what partitions exist for. This is why topics are divided into these additional sub-channels. to allow you, first, to distribute the topic across multiple machines. Thus, communication happens much more efficiently and allows parallel execution in these partitions, but also guarantees that on each machine you have data copying from different sources, and all partitions are replicated, that is, copied to other machines, and you never lose data. This is what partitions are generally needed for. And this is how Kafka uses these additional servers that you initially set up to avoid data loss. But thanks to partitions, you can now also use them, and everything is reliably protected thanks to replication in the right places and thanks to parallel execution. And all this is provided by partitions. But it is very important to understand what this is at all. And what is very important to know is that the delivery order is violated, and only messages with the same key end up in the same partition. And there the order is sequential, otherwise parallel delivery. This is what makes Kafka incredibly powerful. The fact that you can distribute one topic across countless physical computers, you can literally say for a topic: "I have 100 partitions." Accordingly, I can put each partition on a separate machine and say that I have 100 machines through which data for only one topic flies. Moreover, each topic has a bunch of partitions, which you can also distribute across different machines. In this situation, it turns out that you can scale Kafka infinitely. And since modern applications work with unimaginable amounts of data transmitted over the internet, billions of messages per second are transmitted. This is, well, just a huge amount of information. You must be able to transmit this information quickly. One computer will not be able to transfer such an amount of information quickly enough to the other side. Accordingly, all this information must be distributable across a huge number of machines, and then everything flies. And this is why Kafka is wanted everywhere. All modern applications process a huge amount of information. Kafka allows them to scale the throughput of their systems to infinity, in fact. This is how it's done. Imagine that you have a situation like we are solving with the guys at my Java bootcamp via the link in the description. We literally develop features there that must withstand loads from millions of users. And, in particular, we use Kafka and microservices there. And here's a literal example of how we do it. We are developing a social network application. And imagine that you have a situation where you have a post microservice, there is Kafka, through which post messages fly, and they are consumed by a notification microservice on the other side, which we wrote with the guys at the bootcamp, yes, and it sends notifications to subscribers of those authors who publish posts. For example, in our Kafka, there is a topic for posts, it has, say, five partitions, and we successfully process these messages. Everything works like clockwork. But this is in a situation where we have one user. What if we have 100 million users, a huge number, they are very popular, they publish a lot of posts in our social network. And then we discuss this problem with the guys at the bootcamp. For goodness sake. Then it turns out that, first, our post microservice is under tremendous load and must process a lot of requests for publishing posts. This is one point of high load that we need to think about how to solve. There is also Kafka. This is just a program running on one computer. If on the other side 100 million requests every second fly as messages to one computer on which Kafka is running, then this computer, well, it doesn't feel very good. It's also, in general, in shock. And problems start to appear there. We need to make sure that we can handle high loads there in Kafka as well. And besides, there is a consumer of these messages, there is a notification service, which we wrote, and it also processes 100 million messages every second that fly to it from Kafka. How will it cope with this? And so we discuss this problem at the bootcamp. And Kafka helps us do this excellently, and microservices even better. Here's what we do there. We say: "Well, for goodness sake, if we have a high load on the post microservice, then let's just launch more servers." of this post microservice. Let's raise copies of this post microservice on different servers and distribute the load from all users we have across these available servers. Yes, a person wants to create a post, their request to create a post is generally directed to one of the available servers on which the post microservice is running. Thus, the load from all 100 million users is distributed across the entire set of servers we have. This is the advantage of microservices, yes, and we, of course, guarantee that everything is done correctly there, that the data does not store state anywhere. This is what we constantly talk about with the guys at the bootcamp and how we build features of this class that can handle high loads. If you want to join the Bootcamp, the link is available in the description. Be sure to come to the Java Bootcamp, we will work together. We will also review all your code, yes, so that you do everything correctly. This will be done by a practicing developer from, say, Sber, who solves these problems every day in their real job, will review all your code and you will also work with him in a team. So, we have solved the problem of high load on the post microservice, that is, on the producer, but all these microservices still write to one single Kafka, and on the other side, it is still consumed by one single notification microservice. Well, okay. With the notification microservice, since it is also on fire, yes, a lot of data is flying to one server, we can also solve the problem in the same way. We can say: "Well, okay, let's also launch several copies of the notification microservice on different servers, connect them all to the same Kafka, to the same topic, and they will evenly from different partitions, accordingly, process in parallel, yes, this data, the huge stream of data that flies there from the post microservices." That is, there are many post microservices, they all write to different partitions. Accordingly, we parallelize the data stream across different partitions and across the producer microservices themselves. And we also parallelize consumption from different partitions across different machines. Again, we can set up as many machines as we want, but no more than we have partitions, unfortunately, yes, because only one consumer, one physical machine can consume from one partition. Setting up more consumers than you have partitions makes no sense. That is, if we have five partitions, we can set up a maximum of five running consumers of the notification service, specifically on physical machines. If desired, you can simply create more partitions and parallelize it even further. So, we have solved the problem on the producer side, they were burning. We have solved the problem on the consumer side, also a huge stream of information. But we still have one small Kafka server running with a bunch of partitions, through which a huge stream of data flies. Naturally, this server is also in shock that it has to hold all this. And this also needs to be scaled somehow. This is why we have gathered. This is how, for example, Netflix uses Kafka, how Google uses Kafka, Yandex or Ozon, anyone. Precisely because of this. There are also partitions in this topic. And each topic has partitions depending on how you configure it. Accordingly, you can simply take, set up, say, three Kafka servers and say that some partitions will be on one server, some on another, some on the third for this topic. Moreover, on all these servers, there are also copies of different partitions that are distributed across other servers. And now we say that one microservice, running on one server, which is a producer, writes to one Kafka server into these available partitions on this server. And one of the consumers of this data listens to this Kafka server and takes data only from these partitions that are available there. Another post microservice writes to a completely different Kafka server, to different partitions. And another consumer, the notification microservice, takes data from this server. And it's the same with the third instance, with the third server that we set up. And as a result, we have parallelized all the data across a whole bunch of machines. And the throughput of our system has significantly increased. We have relieved the load from a single server, because the data passing through it has been distributed across several machines. Communication channels have been distributed across different computers. And each computer ultimately receives only a fraction of all this load, all this stream of information passing through the entire system. But in any case, it all works as a whole, because all these three Kafka servers know about each other, they copy each other's data. If one of them fails, all producers and consumers switch to another server that contains copies. The load on it will increase, of course, but we can quickly launch another server, redirect data there, and so on, and so on. And this is precisely why, because it is necessary to distribute countless arrays of information across different data channels. This is why Kafka is the most popular choice today in modern development as a message broker, because it allows you to scale the throughput of your systems to infinity. simply by creating enough partitions for your topics. Well, you need to think about this. You need to calculate how much data in this situation will approximately fly through your topic. Literally in numbers, some volume, megabytes, gigabytes per second. Naturally, make a decision about how much we want to parallelize this, how much we want to distribute it across different machines, how many partitions will go to one machine, and how many partitions in total we should have, how many consumers we will have then, and how the data will flow across different computers. But Kafka gives you the tools to think about this, but also to do it. And this is why it is the most popular choice in all large projects today. Well, and not only large ones. Any startup today uses Kafka. It provides delivery guarantees, huge scalability, and super amazing flexibility in settings. One topic you can have at least once, yes, lose a message. Another topic can be at most once, and so on, and so on. It is also important to note that we can talk about exactly once delivery, but this is much more difficult in distributed systems. To deliver something exactly once to the other side, you need to put in a lot of effort. In Kafka versions four and above, this has become possible thanks to distributed transactions. But this is a much deeper topic. And if you are still learning to understand, learning to work with Kafka, then you should not invest effort in understanding how the exactly once guarantee works, that is, exactly once, but it is enough to focus on at most once and at least once delivery and understand how all this works. And I guarantee that if you know all this and can reason about such systems in the context of Kafka and how it works, then you will pass, in general, any interview for a conditional junior or middle where a question about Kafka will arise. I hope this video was very, very valuable for you. It would be great if you liked it and subscribed to the channel. It really helps. Also, come to my Telegram, where I write a huge number of posts in super simple terms on technical topics and talk about all sorts of really technical concepts, explaining them in a truly interesting way. The link to my Telegram is available in the description. Thank you, and see you.