Transcription
In modern applications, a huge number of different technologies are used: cache, message brokers, microservices, file storage, and a bunch of other things. Damn, it's mind-boggling when to use one and when another, but today I'll explain in super simple terms. Why is each backend component needed, what is this backend in general, and when should you use, for example, Kafka, and when Redis? Hello, my name is Vlad. I am a developer at one of the best companies in the world with seven years of backend experience, and I live in Amsterdam. On my channel, I talk about how you can become a super powerful programmer. In this video, we will go step by step through the roadmap of developing your application's architecture, starting from the architecture of the simplest pet project and moving to a system capable of withstanding loads from millions of users simultaneously. And at each stage, we will talk about why and when exactly you should use one component and when another. Let's go! Let's first figure out what frontend is and what backend is. In any modern web application, there are always these two parts: frontend and backend. And in this case, it's everything that users of this application interact with. Yes, for example, when we pick up a smartphone, click some buttons, fill out some forms, see some animations, all this is frontend. And in the modern world, there are mainly two types of frontend: web interfaces and mobile applications. Yes, those tools, in general, with the help of which any person can give some commands to any program, and it will be convenient for them, it will be intuitively clear how to interact with such an interface. And this is exactly what frontend is called: everything you see as a user and with which you can interact with your hands, in general, enter some data, and so on. Frontend is usually implemented in JavaScript, using, for example, the React framework. This is if we are talking about various web interfaces, i.e., what you see in the browser. If we talk about mobile applications, then for developing applications for iOS, the Swift language is used, and for Android applications, in the modern world, either Java or Kotlin is used, which is gaining popularity more and more actively. Now, what is backend? The thing is, you can't just write some program and tell users, "Well, you'll run it in your browser or on your smartphone." The programs we write are often very resource-intensive. They often perform some complex operations. That's precisely why we use them; they significantly simplify our lives because with one click, some incredible actions can occur. Yes, and consequently, they require a lot, a lot of resources: simply processor time, RAM, hard disk storage, and so on, and so on. And therefore, we cannot force users to simply run all these programs in their browsers on their own computers or on their smartphones because they simply might not have enough resources that are in their device. Such devices would be very, very expensive, capable of running all programs. Therefore, instead of running such programs on each specific user's device, developers came up with another concept: they came up with backend. They thought, "We need a very powerful computer for each user." What if we buy one such very powerful, ordinary iron machine, but only with a huge amount of RAM, with a very powerful processor, with an almost infinite hard drive, and a super cool, maybe even video card? Let's put this computer in our office and call it a server. It will still be an iron computer, but it will be called a server simply because users of our application will now send commands over the Internet to this computer to perform some operations for them. That is, they will ask it to serve their requests, and in English, "to serve" is the word "serve," and therefore such a computer was called a server. It's just a computer that serves user requests. As a result, all these powerful programs that constitute the main logic of your application are placed on such servers. And since all this logic is on this server, which is located, as it were, behind the frontend, people decided to call it backend. It's behind, the back end, everything behind that tool, behind the frontend, the front part with which the user interacts. And therefore, it's called backend. And usually, backends are written in a wide variety of languages. Among the popular ones today are Java and the Spring framework, for example. It could be the Go language. It could be C#. It could be Python, and many, many others. The frontend, reacting to the user clicking any button, sends over the Internet using special tools available in JavaScript and React, some data, some information, simply text and numbers, to the backend, to that very computer called a server, on which the program is running where the main logic of your application is written. And this program simply performs some operations with this data that it received from users over the Internet. And then, after performing these operations, it sends the results of their execution back over the Internet in the form of a set of text in a specific format and some numbers to the frontend, where the user sees some successful results. But all this happens very, very quickly because the data is transmitted instantly, and the computer on the backend performs operations, well, just lightning fast, ideally. Okay, but let's imagine that you have developed such an application. You have a frontend where users can click some buttons, send commands to your backend, where your main application is running on some server. But the problem is that users will constantly send you some pieces of data. They will, in general, create their profile, register in your application, save their name, email, and so on. They will create some entities in your system that need to be saved somewhere because users will close the application, open it again, and they want to see their data that they just added. This means the data was saved, saved on your backend, on the hard drive of your server. But then access to it would not be very convenient because it would be saved in a completely disordered format that would be very difficult to parse. Therefore, people decided to structure this data so that later, when the user again asks the application to return it back and show what they saved before, the application can find it very easily and immediately in a structured form. For this, special programs called databases were developed. A database is essentially just programs that can save specific data that you give them to the hard drive of a computer on which these database programs are running, in a specific format convenient for later retrieval of this data and for managing it, interacting with it, linking it, and so on. And the most popular databases in the world today on the backend are, of course, SQL databases. These are databases that use the SQL query language to form commands for inserting data, retrieving it, updating data, deleting it, and so on. Thus, almost any modern application uses at least one SQL database to store a huge amount of the most diverse data in a specific format and access it at any moment. This means that your program, which you wrote on the backend, for example, in Java and Spring, can send SQL queries, i.e., effectively in another language, to a completely separate program, possibly even running on another server, on another computer, over the Internet, so that this database processes this command in the SQL language that it understands and returns exactly the data that we requested in this SQL query to your backend. And your backend then returns it to the frontend, where the user sees it. There are many different SQL databases: PostgreSQL, MySQL, Oracle DB, Microsoft SQL Server, and in general, for you, if you are just starting to learn, the differences will be minimal in their functionality, and in general, the SQL language works in all of them almost without changes. Therefore, you can choose any one for learning that suits you. I prefer PostgreSQL, for example, but MySQL is also fine. The differences start already in the depth of how exactly these programs work internally, i.e., databases. Each of them is structured differently. SQL databases store all the data that you add to them in the form of tables. Accordingly, tables consist of rows, which are pieces of data, and columns represent individual elements of this data. If you want to quickly and clearly understand what an SQL language is in general and how to apply it, what are the main commands, what is JOIN, what is Foreign Key, what is GROUP BY, and so on, then definitely watch my famous video "SQL in 1 Hour," where I explain in super simple terms and very thoroughly with examples and practice with animations how to generally use SQL, and after watching this video, you will be, well, the absolute master of this language. Let's move on. Before this, we said that the frontend communicates with the backend over the Internet. How exactly does this happen? The Internet is generally a simple system in which all computers in the world are connected to each other. You can think of it as happening literally through wires, i.e., between all computers in the world, thanks to various technologies, there is a connection through which you can transfer information from one computer to another. This is essentially the most important thing you need to know about the Internet. The thing is, information can be transmitted in any format. You can transmit it as one large string in which everything you want to transmit from one computer to another is written separated by commas. You can transmit it in the form of a table. You can transmit a whole picture on which you will write by hand what you want to transmit, and so on, and so on. Naturally, depending on how exactly, in what form you transmit certain information, on the other side, on the side of the computer, i.e., the server, for example, which is the recipient of this information, there must be specific code written in the program that receives this data, which knows how to parse exactly this format of data that you are transmitting and extract useful information from it for performing some operations. That is, the program that works on the backend must somehow understand what information it has been sent and understand what to do with this information, what command has come to it. And since people transmitted data in various formats, one person invented that he would transmit it like this, another person decided that he would do it differently. It was very inconvenient. Everyone transmitted differently, and it was very difficult to connect one application to another because one application always sent data in a specific format, for example, as a string, and another application that received this data could only process the tabular representation of this data. And as a result, it was necessary to add logic that processes both this format of data and that one. And people decided that this is, well, stupid and came to the conclusion that the format of data transmission needs to be standardized, to literally agree on how exactly we will arrange strings, numbers in this set of data that we transmit from one application to another so that all applications in the world can easily understand what we want from them. And this is how the HTTP protocol appeared. In essence, the word "protocol" means an agreement, a contract. People agreed that this is how they will transmit data from one program to another, and this is the protocol, how people agreed to do it. According to a certain protocol, we transmit data, and HTTP stands for Hypertext Transfer Protocol. Thus, HTTP is simply a set of rules that you must follow to send data from one program to another, and the other program, the recipient of this data, will understand what exactly you sent it. And this is how all modern programs communicate with each other over the Internet. Mostly all of them use the HTTP protocol. And in the HTTP protocol, there are two concepts: request and response. A request is that structured according to the HTTP protocol data that one application sends to another application for execution. In this case, the frontend sends to the backend. The backend processes the request. That is, it simply receives data in a specific format, looks at what can be done with it, performs the necessary operations with this data, and forms a so-called HTTP response. This is another piece of data structured according to specific rules according to the HTTP protocol, which the backend sends back over the Internet to the frontend. And it is according to the HTTP protocol, structured in such a way that the frontend, which is also a program, can also understand what this data is and how to work with it. But also, when we talk about the HTTP protocol, in the modern world, the word REST often comes up. What is it? The thing is, when we send a piece of data from our frontend to the backend, we imply that one piece of data is intended for performing one operation, for example, "Create user." We need certain data of a certain type and a certain command to create a user to create him directly. Also, in our application, there may be another command, for example, "Block user." This is a completely different type of command, and it expects a completely different set of data, which can also be sent from the frontend to the backend. But this leads to a completely different result. So, on the backend, there are ultimately two handlers for different types of data. The first handler is a handler for data used to create a user. The second handler is for processing data used to block a user, for example. The more types of data you have, the more commands your application can perform, the more functionality it has, the more such handlers you will create. And these data handlers received the name "endpoints," end points. That is, they are located, as it were, at the edge of your backend and are open to the outside world. And any application, knowing how to access this endpoint, can, following a specific data format, give a command to your application. That is why both mobile frontend and web frontend can send the same commands to the same server and get the same results, although one is a web application, the other is a mobile application, but they use the same handler in the same data format on the backend to simply give this command, regardless of what other application calls it. You can think of these endpoints to which you connect from one program to another to perform a specific command and transfer your data there, then disconnect, and then connect to another socket, transfer other data there to perform another command. These are endpoints. So, when we talk about REST, it's just a special approach, how people have again agreed on how exactly they will describe access to these endpoints, to these sockets, what the application that wants to send data to them should know about them, and in what form it should address them so that it is as convenient and intuitive as possible. REST is simply a style of describing your endpoints, a style of creating, a structure of creating these sockets so that it is convenient to connect to them. It's also a way of standardizing, nothing more. People came up with a format that is convenient for describing, for creating these endpoints, so that in any application, you can always easily understand, "This is such an endpoint, and this is such an one," and can easily and intuitively interact with them. This is what REST is for. Thus, REST is literally just a set of rules that you can follow or not follow. It's not a special library, a special language, or technology. It's how you, as a developer, create these endpoints, according to what rules you do it. You can obviously find these rules on a huge number of resources on the Internet. This video does not delve into the depths of how to create REST endpoints, but obviously, you can do another one. So, be sure to like this video and subscribe to the channel if you like the format of explaining complex things in simple words and would like to see videos in the future. Thank you. Exactly as depicted in the diagram, a typical pet project of a beginner backend developer usually looks like this: who, for example, wrote an application in Java and Spring, connected some SQL database like PostgreSQL, taught them to interact, made a couple of endpoints in their application, and wrote some frontend to interact with their application and present this data. But it is precisely because, precisely because this application is structured so simply, with only a few components, only a few simple ideas are used, that people with pet projects are not so often hired as juniors. That is why a project must be much more interesting. And let's figure out now why your project should be more interesting and why this is not enough. Usually, in real systems, all problems in real applications begin with the fact that a real application gets real users. What happens when your system gets really real users? They start clicking all the buttons that are in your system, they start entering all sorts of data that you couldn't even imagine could be entered into your system. So, if your application becomes popular enough, you get a lot of users, and they will all constantly click these buttons simultaneously. And each click of this button leads to the fact that over the Internet, via the HTTP protocol, some data is transmitted to your endpoint on your backend, and it is also saved to the database, or you access the database to get it from there. The thing is, all your programs on the backend, the database, your famous application with all the endpoints that perform all the logic, they are still ordinary programs. It's not some kind of magic. They are simply run on iron computers that are located somewhere. Yes, they are powerful computers, but any, even the most powerful computer in the modern world, has a limit to its power. It has a limited amount of RAM, it has a strictly defined maximum processor frequency, and other indicators. This means, essentially, that at any given moment, any computer can accept and process only a strictly defined set of commands, no more, simply because it will not have more resources for it. And if at some point your application has enough users, and the computer that, for example, hosts your database so that it runs there, is just a program, not powerful enough for such a volume of users and their simultaneous commands to this database through your backend, then at some point your database will start to simply lag under load. It will start to lag hard. The computer works at its limit, and this often happens simply because a huge number of users that you have in your system perform absolutely the same actions. Many people simultaneously need to perform the same thing: access the same data, save a lot of data that is very similar, or something else. Most often, in your application, they will try to get some data that is the same for all users. For example, in a social network, many people visit the profiles of celebrities. Essentially, from the database, at that moment, they extract information about the profile of some famous person, and it is transmitted to the frontend, where they all see it. But if the person is popular, then many people will simultaneously visit his profile. They will all constantly get the same data from your database and heavily load your computer, the server on which your database is running. If your database crashes at some point simply because the computer burns out, I don't know, cannot work at its limit, or simply works very slowly, then all your users will be furious. They will hate your application because it does not return instant results, and consequently, they will stop using it. Modern business development cannot allow this. Applications earn money only because people use them, because they spend time in them, and everything should work perfectly, instantly, quickly. Users should be happy. But if everything works slowly, they leave, you lose money, your business closes. People noticed this problem: that users often read the same data from the database, and they thought, "The database stores all its data on the hard drive. The hard drive is not the fastest computer memory. Accessing it is slow, reading data from it is slow." Yes, relatively, because in a computer, there is actually another memory: RAM. It is much faster. That's precisely why it's called RAM. Reading from it is faster, writing to it is faster, and so on, and so on. And people thought, "What if, in addition to the database, we add another additional storage to our system that will store all the data that we put into it not on the hard drive, but in RAM, so that access to it is much faster?" Thus, the so-called cache appeared. Cache in your application is an additional storage, separate from the main database, in which you store all the most popular data in your system that your users read or update very, very often. Because cache is a special program that stores all this data precisely in RAM. Therefore, a huge number of requests to the cache, which is also essentially a program running on some server, on some computer, will be much faster than if they were all directed to the database. Moreover, such a program can process them immediately, much more, a much larger volume of operations can be performed simply because it returns results faster. And such programs received the name "caches," and it is they that allow large systems to guarantee that your users, when accessing popular data, can get results as quickly as possible. Such programs in the modern world are, for example, Redis, one of the most famous caches in the world, or there is also Memcached, actively used in the development of many modern applications, especially at the level of large companies. In essence, cache is also a database, also a program that stores data in a specific place, also can retrieve it, also can save something there, and so on. But in these programs, in Redis, Memcached, SQL language is no longer used. Special ways of interacting with these programs are used, which are significantly different from the SQL language, and in general, everything looks different. But in fact, there are many libraries in various popular languages that greatly simplify interaction with programs like Redis, for example, where you can send all the necessary data by literally calling one method or get it. And thus, if you want to take your pet project to the next level, bring it closer to a real application, then think about what pieces of data your potential users might access very, very often, the same ones. And if you can think of such, then add a cache like Redis to your application and put this data there, and direct users precisely to the data in the cache if it's there. If it's not there, direct them honestly to the database. But if you got it from the DB and know that it's popular data in the future, put it in the cache so that all future requests are sent precisely to the cache. Your application, thanks to a not-so-complex component like Redis, reaches a completely new level. You start thinking like an engineer. You think that your system might have a million users and that they can all overload your database, and the cache will save it from crashing. But you think, "Is that all?" Unfortunately, that's not all. There are still a huge number of problems that we need to solve. The thing is, there is no magic in the world, and nothing works just like that, and nothing is obtained just like that. That program that you wrote, for example, in Java, your direct backend, your application that performs all the business logic, it is also simply running on some iron computer. Yes, it is very powerful, potentially, but it also has limited resources. One day, your application, your program on this server, might receive a huge number of requests from your potential users again. Yes, you have solved this problem for your database by adding a cache, and it has become easier, but your application is a redirector of commands to various other programs with which it connects. But nevertheless, it is also running on some computer. Nevertheless, it also requires some resources to perform all these operations that you assign to it. If there are very many of them, your application itself, the program in any language that you have written, will also start to work slowly, will start to lag, will start to respond to users with delays, return their data after a while. Users again get furious. It takes a very, very long time. Sometimes your computer can simply burn out and shut down, and then your entire application simply stops. Users send data somewhere, but where to send it? The computer you are trying to access simply does not work. And this is a problem. In the modern world, applications cannot shut down. No business can afford for its application to simply become unavailable to its users at some point, whether it's day or night, weekend or New Year's.
Because all over the world there are always people who can interact with it via the internet. And this is the profit of these companies that provide such applications. And the application should ideally always work. But since it is launched on physical machines, and they break down, overheat, get overloaded, you are not insured against problems, and you need to do something about it. You cannot allow everything to shut down, and people to think and decide: "Here we have our huge web application written, for example, in Java. We process a huge number of commands in it, countless. We process user saving in it. We process some financial transactions in it. We have added social features to it so that people can leave comments for each other, share some posts. We can upload pictures and videos, and all this is done by one single application. It performs very, very many operations. And you have very many users. Each of these users potentially wants to perform different things. One transfers money, another publishes some posts, a third updates their profile by uploading a picture to it, a fourth watches some video in your application. All this is different. But if all these functions are concentrated in one single application, it means that all of them will be processed on one single computer on which this application is running. Because it is one whole application, it is like a single piece, a single monolith. This is where this word comes from. A monolithic application is an application in which all the functions of your system are concentrated, all the features you have, all the endpoints, all data access, and so on, and so on. All processes are embedded in one single application, in one single large program that is responsible for executing them all. And for monolithic systems, when they become very large, when they have a large amount of functionality, you need to allocate more and more powerful servers. But server power always has a ceiling, and in the modern world, we have already begun to reach it. Modern applications require much more resources than any modern physical server can offer. They all have limitations. And therefore, people decided: "If we can load one computer so much in the modern world, because of such a volume of users, because of such an abundance of functionality that we provide, if we can load one computer so much, only because if we break this one monolith into several different applications, all of which will be launched, each on a separate server, on a separate physical machine, but each of these mini-applications will be responsible only for a specific piece of functionality of our entire system." One application is responsible only for users, another is responsible for the news feed, a third application is responsible, for example, for the system of sending notifications to users in our system. And if we do this, it turns out that we send only a part of all user requests to our overall system to each application. But this means that we reduce the number of operations that each application performs in principle. And this means that since each application is running on a separate server, we are distributing the load across different servers. One server is responsible for users, another for notifications. This means that it will only be responsible for this. If the number of requests for user registration increases, then the notification server continues to work as it did before. Nothing changes for it, it does not get overloaded. The server working with users gets overloaded. This is another problem that we will solve. But the notification server remains constant. Thus, you can very competently allocate resources for the operation of specific features. For example, one feature is more heavily loaded than another, and you can allocate a more expensive server with more resources for it. And if a feature does not require a large number of operations to perform, then you can put a cheaper server for it and save money. Thus, simply by breaking this application. And these small applications that, when connected, form a huge system in the end, are called microservices. In the microservice approach, when you divide one large application by functionality into many different applications, which are also essentially the same web applications, the same backends that you wrote before, but now in the form of separate applications, for example, also written in Java, using all those technologies, like Spring, for example, but they exist separately. One program in Java is launched, it is responsible for this. Another, also in Java, is launched, it is responsible for that. It has different code, but it integrates into the overall system. You are essentially assembling a puzzle from different pieces. You launch several programs now, and they work together in the system. Why is it called a microservice? Micro because it is smaller than the entire monolith, like a separate piece, and it was called micro. It doesn't necessarily have to be very small. A microservice can actually be very, very large and powerful, simply because it is a piece of your system. Service because it provides services, provides a service, handles some requests, performs some operations. So, a microservice is created. Now, the point is also that not only the frontend can access different microservices depending on what operation it needs to perform, but also the microservices themselves can access each other via the same HTTP protocol, still over the internet. Because they are just two different programs running on two different computers. I can send an HTTP request from one program to another program. I did this from the frontend to the backend. The frontend, a separate program, sent a request over the internet to the backend, another separate program. Why can't two programs on the backend do the same thing? And so they do. Accordingly, as a user, by sending a command from the frontend to the backend, you can access one specific service, one specific mini-application from your entire system. For example, the feed service, the news feed service. You add some post, let's say, to your system, as a user. And all your subscribers should, for example, receive a notification that you, as the author, as someone they are subscribed to, have published some post. And in this situation, the feed service, the news feed service, would be responsible for what? As an application, it would be responsible for saving this post to the database and publishing this post to the news feeds of your various subscribers. And at the same time, this service, this separate application, would send a request, a separate command, to a separate notification service so that this service, not the news feed service, would handle sending notifications to these users about the post being published. Thus, one application is responsible for some actions, and another for others. And pressing literally one button on the frontend can lead to a whole chain reaction of various operations on your backend, which simply communicate with each other over the internet. And this diagram shows only three microservices, but in modern backends, in various companies, there can be hundreds and thousands of these microservices, and all of them are organized into a single system that, through a web of various requests and integrations, interacts and provides the set of functionality that we, as ordinary users, use. And this is precisely how the problem of loading this huge monolith onto the server on which it is running is solved. We simply break down the functionality and distribute it to different machines, and the load on different machines is much, much lower. Unfortunately, microservices are not a panacea. We are still living in an imperfect world. There is still no magic. The computers on which each of the microservices is running are still ordinary physical computers, ordinary servers that are located somewhere. They overheat, they wear out just the same, they burn out, they get unplugged, and so on, and so on. Thanks to microservices, of course, provided that one of the computers is lost. For example, the server on which the notification sending microservice is running in our system has burned out. It simply failed, burned out, I don't know, the hard drive burned out, the motherboard flew off. And this means that our notification microservice has simply shut down. This means that all requests to it will not work. It simply cannot perform any operations. But all the rest of our application continues to function. This is one of the advantages of microservices: if one microservice fails, the rest of the system continues to function. Imagine what would happen to a monolith if its server burned out: your entire application would stop. But here, in our case, the feed service continues to calmly process all requests and commands. The news feed service also does. Well, notifications have stopped sending, but everything else works. Nevertheless, in modern systems, guarantees that all functionality continues to work for users are very important. We cannot allow, again, half of our application to work and half not. If computers continue to break and burn, it seems impossible to guarantee. In fact, people haven't invented anything brilliant, but they managed to solve this problem. They simply said: "If I have some application, I put it on some server that can potentially shut down for any reason, even a hurricane will carry it into space, and it obviously won't be able to work there, then, damn it, what if instead of running my application on one single server, I take this very application and run it on ten identical computers?" I will run this same application on ten different computers, simply copies. If one of these computers breaks, then I have nine more identical applications running on different computers that work independently of the first one, that continue to do something. If I lose one of my computers, the entire system continues to function. And this process of copying your application to other computers is called replication. A replica is a copy of your application running on another computer. You don't write separate code on another computer. You don't do anything like that. You write the code once and simply copy this application that you have created to different computers and run it there. And all this gives you a huge number of possibilities. Firstly, your system can never completely fail. If you have enough computers, enough replicas of your application, then the probability that all of them will suddenly break simultaneously is very low. If one breaks, the others will most likely continue to work, unless, of course, they are all connected to one power strip and you have a short circuit. But that's already a foolish replication strategy. The same thing can happen with your database. It's a program running on a separate computer with your cache. It's also a program running on a server. Therefore, the same rules apply to them. Instead of losing the cache in your application and making users wait longer, let's replicate it on multiple computers. If one burns out, we have plenty of others that continue to serve user requests. Moreover, we used to accumulate data. Our application used to shut down, the computer burned out, we could lose all the data. Imagine, it's a disaster. But when we accumulate it on a bunch of computers, one burns out, we have all the others left. We can even install a new one and transfer the data from the existing working ones there. Then we definitely don't lose data. The same with the database. People thought: "Damn it, if my database server burns out, if the hard drive fails, then I'm screwed. All the data for years that users have added to my application will simply disappear forever. I cannot allow this. It will be a disaster. All my investments, all my business partners will send me away. Just if this happens. Therefore, I will install many computers and replicate them on all of them, transfer all the same data there, and connect them into a single system." And in databases, it's a bit more complicated, and usually there is one server that accepts all write requests, for saving some data to your database. It is called a master, a master server, or a master node. Other replicas also connect to it, which are used for additional data storage, but they also, since they store copies of the data, can handle read requests. That is, you can distribute many read requests from your database across multiple of these database servers. Thus, you reduce the load on one server for reading. Unfortunately, writing is usually done on one machine. These nodes in the database, replicas, copies of the database that can only handle read requests are also called slave nodes. That is, there is a master node, the main, main node that saves data, and it also copies this data to all other machines so that up-to-date copies are always maintained on all replicas. And replicas that handle read requests are called slave nodes, meaning they are secondary. In all other cases, in working with microservices, there are no masters and slaves. Your application is simply launched on a bunch of different machines. But it's the same application, the same version, with the same logic, with the same code. But it simply runs on different machines. All of them can accept both write requests and read requests. It doesn't matter at all. In databases, simply for ACID guarantees, you need to accept write requests only on one machine. And this is precisely the idea of replication. Nothing brilliant, just put more computers so that if one breaks, all the others support it in its catastrophic failure. And now, when something breaks, we are still okay. Our application continues to work completely. And that's not all. But you will tell me, Vlad, I launched my program, my backend, on five servers. But these are different computers with different addresses on the internet. And accordingly, to access a specific one, I need to send a request specifically to it. When my frontend sends data to my backend, how does it know, for example, when accessing the users service for registering a new user, which of the five computers, which server on which my application is running, to address? Which specific one? This is a fair remark. Indeed, if there are five, I need to know which specific one I am sending the request to. Well, people solved this problem. They invented a special program called a load balancer. This is indeed a special program in which you can register all your replica computers for a specific service and say: "When a request comes to my load balancer from a user to this specific microservice, let the load balancer decide which specific physical machine, which real computer server, this request will be directed to, because they are all registered in it." The load balancer, as a program, stores information about the IP addresses on the internet of those computers that you have placed in it. And it can accordingly decide which one the request will be directed to. That is, it knows about all of them, and it makes this decision. Thus, your frontend, for example, when sending a request, knows nothing more about your microservices, about your backend, and so on, and so on. It knows that it is addressing this load balancer program, which will then decide where to direct this request further. Thus, the load balancer knows about endpoints and knows about specific servers on which certain microservices are running. Not only can the load balancer send a request to a specific microservice, not only can it decide which specific computer, on one of which this microservice is running, the request will be sent to, but it can also evenly distribute the load among all these servers. Imagine that you have a very large number of user requests for registration in your application, very, very many. Now there is a peak of activity. Everyone wants to register in your system, and you have five servers. And you could say: "Let one process all requests, and the other four are on standby." What if the first one fails, then there are four more in reserve, and if necessary, they will handle the remaining requests. But this is not rational, because all four others will simply be idle, and you will pay money for them, for these servers, and for the fact that the application is running on them, and for the fact that the electricity is being consumed, and so on, and so on. So why not, while requests are flying to this application, distribute all these requests evenly across all these machines so that all machines are working and each machine receives only a part of all the traffic that is flying to this entire microservice? Because on all of them, the same copy of the same application is running. Accordingly, they will all perform the same logic, regardless of which specific request came to them from this user or from this application. It's just different data. This is also what a load balancer can do. Not only does it know about the specific machines to which the request should be sent, but it can also evenly distribute these requests among all available machines for this microservice to which it is addressed. There are various programs that provide this type of functionality. The most popular load balancer, perhaps, is Nginx. But its configuration is usually the responsibility of DevOps engineers. Although you, as an engineer, can also do this one day if necessary. For example, if you work in a startup and you don't have a DevOps engineer, you will definitely be doing this. But precisely for this, load balancers are needed in your programs. But again, a load balancer is just a program that is also running on some server, and obviously, if this program fails, then all your services will be unavailable to the frontend, because the frontend now only knows about the load balancer. It does not go directly to the services. It has this layer in the form of a load balancer. Therefore, you cannot allow the load balancer to run in a single instance on one single machine. Because if it breaks, everything breaks. Therefore, load balancers are also usually run, at least a couple of them, so that if one breaks, the second can take its place, and the operation of the entire system can be continued in normal mode. Replication. Okay. Now imagine that in your social network application, a user appears who is a celebrity. He has 10 million followers. And now he decides to publish some post. And this means in your system that all his subscribers must be notified that a new post has appeared in their news feed. How does this work on our diagram? The author of the post sends a request to our service called the Feed Service. It processes this information and, among other things, because it is not responsible for notifications, sends a request to the Notification Service, which is supposed to perform the logic of sending all these 10 million subscribers all their notifications, which will then be displayed in their browser or on their mobile application, for example. But what does this mean? It means that when the author publishes a post, during the communication between the Feed Service and the Notification Service, the author will have to wait a very long time until all requests from the Feed Service are sent to the Notification Service. And there will be 10 million of them. Moreover, communication over the internet is unreliable. Networks in general are unreliable. Communication sometimes breaks, fails. You need to try again, or everything fails due to a timeout, or something else. This is the internet, you can't rely on anything. This means that when a person literally presses the post publication button, they see a loading spinner and wait for a very long time, although they only need to save the text to the database, while the Notification Service deals with all the bunch of requests that are flying into it. And the Feed Service waits for responses to these requests that the notifications have been successfully sent, and only then will it tell the author of the post that the post has been successfully published. But there are 10 million notifications to send. This means you have to wait for 10 million responses from the Notification Service, which is very, very long and slow. And then users in your application simply hate publishing posts, especially celebrities who need to send notifications to 10 million people. And then people thought and decided that if we create a new program, just a new program, which we will also launch on some computer and call it a message broker. This program will work as follows: whoever wants to send some piece of information, some service puts this piece of information into this program. And the service that receives this information listens, waits for this piece of information on the other side of this program. That is, it connects to it from the other side. Accordingly, this program will put all the information that is published into it into a queue. That is, each individual message will be put into a queue for processing. Thus, if, for example, the Notification Service can process 10 messages per unit of time, and the Feed Service can send 10 million, then all other messages that cannot be processed right now will simply wait for their turn within this message broker program. A message broker can be, for example, Kafka. A very popular modern message broker. What are these messages? They are literally the same pieces of information that were previously HTTP requests in the form of letters and numbers presented in a certain format. Now they are presented in a different format so that they can get into this message broker. And the point here is that the Feed Service, after it has formed one large message containing all the information about all the necessary recipients and all the information about the steps that need to be taken to send all notifications, simply publishes it once into the message broker and immediately forgets about it. The message broker guarantees that the message that was published into it will not be lost. It is stored within this message broker program. And now the Feed Service, knowing that the message is definitely saved within the broker, can go and do its own thing. It doesn't know when the Notification Service will retrieve it from this queue and process it. But it doesn't matter. As a user, you can receive notifications a little later that a post has been published. The user experience for you will not change much in this regard. Nothing terrible will happen. But this allows the Feed Service to immediately return a response to the author of the post that their post has been published and not wait for the Notification Service to do all this huge work, because it can be done asynchronously, in parallel, independently of the Feed Service. This gives you a huge number of possibilities. Moreover, message brokers provide some guarantees. For example, Kafka never loses messages that have been published to it. If you put something there, you can be sure that this information will never be lost, and someday the service that is on the other side, or the consumer service of these messages, will retrieve this information and process it in some way. This is precisely why message brokers were invented: to separate services from each other and weaken the dependencies between them. The Feed Service no longer needs to wait for the long work of the Notification Service. It can simply continue to do its own thing, knowing for sure that the commands it gave to the Notification Service will definitely be delivered someday, thanks to, for example, Kafka. These are message brokers. If you already know Java and dream of moving to a city like Amsterdam, but can't find a job, then come to my Java bootcamp where we will study and work with all the necessary technologies that a truly professional European-level developer needs. This includes Spring, Kafka, Redis, Hibernate, PostgreSQL, and most importantly, nine microservices, all of which you will write. Not just write, but implement really cool features like a news feed, a link shortening service, or a notification system that will inform your users about important events in your application. Moreover, these features will be designed to withstand really high loads from potential users. You will not just follow my guides, but truly understand why certain solutions are applied. And why will you understand this? Because you will work in real development teams with other participants according to Scrum rules, using real tools that you will use in real work. And also, in your team, there will be a real developer who will help you throughout the entire bootcamp. This will be a person who actually works in a company like Yandex, Amazon, or MTS. You will have calls with him several times a week. He will constantly answer your questions, at most within one day, and regularly review all your code, absolutely all the code you write in the bootcamp. The review will be done by a real developer. And after completing the bootcamp, you will leave with a CV and a real project in your portfolio, with which you simply cannot be refused a job. It will be impossible, and all recruiters will want you in their company because you will have not just a pet project, but experience working in a team with real engineers on a super powerful application with super cool features. Come to the bootcamp, the link is available in the description. Now let's imagine that your application has hundreds of millions of users who generate a huge amount of data daily, which you store in your SQL database, for example, PostgreSQL. And everything is great until the space on the hard drive of the computer where the PostgreSQL program is running fills up.
This is just a hard drive, the memory on it can run out. And what should be done in this situation, you will tell me. We can install a larger hard drive, we will have more memory. I will say, "Okay, but then it will fill up again. You will need an even more expensive and larger hard drive." And right now, if you go looking for the largest hard drive in the world, you will find it one way or another, but its size will be limited. And the size of the data that is published in your application will be limitless. There will never be any stop. Users will always generate more and more data. The problem with SQL databases is that the data stored in them is very difficult to break into pieces and distribute across different computers. Why is this difficult? Because it cancels out transactions. It is very difficult to ensure ACID guarantees, if you have heard of them. And therefore, people invented databases that allow all the data stored in them to be easily distributed across several computers connected in a cluster. And such databases are called NoSQL databases, meaning any databases that are not SQL. The most important difference between them and ordinary traditional SQL databases is that they do not provide ACID guarantees, i.e., they do not allow you to perform ACID transactions. And you always need to be prepared that the only guarantees these databases give you are the guarantees of Basic Availability, Soft State, and Eventual Consistency. Of course, we need to talk about them separately, but one way or another, there are countless diverse NoSQL databases of different types for different needs that you can use if you need to store a huge amount of data in your application that does not fit on a single machine. For example, there is a popular database like MongoDB. It stores all data in the form of documents that can be easily accessed by key and stores countless diverse information there, also without being tied to any strict schema. That is, there can be literally the most diverse data. There is Cassandra, a columnar database for storing a huge amount of information. It is used, for example, in Big Data solutions. There are graph databases like Neo4j, which are used, for example, in social networks for fast searching between entities. For example, how to find friends of a specific user in a huge user base. This is brilliantly applied in social networks. But the main point of these databases is that they distribute all the data stored in them across several machines quite easily, out of the box. You don't need to do anything. These are special programs that know how to do this for you. You tell them how many servers you have and where they are located, and they evenly distribute the data among them. And if new computers appear in your system, they also automatically ensure that some part of the data will also be redirected there. But due to their distributed nature, you need to remember this: you will not receive any ACID guarantees.
Accordingly, when should you use SQL databases and when should you use NoSQL databases? I would say that by default, for your educational projects, for organizing any startup, or something else, you should always initially use SQL databases. And only then, when you realize that you are hitting a wall regarding which NoSQL database to use and where exactly, and how you will work with it in your application. This is not a decision that can be made randomly. Therefore, focus on SQL databases. But it is useful to know about NoSQL, to know the BASE guarantees, how they differ from SQL, but you don't necessarily have to use them in your work. They are applied only when you know for sure that they will help you. Moreover, for example, Redis is a distributed key-value database, so it is also a NoSQL database. Continuing the topic of SQL databases and their limitations, it should be said that if you are using a lot of memory with ordinary data, then imagine what will happen if you allow users to save ordinary files, pictures, videos, etc., in your SQL database. After all, files can be of unlimited size. And if you allow a user to simply and trivially upload anything into your database that is running on a computer with a limited hard drive size, then what will stop a user from creating a file that is larger than your entire hard drive, uploading it to your application, and crashing the entire system? For this very reason, never allow users to save their files in your database. Okay, you could set a limit, but that still doesn't solve the problem. Users will save a lot of files, and they will fill up your storage very, very quickly because files are always much larger in size than ordinary strings or numbers. And in this regard, people invented so-called BLOB storage. This is file storage that you use precisely for storing files of enormous size. How do they work internally? They use a distributed file system. These kinds of concepts are very complex things and are not the topic of this video. I just wanted to show that you should never save files in a database, in an SQL database, and that you should always save them in so-called blob storage or cloud storage or object storage. This can also be called. What popular ones exist? For example, Amazon S3. There is Google Cloud Storage, MinIO, and many others. The essence of this storage is that it is a special kind of database specifically for files: absolutely any pictures, videos, binary files, archives, anything. Users put them there, and this storage, in response to the saved file, returns you a link to this file in this very storage. You then put this link into your main SQL database. And now, when a user wants to download a file, they first get a link from your database, and then, by this link, directly accessing the file storage, they download the entire file and get it on their screen, for example, a picture. This is how file storage works, and this is precisely what you should use for saving absolutely any files. But, of course, it's not practical to constantly make users go to your file storage over the internet and download every picture over the internet every time they reload the page. This is a huge amount of traffic, it's very slow, it's extremely high loads on your system, and it simply costs money. And therefore, people invented a technology called CDN, or Content Delivery Network. This is simply a program running on a huge number of computers around the world. You don't run these computers; there are special organizations that provide this CDN as a tool. You pay for it, get access to all these servers they run around the world, and you can use them as a cache for your files. That is, when a user downloads a picture from your object storage for the first time, they download it from this object storage quite honestly. But after downloading, they put it on some server in this CDN that is geographically located near this person. And accordingly, all subsequent requests for the same picture will go not to your application, but to this server in the CDN. Consequently, the speed of this file will be much faster because the server is geographically closer to the user, and the load on your system will naturally be reduced because now you are not serving the user's request for this picture, but some third-party organization that provides you with services for using the CDN. Thus, if users in your application frequently download pictures, videos, etc., and interact with them, literally watch them in your system, then make sure you use a CDN to store these files there.
Now let's look at our diagram again. We have a huge number of diverse programs for completely different purposes. But all of them are running on different computers, servers, which are just iron machines that are somewhere and plugged into an outlet. As it was before, people, some organizations that develop applications, bought these huge iron servers, brought them to the office, plugged them into an outlet, configured them, and ran their applications on them. But imagine, tomorrow you need a new server to make a replication. You buy it again, bring it to your office, plug it into an outlet, configure it again, and so on, and so on. And so for all the programs you have, because they must be run on different computers. Because if one computer burns out, at least the others work. And if one component of your system fails, they all fail at once. Therefore, they cannot be run all on one computer; it is unsafe. As a result, because you literally had to configure everything manually, bring, buy, set up, you couldn't do it quickly. It's always a limitation, and you might simply not have the resources for another new computer, especially an expensive one. Especially if the load decreases, why do you need such an expensive one? And tomorrow it increases, so you turn it on, then you turn it off. The resources you spend on servers are used very inefficiently. And therefore, Amazon thought, "What if we build one huge building where we put a huge number of iron, massive computers with super powerful resources and rent them out to people who need servers?" Thus, AWS appeared. The cloud is not some ephemeral cloud; it is a huge building where a lot of computers are located, where you can call or send a message and say, "I need a computer with such resources so that I can run such an application on it." And they will tell you, "Okay, we will turn it on now, configure it, you just run the application there, and everything will work for you over the internet." So, wonderfully, you don't need to go anywhere, you don't need to buy anything except a subscription to this service. You don't need to ask anyone to configure it; they handle everything. If it burns out, they will automatically give you a new one because you are paying. And so, clouds appeared. Clouds are just a nice name for this service of providing you with servers for rent. And now you can run all your programs not at home on a bunch of different iron computers, but simply in Google Cloud, which has its own cloud, AWS, Heroku, Azure, and so on, and so on. All of them provide services for renting literally servers and various other services. They provide diverse machines, but one way or another, you can run all your infrastructure, all the programs you have, on someone else's machines and simply pay as much as you use resources at that specific moment. You can dynamically ask them to add machines to your system very quickly to handle large loads, and then dynamically remove these machines in literally a few seconds. They have a magnificent infrastructure and magnificent service. As a result, you scale your business and your system very, very quickly because you simply don't worry about how to configure huge iron servers and install everything on them. Amazon employees do it for you; you just pay them money. And as a result, all your infrastructure is running in the cloud. This is a cloud-based architecture. When they say that everything is simply running on computers that we rented in the AWS service.
Well, you've launched your system, everything is working great. But at some point, one of your users gets a huge red error when they try to do something, and they can't bypass this error; it completely blocks them from using your system. They write to your support and say, "My application is not working." You swear and go to figure out what happened to them. But how do you do it? The user won't tell you anything specific. They will say, "I have a red error on the screen, and I don't know what broke on the technical side." It could be Aunt Tamara the accountant, who doesn't even know that programming languages exist or anything like that. She just wants your system to do what she asks it to do. She's not interested in your backend bugs or anything like that. Then how exactly will you figure out what exactly broke? You can't interrogate her; she won't tell you anything useful. And precisely for this, to solve this problem, people invented logs. Logs are literally a file into which your application constantly, when any action at all is performed, a specific line in your code is executed, it literally writes text with a specific message about what exactly is happening in your system at this moment in response to a specific user request. At this stage, when this method is called, when this method is called, when we access this data storage, when this happens in this, for which user, what data we received, what exception occurred for which user, what is this exception, what is its stack trace. This is text, it's just textual information that your application, each of your microservices, publishes into one large file and simply saves this information there so that later, when a problem arises for a user, you can open this file, find the time when a certain line was recorded, and when a certain problem occurred for your user, and see what exception was there at that time and simply study its stack trace, figure out where it crashed in your code and why, and solve this problem. This is the whole idea of logs. But, of course, storing logs in one huge file is inefficient because they gradually become very large, it's difficult to search through them, and then there will be a lot of them; it will be gigabytes and terabytes of logs in large systems, possibly even petabytes sometimes. And therefore, people simply invented a system that will parse all the log text in real-time and allow you to search very, very quickly by any keywords in these logs. There is a special database that knows how to do full-text search, meaning you feed it a string, it breaks it into pieces, and by any of these pieces, you can very, very quickly find the full string in this database. Such a database is called Elasticsearch. Accordingly, what did people invent? They said, "Let all our services that write logs, instead of writing them to one huge file, write all these logs, simply lines of text, directly into this Elasticsearch database." It will index them, meaning it will literally break them down into these many individual words by which all these lines can then be searched. Accordingly, now all logs are stored in this database, and when a problem arises, you can simply send a query to it, not in SQL, but in a special language that this Elasticsearch database supports, to search for specific lines, for example, within a certain time interval, because these lines contain information about the time when they were recorded there. And for convenient work with this database, another tool was invented, Kibana, which provides a convenient user interface for performing these very queries to this Elasticsearch database and searching for relevant information there. And this is precisely what logs are for. Most often, in your real projects, in professional activity, you will encounter the Elasticsearch + Kibana system for collecting logs from your entire system, where you will spend a lot of time when debugging some production problem that has arisen for your users.
But, of course, besides collecting logs, when your system is running, you also want to know: "My program, some application that I wrote, is running on some computer. How intensively does it use the resources of this computer? How many users are accessing my application per second? What will happen if their number doubles? Will my application crash or withstand the load? How many events per second does my system consume?" And so on, and so on. You want to know all this constantly so that at any moment in time you can look at your system from above and see where there are bottlenecks, where the loads are too high, your system is running slowly, where everything is fine, and where you are close to filling the hard drive, where there was a spike in load, why it happened, and so on, and so on. Analyze it retrospectively and at every moment in time, you can generally monitor the performance of your entire system. You simply want to collect metrics about the operation of your entire application, all its components, services, various databases, caches, and so on. For this, people invented another special database that knows how to collect metrics, meaning simply some information, again presented in a specific format, about the operation of your system, which each tool from your application can send to this database using some special library. Such a popular database, for example, is Prometheus. Using special tools again, you can send information there about how many times a specific method in your system was called, literally in the code, how many times your method was called during one specific second. This database stores this information in a specific format so that then, based on this information, it is very easy to search within a specific time interval. That is, you want to know for today, how many times someone called a specific method in my code. This database allows you to do this very quickly. It also has a special query language. But so that you don't have to learn another query language, because you would have to learn for Elasticsearch, and for Prometheus, and you need to know SQL, and so on, and so on. This is inconvenient. And people invented another tool called Grafana, which simply provides a convenient user interface for working with this Prometheus database. Grafana can build various graphs based on the data already stored in this Prometheus database, at your request, analyzing how the amount of data has changed depending on time, building various diagrams, analyzing this data, and collecting it in a convenient format for you so that you can always analyze what is happening in my system right now. You can literally open Grafana, and you will have a bunch of graphs that you yourself told it how to build, and they will change in real-time based on the data that is currently being updated within the Prometheus database to which Grafana is connected. And you can just sit and watch how it's performing right now. And if something happens, Grafana will also send you notifications that, for example, the load on some microservice has critically increased, and you need to react urgently. But you can immediately go into Grafana, see what exactly went wrong. And metric collection is one of the key tasks in any modern backend. And therefore, you obviously should at least hear that Prometheus exists, that there is metric collection into this database, and that Grafana exists to conveniently view this data in the form of graphs, diagrams, and all sorts of things. I hope this video was very valuable for you. It would be great if you liked it and subscribed to the channel. Also, come to my Telegram, where you will find a huge amount of unique content about my work in Big Tech companies, technical articles, and also travel, lifestyle, and much more. Thank you, and see you.