Transcription
Microservices are everywhere today—in every job listing and every project. The ability to develop them is an essential skill for any modern programmer. In this video, with its fantastic animations, we will discuss microservices in great detail using very simple language. I guarantee that by the end of the video, you will have a wonderful understanding of what microservices are and why they are important.
Hello, my name is Vlad. I am a developer with seven years of experience in backend development at one of the best companies in the world, and I live in Amsterdam. On my channel, I share how to become a powerful programmer in the shortest time possible. In this video, we will cover everything related to microservices. We will start from the very beginning—where they came from, what a monolith is, the problems associated with monoliths, how microservices emerged, their advantages, and also the challenges they present.
We will also discuss the crucial question of whether to use a monolith or a microservice architecture in your projects. This narrative will be accompanied by visuals to maximize your understanding of the material. Let's get started!
First, let's figure out what problem microservices solve. To understand why they are needed, we need to start with the concept of a monolith.
Let's break it down: any modern web application has users who interact with it. In today's world, there are generally two ways to interact with web applications:
1. A web frontend, which is literally what you see in your browser—buttons, forms, input fields, etc. This is usually a separate application written in JavaScript using a framework like React.
2. A mobile frontend, which includes the same buttons, forms, and input fields, but on your smartphone. This is still frontend development, typically done for iOS using Swift and for Android using Kotlin or Java.
These two frontends are essentially programs written in specific languages that users run on their devices—either on their smartphones or computers. When a user opens a browser, they see this frontend, which interacts with the backend (often referred to as "the backend").
The backend is the main program that executes all the logic of the application we are discussing. It contains all the core functions, interactions, and data storage. When you click something on the frontend, the data is sent over the internet to the backend, which is often written in a different language, like Java, and runs on a separate computer known as a server. This server belongs to the company developing the application.
The backend is called "backend" because it operates behind the scenes; the user does not interact with it directly. Everything happens over the internet, invisible to the user. It seems to the user that they clicked a button and something appeared in their browser.
In addition to the backend, there is likely a database—a separate program that can store data sent over the internet to the hard drive of a computer where the database is running. This is how interaction is organized in any modern application at a high level.
Since the backend contains all the main logic of the application, it can be quite a massive program with a lot of code and diverse functionality. If you look at modern applications, you will see that they have many different features; each button performs a specific action, and so on.
Historically, when developers wrote the backend for their applications, they often used Java and the Spring framework to organize web interactions over the internet. Within this single program, they wrote all the code for various features. For example, if we are building a social network application, this single Java program running on the backend would handle the logic for creating posts, sending notifications to users about comments on their posts, managing comments, creating news feeds for users, and implementing a recommendation system to suggest content that interests users.
The web frontend and mobile frontend interact with this application over the internet, where all the functionality resides. These functions can also interact with external components. For instance, the recommendation logic might interact with a machine learning model to provide results, while the notification logic might use an email service to send notifications.
This backend program is structured so that all its features are contained within a single software project—essentially a single codebase. You can either run the entire application or not run it at all. This architecture, where everything is together in one application, is called a monolith.
A monolith means indivisible; it is all together as one large piece. In this illustration, the monolith appears quite simple, consisting of four major features combined into one application. Unfortunately, modern applications are much larger, with significantly more functionality, and the monolith becomes enormous—a massive codebase with hundreds of thousands of lines of code.
As the monolith grows, it encounters more problems. Let's discuss the issues associated with monoliths.
The first and most prominent problem is called tight coupling. The individual features, which are logically separate and not interested in each other, are actually contained within a single codebase. This means that different components responsible for different parts of your system are tightly linked. They literally call each other's methods and often use components from other modules in their logic.
As a result, these connections become very strong because they interact with each other through direct code calls. If I, as a programmer, want to change some functionality in my system—let's say I want to modify the news feed feature—I might make changes to its code. However, because it is tightly coupled with other components, I often find that I need to make changes to several other components as well, even if they are not logically related to the news feed.
This can significantly extend my development process. I have to do more work and understand how all these components work. The monolith is a massive project with a lot of different functionality, and I need to have a deep understanding of the entire project to ensure that my changes do not break anything.
This is a problem of responsibility for different components. As a developer, I am responsible for everything in this monolith. The larger the monolith becomes, the more features are added, and the more I need to know about it, even if I am not the one writing all the code. I might have a team of 100 people contributing to it, but I still need to be aware of what they are doing.
The worst part is that when I make a group of changes across different components due to a chain reaction from a single change I wanted to make in the news feed, I not only change that logic but also have to ensure that all my changes work correctly. As a developer, I can make mistakes. Even if my original changes are correct, there is still a whole group of changes that might contain errors.
When you change the code of a working application, there is always room for mistakes. Any change needs to be tested carefully. However, the problem is that the connections between components in a monolith are so tight that you might not even realize that one of your changes affects other components.
You might check several components to ensure they are functioning correctly, but if you forget to check or do not notice that you need to verify another component, you could end up introducing a new bug. For example, you might change the logic of the news feed without realizing that it affects the notification logic, and this can be very difficult to trace.
When people develop such applications, they often spend a lot of time rechecking the functionality of all components because they are so tightly coupled. Any changes in one part can cause issues in others, leading to a decline in the overall quality of the application.
You might think your component works correctly, but later users report that something else has broken. You look into it and are puzzled as to why it is not working, even though you did not change that part. It turns out there was a complex dependency between different parts of your system that led to the failure of the notification logic, even though you did not touch it.
There is, of course, much more to discuss, but even what we have covered so far is frequently asked in interviews about microservices at top companies in the CIS. That’s why I have prepared a completely free cheat sheet with the most popular interview questions on microservice architecture. You can grab it via the link in the description on our wonderful website called Java Junior Library.
There, in addition to this cheat sheet, you can find a wealth of materials to prepare for interviews, including Docker, PostgreSQL, and much more. This library will be constantly updated, so feel free to check it out and prepare for interviews at the best companies in the world.
Now, let's talk about another significant problem with monoliths: deployment. What is deployment?
You have your application in a certain version, let’s say version 1.0, and you have made some changes to it. Suppose you tested everything, and it works perfectly. Another programmer on your team also made changes to a different component, working independently on the same application.
Now, you need to deliver this new version to users so they can use all the features you added. To do this, your code is compiled into a single executable file that can be run on a computer. For Java, this is done using a tool called Gradle, which turns your Java code into a .jar file. This is similar to an executable file for a computer game.
After Gradle compiles your code into this file, you need to look at your servers. Right now, you have one server running the old version of your application, which contains the .jar file of the previous version. You need to deploy the new version of your application.
The simplest option is to set up a new server, load your new file (the updated version 1.1 .jar) onto this new server, and run it. This means your updated application is now running on this new server. You then redirect user requests that were previously going to the old server to the new server by changing the configuration to point to a different IP address.
This process of delivering and deploying a new version of the application on a new server for users to interact with is called deployment.
The problem with monoliths here is that the new version you compiled includes changes not only from you but also from that other programmer. If they introduced a bug in their changes that does not affect your changes, the new version you deployed will still have a problem.
When this version reaches the users' servers, they notice the issue and start complaining. The first action for developers is to roll back the new version to the previous stable version. But what does that mean for you? It means you roll back all the changes you made, even the ones that were working perfectly.
Since everything is bundled together in a single monolith, you are forced to deploy all changes at once. If one of them breaks, you have to roll everything back. This means that when you roll back, you take away not only the non-functioning features but also the new working features that users might have started using. This is not a good situation.
The most frustrating problem with monoliths is the issue of scaling. Look at our architecture: we have this huge backend running on a server. A server is simply a powerful computer capable of running the application that users access over the internet.
These machines cost money. We either bought or rented them, and they are expensive. Let’s say this server costs $3,000. Now, suppose today one of our famous users posts a video that goes viral among their 100 million followers—let's say Lionel Messi.
This post generates a huge amount of traffic, and the load on the news feed logic of our application increases significantly. A lot of resources are allocated to handle this load, but resources are always limited on this computer. You cannot have more RAM than what is available on the computer running your application.
When the news feed experiences difficulties, the entire application suffers because if the news feed is struggling, so is the server. Since we have a single monolith and all other features are running on the same server, resources start to run low.
The logical strategy in this situation is to take your application, this monolith, and set up additional servers. You buy several new servers, each costing $3,000, and run copies of this application on all available servers. This way, you distribute user requests evenly across all these servers using a load balancer.
It seems like the problem is solved, but in reality, the component experiencing difficulties is the news feed. All other components in your application are not under load, but because it is one large application, you need a powerful, expensive server to run it.
Modern companies want to save resources; they do not want to spend so much money on maintaining the entire infrastructure. They want to reduce costs significantly. They think, "If only one feature out of four is under load, how can we add new servers without spending money on all of them?"
The monolith does not allow for that because it is a single piece of software. When modern applications became so large that it was impossible to manage the aforementioned problems, people invented the technology of microservices.
What they did was quite simple. They looked at their monolith and saw that it consisted of several parts that were logically separate but tightly coupled due to the architecture of being in a single project. They thought, "What if we take the logic of each component separately and instead of keeping them as a single project, we create four separate programs—four separate Java applications in their own Git repositories?"
We would develop each of these components in separate projects, with code dedicated only to one feature, such as creating posts. The code related to the news feed would only handle the news feed, and nothing else. Each of these applications would then run on its own server—literally on a separate computer from the others.
However, since these components still need to interact with each other to form a unified system, we organize this interaction over the internet using requests from one application to another, using protocols like HTTP or additional programs like Kafka, which facilitates communication between different applications running on different computers.
In the end, you still have one large system, one large application that performs all the same functions, but the exchange of information between different components now occurs over the internet instead of through direct method calls within a single project.
Nevertheless, it remains a single system, but each component exists in isolation from the others. Since each component is essentially a separate application serving specific purposes, these separate applications are called microservices because they provide services to the components that depend on them or to users.
Moreover, they are much smaller than the monolith. However, any microservice is not necessarily a small program; it can contain a lot of code, but it is still smaller than the entire surrounding system. Additionally, this code is dedicated to a specific function or domain and does not perform any other actions.
As a result, this architecture has numerous advantages over monoliths. One of them is development flexibility. Now, if I, as a programmer, want to make changes to my news feed, I simply open the news feed microservice and make my changes there.
Since this component is no longer tightly coupled with other components and interacts with them over the internet, I do not need to worry about how they work or how they are implemented. I only need to test my changes in this microservice to ensure they work correctly, and since the other microservices have not been changed, their code remains the same, meaning there are no new bugs in them.
This makes development much more flexible. I can now make changes to my codebase, which is very large, more quickly because I am making these changes in an isolated component without affecting the quality of the overall system.
Moreover, your system becomes flexible in the sense that each microservice is essentially a separate application. If, for example, one of these microservices performs a specific logic for which other technologies are better suited than those used in the other microservices, you can develop the recommendation system, which uses machine learning, in Python instead of Java.
This allows for diversity in technology choices that are better suited for specific tasks, making our development more flexible. Our architecture becomes more adaptable, allowing us to do many different things without being tied to the tools we use in other areas.
The best part is that now any other developer who wants to make changes to a different component, such as the recommendation system, can do so without affecting your work. For instance, if you have another programmer in your team who specializes in machine learning, they can update the code for the recommendation system, and their changes will not impact yours.
This isolation of microservices is incredibly convenient. It speeds up and simplifies collaboration on such a large, complex system for multiple programmers working together in one team.
This leads us to another significant advantage: the isolation of microservices from each other, especially evident during deployment. For example, if you, as a programmer, update the code for the news feed microservice in our social network using Java and Spring, and another team member updates the recommendation system in Python, you will compile your news feed microservice using Gradle into an executable JAR file, while the recommendation system will be compiled using a different tool related to Python.
Now, you have two separate executable files with different changes, which can even be different versions depending on how often these microservices are updated.
With microservices, your architecture has split the monolith into many other microservices, each running on separate servers. When you deploy a new version for users, you launch a new server, load the executable file onto it, run the new version of the application, and redirect user traffic to this new server while shutting down the old one.
You do the same for the other microservice on a completely separate server with a different file, redirecting user traffic through a different configuration. Everything is isolated from each other.
Now, imagine the same situation as before: the machine learning engineer makes a mistake in their code, and the recommendation system has a bug. Users are unhappy. To fix this, they simply roll back their microservice to the previous version, while your microservice remains unaffected.
This means that due to the isolation of changes, you are no longer tied to other developers when they encounter problems. You can still provide your users with quality functionality, even if one part of your system is unstable while others continue to work correctly.
This is a significant advantage of microservices because your users love and pay for these applications to have rich functionality. If you periodically take features away from them, they will be very disappointed. Now, you only take away broken features while keeping the working ones, and you no longer have that dependency.
The isolation of microservices is incredibly beneficial in this regard.
Now, how does scaling work with microservices? Since we mentioned that each microservice is a completely separate application running on its own computer, imagine a situation where the load on the news feed feature increases.
The load is actually on the news feed microservice and the server it is running on. Only that server is experiencing the load. Moreover, since these applications are smaller in terms of functionality and exist separately, the servers needed to run each microservice are cheaper.
Instead of costing $3,000, they might only cost $1,000. This makes it cheaper to run your entire application. Furthermore, when the load on your microservice increases significantly, you can add new servers to provide more resources to that microservice by running copies of it on those servers to handle the increased load.
Since the traffic has increased, you will be adding servers for the news feed microservice, while the notification system continues to run on the same set of servers without requiring additional resources. This is very advantageous for the company.
This is why modern large companies are transitioning to microservices; it is simply much cheaper for them than maintaining a monolith that needs to be scaled. Modern applications are enormous, requiring many expensive servers, and microservices significantly reduce the cost of maintaining your system.
Additionally, when it comes to reducing development costs, microservices greatly benefit the team of programmers developing the application. Since each microservice is a separate program existing in isolation from the others, you can have one team working on one application, another team on another, and a third team on yet another.
These developers only need to understand their microservice and how to improve it. They do not need to dive into the code of other microservices. This makes it easier for them to understand their work, reduces the time spent on connections, and allows them to deliver features faster, which accelerates business growth.
Overall, developing such a system in a team becomes much simpler because each engineer focuses on what they need to and is only responsible for a portion of the functionality, not everything at once.
Of course, using microservices significantly increases the reliability of your system. Imagine a situation where you had a monolith, and suddenly the server running that monolith—a physical computer—shuts down, whether due to a power outage or a hardware failure. Your application stops working, and all features become unavailable, leaving users frustrated.
In the case of microservices, if the server running the notification microservice fails, all the servers could go down, and your notification microservice would stop working. However, since this is a separate microservice running on separate computers, other parts of your system continue to function.
Your application may degrade slightly—notifications might not be sent—but all other features continue to work correctly for users. They might notice a drop in functionality, but the entire application does not shut down. This is crucial; it is better to lose part of a system than the entire system because the other parts can still generate revenue or serve other purposes.
With microservices, your system remains operational while you fix one of its components. Users may not even notice that one of the components has failed, which is incredibly advantageous.
Now, the question that many beginner developers are asking: if I am developing my project, should I use a monolith or microservices?
There are two scenarios. If you are developing a startup and want to create an application that could potentially earn money, you have a business plan, and you have a supportive team, start with a monolith. It will be much easier to develop.
Creating microservices is a much more complex architectural task, and it will slow you down when developing a startup. A monolith can be developed more easily because you do not have to worry about organizing connections and distributing data across your system. In a startup, where speed of development is crucial, a monolith is the way to go.
Later, when your startup grows into a large application and it becomes challenging to manage due to the monolith, you can break it down into microservices.
However, if you are working on a project to impress an employer, to showcase something in your resume, and to create something impressive, it is worth investing your time in learning to develop microservices. All modern large companies use microservices in their work. If you can demonstrate that you already know how to do what they are currently doing, you will have a better chance of getting hired.
That said, developing microservices is significantly more complex than working with a monolith. In this regard, having the support of an experienced mentor would be beneficial. A mentor can help you establish a clear architecture based on microservices and review the code you write.
It would also be great to have a development team with whom you can work on this complex project involving multiple microservices to achieve a complete and outstanding result that you can proudly show to others.
You can find all of this—mentorship, code reviews, great lectures on microservices, and cool features involving Kafka and Redis—on my Java bootcamp, linked in the description. Many people have landed jobs at top companies in the industry, such as Sberbank, Yandex, VKontakte, Tinkoff, and many others, with salaries ranging from 100,000 to 200,000 rubles per month as juniors.
They get their first job because they work on an impressive project, adding it to their portfolio, which includes microservices and all the latest technologies.
Now, let's talk about the problems of microservices because they certainly come with their own challenges. Developing a microservice architecture is much more complex than a monolithic one, and that is a fact.
This is why the requirements for developers are increasing; it is simply a much more complicated development process.
Look at the architecture we had before with the monolith: we had a frontend, a backend, and a database. That was it. Everything worked beautifully. Then we decided to break the monolith into microservices, and here is what we ended up with: not only do we now have four separate applications instead of one, but we also need to run them on separate servers.
We must organize communication between them over the internet. Each microservice must know the internet address of all the others to send and receive data. Moreover, the interaction might not only use the standard HTTP protocol but also Kafka, which is another program that needs to be set up on a separate server.
All of this still interacts with our database. If you look at it, the architecture of our system has become much more complex. It requires administration and monitoring of each individual microservice. Any of these servers can fail, and we need to keep an eye on them and analyze what is happening in real-time.
Modern systems can contain not just four microservices but hundreds, and they may be written in different technologies, using different configurations for deployment, and so on. The architecture and maintenance of your system become significantly more complicated when using microservices.
It is simply more challenging to discuss and reason about how something works. There is also a potential communication problem. Since your microservices do not interact directly but communicate over the internet, this can lead to errors.
Why? Because data transmission over the internet is not reliable. No network guarantees 100% delivery of the data you send. Something might get interrupted midway. Therefore, when your microservices interact, you, as a developer, must always be prepared for the possibility that a request over the internet might fail or that you might not receive a response because it also travels back over the internet.
You need to ensure that if this situation occurs, you make additional attempts to resend the request, and so on. Additionally, data can be lost during transmission. If you do not ensure that critical data sent over the internet does not disappear due to network issues, you could face a major failure.
This is why developers often use Kafka, as it guarantees that data will not be lost. But that is a different story, which we will discuss in a separate video about Kafka.
Another important problem that can exist in microservices is data consistency. Imagine a situation where a user enters some data through an input field in your application. This data goes to your backend, which consists of many microservices.
For example, the data arrives at the posts service, which saves it to the database. However, this posts microservice also sends a message to Kafka with information about the post. The operation is completed, and the notification microservice consumes this information from Kafka.
But a problem can arise if your code is written in such a way that the consumption of data from Kafka occurs before the data is saved in the posts microservice. The notification microservice might try to access the database expecting that the posts microservice has already saved the data, but it is not there yet.
This is a simple modeling of problems, and if you have an SQL database, you will likely handle this with transactions, and everything will be fine. However, if you are using a NoSQL database, the situation becomes much more complicated because NoSQL databases do not have transactions.
In microservices, because they can access the same data source from different places, problems with consistency can arise. You might expect to receive something but do not get it, leading to inconsistencies in your application logic.
This is a very complex problem that can arise anywhere. If you follow the best practices, each microservice should have its own separate database that is not connected to the others. This makes data access even more complicated.
If you need data from one microservice that is stored in another microservice's database, and you need to update data in multiple databases at once, you face challenges because transactions only apply to a single database.
This is where patterns like Saga come into play. The development process becomes much more complicated when you start thinking about these issues. This is why developers today earn so much money—they are solving complex problems that require significant expertise.
The most frustrating aspect of microservices for me as a developer is debugging. In the previous monolithic architecture, when a user clicked a button on the frontend and encountered a big red error, they would report it, and we had two places to check: either the backend or the database.
We would check the logs on the backend and look at the database to find the problem. But once we transitioned to microservices, when a user reports an error, we have to figure out where exactly the problem occurred.
Their operation might have passed through multiple components in our system, and we need to determine which one caused the issue. Is it in this microservice, another one, the database, the load balancer, or Kafka?
There are now many more points of failure in the system that I need to consider. If I want to debug, I set a breakpoint in my application, but the execution might jump to another microservice running on a different computer, making it difficult to track the same request as it travels through the entire system.
This leads to the need for tracing logic for requests, and so on. The development process becomes even more complicated. There are libraries available for tracing requests that pass through your entire system, but you still need to read about all this and learn how to work with these challenges.
This is why developers are highly skilled engineers working in great companies, receiving numerous benefits and high salaries. They invest time in researching complex problems and learning how to handle them, ultimately solving high-complexity tasks for large companies.
To learn all of this, come to my Java bootcamp linked in the description. It will be great!
I hope this video has been very valuable for you. It would be wonderful if you could like and subscribe to the channel; it really helps. Also, join my Telegram channel via the link in the description, where you will find a wealth of super cool, detailed technical posts with great animations, all explained in simple terms on various development topics.
It will be fantastic—subscribe to the Telegram channel. Thank you, and see you next time!