Transcription
Greetings, friends, and in this video, we will talk about the development, the evolution of web technologies, and how we got from the kind you see on the screen to this kind. We will talk about the technologies that are used, how they are connected. I think for people who have recently started programming, web development, there will be a lot of interesting things, and it will create a comprehensive picture. A view from the outside on modern web development, on the tools that are used, and approaches.
First, I want to note that the diagrams that will be drawn do not reflect the real data flow, they do not reflect the sequence of technological development in the order in which they developed in real life. This is just a visual, schematic diagram that will allow, again, to see how all this developed. But the sequence will be not so much historical as logical. So, less words, more action. Let's watch a short advertisement and move on to the video.
And before the video starts, I want to introduce you to the Selectel service, which provides cloud services, servers, equipment, network services, and much more. And in particular, I want to tell you today about databases as a service, so-called Database as a Service. This is a managed service that allows you to get ready-to-use database clusters in the cloud as quickly as possible. That is, Selectel ensures the setup, maintenance, backup, and reliability of these clusters. This will help speed up development and product release to the market, meaning your employees will be able to focus on product tasks first and foremost and not be distracted by infrastructure support and database maintenance. The service automates the processes of creating and supporting fault-tolerant clusters, performs regular security updates of the operating system and databases, and you, in turn, save time on configuration and maintenance. It is also interesting that you pay for consumption. The service works on a pay-as-you-go model. You pay only for the resources actually used. And now you can create a cloud database cluster with a 30 percent discount. And if you already use databases, the price will be automatically recalculated. Register in the Selectel panel and create a reliable cluster. All links will be in the description.
So, friends, the development of the Web as we know it began in 1990. Since then, 32 years have passed, and the web has come a colossal way of development. It is about this that we will talk today. Initially, the web was just HTML documents, ordinary HTML pages, thanks to which scientists and other interested people could exchange information. All this works over the HTTP protocol, which is the application layer of the TCP/IP or OSI model. At the same time, each such available page on the Internet has its own address, and the connection between these pages is achieved through hyperlinks, which allow you to go from one page to another. So, to summarize again, initially, this World Wide Web was about exchanging some documents, pages that are connected by hyperlinks. The idea, the concept is as simple as possible. It is obvious that any technologies become more complicated, and HTML alone was not enough. We wanted to add some visual component to the page, change colors, fonts, change sizes, positioning, and then CSS, Cascading Style Sheets, appeared, with the help of which we can do all the previously mentioned things on our HTML pages. But again, this is just the appearance of our page.
It is obvious that sooner or later, with such a powerful tool for data exchange, we will need not only to display some information but also to collect it, various forms, controls with which we can get information from thousands of kilometers away. But sending some information is not enough; this information needs to be stored somewhere. And here we come to backend technologies, to server-side technologies that will work with data. But initially, the HTML page itself does not hang in the air. After the user enters the address of some application, the address of some site, and presses Enter in the browser, an HTTP GET request is sent to the server. This can be a simple web server or a more complex server. In turn, it knows which HTML file to send to this address, and it simply sends it to the user, after which the user will see this HTML rendered in the browser. They receive a ready-made page with which they can already work and interact.
In order to serve static pages that do not contain dynamic data, we only need a very simple web server that can serve static files. That is, it listens for requests and, based on these requests, identifies the files that need to be served: HTML files, index.html, contacts, about, main, and so on. And it serves these files. But a couple of minutes ago, we came to the idea that we want to collect data, and therefore this data is no longer static but dynamic. That is, we can collect various data and provide this data to the user. For beginners to understand better, let me give an example. For instance, you open a page on Wikipedia where there is just text; this is static data. But when you open a page where you can leave a comment, and by sending this comment, it is saved somewhere in a database, and this data is always shown to you differently depending on how many comments have been written, then this is already dynamic data. It is understandable that on Wikipedia, you can also change some data, but let's assume that the article is always static. So, let's summarize: the user opens a page, there is a form addressed to them. In this form, they enter data, for example, their initials, press some button, and at this moment, a request is sent to the server. This is a POST request, an HTTP request. If for retrieving some data we used GET requests, then for sending some data we use POST requests. At this moment, our server, the core of our application, must save this data somewhere. And here we come to the concept of databases. The server can both save some data to this database and read from it. That is, we save some data, the server reads new data from the database, creates an HTML file with this data, with dynamic data, please note, and returns it to the user.
At the moment, for working with documents, we used HTML files and CSS files for describing styles. And here we have some separation. We separate the server and the client-side, which the user sees in the browser. Along with HTML and CSS, JavaScript appears. This is a programming language that allows you to add some dynamism to the page: opening modal windows, some complex animations, form processing, form validation, and much more. If for some simple pages we could get by with almost no JavaScript, then with the complication of these pages, it is already impossible to do without JavaScript, because it is responsible for the logic and dynamism on this page. In addition, JavaScript allows you to add such an important concept as asynchronous requests. So, let's recall the flow without them. We send some request to the server, for example, data from some form. At this moment, we are in a waiting mode because the data does not travel over the network instantly. And as we determined earlier, the server generates a completely new HTML page for us and returns it in its entirety. After that, the user must download this entire page over the network. This option is not very good, since all pages in our application, as a rule, contain some common parts: a navigation panel, some menu, a footer with contact information, and so on.
Here we come to the concept of asynchronous requests. With the help of JavaScript, we send this asynchronous request, and at this moment, the browser waits for a response from the server, but the page does not update. We are simply in waiting mode, but the page does not update. This is a very important point. And the server no longer returns HTML; it returns JSON, or XML, or any other type of data. But it's important to understand that it doesn't return a ready-made HTML file; it only returns the data we requested. JSON is just one of the most popular data types for client-server interaction. And in order for the user to see new data on their page, we don't need to download the entire HTML page. We can take the data from the server that it sent us and insert it into the page in the right places. And we do this exclusively using JavaScript, from the request to inserting this data into the right places. If before we had many HTML pages, now we have very, very many JavaScript files, which contain all this logic for interacting with the server and processing data from the server.
Let's recall this slide. At this stage, when I was talking about it, the server and the client were roughly the same thing. That is, our application generated an HTML file with dynamic data, the server generated it and sent it to the browser. So, again, at this stage, we can consider the application as a single whole. There is no separation into frontend and backend as such here. But as soon as we introduce the concept of asynchronous requests and processing all these requests with JavaScript, there is a clear separation of the client, i.e., the frontend, and the backend. Now these are two separate applications, and development is done by two separate teams: backend developers and frontend developers. If before we said that we have many HTML pages that contain some style sheets, CSS, some JavaScript, then the structure looked like this: we have many pages, we have a lot of CSS, we have a lot of JavaScript, and each page links to CSS files and JavaScript files. But with the advent of asynchronous requests, we realized that there is no point in a large number of HTML files, because we can replace all the content on the page using JavaScript and the data that the server sends us, for example, in JSON format. Applications like this, which contain only one HTML page, are called SPAs, which stands for Single Page Application. That is, it's just an application with one page. For example, we opened this page, and user data was loaded. Then we clicked on another link, an asynchronous request was sent to the server, and we saw, for example, a page with information about the user's payment system. Within one page, we simply replaced the data that came back from the server.
But then the question arises: where do we get this initial HTML page from if the frontend and backend are now two separate applications? The backend is an application that interacts with the database and returns some data, and the client is what we interact with in the browser. Because 5 minutes ago, I said that we need some server that will serve that HTML file for a GET request. And 5 minutes ago, I also said that a very simple web server that will simply serve static files would be enough. For example, it could be Nginx, or it could be Apache, which can be configured in 5 minutes and can serve our HTML, JavaScript, CSS files to the user in the browser. So, the logic here is as follows: the user enters a URL in the browser. At this moment, an HTTP GET request is sent to the Nginx server. This server can serve static files, and it sends HTML, CSS, JavaScript files, some images, everything related to this to the user's browser. This is a simplified scheme.
So, we have understood that the frontend is now a fully-fledged separate application, and the web server, for example, Nginx, is responsible for serving static files. And our actual backend application server, which contains the core of the application, business logic, interaction with the database, is a separate application. Moreover, many clients can use one such backend. It can be a mobile application, a web application, or just some of our partners who interact with our server via API. That is, the number of consumers can be arbitrarily large. We interact with this server in a request-response format. That is, we tell it some command, for example, "Give me user data," and the server, in one of the formats again, JSON, XML, it can be any format, returns this data.
So, at the current stage of our system development, the frontend and backend are already quite complex, separate applications. On the frontend, as we have already found out, we have a lot of CSS, a lot of JavaScript files, and there may be files of various other extensions: PNG, JPG, fonts, and so on. And processing all this manually becomes difficult. And here, bundlers like Webpack, Rollup, Vite, and so on come to our aid. Before telling which files need to be served, we pass our JavaScript code and so on through this bundler. The bundler prepares everything in a convenient form for production, compresses files, using Babel, it can prepare our code to work in older browsers, and performs many other secondary tasks. So, the scheme looks approximately like this: we pass our JavaScript, CSS, and other files through a bundler. At the output, we get a so-called bundle, which is all the code of our application. And this bundle is then served by the web server, and interaction with the backend occurs in a request-response format directly on the frontend.
The backend infrastructure also becomes more complex. At the moment, we have the core of the application. In this core, we have described the business logic. There is a database with which this core interacts. The database is usually initially a relational one, like MySQL or PostgreSQL, and such a database is a set of related tables where data is strictly structured. For a certain type of data, this suits us. But at some point, we want to store chaotic, weakly structured document data. One of the types of NoSQL databases is document-oriented databases. The most frequently used is MongoDB. In addition to classic data storage, we may also need fast access to some data in the form of a cache. And here, Redis or Memcached are most often used. For example, we want to store user sessions, JWT tokens, or other frequently requested data that will be much faster to retrieve from the cache than to make a request to the database. And thus, we improve the performance of our service.
The project infrastructure is becoming increasingly complex. Deploying an application now takes a significant amount of time. Tests are written, linters are run to check your project's code style, assembly is performed, and so on. We would like to automate this process somehow. And here, for process automation, we use CI/CD pipelines, in which all types of tests are run, and our application is deployed. Here, Jenkins, TeamCity, GitLab CI, GitHub Actions, and many other tools designed for CI/CD can be used as tools. At the same time, the application is packaged into a specific container, a Docker container, within which all the necessary environment is configured for quick deployment. For example, it could be Node.js, the correct version, a configured database, and so on. And so that we don't have to configure this environment from scratch on every machine where we deploy our application, we configure it in an isolated container, thereby simplifying the process of deploying our application. In short, to summarize, a CI/CD pipeline is the automation of those actions that I listed earlier. By the way, there is a video on the channel about CI/CD, and a hint will pop up now, and a link will also be in the description. You can watch it.
Moving on. At some point, the core of the application becomes incredibly complex. We have different areas of responsibility. In addition to the online store that we initially developed, we have a fintech department, we have a news feed. These are huge, separate pieces of business logic that are not connected to each other. However, the core of our application still represents a huge monolith, and supporting such a monolith, like any large system, becomes difficult. And it is obvious that we can create a distributed system where each service represents a separate area of responsibility. That is, from a large monolith that includes the store, fintech, and news feed, we create three different services. At the same time, we also have a separate authorization service through which we gain access to all other services. The system becomes distributed. Each service has responsible people who manage it. Each service has its own CI/CD pipeline, which makes the deployment of each service independent of others.
There are also other advantages of distributed systems. For example, if the online store fails for some reason, then another part of the users who are interested in the news feed can continue to use the service. But it is obvious that if some authorization service fails, then everything will break. However, fault tolerance is increased in any case. But there are also other advantages of such distributed systems, which you can read about on the Internet. This topic is always a hot topic. So, here we have smoothly approached the microservice architecture. I think many of you have heard this term. We have decomposed services, and in fact, they are independent of each other. However, this is still bricks, pieces of some huge, large project, product. And we want to have a single entry point to this product. And we create so-called API Gateways, through which we access the rest of the services. That is, we send the request to this, and it already knows which service to redirect this request to.
Our infrastructure is becoming more and more complex, and we have a need to store a large number of files. And here we come to the concept of S3, Simple Storage Service. This is a special service with which we can store large volumes of data. Similar services are provided by many large IT companies: Amazon, Yandex, and so on. But S3 is very often associated with Amazon, because, as far as I know, they were the first. In addition, sooner or later, you will need analytics. The application is growing, the number of users has increased many times since the product launch, and you need to store some data about users, about what buttons they click, what they most often enter in the search query bar, for example, which products are bought most often, and so on. And for storing such analytics, simple databases are not very suitable, because we need to store huge, just huge volumes of data, and at the same time, we don't have a particular need to change or delete them. We only need to add and be able to read this data quickly. That is, the key here is reading speed. And here we have one of the databases designed for analytics. One of the most popular is ClickHouse. This is precisely a database designed for fast access to analytical data.
Also, since we are using a microservice architecture, there is a possibility that a message sent by a user from the client might get lost somewhere in one of the services for some reason. For example, the service was restarted, or something broke, and we want to guarantee that with this multitude of services, our message will not get lost. A message broker, such as RabbitMQ, can help us with this. In addition, we can perform some delayed actions, not right now, but with a delay. For example, it can be compared to mail. We go to the post office, hand over the item, we are given a ticket, and we wait with this ticket, not worrying about anything. And we can be sure that the package will eventually reach the recipient. We also have a need to perform some actions according to a calendar at a specific time. For example, every night we go through the database and delete users who haven't logged into our system for more than two years, or every morning at 9 AM, we send users a newsletter to log into the application and receive some reward, as is usually done in games. And at this moment, we add some scheduler that, according to the schedule, at a specific time, according to the settings we give it, can perform some actions, some operations.
At the moment, we already have a sufficiently complex system with a very complex infrastructure, on which probably more than one team works. Hundreds of thousands, if not millions, of people use our service, and it is very important for us to maintain the operability of our service constantly. Imagine a situation where all computing resources, all server capacities, are stored in one data center. And if, for some reason, the power goes out there, or something breaks, some emergency occurs, then our entire service stops working, and we lose a lot, a lot of money. And to avoid this, we need to balance the load between different data centers. For example, we had one, we made four, and diversified these risks. That is, now if one of the data centers fails, there will be three more backups. If two fail, there will be two, which is more than enough to guarantee the stability of our service. We need to configure the load balancer so that it can distribute this load between data centers according to certain rules. And it is obvious that this should apply not only to the server-side of the application but also to the client-side, where Nginx serves static files. Ideally, this load should also be distributed between different data centers so that users always have access to our application through the browser.
At the current stage, we have a complex application. But if you look at the backend diagram, it looks much scarier and more complex than the frontend diagram. And many people think that the frontend is something simple, like layout, moving buttons, making forms. But modern frontend is also quite large, complex systems. And here we encounter essentially the same problems that we had on the backend. And we are talking about a large monolithic application, and the problems of a large, difficult-to-maintain application are also solved by a microservice architecture. We have one large mono-repository, which will store all microservices. The font will be quite small now, so I will explain everything verbally. We have services. It could be, again, the same online store, news feed, fintech, now three frontends. Essentially, these are all independent services that are not connected to each other in any way. However, the problem arises that these are services of one large product, and there will probably be something common: common components and buttons, inputs, and common forms, common modal windows, some data processing functions, common modules, common configuration, and other aspects that may be duplicated from service to service. And in addition to microservices, we also store these reusable packages in this mono-repository: UI kits, which are buttons, inputs, and so on, what is used in all projects, some common auxiliary functions, modules, common configuration. And services reuse these packages within themselves. Thus, on the frontend, we get exactly the same distributed system where each service represents a separate project that can be easily deleted, replaced, and if one of the services breaks, other services will not be affected.
And on the frontend, in modern realities, development is usually carried out using one of four popular frameworks. If it's tiring to list them on the backend, then on the frontend, there is a limited number. And so that I don't start getting tomatoes thrown at me, I emphasize that it is precisely popular frameworks that have a limited number. There are quite a lot of frameworks that are used by a small number of developers. So, as the basis of the project, React, Vue, Angular, or Svelte are usually used. For bundling, Webpack or Vite are most often used. In modern realities, there is also TurboPack, as well as Rollup, esbuild, and other less popular bundlers. For working with data and state management, Redux, Effector, and others are used. For server-side rendering, which I have said almost nothing about today, frameworks like Nuxt, Next.js, or a custom solution can be written on Node.js. Server-side rendering is also used for full-text search, which is quite difficult to implement on your own. Usually, tools like Elasticsearch, one of the most popular tools for full-text search, are used.
Also, since we use a microservice architecture, there is a possibility that there are even several dozen services, and we need to conveniently manage all the containers that are responsible for each service. And Kubernetes will help us with this. For data exchange between some consumer, in this case, the frontend, and the backend, REST API is most often used. SOAP, GraphQL, or gRPC can also be used. I also forgot to add WebSockets to this list for real-time interaction. By the way, there are several videos on the channel about WebSockets, also videos about GraphQL, also videos about REST API. You can watch and learn more about what these methods of interaction between client and server represent. Also, as a rule, in small, large applications that many people use, there is always processing of a large amount of data. Analysts handle this data. And here the most popular tools are Hadoop and the HDFS file system. This is a separate area that includes solving many interesting problems. For example, finding specific data, extracting this data from some text, finding duplicates, processing data, building graphs, diagrams, collecting data from different sources and bringing it to a unified form. And the keyword here is Big Data, i.e., large data. This also adds its own specific characteristics. Data processing occurs according to certain patterns. For example, we will have to do a lot with this data. Sometimes they are sold, they are very expensive. Sometimes this data is transferred to neural networks that learn on this data. In general, there are also a million possibilities here.
And with this, we conclude the video, friends. I hope it was useful and informative. You have clearly seen the scheme of what modern web applications look like. I try very hard to make the videos visually pleasing and understandable. I hope you notice these efforts and don't pass by without liking and writing a comment, because this helps promote the videos and helps me, in particular, to create content for you. Well, that's all, friends. Thank you all, good luck to all.