Transcription
Grafana is a system for visualizing your application's metrics. When you are developing something, it is important to monitor what is happening inside, how loaded the server is, how quickly requests are processed. Even if you just have a smart home, Grafana is an excellent choice for displaying the status of a smart light bulb in a smart bedroom. Grafana can display not only graphs but also tables, histograms, geo maps, and a bunch of other ways to visualize your metrics. Grafana can pull data not only from Prometheus but also from ClickHouse, InfluxDB, or Elasticsearch. And by the way, in this video, I will show you both Prometheus and ClickHouse without leaving the office, or rather, right in Grafana. You can also configure alerts, and I will show them today as well. And if you don't want to rent your own server, Grafana even has standard solutions with a very comfortable free tariff. In general, sit back and get comfortable. My name is Slava, and let's get acquainted with one of the most popular metric visualization systems. Traditionally, the test project will be a mini advertising server, about which there is already a whole playlist on the channel. We will install Grafana using Docker, so we will add it to the docker-compose.yaml and run the command docker-compose up. And while the Grafana image is downloading, please subscribe to the channel to see more technical videos in your feed. The image has downloaded, the container has launched, and now we can open Grafana in the browser to make sure everything is working. We enter the address that we forwarded for Grafana. The default login and password are admin, admin. It immediately suggests changing it, but we won't do that, we'll skip it. And voila, here is Grafana, already installed and working. Next, let's connect Prometheus, which is in a neighboring container. To do this, we need to open the data sources settings and add a new data source, Prometheus. Since all of this is in the same docker-compose, we can access Prometheus through the name prometheus, so the address is 9090. We can leave everything else unchanged here. We save, test, and voila, the data source is working. And let's start by displaying, say, the request rate, i.e., how often we receive requests to the rotator. To do this, we need to create a new dashboard, i.e., a place where we will accumulate all the metrics. Let's say rotator. Actually, Grafana has a cool graphical interface through which you can directly see all the metrics that are already added here. But before displaying this panel, let's make sure that we have some data at all. For this, I will launch bombardier, which will make a certain number of requests to the rotator. I can already start displaying this metric. I will display the metric using code. To display the rate of some metric, we need a function of the same name, rate, to which we pass our metric, i.e., http_requests_total, i.e., the number of requests. And then we need to choose the interval over which the values will be averaged. I will choose one minute. And here are our metrics. Let's set a slightly shorter time to see the curve on a different scale. Grafana has a cool feature that instead of showing figures like 50,000, you can explicitly specify what unit of measurement you have. And among the units of measurement, there are, of course, just seconds, and there is a format like Short, when the data is displayed in such a convenient format. And you also see that this incomprehensible, scary string is displayed instead of some single status. Let's try to explicitly specify what data we need to display. I want to see only status 303 there. Next, let's display the request processing duration. To do this, I create a new panel, and for it, I will need the same metric, http_requests_total, but now I simply select 90 percent. Voila, I have some very small number here. Remember that I said that you can choose the unit of measurement, and this is exactly what we will do now, and we will choose seconds here. Our request is processed in 5 microseconds. So, we have learned to display graphs. Let's display the rate as a separate counter, independently. I just copy the query, create a new panel, paste this query into it, and select the panel type. Among all this pile of types, let's say stat, it displays the last value, although I can configure this. And if I now send bombardier to hit the rotator further and refresh the panel, voila, 9,500 requests. Well, there is a slight delay there, because Prometheus collects data at certain intervals, but you generally understood the idea. But what happens if I launch a second rotator? For this, let's give the vde configuration the ability to launch multiple entities. And forgive me for this decision, I will directly modify the ports in the code and launch the second rotator. Prometheus still doesn't know about it, so I will add it to Prometheus as well. Now I will restart everything. Voila, Prometheus is now collecting data from two rotator instances. What if I now launch bombardier to send requests to the second rotator? I return to the dashboard, and as you can see, my data is duplicated. What should I do with this? If I want to display only one curve for a specific instance, variables will help you with this. So, what you need to do is set variables at the dashboard level, so you need to open the dashboard settings, and on the left, you will see variables, which is exactly what we need. We add a new variable. It has a name that will be available in all metrics. And now you just need to enter a query that will give you values for, essentially, just a dropdown list. Prometheus has a very cool function called label_values, which will help you get all possible values for a metric. I will take http_requests_total as a basis. I want to get all instances that I have, and here they are. Grafana immediately displays them. I save, and now, after I have created this variable, it has not only appeared at the top, but I can also use it in queries. I go down to the queries themselves and add it to the filter by instance. Voila, a single graph is displayed. Let's add this to all panels now. With a light movement of the hand, I can change the rotator instance whose metrics I am interested in at the moment, and literally on one dashboard, I will immediately see the metrics of different rotators. I will tell you about another convenient feature in Grafana. For example, you have uploaded some update or started some load testing process, and you know that you started it at 16:53. You need this to understand at what point your graph should change. And on the one hand, you can remember that at a specific time you started some process that will somehow affect the graphs, but this is not very convenient. What does Grafana offer? Grafana offers you in this case to simply set an annotation. You know that at 16:53 some process started, and you click here, for example, write that bombardier was launched, save it, and now a vertical line will be displayed on your graph, after which some data will appear. Voila, very convenient. Naturally, everything is saved, and after a reboot, you will see this vertical line in the same way. An annotation is a very visual way to show that something should change after a certain point in time, for example, you have uploaded an update. And in the process of our playing with Grafana, we have collected a lot of statistics. ClickHouse, can it be displayed in Grafana? Of course, it can. To begin with, you need to connect another data source, which will be ClickHouse. But here's the catch: if you click on this data source and try to enter ClickHouse, it won't be in the list. This is because it needs to be installed additionally. It's very easy to do. You open your Grafana settings, open the list of plugins, and enter ClickHouse in the list. Click the Install plugin button. The plugin is installed. And from the documentation, you can create a data source in the form of ClickHouse. My container is called clickhouse-db, so that will be the hostname. The port is the standard 9000, because we are inside the Docker network. The username is default, and the password is ver123. We save, test, and the data source is working. Let's go to create a query. We return to our dashboard and create a new panel. And our data source will not be Prometheus, but ClickHouse. And Grafana even offers a very convenient visual builder so that you don't have to write SQL queries. So, the database is rotator, the table is statistics, I want to display a graph of the sum of requests. Voila, I run it, and here are our requests. And if I want to display the browser as well, then I open the editor and simply add the query for the browser. I run the query, and it also displays that it's Fast HTTP, everything as it is stored in the database. Of course, you can display this not only as a graph but also simply as a table. Voila, the sum of requests, http. A big and important feature of Grafana is alerts. Thanks to alerts, you can set up automatic notifications if something happens to your service. And therefore, let's set up alerts to a Telegram channel if any of our rotator instances suddenly go down. And for this, we will need three things: a Telegram channel where alerts will be sent, a bot that will send them, and the channel ID. Let's start by creating a bot. You've probably seen this procedure a million times on BotFather, but I'll show it to you again. First, we create a new bot. It will ask for the name of this bot. I'm calling it youtube_grafana_tutorial. Then you choose a username for the bot, and it gives you a token. Then you need to create a group where your alerts will come. I will immediately invite the bot to it. And now the last part: how do we get the channel ID? There are many ways, but I consider the following to be the safest. First, you need to send something like "test" in the channel and mention your bot. Then you take and open the following address, where you need to replace this part with the token for the bot that BotFather gave you. I replace it, go to the address, and in the browser, you will see an update for the bots to which you will find your channel. Here is its ID. Then we take the chat ID, take the token for the bot, and go to Grafana to create a notification channel. We can create it in the alerting section, in the contact points tab. Here we add a new contact point, let's call it, say, Telegram, and in the list of integrations, we choose Telegram. We specify the chat ID that we took from here, and the token that BotFather gave us. Now we can test the integration and send a test notification. We sent it, and voila, here is our bot, which has already written something to us. Everything went successfully, so we save. Now we return to our dashboard. We need to add a graph that will be responsible for displaying the status of our rotator. For this, Prometheus has a metric called "up". This metric outputs one if the service is alive. It also has a filter by jobs, and since we are only interested in the rotator, I will specify job=rotator. And as a result, I see that both of my instances are alive. We save the panel, and now we open its editing again and add an alert. Here, a very convenient graphical interface opens for configuring our alert rules, and we need to set it to trigger when the value drops below one. Then we create a folder and I will set a very quick check so that we get a result faster, every 10 seconds. An alert has three states. The first state is when everything is fine, the so-called OK state. The second state is when something is wrong and the alert should send notifications. There is a third intermediate state when the alert is collecting data, the pending state. And here we can specify how long our alert will be in the pending state, i.e., how quickly Grafana will consider that something has actually happened and notifications need to be sent. Let me set it here for half a minute. In principle, we have configured everything, and the alert can be saved. And now everything is fine. And before we shut down the rotator, let's go back to the alerting settings and specify the policy according to which we will send notifications. I will edit the default policy, i.e., the one that will be applied to all alerts. And here, as the default communication channel, I will choose Telegram. Now I return to the panel, and we can go and stop the rotator. We have stopped the rotators. Everything, in theory, from this moment on, we will also record that the service has gone down. For this, we will wait a bit. Here, our instance has gone down. Since we check every 10 seconds, Grafana is already saying that we are in a pending state. In general, what is this pending for? It is needed to avoid cluttering your alert channel. The thing is that there are situations when you have either a false positive alert in the best case, or your system will try to restart itself and everything will be fine, and you want to wait for half a minute for the system to try to recover itself. This is why you set a certain delay. If everything recovers within this delay, the alert will not be sent. If it doesn't recover, then all communication channels will be affected. Here, half a minute has passed, Grafana realized that the rotator has not woken up and switched the state to firing. We switch to the Telegram channel, and yes, here is our alert that our instance on port 884 has gone down. Now let's start it again. Let's go back to Grafana. Voila, the instance has woken up, and everything has stopped. And Grafana has sent a message that everything has been resolved. But did you know that you can monitor not only these boring system indicators? Of course, using the same principle, you can set up alerts for ClickHouse and monitor, for example, the amount of money per minute. You just need to turn your business indicator into a panel in Grafana, and you can easily set up alerts for it. And with different alerting channels or contact points, you can send some alerts to one chat and others to another. Grafana is an incredibly flexible and powerful visualization system. It covers the needs of both a smart home and some huge distributed cluster. I hope the video was useful to you, but instead of the usual conclusion, to my surprise, I want to thank my first paid subscriber on Boosty. Honestly, I planned to announce Boosty a little later, but the first donation preempted my plans. Therefore, I present to you the final screen with sponsors, which I will show at the end of each video. If you want your nickname to appear on this screen, then I will sincerely be glad for your subscriptions, as well as one-time support, and all the money will be directed towards the development of the channel and accelerating the release of videos. And now, thank you very much for watching. Good luck to you and transparent operation of your microservices. For the next video [music]