Transcription
[music] Yes. [music] Yeah. [music] This is [music] Yeah. Yeah. [music] Good evening. My name is Nikolai Smirnov, and today we have an introductory lecture on backend development. I work as an engineer at Yandex.Lavka, specifically in the division that handles Yandex.Lavka's search. And what will we talk about today? We will be building a service from scratch. We will develop this service, the Yandex.Lavka search service, from scratch, as if it didn't exist yet. At the same time, we will follow the development of this service, specifically the development of the idea itself, from its birth to its realization. From beginning to end, we will touch upon the topics of future lectures that you will encounter on your learning path in the summer schools. These will include databases, software architecture, and infrastructure. We will talk about testing and alerting, about how to write metrics, and also about deploying your software products to production. Naturally, a small disclaimer: firstly, this lecture will not rely on knowledge of any specific language like C++, Python, or Java. It will be quite general knowledge, but there will still be some code snippets. Be prepared. And we won't dive too deep. I really hope that in future lectures, you will explore these topics thoroughly. Let's follow the path of an idea, the path of a feature. Specifically, the feature will be represented by us in the form of a new service, the Lavka search service. From the idea to the second step of a hypothesis, building an MVP, minimum viable product, a, minimal viable product. Then we will develop the architecture of this MVP. We will talk about how we write code and tests, and deploy, launch our already finished solution to production. Well, and then, naturally, we need to analyze whether this idea has truly taken off. Let's turn to the initial data of the task. Specifically, we want to imagine, let's pretend that there is no search in Lavka. But we need to create it, this service. On the input, we have 100,000 products. This is approximately the assortment of Yandex.Lavka. And the search works in such a way that the user enters into the Lavka application, may even enter an incomplete search query, meaning an incomplete query is an incomplete query. For example, you can enter "mol", and the search should guess that it's "milk". And on the output, we should get a list of products. Preferably, it should also be sorted by decreasing relevance. That is, the products that most accurately match the query should be at the top, and those that we are less sure about, at the bottom. Yes, but first, we need to stop and ask ourselves: why are we making this feature? Usually, this question is preceded by the formation of some product hypotheses. They are formed in this form: If [some action], then [expected result], and обязательно [an explanation] that we attribute to this expected result. Why will this result occur? And let's formulate the first hypothesis for the Lavka search service. If the user uses the search bar, then the time to fill their basket will be reduced, specifically the time until they receive these products, in general, on their table. That is, they will fill their basket faster because they won't have to find products in the extensive catalog. 100,000 products is quite a lot. Even though they are divided into good categories, it's still sometimes quite difficult to find what you need. And the second hypothesis: if the user uses the search bar, then they will likely buy products they haven't bought before. Why? Because the search returns relevant products for the query. And in the catalog, as you know, there is some principle of product division. And if, for example, you search for "gluten-free" for something, you might not find such a specific item in the catalog, but in the search, you might find something gluten-free that you haven't tried yet, right? And the first action that we take after deciding to test this product hypothesis is to figure out how to test this hypothesis with the minimal possible effort. MVP can be jokingly expanded like this, yes, if you were to implement a full-fledged, good text search with various typo tolerance options, so that it understands search queries well, it would take, well, a rough estimate, 3 months of development and implementation. We want to do it faster and test the product hypothesis. Perhaps everyone is already used to the catalog, and no one will use this search. We can do the same in 2 weeks. If, by chance, no one uses the search, we will save 2 months. You must admit, that's not bad. What kind of MVP? What will the MVP look like? Lavka has an application, the Yandex.Lavka application, this is the Frontend. Here is a very simplified diagram, but you will grasp the essence. The frontend has a screen. On the main page, we will need to create some additional elements, a search element. In general, we won't touch the frontend. This is also work that needs to be done, and it falls within the scope of frontend development within these 2 weeks. What do we do? We make a request from the frontend. The frontend usually sends its requests directly to the end service that provides the final functionality, and there is a gateway. This is backend for frontend, bff. Maybe you've heard of such a term. And this service distributes the request to the appropriate end service, can aggregate data from several services, and return results to the frontend. And this is what happens here. A search cube is drawn here. This is not yet a separate service. Whether it will be separate, we will discuss on the next slide. For now, let's just imagine that there is some functionality, search functionality, and it will be implemented as simply as possible. It's just a select, some ID, a product ID from some table that we will manually populate in the database. Name, like, and our template. So, here, of course, we can slightly complicate this SQL query. For example, build the condition in such a way that the search query is broken down into several parts, normalize these parts, cut off endings, and so on. But the essence, I think you've grasped, is that it can be done quite simply. Moreover, many databases even allow fuzzy search, meaning they account for typos. This is built into the core functionality of these databases, such as PostgreSQL, for example. And that's it. The MVP, one might say, is ready now. Yes, I forgot to mention that besides the IDs that we returned from this search function, we can't send them to the frontend. We need to, first of all, check if these IDs are actually in stock and can be returned in the search results so that the user can order them. That is, there is a stock service, which checks availability in stock. And there is also a catalog service. The very catalog service that renders the storefront, where there is dairy, eggs, and so on. This service knows everything about the products, it has information about the image, the name. That is, it is used precisely to enrich the product card that will be displayed on the frontend with information from the ID. Such an uncomplicated MVP. Next, we need to determine how we will implement this MVP, which components are already ready in the system. The Yandex.Lavka system is quite large. How to integrate this search service? Well, let's see where we will integrate this search functionality. There is, as I mentioned, the catalog service. This is a quite large service, it has a lot of functionality, many developers, a lot of code, and so on. And it has its own database. It seems like it would be great to add another endpoint there, meaning another context. This is a handler. We also call endpoints handlers in Yandex. Get used to this name because I sometimes use it without explicitly mentioning that it's an endpoint. And the search endpoint or handler will be visible externally, and at the same time, we need to place a similar endpoint in the backend service that will return the IDs. Where to place it? In the catalog, or write a separate search service and create a separate database for it? Let's consider the problems of adding new functionality, new feature functionality, to already existing services. This service, as a rule, has many developers, more than ten, perhaps. A large codebase, meaning it's quite difficult to understand, to figure out where to integrate. A high chance of rollback of changes. If there are many developers, they write many things. And at the moment you release your feature, a dozen other features might be released along with yours. And rolling back due to one of the features not working correctly is quite easy and typical. Rollback means rolling back from production. Compiles slowly. This is also, of course, a characteristic of large codebases. But what to do? Development time increases. Testing takes a long time. Since there is a lot of functionality in the service, it must be thoroughly checked, including regression testing of all existing tests, to ensure that you haven't broken anything with the new functionality. And one of the last, but not the least important, points is that your handler, your endpoint, which you have integrated here, might experience a sudden avalanche of load. Everyone will really like how you implemented the search. Everyone will start using it, and the current resources of the service will be insufficient. So, along with bringing down the search, we will also bring down the catalog service, and the Lavka application will not work at all. Yes, here are some rules of thumb for placing, excuse me, features in a separate microservice. This is exactly what we did. First, and perhaps most importantly, is that the functionality can be separated from other services, meaning it is independent. For example, payments, search, discounts, advertising. Of course, you can integrate them into another service, but it's clear that they have little overlap with the existing services in the Lavka infrastructure, so it's better to move them to separate services precisely to avoid the problems we discussed. Scalability is an important thing. Since the search service may be used by more and more people over time, it is necessary to allocate more resources specifically to this functionality. Therefore, a separate service allows for very flexible scaling of resources only for this function of the entire application. This can also include separate services that process images, for example, heavy operations, apply models, yes. And this point does not specifically concern our example, but it is also an important rule of thumb that functions and services related to security are usually separated from other large services. It's understandable why: personal data and access keys need to be isolated very strictly. To do this carefully, it's better to put such functionality into a separate service. So, we decided that we will write our own separate product search service. What database will we use for this service? The choice of databases is dictated, in fact, by many factors. And one of them is the technology radar or the list of technologies used in your team. What is this? Well, in fact, it's a list that includes the technologies, the tools that you can use. The technology radar is a more detailed map of the technologies that are used, that are being considered, that have already been tried and discarded. So, it's a map where you can see what has already been tried. You can refer to this radar and see if you will be reinventing the wheel and implementing something that someone else has already implemented and, for example, unsuccessfully. Therefore, when choosing a database, the technology radar is important, and we look at it first, but we also consider the team's expertise, because usually in the list of allowed technologies, there may be several databases, and you can choose any of them. Well, you need to look at what you know how to work with, and can quickly write this MVP in 2 weeks. There is support in the infrastructure, meaning you have that, I don't know, cloud, where this database will be supported simply as a service. This is a big plus in implementation, in speed of implementation. And probably the most important, of course, the key factor is whether the database suits your goals, your task. We need to search for text, right? That is, text search is, in fact, a typical task. And, in general, it can be solved in different databases. We can search in MongoDB, there are also special plugins in PostgreSQL that can search well for fuzzy queries. There is a specialized database, OpenSearch. Someone has probably heard of Elasticsearch. Well, it's the same thing, only its open-source implementation. And it seems that in this case, it would be ideal to set up OpenSearch, but OpenSearch was not considered specifically at that moment in the Lavka team, because there was simply no one to set it up and no one to maintain this database. Therefore, let's assume that the search service, a small disclaimer, in fact, what I am telling you is very convincing that this is exactly how search in Lavka developed. It's slightly modified to adapt it for the lecture, for a simplified view of this story, yes. Let's assume the developers chose PostgreSQL simply because it's quite easy and convenient to interact with PostgreSQL, and it's already there, you can easily get a database. There are already instances of this database in other services. Next, we need to decide what language we will use to write our service. And here, of course, is the field for holy wars. Holy wars are precisely about choosing a language. Some say you should write in C++. Why? Because it's super performant. Some say in Python, because you can do it quickly, and some say in Java. There are many criteria, just like when choosing other technologies, but the main ones are, of course, what you know how to write in, what your team knows how to write in. It will be bad if for an MVP you choose a language that no member of your team is proficient in and you will have to learn additionally. In addition, the expected load is taken into account. Load is usually divided into CPU-bound and I/O-bound. In our case, we will simply fetch data from the database and return it, essentially just transferring JSON. Therefore, here, of course, something that is not demanding on computational resources, on CPU, and can be written in Python would be suitable, because Python, as we know, is an interpreted language and is often chosen precisely when you need to write something very quickly, when you don't need to beg the compiler to let you compile a binary. In Python, everything is quite fast and simple. And moreover, in Python, often such specific packages are used, which, for example, are needed for your search or for some functionality in search, for example, when searching by image, some package that is only used in Python, there is for Python, and it can be conveniently and quickly adapted in a Python service. Yes, well, there are many criteria, in fact. As confirmation of which language to choose, I will simply give an example that I have. The guys from Taxi rewrote a service that was written in Python and generally worked quite well. It handled 100 RPS using one and a half CPU cores. At the same time, the same functionality, under otherwise equal conditions, here, of course, some conditions might not be met, but in general, the same functionality, on the same computational resources, delivered 400 RPS in C++. No offense to Python, no offense to anyone who writes in Python. I also love and respect Python very much. Yes. So, what's next? We've chosen the language. Actually, in Lavka, search is written in C++. Here, perhaps, the role was played by the fact that this search, naturally, as a hygiene feature in any retail, in any service that sells products, must be highly loaded, firstly, and return results quite quickly. Therefore, it was written in C++. All technologies have been chosen. You, I may be talking for a long time, but in reality, everything happens quite quickly. These issues are resolved much faster than I am even saying. How to agree with frontend developers on what data you will return and what data they should send you? Interfaces are used for this. This is a very important part, in fact. And for interaction with the frontend, we use OpenAPI descriptions. You've probably heard of Swagger. This is it. It describes, this description has a fairly simple structure in a YAML file. And in this structure, we describe the endpoints to which the frontend will send its requests. And here, V1 search is an example of such an endpoint for a handler. We describe the parameters we expect as input. We expect text. This will be a simple text query. And we describe the response from our endpoint. Here, in a shortened form, I describe only a successful response. A 200 OK response, which will contain, in fact, the search results themselves, namely an array of product identifiers. Based on such descriptions, based on YAML descriptions, it is very easy to generate almost ready-made services. That is, they will be stubs, of course, but with the development of low-code, you can prototype very quickly. In fact, such services are simply based on the interface. Moreover, you can ask the interface to generate them. Yes, this is not the only description format. Inter-service communication, as a rule, uses gRPC, a binary protocol that is more efficient for sending requests. But with the external world, for simplicity, OpenAPI is most often used. When designing an interface, you must always think about API extensibility, meaning you will develop it later, add some fields. You need to think about this, but it's better not to get too bogged down in what the service will be like, what the interface will be like in a month, two, or three. So, it's desirable not to overcomplicate here. Versioning must definitely be provided so that you can support different clients, if there is such a need. For example, there are different applications, there is the Lavka application, and there is the Food application. And they have different release cycles, so sometimes it's necessary to support different interfaces. And yes, and remember that the service can be scaled both horizontally and vertically. With horizontal scaling, we add new instances of the service. And if the service interface supports the principle, and this principle contains the main, let's say, the main postulate, that the service does not contain state, then such a service can be scaled horizontally quite easily. And vertically, of course, is scaling by increasing resources, increasing the performance of your service, without adding new instances. Okay, we've seemingly figured out the service. Here I wanted to talk about backward compatibility. It's quite a common task, in fact, when you need to extend your interface, meaning to add something that was not previously provided. Well, first, yes, there is versioning, which I mentioned, and there is also the possibility to extend interfaces using optional elements. Optional elements don't hurt or help anyone. That is, clients are happy, and the server that receives this message and exposes this interface is also happy because it doesn't expect anything to be necessarily sent to it. But there is also a downside to optional elements. Sometimes you want to strictly say that certain elements must be present. And let's consider what the sequence of actions should be if you want to add a mandatory element to your interface. If you add it to the request, meaning to the request that comes to the server, then first this updated interface needs to be updated on the clients. And so that clients understand that from this moment on, they must send this field. So, recompile all clients so that they start sending this mandatory field, update all clients on the servers, meaning on the Lavka application, for example, update everyone. Then this mandatory field. The server, at first, does not yet have the updated interface, it will simply ignore it. This is how the format with which OpenAPI communicates between the service and clients is structured. It will simply ignore these mandatory fields. Then, when all clients have been updated, you can roll out the server. And then it will strictly check that all clients are sending it this mandatory field. If we want to extend the response, meaning the response from the server, then it's exactly the opposite. First, we roll out our service, and it starts sending the mandatory field to the clients. Clients ignore it, and then the updated clients gradually start to perceive this new mandatory field. Okay, we've described the architecture, but it's important, usually one or two people do this in consultation with each other, but most often, after the architecture is described, a certain procedure is needed to understand that nothing has been missed or forgotten. This is an architectural review. It's very similar to code review, meaning it's also a document, often called an RFC, request for changes. It's in text form, preferably with pictures, if you want it to be read, it's better to accompany it with pictures, and it's sent to the code review system, and there people will make suggestions, criticisms on your document. And thus, a better solution is born. And if something is truly improved at this stage, you will actually save a lot of money for your employer, because a better solution at the architectural review stage will pay off by not having to redo it later. There is such a rule of 1, 10, and 100. That is, fixing errors in production is much more expensive. Here are examples of how architectural reviews are conducted at Uber and Yandex.Taxi. Well, in general, I think it's pretty much the same as code review you've seen in your own products. Okay, we write code, go through code review of what we've written, adhere to code writing rules, usually code formatting, meaning we use code style guides, linters, formatters. And we will skip this part, it's important. You will be doing this in the backend development school. I want to talk about tests. What tests do we write and when? Surely you have all seen this testing pyramid. This pyramid contains layers. On the bottom layer, the largest, thickest, fattest, are unit tests, then integration tests, and the peak is end-to-end tests. Sometimes they are also called system tests. Some even place end-to-end tests on a small peak, even higher, but that's not the point. The main thing is that there are certain levels of testing. And let's understand what we expect from these levels and what tests are generally written at these levels. Firstly, we want most tests, good tests, to run from your machine. This is actually a very critical requirement for tests on your machine or on a virtual machine. That is, there should be some environment where you can test your hypotheses, program something, and check that it works. If this is not the case, then, in fact, the development cycle increases significantly. Unit tests must be present, which will cover complex algorithms in your code. If, of course, there are no such algorithms, then the testing pyramid sometimes turns into a testing barrel. That is, there are actually few end-to-end tests. Many integration tests, because they are important, I will explain why. And few unit tests if there are no complex algorithms in your code. What about integration tests? I'll let you in on a secret: in the search service, we actually write a lot of integration tests, because they cover all endpoints. Here, handlers are indicated, meaning they emulate how the service will behave in production. That is, they send some messages from the outside to the service, the service responds, and the tests check that what is expected is indeed returned at the endpoints. And also, of course, there are main usage scenarios in these tests, integration tests, and negative ones. System or end-to-end tests, as a rule, are run in a test environment, meaning a special stand where the entire Lavka, all services, are deployed, and you need to check your final feature in the finished product. At the development stage, if you are developing a new feature, then tests are usually manual. That is, you just need to click through the application and see that the search actually works. Later, if the feature is established, then autotests are written for it, and usually test automation engineers are involved. And the test becomes a regression test. That is, when a new release comes out, your functionality, your cool feature, is also checked. Performance testing and production autotesting deserve a separate mention. Performance testing is a general concept, it's not always done, not everywhere. But if you are writing some kind of infrastructure or core service, a critical service, then it must be tested very well from a performance perspective. Why? Because many services will depend on it, and the speed of this service will affect all other services and the overall solution, either negatively or positively. In performance testing, on
In reality, there are many subtypes. And one of the types is load testing, when a load is applied to the service and the load is increased until RPS - which is requests per second. So, we are continuously increasing the number of requests per second with our load generator. For example, this could be Yandex Tank, a system that allows you to smoothly increase the load and find the point at which the service can no longer cope with the incoming load, and starts responding with either timeouts or errors. This point is called the breakdown point. It is important to determine the breakdown point in order to roughly understand what load your service can handle on which resources. There are other types of performance testing, for example, stress testing. Specifically, our services are deployed in several data centers at once. And one of the performance tests, called stress testing, involves removing one data center and seeing if our service survives at all in the service of the store, whether it will work and redistribute the load. There are also other exercises, chaos exercises, when a search service or some other random service is completely turned off, and the solution, the complete solution, namely the store service, is tested for viability. Yes, I forgot to announce that everything you are listening to, you are absorbing and analyzing, and at the same time, you are free to ask questions. You can do this in the chat. We will take a short break approximately in the middle of the session, and for some of the questions, the most interesting, liked questions, we will see how to answer them. I will try to answer them, so please write your questions. Okay. About the environment for tests. We remember that we have our beloved laptop, which we work on, or a stationary computer, and a virtual machine. Often, in these environments, unit and integration tests must necessarily be performed. And if they fail, it's sad, but it's better for them to pass and pass quickly in order to get fast feedback on what you have coded. After everything works locally, you have checked your code, the code of your feature is committed to the common repository. And with the help of processes called continuous integration and continuous delivery, first, it is delivered to some kind of test environment. This can be a test stand, or it can be a specially allocated environment for testing to run unit and integration tests. But if a specific area is allocated, namely the testing area, then your changes are uploaded there and all possible tests are run to exclude the probability of an error in your code and in the code of someone who has also gone with you to this testing. What happens in this environment? All those tests that are unit and integration tests, which I mentioned, and system tests can also be performed, including manual ones, meaning this is a stand where you can click through your new functionality. And performance tests are also launched here. Sometimes separate servers are allocated for performance testing so that they are comparable in performance to what will be running in production, not measuring the temperature on Mars, but making performance measurements close to production conditions using the same continuous integration and continuous delivery processes, your code then goes to the pre-stable environment. This is already production, but not production for everyone. That is, naturally, all tests should have passed by this time, and the release should be deployed to some specific server, which is the pre-stable environment. And often, smoke testing is carried out precisely in this environment. Smoke testing is a small test that checks that the service has started up, that it responds to some service messages with 200 OK. And after that, if everything is fine, and a certain percentage of traffic is sent to it, usually a very small percentage of users get to the pre-stable environment. And in general, monitors are used to check that the new service that has been rolled out behaves exactly the same as the rest of the stable version, which has not yet been updated. At the same time, no new errors appear. Next, it is rolled out to stable, to production. So, we have put a comma, not a period, in testing. Let's talk about how everything rolls out to production. First of all, we need to think about how we will roll out our feature to production. The first thing we need to estimate is how many instances of this search service to request from the infrastructure. We can go with one service per data center, meaning three data centers, three services, but will the incoming load on our service be enough to withstand the influx of users? Well, we need to make some forecast. To do this, logs and monitoring are analyzed, which are currently being observed, and at peak hours, what is falling on the service. Here we mean the store service. In principle, how many users per second are making some requests. Then, after analyzing these requests, we can build some forecast. We can also assess current performance, and the results of load testing help us greatly with this. Yes, we can approximate them and estimate how many resources we need to withstand the load that we have forecasted. Okay. Now about deployment. This is a very important slide, because, well, Yandex has a quite serious structure with geo-redundancy for the services that are deployed to production. And for example, here is a very simplified diagram of how requests reach the service pods. Pods are essentially instances of a service within the cloud. A request goes, yes, via Yandex.ru, we find Yandex.ru, we resolve this DNS name to an IP address. Via the IP address, we go to the correct server, already in the correct data center, which then balances. This is an L3 load balancer that balances to the data center to which the request needs to be sent. There are complex mechanisms for how balancing is done. The essence is that our request reaches the data center, where an L7 load balancer is already working. L7 and L3 probably don't mean anything to you, but in fact, if someone has heard of the seven-layer OSI model, it's exactly that. L, respectively, is at the seventh level, L3 is at the third level of this model. What happens next? When a request reaches this center, it understands, by the name we are requesting, which service we need to request, and sends it to a specific pod, balancing the load between these pods. If the service needs, and this happens very often, to go to some other service, then again, an L7 load balancer comes to its aid, which can distribute the request, either within this data center. If this service is not in this data center, then it distributes the load to the neighboring data center. But often, requests between services are within the same data center to save traffic. Naturally, this is a dynamic system. This means we cannot make it so that all service instances are always fixedly attached. Therefore, when raising or lowering a service, there is a storage database, a service registry, service discovery, which stores information about which services are running, on which hosts, where load can be directed, where it cannot, which pods are ready to accept load, and which are not yet. That's roughly it. Well, we also need to draw one more layer, Datacenter 3, and the work of our search service happens. It lives in three data centers like this. How will we roll out a new feature to this large scheme? It happens gradually. The first server to receive the new feature will be the pre-stable server. Usually, it's one server among the servers. And as soon as we see that everything is fine according to the instruments on it, well, here you can see that the rollout has occurred, and sometimes some errors arise when lowering or raising a new service instance. These errors are retried, so for the user, everything happens seamlessly. They don't see these errors. And at the same time, the new instance is rolled out first to pre-stable, and then you can see that such a wave rolls out to stable, meaning to all remaining N minus one servers. The feature itself is rolled out, meaning its code lives in production. It's running, but no one is calling it yet. Why? Because, as a rule, new features are written under flags, meaning they are not immediately enabled. Why is this done? Well, firstly, there is a tool that manages this. These are flags or feature flags. They are also called feature toggles. There is an admin panel that allows you to manage feature flags, these flags, and smoothly enable them. This means we can enable a new feature not on all servers, but only partially for a specific audience and see if everything is okay with performance, with how users interact with this feature, whether there are any errors, or if some usage scenarios were not taken into account. For example, users start entering very long strings, and we didn't anticipate this. Such errors can be caught on live users. So, what else do feature flags allow? They allow for gradual rollout, they can be quickly disabled if we see errors occurring, just click in the admin panel and disable all feature flags that cause suspicion. And, accordingly, the feature is gone. They go to fix this feature. And they are also used for A/B testing. We will talk about it. But now it is important to understand that, as a rule, analysts are involved in these tests. That is, they design the test, divide users into groups, and make a decision on whether a feature is good or bad. However, the backend developer must support everything that the analyst has designed in the code. This means some work also needs to be done. And what is this work? Let's imagine that two users with different identifiers make a request to open the main page of Yandex.Store. User ID 111 and 22. At the moment of the user's request, the service, well, not the search service, but the preliminary service, which we designated as a gateway on the diagram, meaning this registrar service, does the following. It goes to a separate experiment service, passes some metadata about the user, for example, their identifier, where and from which side the request came, and some other data. And the experiment service, based on the configuration set for it, determines whether to show the new functionality to the user or not. Well, for one user, it decided not to show it, for the other, it decided to show it. And, accordingly, they see different pictures in their application. Consequently, they act differently. Most often, users are segmented simply by a hash of the user identifier. Why? Well, in order to fairly distribute some users into one group and some into another. Yes, and the rollout sequence with the help of the experiment service usually looks like this. That is, first, only you, in general, see your cool feature, you see it, you set it up for yourself so that it is visible to you. Use it, click it, check that it works. Then roll it out to the team. The team usually consists of the most loyal users, they provide you with feedback, say that it's great, we need to roll it out to production. Well, the next stage is A/B tests, where users are divided into equal groups. A is the control group, which has the old interface, this feature is not present. B is the test group, and the feature is enabled in it. What does the analyst do? They look at how users in group A behave in relation to group B, or vice versa. If the A/B test is deemed successful, meaning users in group B are happier than users in group A, then the feature is rolled out to the whole world. Here is a small piece of code that shows how the call to the experiment service goes. This is muched experiments. It returns an array, or rather, a list of experiments that are enabled for this user. To determine if a feature is enabled for this user, their identifier, category, and some other parameters are passed. There is an experiments admin panel where you can finely tune that this experiment will be rolled out, for example, to certain platforms from a certain version to another, it will last from a certain date to another. And, for example, it will only run in Russia. And the experiment service says: "If the experiment is enabled, then under this feature flag, you do your cool features." Yes, an important question. Well, maybe not very important, but a hygienic one. What happens next with these ifs in the code? They clutter up the code. And, as you understand, it can be quite difficult when there are many flags that enable or disable certain experiments. As soon as an experiment is deemed successful, well, there is a good rule to roll out, to delete these flags. But often this is done on subbotniks, meaning when developers simply gather and delete such flags in bulk. How to understand that the service is working? You rolled it out to a group or to your team, how to understand that everything is fine with it? Well, usually, the duty engineer monitors the health of the service, the health of the feature, and does this when the rollout has already occurred to a certain group, perhaps to a small percentage of users. They monitor using monitoring. The service has some signals that it sends. These signals are sent by automation and are collected again by a service that stores metrics. Naturally, there are quite a few such signals, and you can't keep track of all of them, so there is automation that monitors these graphs. But let's look at why monitoring is done. These are, of course, the most basic characteristics of pods. This is how much resources the service occupies, meaning how many megabytes, gigabytes of disk space it uses, how much RAM it occupies, CPU cycles, for requests, the response code is checked. If the service suddenly starts returning 5xx or 4xx errors, then in this case, there will definitely be an increase in some indicators on the graph, and an alert will occur. There are other specific services, such as queues. For them, different characteristics are evaluated, for example, the size of the queue, the time a task spends in the queue. And, in addition, we can write our own metrics, for example, the average length of a search query or, well, this is probably not a very useful metric, but for example, the number of items returned from the search results is an important metric, because the average value will actually be a good indicator of the health of the search service. If this value decreases sharply, it means that the search is starting to fail. At the same time, it may not be returning 5xx errors. Yes, let's take a short break, and continue with DevOps. And if there are any questions, I will try to answer them now. So, a good question. What part of development usually falls on a junior and their responsibilities? Well, let's put it this way. A junior is a person who, as a rule, works with a mentor, meaning they are accompanied and told what needs to be done, shown code examples, and advised on how to solve a particular task. Therefore, well, it happens that new features are also developed by juniors, if it is very clear how they can be implemented, right? That is, if the mentor himself understands what needs to be done, he just needs to present it in a clear form, give it to the junior developer, and they will implement this functionality. Yes, as a rule, juniors interact little with other teams. This is probably the prerogative of more senior developers. Well, in general, these kinds of tasks are quite often solved by juniors. And there are very interesting tasks, in fact. If they are well isolated from other tasks, you can get a cool task, at least during an internship at Yandex, and add it to your portfolio. A good question also. How to synchronize the state of pre-stable and production? If we are talking about a service that has a database, then it often happens that when deploying to pre-stable, it is necessary to make some changes to the database, so-called database migrations. Well, they are often done in isolation, right? That is, first, we update the database, doing it in such a way that the changes are backward compatible. Yes, indeed, this can be quite difficult. But rollback scripts for migrations are also provided. When something goes wrong, and we haven't foreseen, for example, some backward compatibility, then the migration rollback script is executed. If we are talking about the state of the service itself, then we try to write stateless services that do not store state within themselves. This is precisely in the spirit and allows for easy scaling and independent updating of pod services. What is the approximate ratio of test code to production code in the store? By production code, I mean, if I understood the question correctly, then how much code is written and is already active and accepted as an experiment, and the code that is under flags. Well, 90%, I think, of production code, and the remaining 10% are under flags. I don't know if this satisfied your curiosity. So, why a separate center for BD? You can do everything on MySQL and you won't need a separate server. Well, look, modern realities and the realities of working at Yandex dictate new rules for working with databases. That is, most often, service developers do not set up their own databases, but use databases as a service. That is, it's a cluster of databases, to which a request is simply sent, it knows how to distribute the load across different instances of this database, does it quickly, and this is often how it's done in large companies. If we are writing some startup, then sometimes you need to set up your own database and, in general, run something on it. DevOps is quite difficult. And in fact, it's quite far from backend development, to be honest, at least at Yandex. What is used in the store for feature flags? For example, Unleash, not Unleash. Their own admin panel and their own experiment service are used. Well, because I think that when Unleash was written, this is my assumption, I haven't used this service, but at Yandex, the culture of conducting A/B tests is very old and probably one of the most mature, if not in the world, then definitely in Russia. Well, okay, let's move on, because time dictates that we move on to the next episode, namely, we will continue with DevOps and talk about who monitors the graphs. Automation actually monitors the graphs. And for every graph that we would like to monitor, there are some specially configured systems that monitor this graph. If the graph crosses some boundary, exceeds or falls below the permissible level or some permissible range, then alerts occur. These alerts can be of different levels. The most serious, which requires you to run and fix immediately, is a disaster. As a rule, automation is also connected to it in the form of calling the duty engineer, a robotic system that tells you aloud that you have an alert: taxi contractor Order sping stable, something something. And you, as the duty engineer, must react to this signal, go, connect, see what happened there, and start fixing it. There are other levels, the reaction to which is not so serious, but nevertheless, they also do not tolerate being left unfixed. This is the critical level, which indicates that something bad will happen soon. And the warning level. Usually, such warnings can be postponed. If you catch a warning in the evening or at night, then usually you don't react to it, but fix it in the morning. Yes, naturally, your phone is not your only friend, but also Telegram and other communication channels. They will reach you through all these channels, even if you are sleeping, if you are on duty, you will be woken up. If you are completely asleep, then the second duty engineer is definitely not asleep. How to understand the problem? You have found out that some alert has gone off, and you need to diagnose this error. Logs help the most here. In fact, this is the first tool that allows you to dig in and see what is happening with the service. To do this, we simply look at the service logs. There are different systems for aggregating these logs. Some time ago, we used this system. Now we have switched to another one, but it doesn't really matter. The main function is that we can see current errors on the service almost in real-time and analyze which errors have increased, and what happened. Yes, other tools also help with diagnostics. These are back offices, technical admin panels. As you know, any service has some public IP that it exposes externally so that clients can access it. If we are talking about a search service, this is such a handle, as I said, for Yandex.ru, this is an endpoint, that's what the handle is called. Public handles are where our clients go. But every service also has private handles. These private handles are most often used either by other services or by admin panels. Technical admin panels allow you to perform some safe actions with the service, look at the state, restart some component, or view some specific logs. And for this, you need to provide such technical handles in your service. The Yandex.Store search service also has technical handles. For example, through them, we periodically check if our search is working correctly and what kind of results it provides. That is, how high-quality it is. This happens periodically. We simply send requests to this handle, as users do. Okay, we have figured out the problem with the logs or with other tools. We fixed production, and we go to celebrate this moment happily. But no, in fact, there is a process called incident management or incident analysis, in which, in fact, it is a meeting where engineers who are well-versed in the service and infrastructure meet and discuss the problems that have arisen, emergency problems that have led to losses, for example, some service losses or financial losses for the service. What happens at these meetings? Well, first of all, the chronology of events is reconstructed. What happened, what was happening? Why? How did the duty engineers act, what did they do, how was the error correction done, how was the analysis done? Why is this needed? Not to blame anyone. Not for the sake of blaming. Like Vasya overslept and didn't hear the alert call or did something completely wrong or took a long time to figure out where to look, which logs. No, not for that, but to find the causes and not repeat, not step on the same rake in the future. That is, so-called action items are written down, that is, actions that will prevent the occurrence of this error in the future, if such actions are possible at all. And an important rule at such meetings, I say "meetings" loosely, at these meetings, is that no one is blamed. Yes, and this is actually a very important factor, because otherwise problems will simply be swept under the rug. We don't do that. We do everything openly and friendly. Okay. We have dealt with the unpleasant moments that could arise during feature rollout, and understood how to diagnose them. Now we need to understand if this feature is useful. To understand this, we need to calculate some business metrics. These business metrics are abundant, in fact, in any mature solution. Often, there are hundreds, or even thousands, of these metrics. But sometimes they are still not enough, for example, if you are creating some new unique functionality, some metrics for this functionality can be recorded separately. The metrics service exists for this. The Matrix Client in this piece of code is precisely a client to such a service, the metrics service. Well, and here it writes that search was used in this session. Then, based on these metrics, a graph can be built, and we can monitor how popular the feature is. These metrics can also be aggregated and used in more general cases when evaluating, including user behavior, where these metrics are present, how they are collected. After you have rolled out the feature, and the metrics that are important for your specific feature, namely for the search service, remember when we formulated hypotheses, that it is important for the user to add items to their cart faster. And the second hypothesis was related to the fact that the carts will actually be more diverse. That is, users will not stay in their bubble of familiar products, but will come out of this bubble and pick up new products from the search. These metrics, as key user behavior metrics, are tracked in A/B tests. Here we even see an A/B test. That is, we have a control group A and two more test groups, which can actually differ in some settings. For example, one group will have search that takes into account typos, and another group will not take into account typos. And here, based on key metrics that are collected based on, in fact, both the metrics that we send from runtime, and the metrics that are taken into account indirectly by user behavior. That is, each user session is evaluated, all sessions are aggregated, and we look, for example, at the average number of items of goods, meaning from different categories, that users ordered in group B. If there are more of these goods, meaning the results are more diverse, then
We will see here a green, green number in the header. This is how these metrics are obtained. And based on product metrics, a decision is made about which group won, and, accordingly, whether we roll out the experiment or not. But you can't just take and roll out some feature, because when rolling out to a large number of users and with the evolution of your service, that is, it develops, the load on it grows, users organically become more, because they are added to the stores, they become more active search users, the document base itself grows, that is, there become even more goods. And we see that the graphs show an increase in search response time. This is bad, and you won't be able to check this in any way, you won't be able to foresee it in any way, if you don't launch your feature on real users. Therefore, diagnostic tools exist. Specifically in this problem, checking stack threads is most often used. You can see what the threads are doing, why they are slowing down. Or you can use profiling as a heavier tool and take, for example, a flamegraph. And see that all application threads are almost entirely busy reading something from the database. Well, here, perhaps, we simply lack database resources, and they can be increased. But in fact, there are different ways to optimize working with the database and speed up queries. For example, this is the optimization of SQL queries. You can create an index on frequently requested data. And thereby you will simply significantly speed up the execution of your SQL queries. Often this is a cheap way to increase your database performance without resorting to horizontal scaling. And such rules, not rules, but some recommendations. Avoid functions inside. If you are making an SQL query and writing an SQL query with this part, with a part, use pagination, that is, request not all data, but partially the first, second, third part. This will provide a much faster response to the user. Yes, you can also evaluate the execution plan using XPlain and make some optimizations based on what you see in this plan. And then follow more cunning methods, which, of course, fit more into the realities of small projects when you deploy the database yourself. For example, you can partition tables, divide large tables into smaller and manageable ones, for example, divide a large table with logs, which you need to access, into parts by months and when making a query, search not in the entire table that covers all time, but access a small table for a specific month. You can use sharding, that is, when you physically divide your database into several databases, and send queries to one database, then to another, then to a third, based on a certain criterion. This criterion, well, here jokingly, criteria like beautiful ones go to this database, smart ones here, and smart and beautiful ones go to the database in the middle. In reality, of course, most often, a criterion not dependent on, let's say, on the criteria of other objects is used for sharding. It can simply be a hash of the identifier. Clustering, your database instance can be clustered, that is, make, for example, an Active and Standby cluster. In Active, you will have read and write queries, and in Standby, only read. You can read from this Standby. You can build different types of clusters. But I repeat, that in large companies, you often don't need to deal with this, because the DBMS cluster is already provided as a service. You simply access a single entry point, and the cluster is already behind that entry point. What can we do? As developers, without resorting to database-side optimizations, we can use caching. If we know that products in the database are not updated very often, we can, for example, for popular queries like milk, eggs, bread, build a cache and keep it directly in the service's memory and provide a very fast response without querying the database. This also significantly reduces the response timings of the search handler. Uh-huh. Yes. But let's move on. We decided that it would be great, in fact, to put all these 100,000 products that Yandex Lavka has into memory. They also fit perfectly in memory of a single service instance. And the main thing is to structure this memory correctly. And there are algorithms that allow you to keep your index, by which you want to search, in memory in a fairly compact form and search in it quickly. Such an algorithm in the Lavka search service was the Aho-Corasick algorithm. It is also used in the `grep` utility in Unix. And what does it allow us to do? Well, first of all, it allows us not to access the database. We exclude it from our architecture. We move to a new architecture. But in order to fill this index, the index by which the Aho-Corasick algorithm searches for products by search query, it is necessary to get this index from somewhere. If we don't have a database, let's go to the catalog service and asynchronously fill our index from there at the start of the service. This can be done. It's done quite quickly. This index is filled over the network. All these 100,000 queries take about tens of seconds when the service starts, and such a process is performed. Yes, this is actually a great optimization. And to be honest, this is exactly how the Lavka search service looked in its first version. It didn't use a database, but used Aho-Corasick and an in-memory index. But let's imagine that our search is working, working quite well, stably, handling the load, resources are sufficient, but something doesn't suit you about it. For example, you use fuzzy search in the database in this index, meaning you can search by parts of the query, but for the word "liver" it shows "cookies" or for the word "lemon" it shows "lemonade". Well, in general, you can expect such answers from the search, because you assume that you have instant search, and the search should guess what the user will continue to type, but you would like to make it so that a user searching for "lemon" gets lemons at the top of the results. And for this, it's not enough for us to, first of all, understand this problem, and for this, it is necessary to evaluate the output with our eyes. Well, or with the eyes of people, or with the eyes. Now LLMs, large language models, are widely used. With the help of these tools, tools in the form of assers, which know how to label, in Yandex tasks there is such a service, they know how to label pairs of product and query. That is, they can say: "Well, yes, showing lemonade for lemon is quite relevant." At the same time, if we show lemonade for lemon, it is less relevant. That is, they can introduce some gradation, and with the help of this gradation, you can derive different metrics. For example, a search metric that we often use is the proportion of relevant results at a certain position. The graph actually shows the amount, not the quantity, but the proportion of relevant results. About 90%. In the top search results, we can measure such a characteristic. There are other search metrics such as NDCG and Pfound, which indicate how well the results are ranked, and the user will see what they were actually looking for. We also use all these metrics, and they need to be implemented to understand the scale, how bad or how good the search is working. What is this for? To take the next step, a qualitative step in the evolution of search. Namely, let's collect user logs, click history, that is, which products in our basic version of search users actually clicked on. Well, that is, if they searched for "lemon", and it was in some twenty-fifth position, but they clicked on it, we would like to take into account the clickability of this product. We will collect these logs. Then we will collect a dataset from these logs, which will contain some features. This is a standard task. There are some features, there is a target, that is, whether the user will buy this product or not, and we will train a model to predict, based on these features, whether a particular product position, a product in the search results, will be purchased. And we will teach our results to be ranked by this score. And let's see how to do this. This, in fact, will greatly improve the quality of search, especially if we know how to look at it. And in order to, to monitor this quality. To take such a step, it is necessary to implement, the simplest way to do this is to implement a DSSM model. Perhaps not the simplest, in fact, but it is a truly lightweight, good approach. It is a lightweight neural network model that allows you to compute the embedding of a query, that is, a numerical representation, a query vector, and can also compute a product vector. And at the same time, for those queries and products on which a purchase occurred, this neural network will try to bring these vectors closer in the vector space, make them such that their cosine similarity is large. And this model was implemented as the next step in the development of Yandex Lavka search. It was able to do this and still can. We use it. Good embeddings of query and response. And the backend developer's task, in fact, including the backend developer's, is to implement the use of this model in the search service. There are actually many interesting things here that can be analyzed and seen how it will work from the backend perspective, but this goes beyond the scope of our lecture. Everything I wanted to tell you, I have told you. Let's summarize some conclusions. We have built a new service and carried it from the idea, from the emergence of the idea to the development of the first MVP architecture. Then, hand-in-hand, we carried this service through the stages of the feature lifecycle by writing code, by testing, by deploying the service to production. And even more so, we saw that this feature is really cool, and compared user behavior in an A/B test. Subsequently, we used the behavior that users showed us in this feature to implement a component in the service's search. Thank you very much for your attention. Let me look at some more questions and try to answer them. So, who is responsible for deployment, the team or is there a separate team for this? Well, look, there is a division of responsibility. There is infrastructure that needs to be prepared. You need to invest, in fact, in making it convenient to roll out releases. This is handled by a separate platform team. That is, it provides some tools that allow you to conveniently, qualitatively, and quickly deliver your product to production. What is required from you is precisely to use these tools, and no one will roll out your service for you, your feature. That is, the developers themselves, who are responsible for the service, roll out releases of their service. What language does your team mainly use? The search service is written in C++. We use Visual Studio Code. In fact, we have many internal indexing tools, but their analogs are also available in open source. We also often use Python, as ML engineers, we write a lot in it. To build data pipelines, SQL is used, roughly speaking, but we have our own SQL, which is called YQL. That is, distributed storage in the form of MR tables, Map Reduce tables. We also use, if we talk specifically about backend development, then it's C++, Visual Studio Code, and now very popular Vipe-coding. So, yes, does Vipe-coding exist at Yandex? That's the next question. It exists, and how. Uh-huh. Let me see what else is interesting. Uh-huh. Another important question. Well, here, perhaps, I'll rephrase it a bit. Is the rollout to production iterative for each service, or do all services get released at once? So, as services are, this is a plus of microservice architecture, its minus is that it is quite complex, it needs to be managed, a good platform is needed to use it. But the plus is that services develop independently. We considered how to ensure backward compatibility in interfaces. We try to maintain this backward compatibility. And therefore, services are rolled out independently. That is, something updated is rolled out in one service, then something updated is rolled out in another service, and then they are combined with these flags. The new function works if it affects several services simultaneously. But yes, in general, services are rolled out by the team as many times as they want and completely independently of other teams. Who in the team defines business metrics? Well, most often business analysts, yes, analysts are involved in this. I don't know, at least in Lavka. Well, and in general, I think this is a characteristic of Yandex. ML engineers are very versatile soldiers, as they say. They sweep, sweep the floor, and fly on a broom. This is how we do everything. That is, both backend developers, data engineers, engineers, generally speaking, this is how it happens. And business metrics also have to be invented sometimes. Uh-huh. Another good question, related to ML. If a service is written in one language, and an ML model in another, in what form is it integrated, as a separate service? Yes, it's quite a sharp question, probably in other companies. At Yandex, it's simpler with this. Most ML services. And in fact, the ML service itself is the same service, but it has some internal logic that essentially needs to perform inference, that is, predict something using your model. And this preparatory stage for inference is actually done by some framework. It's internal, I can't say anything about it, it's not open source, but many typical tasks that are necessary to perform inference are performed within this framework. Well, that is, what is it? It's the preparation of these features, based on which the model makes predictions, their formation, passing them to the model, processing the response. Another important point, in fact, is that the response needs to be logged in a unified format, which can subsequently be used for training, for forming a dataset. All this falls on the shoulders of the framework, which is written in C++. And therefore, among other things, we write ML services in C++. Everything is simple here. Well, okay, we've delved into this a bit, of course, it's a good topic. I like it very much. But there are many more interesting topics ahead of us that await you in the backend development school. I am sure you will find it very interesting. I was very glad to spend the evening with you today. And everyone have a good evening and good luck with your studies. [music] Yes. Tak.