Transcription
So, good evening, colleagues. I am glad to see you at the open lesson of US. The topic of today's meeting sounds ambitious. Trends in stress-resistant design of Composable Architecture. Why do we use stress-resistant? Well, because modern systems, they experience colossal pressure. You see this. Business demands features yesterday. Load grows unpredictably, and now AI needs to be implemented everywhere. Even generative AI, even better. Today we will analyze how to stop building fragile systems and move to a reliable component architecture. And to start with, a technical point, please, write a plus in the chat. This is already a tradition, if I am heard and seen well. It is important that you clearly see the diagrams, as there will be a lot of visual information. Uh-huh. Thank you. So, let's get acquainted. My name is Andrey Nosov. I work as an AI architect at RAFT company. My background is 13 years of system development, with the last few years focused on generative artificial intelligence. I have a PhD degree, I lead a course, and I am an architect at OTUS. My task today is to share practical experience from the field, not just theory from textbooks. The rules are simple. We are here to learn, so active participation is welcome. Write questions in the chat as I talk, ask them during special pauses. And I will constantly keep an eye on the chat. I will try to answer all of them. And even if something remains off-topic, I invite you to my personal chat, where I will definitely answer all your questions. A recording of the webinar will be available, it will be sent to your email, so if you miss something, you can rewatch it. Well, and let's get acquainted with you now. Tell me a little about yourselves in the chat, what is your position, for what purpose did you come. It will help me, actually, to set the right accents, to delve into the code somewhere, to talk more about business processes. Great. We'll wait literally a couple of minutes. Java, Python, everything is fine. So, no analysts today. Well, okay. Uh-huh. Well, let's move on. A few words about the platform we are on. This is OTUS. OTUS is not just IT courses, we specialize in training professionals. We have quite a lot of programs designed for all levels from junior to lead. And the main feature is that we quickly adapt the program to real market demands. Therefore, we teach only what employers require here and now. An important point for your career. We have an educational license, which means that after completing the course, you will receive an official certificate of advanced training. Not just a pretty piece of paper, for many, the chat is also a significant argument. And for yourselves, as confirmation that you could do it. Yes, I can. The directions we have are very diverse, including programming, testing, management, and Data Science. But today our focus is at the intersection of architecture and AI. So, let's return to the topic. Composable architecture as a trend. Why has it become important now? Let's figure it out. Here is our roadmap. We will start with a diagnosis of why current approaches are faltering. Then we will lay the foundation for a new architecture, well, relatively new, because this trend has existed for quite a long time, but is now gaining maximum popularity. We will analyze the methodology, how to cut a monolith, how to do it correctly. We will talk separately about AI, as it is the main driver of change that currently affects almost all industries. We will delve into engineering details and finish, of course, with people and processes who will live with all this. A couple of points, yes, why should you spend your evening on all this? Firstly, from a technical point of view, I want you to learn to see system boundaries a little differently and thereby broaden your horizons. It's not quite a set of tables in a database, but rather independent business blocks, PBC. This is what I will be talking about today. This will save you, first and foremost, of course, from endless coordination of various contracts. Secondly, career-wise. The ability to build anti-fragile systems that don't crash from every sneeze and are ready for AI implementation. This is what commands the highest salaries on the market right now. This is a transition from the role of a firefighter to the role of a strategist. There are analysts. Ekaterina, hello. So, let's start with the prerequisites. Numbers, essentially, do not lie. Let's look at them. On the left graph are data from Dora. What is Dora? Dora is DevOps Research and Assessment. A program that allows you to see where the business is losing, how much is being deployed, and so on. So, until 2021, we see growth. The more often we deploy, the more money the business earns. The dependence is linear. But now, in recent years, we have reached a plateau. Accelerating releases has led to a multiple increase in revenue. Why? The answer is on the right. In a typical enterprise, 60% of the budget goes to so-called integration glue. This is code that sells nothing to the user. Mappers, adapters, endless intermediaries, fitness functions from one format, translation from one format to another. We spend money fighting complexity, on creating convenience, not on creating value as such. There is a bit of a paradox here, creating convenience, not value. Value for the user, convenience for the developer. The whole problem here lies, of course, where? In mathematics. This is the so-called combinatorial explosion of classic microservices. Complexity grows quadratically. It reminds us, in fact, of the very complexity that is inherent in basic transformers. But here it looks a little different. For example, if you have 10 services, you can still keep the connections between them in your head. But if there are 100 or more services, you will ask, where have you seen such systems? In fact, in an enterprise, it's quite a real story when you have 100 or more services, then the number of potential connections goes far beyond 4,000, and, essentially, you definitely can't keep them in your head. What does this lead to? To integration paralysis. To roll out one feature, you will need to coordinate contracts with five teams, at least. Well, for us, for example, it's like that. As a result, we fall into what is called the danger zone. Decomposition will bring us more pain than benefit. And this is direct proof that microservice architecture today is a rather problematic part for business, which does not bring money in the moment, but still requires costs. And what we ultimately build is called a distributed monolith. This is, essentially, an anti-pattern, the worst of both worlds. On the left, we see an example of temporal coupling. You build chains of synchronous calls. Service A calls Service B. B calls C. If Service B fails or slows down, the entire chain goes down. In fact, it strongly resembles basic orchestrations used for AI agents like N8N, Flowwise, and so on. What happens here? What happens here is that in the MVP stage, everything works well because the load is small, but as the load increases, availability multiplies and drops, latency sums up, and the very combinatorial explosion occurs, which leads to the paralysis of the entire system. On the right, you see the same thing, only data coupling. A favorite mistake we make is a shared database for multiple services. You cannot change a table schema without breaking neighbors. As a result, deploying one service requires redeploying all others and blocks development. It's cheap only in the initial stage. Well, I am glad to welcome everyone who has just joined us. We are just approaching the part where we will talk about the so-called theoretical triumvirate. And how to get out of the situation we just described? Essentially, we will rely on three books simultaneously, three principles. I said that the theory will be more practical. It is. The first principle is information hiding, introduced by David Parnas. Here we are talking about the fact that a module should hide complexity, not just hide variables, but hide entire design decisions, so that their change does not break the external world. The second principle is the well-known DDD, Domain Driven Design. The boundary of a service here is determined by language. For example, an order in logistics and an order in accounting. Here they appear as different entities, which is logical, even if they are called the same. This principle was introduced by Eric Evans in his book DDD. And the third principle, immutability, according to Tufte, is the most important principle for distributed systems: data within a service. Well, let's imagine that it's like a person's opinion, it's not constant. The more changeable a person is, the less constant their opinion is. And data outside is facts, events. Facts cannot be changed. This is history. All reliable systems are built on this. And here we have come to talk a little about PBC. What is PBC? PBC is packaged business capability. How does it differ from microservices? Well, let's look at it on a conceptual level. A microservice is often cut technically. That is, an email sending service, a logging service, some technical details. PBC, however, is cut by business. Cart, checkout, search. These are not full-fledged microservices, they are often much broader than microservices, but are packaged along business boundaries. The main difference of PBC is that it is completely autonomous. It needs to be maintained autonomously. It owns its data. No shared databases, communicates through business events. A test of maturity that you can conduct is simply to throw away your PBC and replace it with some SaaS. And all this is possible, perhaps, even in one sprint. If you succeed, yes, you have done everything correctly in designing your architecture, at some stage objections will arise, yes, perhaps it will be an initial overhead, but let's discuss it. We have a pause for this. Are there any questions about the problem, the difference between PBC and microservices? Write in the chat. Let's see what we have. So, hello everyone. Once again, the first question. Will there be practice in the webinar? Rather, today there will be a discussion. That is, we will not be coding as such, but the practice will be more quiz-question oriented. So, no questions? Let's move on then. Let's move on to how all this works under the hood. The main law that reigns here in composable is no remote reads. Remember, we never make synchronous GET requests to other services to get data for work. Why? Because the network is unreliable. Instead, we use the local projection approach. If the orders service, for example, which we mentioned earlier, needs user data, it will not access the user service, it subscribes to the user updated event stream through, say, Kafka. And Kafka will appear often today. And saves the necessary data in its local database. Does everyone understand where the entire composable architecture came from? If there were no Kafka, most likely, this direction would not have arisen. Today, it is the bus that supports the entire approach. Data is duplicated, but orders, but the orders service, yes, it can work even if the users service is completely down. This provides that very autonomy. Within the PBC itself, within this capability, we use an architecture that can be called an onion architecture. To protect business logic, we use such an architecture. The first layer is ingress. This is the adapter layer. It accepts HTTP, validates JSONs. We don't put business logic here. The second is the core. This is where the pure logic is. Decisions are made here. This layer knows nothing about databases, about data, how they look, in what formats, and so on. Kafka is located here. The next layer is state and outbox. When the domain has made a decision, we save the state to the database and simultaneously write events to a special Outbox table. This is done in one transaction. And the last layer of our onion is egress. Output. A separate process. It will read the Outbox table and send events to the broker. This guarantees that we will not lose messages. The pattern usually used here is called transactional outbox. So, well, now, let's see, if there are any questions, what is Outbox or what is BBC in application? Well, let's see, don't hesitate to ask. So, who is the father? PVC is a type of microservice. Pat transactional. PBC is not a type of microservice. Imagine you have a search service. Is it a microservice or a set of microservices that, say, in a Docker Compose, probably the second, not the first. So, is it the only approach against failure? Well, let's talk about this first, yes? So, look, the key difference of PBC is that it is a business function. That is, there is an analyst who brings some value, some service that is necessary for business to use at different stages. Such services can be both small and large. So, at the development level, PBC can be, for example, monitoring services. That is, it's not a single, say, stack, although the stack is known, yes, Prometheus Grafana, it is also a kind of PPC. It can be expanded a bit wider, yes, by using an Open Telemetry collector, LCKY, and so on on top. And at the same time, the PBC will grow autonomously. You can make a PBC of another order. For example, if you need to create the same search service, telephony service, some other service, which will consist of several microservices, essentially, this is already a ready-made, full-fledged, large development. The question arises, what is the difference between PBC and, well, tell me, from the solution we are creating? Here, the difference is practically zero, because in good practice, a full-fledged solution, there is probably still a difference. Look, a full-fledged solution is made and delivered, and PBC is more like components for creating a unified platform that will keep your user engaged on a permanent basis. That is, PBCs are like LEGO bricks that you buy again and again, and they can be assembled into even larger corporate standards, yes. Is there only one approach against failures, or are there others? There are also approaches related to orchestration in CI, so-called, when we use a monolithic, if we take a system, then a monolithic model, which can perform many actions simultaneously. Such an approach still exists, although it sometimes fails. Next, the second point. So, the world is divided into two opposing camps. Is everything divided into particles or everything assembled into one? Essentially, it comes from entropy, yes? And these are the two main approaches: create one huge monolith that works, is fully fault-tolerant. It's easy to maintain. It's always in a predictable state, scales, for example, either vertically or horizontally. The second approach is probably more characteristic of small businesses, and also for medium ones, when we don't know how this world will develop, and we don't invest in something big right away, but do it in parts, bit by bit, we have a chance for great flexibility. These are always scales. Corporations usually choose the first approach, and small and medium businesses choose PBC, because it allows them to be competitive. Module-component, module-component pu-pu-pu. Well, yes, all this can be called that, but it's called, actually, I often confuse PBC with component, and it's a capability. And capability. Here, probably, the most correct word that can be found is capability, because it is precisely these capabilities that allow our business to develop. How to solve the PBC problem if the domain area is not fully known or incorrectly defined during analysis or design. In that case, look, there is, again, let me give an example that I just gave. There is an abstract PBC that solves all problems. For example, a PBC pu-pu-pu based on a fitness function like a gateway, yes. It is clear that this function will be needed almost always. That is, it supports load balancing, is related to role models, and ensures security. Such a universal soldier, yes. And we understand that in our solution it will be accurate. We don't understand what logic it will cover. In this case, your question is probably more about what is related to core logic and common functions. Common functions are quite easy to isolate. The most important thing is to have an understanding of what we will do and to what extent. And as for the core logic itself, it relates to how we have defined, how correctly we have defined our own development strategy. For example, we are a company that deals with integrations, so it's obvious that we will most likely need to focus on integration logic. Or we are a company that deals with chats. It's obvious that we need to focus on developing some state machine that will allow us to manage context and dialogue. And so on. How does PBC differ from a distributed monolith? In the same way. That is, a distributed monolith is a set of microservices, and PBC is capabilities that can themselves consist of several microservices. About deadlines burning, analysis not completed, something needs to be handed over for work, but there's nothing. Oh, that's a good approach, but it's a bit off to the side. This is a question about how you initially plan your development. For example, you can plan it in stages and, at each stage, accordingly, deliver some artifact. This is usually the favorite method of clients. Well. The second point, if you have a product with a very large area of uncertainty, then in this case, you can simply take a pre-project and not take on the risk that your product will not be made. Such cases happen often, actually. For example, we are doing something that our developers have never done before. That is, there was no path, no experience. Naturally, there is a high chance that it might not be done. This is where the pre-project stage exists. Yes, no, Anya, it's not that it's measured by microservices. We've been delayed too much on this pause. Let's move on quietly. I'll answer the last question and all the others a little later. It's not related to the size, whether it's one microservice or several. Capability relates to whether it performs its business function successfully or not. That's the main focus. It can even be created from one microservice. For example, you just made a PVC based on ingress. You'll have one there, all of it is one microservice, but it performs all the business functions that you need. Or you've made something complex, like telephony, where 14 microservices are involved, and you combine them all into a single, not even a solution, but a reusable thing. Another main difference is a reusable block. That is, you can connect it, disconnect it, your entire system doesn't crash, you can transfer it to any other systems. It's absolutely independent. Well, let's move on. The next question we'll look at is, for example, we have, we've just come to these practical parts. We have a huge monolith. How do we correctly cut it into these PBCs? Yes. Well, let's look at the methodologies that architects generally use. Here you can use the first tool, static. This is a capability map. That is, we draw a map of what the business does. For example, level one e-commerce, level two sales, warehouse, support, level three checkout, stock management. Well, these are the leaves of the tree, yes, the tree. And there are candidates that we immediately see as PBCs. The second approach is dynamic, called event storming. We look for boundaries in time, we look for so-called pivotal events. For example, an event like ordered. Before this event, it was just a cart, you could delete, change, do other things. Afterward, it's a legal document, an order. And this event essentially changes the status of the data and the owner of the process. So, here is the boundary between services. Thus, these two approaches define how to cut by these capabilities. Again, you can do it by mapping. That is, we take and cut by useful functions. The second is change over time, that is, how our service will change over time and by doing so, whether we can increase it or grow it independently, so that it grows in the same direction. Well. Now let's look at how to technically cut out a piece of the basic monolith. There are three strategies. The most common, probably, but from my practice, again, I'll preface this, is the strangler fig. This approach is also the safest. That is, we place a proxy like a gateway in front of the monolith. We redirect new requests to the new service, the old one stays in the monolith. Gradually, quietly rewriting, we strangle this monolith and get an independent service. The second is a rather crude method, called tactical forking. Here we copy the monolith code into a new repository and delete everything unnecessary. Fast, but dirty. And the CDC change data capture approach. If touching the code is scary, which often happens, then we can connect to the monolith's database logs, for example, through Debezium, and stream changes to Kafka, which I've already mentioned today. What happens here is that the new service builds its state based on this data, without loading the monolith's database. These are three approaches that we can use to cut out. Now let's move on. In a distributed system, if you have bad contracts, that is, relationships between services, essentially, you're done. As crude as it sounds. Contracts are everything to us. For synchronous calls, we are currently moving to the Open API standard. It is used almost everywhere, especially in the field of generative artificial intelligence integration. It is extremely convenient and universal. But there are still, of course, artifacts where it's not used, and there you'll have to struggle. For events, of course, we use asynchronous communication. Not all services, again, are subject to it, and you have to write additional functions that allow services to function. But the most important thing is the protection mechanism that we apply with contracts. Consumer-driven contracts. Such a mechanism can be, for example, the Pact mechanism. How does it work? Let's see. The consumer writes tests that say, for example, I expect the ID field in string format. The provider is obliged to run these tests in their pipeline before deployment. If they change ID to a number, the pipeline fails. This is ironclad protection against integration failures in production. Not the only one, but one of them. And now let's move on to the hottest topic. AI. Many are now trying to integrate it into their business processes. We will integrate it into the architecture. LLMs are Large Language Models. The main mistake that, of course, pursues everyone here is the so-called divine prompt. That is, developers try to cram all the context, docs, database, data, history, and all this into one request to ChatGPT. Plus, if you have the all-knowing model, which I already mentioned, everything is also streamed to agents. That is, you have one large model with such a super-complicated prompt. Of course, this is a dead end. Why? Because the feature space, it works much better on probabilities for atomic tasks than for distributed ones. Here arises the famous "lost in the middle" effect, which was described in one of the articles published by Apple in June of this month. It describes approximately the following effect. It describes: the model remembers the beginning and end of the prompt well, but forgets the middle. Imagine, once, when I was studying, they told us that people think approximately like this. That is, they always think and hear and see only the beginning and the end. And everything else in between, they fill in and generate from here all cognitive errors and misunderstandings. In reality, this is not the case. But with the model, it is. And consequently, this loss of data leads, of course, to hallucinations. Secondly, the cost grows quadratically here. And it grows quadratically with the length of the context, as we know. Transformers love this, so it hits a serious dead end in terms of computational power. And thirdly, not that it's the least significant, but debugging such all-knowing divine prompts is impossible. Well, essentially, slightly tweaking one thing leads to a shifted error in another. And this will be constant. If someone has ever configured frameworks like Rasa, they have definitely encountered the fact that after the hundredth intent, it starts to produce a lot of artifacts. And here it's approximately the same story. Why did I remember? A client recently came from the past and said that he needed to configure this Rasa, when you can already use much more informative, new, and convenient systems. Well, let's go through your questions now. So, and so you understand, the only thing is that a business function can be the creation of one report, which under the hood has a lot of ETL, mathematics, some visualization. The entire process involves hundreds of entities, and at the end, only one report generates something. It's all true, but if, look, Vlad, if ETL can essentially also be a separate PBC, that is, when we design a service, it can exist without an ETL process. That is, what does the ETL process lead to? To the fact that we can visually see certain functions. And if we design it so that our business function is precisely, well, ETL is unlikely, of course, it can be hidden. ETL always comes with some visualization, most likely. So, it relates more not to
Here is the translation of the provided Russian text into English, following your rules:
The text, and to the front, rather. And in this case, it is possible, of course, to distribute them and distribute the logic correctly within the core system. So, Alexander, in your opinion, how advisable is it to focus on AI? There are many gaps, and no matter how much they try to push. But since neural networks are not developed, it results in the use of raw technology with all the ensuing consequences. Neural networks are not a raw technology, Alexander. They are as they are. That is, it is a probabilistic technology. So, accordingly, we can solve many tasks probabilistically better than with an immutable approach, right? And what does this mean? Pushing them everywhere, of course, is probably a trend now, but not the most correct task. But further on, we will look at how they can be packaged to truly bring benefit. Looking ahead, I will say atomically, because with an atomic approach, there is much less possibility of making a mistake and a better opportunity to localize it than with the approach of huge, voluminous, all-encompassing models. So. And if I have answered your question or not, well, put a plus sign somehow. Like this. Understood correctly? Long prompts are not needed, one short one is better, then the result of the first prompt. Insert the next prompt. And so, with several simple prompts, we replace one complex one. But this is the method you proposed, actually. But you will be feeding the models the full context. It is much better and more convenient to build models on the multi-agent approach that currently exists, where each agent, say, has its own responsibility and, of course, its own prompt. What makes it more stable? That is, it solves its own task, and as an event, well, I'm slightly jumping ahead, as an event it passes it to the orchestration system, and the orchestration system then combines all these events, getting a general answer. So, with the method you proposed, collecting the same bag, but with small bags might not quite work. Uh, but usually, this is how it's done. Usually, this is called the approach with variables. That is, each variable contains some sub-prompt or a set of data. And up to a certain level of complexity, it works, but it still leads to a combinatorial explosion in the end. Like this. Well, let's move on. Now let's see how, actually, the questions about AI have started. Let's see how to do them correctly. Native architecture. I propose to consider an AI agent. Each agent is, in fact, a capability, a business capability. It has a clear role, its own clear, small system prompt, which covers its small function. It has its own memory, its own RAM, nothing from the outside intrudes, and it has its own tools, that is, AI tools, SQL, and so on. For an external system, the agent looks like a black box with a deterministic interface. That is, on input, say, a JSON or a request falls with a task. On output, a JSON with an answer goes out. All the magic and uncertainty is hidden inside the agent's capsule. But look again, this capsule becomes granular, atomic. This means that if an agent performed some micro-small task, we can, well, not suffer too much in the end. If this granular task is not performed, it will also be performed, because it is easy enough to replace. That is, the more atomic the task, the lighter it weighs, the more instances can be hung on top, so that, figuratively speaking, another soldier quickly replaces one. This is what I said, often one agent is not enough, right, functions are much more extensive. Here I have given functions, well, very little. Usually, now they experiment with agents up to 8-9. The last experiment was. 89, figuratively speaking, specialists solve a certain path of tasks and do it successfully. Of course, the orchestration looks quite interesting, although we will get to that too. I will say that you won't have to track much there. So, let's stop at the fact that one agent is sometimes not enough for us to complete a full task. And we build so-called swarms. The first swarms, multi-agent ones, appeared again based on simple orchestration. That is, we give a linear call, and it goes to parallel processing, passes through each node, through each node, and in the end returns an answer. What is dangerous about this? Linear orchestration control is that any fallen node drops the entire pipeline. Well, someone couldn't handle their task among the agents, and, accordingly, the entire task is ruined, and we can start it again. At best, retry logic, and at worst, we simply send a response that the task was not completed. As for multi-agency, the basis here is that it is very important to perform some tasks in parallel. That's one. And two, it is very important to perform tasks independently so that we don't have this domino effect. And let's see how this can look in a standard architecture. That is, we have a so-called router agent. This is no longer an orchestrator, it is an agent that contains a set of events. What does it essentially do? It classifies the task. It has a small classifier inside. When a task falls into it, it selects an expert to send it to. Or it selects a sequence of experts to send it to. That is, they can be in order. It sends impulses, right, for example, an event, the event is sent, the expert is activated, gives him an answer, only then does it pass the task to the next agent. Or it can blast to everyone in parallel and wait for an answer from each. The second, third, fourth are the layer of our workers, which each contains an atomic prompt, which is easy to maintain. You see, here we no longer execute just a request to the LLM, right, and we think: "Uh-huh, excellent, we've overloaded the context at some point, with our super-tasks, and the model either returned nothing or hallucinated. This happens much less here, because it is quite difficult to overload the context with small atomic tasks. It's possible, but difficult. And here we have three different agents. one, for example, classifies the task, one writes the solution, one creates code, one conducts research. And what's more, they even have a critic agent on top of them, who reviews everything they've done. I've drawn it here, right, and if it doesn't like something, right, it also sends it for retries before sending the completed event to the router agent. Here, the cycles we build between them are very important. That's why you asked, right, at the very beginning, why do we use AI systems? And systems, despite their probabilistic nature, sometimes the probability contains a much larger percentage of success than real action. For example, statistically, we can perform actions successfully in 40% of cases. And with the help of various probabilistic models, we can achieve 80%, that is, twice as much. And this happens, in part, due to such a mechanism as cycles. That is, the model rarely produces ideal code the first time. This is a known variant, but it corrects its errors excellently. That is, we can do this with small retries, small additions, and so on. That is, this is also regulated at the prompt level. That is, if an event of unsuccessful execution comes in the first time, or if, for example, the critic says that it's all bad here, then we can activate the second stage in the prompt and have the task decomposed in a different way with it. As we remember, there are many ways to decompose the task itself, and they are applied as prompt engineering. This is how PBC, each of the agents here is, in fact, a PBC. And each of the agents can be extracted as a separate block and reused in other solutions. This is how PBC can essentially exist and build entire systems like LEGO. That is, someone can be removed from the swarm, someone can be added, instances of the same researcher can be added if we understand that he is still a bit slow. Well, that is, well, figuratively speaking, he is not slow, but his model is weak. And so on. Moreover, we can give each of the agents its own model. That is, here they go into orchestration, that is, each communicates with one model, but we can not spend money, hang local models on each, and that's it. It will also be quite cheap, because not every agent will work every time. This means we will save on electricity, at least. Well, in general, on money, of course. But let's move on to production. That is, we are talking about PBC. In AI production, we need to control. And here there are also its own patterns that exist in PBC. The first of them is a security sandwich. That is, an incoming request from the user first passes through guardrails. We remove any personal data, check for injections, which is important. A new term, right, input injection. Injection is when they try to hack your system, for example, if they want to somehow extract data from it and make it hallucinate. That's why input guardrail is a must-have today. And by the way, one of the PBCs that we use in our company is called High Race. This is essentially a component that allows us to guess all these injections in time. By what means? By mathematics, because first large networks are hacked by red teaming, and then all conclusions from this mathematics are put into guardrails and used as this PBC block, which allows cleaning both personal data and checking for injections. Without this, there's no going anywhere now, because everyone likes to play with neural networks. So, what happens next with the incoming request? After guardrails, it goes to the model. The model's response passes through another layer of Output Guardrail. Here, in the output, we check not so much for an attempt to hack the language model, but for the quality of the output format. That is, here, frameworks like W, Ragas, and so on are usually used at the output, which allow measuring metrics of evolution, faithfulness, then metrics of relevance, contextual relevance. We no longer need the answer to be just relevant, it must also be maintained within the context of our request. And, of course, formats. That is, if we asked for JSON, then please give us JSON. If this doesn't happen, then either the library or the guardrail triggers. And one more point that is very important in such systems is the so-called T strategy. It is usually used for savings. And the point here is that we don't use large, huge models for everything. Although, again, as I mentioned, there are two approaches and two extremes. Either we make the world highly atomic, or we make the model all-encompassing. Yes, our large banks today use the all-encompassing model for all cases. Confirmation, for example, Yandex, Gigachat, and so on. This is also an approach, but it is much more difficult, of course, for small and medium businesses to maintain, because, well, it is practically impossible to control its output. And the approach of small and medium businesses is more about solving tasks with some local models, gems, mini-lamas, queens, which weigh 0.6 billion, 0.6 billion, and so on. Well, everything that today, up to 12 billion, fits easily on proprietary gaming cards. So, what are large models actually used for here? Large models are mainly used for reasoning. Reasoning is not only useful but also long and expensive. Therefore, here, firstly, a large model. Why is it also long? Because, if you've noticed, right, they start to output this thing, how it thinks, how the model thinks. Usually, this is just a marketing mechanism that simply eats up your latency. And for most tasks, it's not needed. Well, that is, figuratively speaking, if you call by phone, you won't even see this reasoning, and you don't actually need it when you communicate. With this atomic approach, you can save up to 60% of your budget. So, we've come to the questions again. So, look, Dmitry, I specifically didn't translate some terms because they don't exist yet in the Russian reality, so that you can calmly find them in the sources, in the primary sources, as they sound. Only for this purpose. So that you have access to information. What solution is used for input guardrails. Well, as a rule, it's a regular mathematical model that checks the input, or a set of regular expressions can also be used, but this is a very basic level, which allows calculating, for example, a phone number, or a full name, and so on. But at a basic level, when it's necessary to determine compliance, mathematics comes into play. That is, for example, it is necessary to determine how close this request is to a request that breaks the model's feature space. This is a very complex task, actually. Much more complex than modern RAG systems and so on, requiring greater accuracy. Imagine, right? And you can't afford to make mistakes on guardrails, otherwise your model will be stifled and will stop responding adequately. Which, by the way, happens very often, well, I won't name Yandex GPT, which has heavily suppressed ethics, meaning any of your requests can easily fall into this ethics. How costly is the agent approach? It is as costly as your business develops. That is, what is convenient about this approach? You invest more if your business is larger, if your business is small, then you invest a little at first. That is, each agent grows depending on the load. And, as a rule, its problem lies in two cost categories. The first cost category is TCO. That is, you need to scale the infrastructure for this agent under load. That is, your load increases, the agent's infrastructure grows. The second point is related to the fact that, well, AI systems today, the market is quietly, actually, moving onto these rails, when the realization comes that under AI systems, infrastructure, and human infrastructure of specialists who will keep their finger on the pulse. And, accordingly, the more agents you have, the more people you will need who will support them in accounting. Well, what does it mean to support? It means simply collecting certain logs. That is, the infrastructure is also supported, monitored by monitoring tools and so on, when the infrastructure, for example, goes beyond its limits. It's the same story with agents. That is, when an agent lacks capacity or feature space, a specialist is needed for intervention, which will allow expanding the agent's feature space so that it can live on. That is, these are the two cost categories. The first is infrastructure, developing in terms of hardware, and the second is infrastructure, developing in terms of enterprise infrastructure, which you are growing. And the larger the enterprise, the more infrastructure. Smart router. What is used as it? Various systems are actually used as it, but usually it's just a classifier. That is, you attach a classifier, the class of the request is determined, and this request is immediately routed according to logic, or not immediately. That is, you set the logic yourself in what sequence it is routed. Usually, such classifiers exist on LangGraph. That is, you write in LangGraph, the control logic, there are quite a few tools. LangGraph is a library for writing agent control logic. Is the recognition of how the model thinks purely marketing? Honestly, yes. There are several articles confirming this, which have been published. For example, the well-known Apple, in June-July, published several articles and showed that pure statistics, it doesn't care, it doesn't think, it just outputs maximum similarities, and that's it. And it showed this very beautifully using the example of DeepSeek. Therefore, it is so, yes, a classifier is NLU. Well, including Natural Language Understanding in this case, it can be, yes? That is, it can be just a classifier built on regular expressions. It can be any classifier that allows you to understand whether the request relates to the system or not. And, look, not only NLU, but also NLG, by the way, because our generative model can also classify. Well, that is, give it a request, for example, as input. Please tell me what this is. It will accordingly give you two classes, this or that. So, there are also many approaches to classification. So, well, let's move on. We've finally gotten to the engineering things. I won't say it's hardcore for engineers, but there are moments that will also be useful for you. How does all this develop and work, how can all these PBC functions exist? Actually, there is also mathematics here. Let's start with a theorem called CP. The CAP theorem is the basis, but in reality, its counterpart, the CAP theorem, is used. Let's decipher it. It says that if the network is partitioned, you choose between availability and consistency. In a composable approach, in a composable architecture, specifically in PBC, right, we always choose the advantage related to availability, that is, availability. The system must respond to the user, even if the data is slightly outdated, even if it slightly deviates. But pay attention, right, if it doesn't respond at all, you start pressing the button hard at best, right? But if it responds at least with something, and even jokes about it, for example, you can quite continue to perform this task and even possibly solve it, but a little later. If our network is in order, then latency comes second. That is, we can tolerate latency for a long time, right? That is, a person waits, can wait for an answer, but, say, can even sit down like a cat and wait. But in reality, of course, attention laws say that a delay of more than 3 seconds leads to attention scattering and loss of interest. And finally, of course, we choose eventual consistency. That is, consistency in the end. That is, the system is first available, then we get an answer with a small delay. And in third place, we get this answer consistently. Such research related to user behavior led to the emergence of this approach. PELS is an inverted CAP. How to live without transactions? Distributed transactions 2PC should not exist in microservices. They are simply not scalable. And usually, so-called sagas are used for this. This is also an approach that is implemented in BPC. Imagine, well, how it works. A chain, for example, an order is created, payment is made. The goods are reserved. If an error occurs at the stage of reserving goods, well, there are none in stock, we cannot roll back the payment in the bank's database. We launch a compensating transaction, a refund, and this is a semantic cancellation of the action. The most important requirement here is idempotence. If a message about returning money comes twice, we should return the money only once. But this is logical, right? This is another principle that is embedded in PBC, the principle of idempotence. And one more thing, how to debug all this. Regular logs won't help us when we have 50 services and Kafka. We need a so-called distributed tracing approach. In OpenTelemetry, which I mentioned today, we will not just pass traces, trace IDs, we will build a whole directed acyclic graph. Each event here will know its parent ID, and this will allow us to answer not just the question of where something failed, but the question of why we even reached this point. We see the entire history of the user journey. And thus, the investigation becomes successful. Let's look at all these approaches using the example of industrial IoT. For example, we have 1,000 sensors, and each sends, say, another thousand events. We have 100,000 events per second. If we used stateless REST plus PostgreSQL here, then at 100,000 events, it would most likely die regularly, because the database would simply choke from writes and locks. What can be done here? We can switch to composable streaming, the very same PBC. PBC here will look like Kafka plus Flink. Fortunately, these are both systems belonging to Apache, Apache Flink and Apache Kafka. The idea is that computation and data live together here, in RAM, as colocation. There are no network approaches to the database. The state is stored in Flink locally and asynchronously flushed to disk. The result is practically no delay. The database is no longer a bottleneck at all. The system scales linearly. This was a complex block. Sagas, like the theorem, tracing. Let's look at the questions now. Sagas, of course, are an old approach, but today it has been adopted. Why? Simply because it allows solving its main problems related to distribution, and according to the indicators of distribution itself and the history of this distribution. Without them, it would be quite difficult to restore the chronology of events, it would be point-based. Precisely because of this. So, well, we are approaching another part. Why implement and do all these things? So that they are useful, first of all, to yourselves, so that it is comfortable for you to work and for the people for whom you are doing this. Because, looking ahead, if we grow our AI into an artificial intelligence system, replacing these package business values, then at some point we risk assembling not just one character from these small cubes, like in LEGO, but a whole city that will help whom? People. and organizations. That is, essentially, PBCs are designed to assemble, reassemble the idea of enterprise AI as a whole today. This is their global vision. Platforms have not coped with this. Platform solutions are now fading into oblivion. There are millions of them, if you look, and they all have one small drawback. A platform solution, as soon as it appears, immediately falls into technical debt. That is, technical debt grows immediately with the platform. It's not so linear with BBC, and that's why great hopes are placed on them in building enterprise AI. Where does all this come from? A little history. There is such a law, Conway's Law. The architecture of a system. It states that the architecture of a system repeats the structure of communication in a company. If your development department, frontend, DBA sit separately, you will get a monolith with a lot of layers. Applying the inverse maneuver, it is called inverse convey man. Do you want a modular architecture? First, create modular teams. A Stream Aline Team is a team that owns a PBC, that is, the entire PBC, from the database to the frontend. They have full autonomy. And to avoid going crazy from complexity, there is such a Platform Team. They create this platform, it's like 8S, CCD as a service, and many others. Thus, a whole system emerges that is based on understandable blocks, business blocks of management and business growth as a whole. What does the Platform Team Platform Hill do? It creates so-called, well, another term, TVP. This is a signable platform, that is, a platform as a product. This is not quite a platform in the modern sense, right, when we just put a bunch of microservices on it and shout loudly: "Live." No. Here, the point is to extract these cubes and reduce cognitive load. That is, a business developer should not write Helm charts from scratch. It works like this for us. A developer writes console commands, for example, something like `raf new PVC name checkout`, and after 4 seconds, they have a repository ready, a pipeline configured, monitoring connected, basic code, and they immediately start writing business logic. This is what we, for example, strive for at the most basic stages, in practice. By transferring this logic to all other departments, stages related to presale arise, that is, automation obtained from a phone call and decomposed into Jira. Why is this for business, essentially? Well, of course, it's return on investment. That is, at some point, everyone is already realizing that artificial intelligence is not a toy that can be put away and then thrown away, but everyone starts expecting investments from it. And for investments to happen, it is necessary to form these atomic things, which will have value and cost. That is, even if you have a complete solution, well, you are such a brilliant Archimedes, right, but it doesn't work the first time, then the valuable blocks that remain after this solution will easily form the basis of other solutions. For example, wheels remained, some kind of catapult launcher, a couple of ropes, and some other Archimedes will assemble a complete throwing weapon from this, for example.
Here. And what does this lead to? Returning to the very beginning, when we measured the DORA, yes, with you, it will also continue to grow as soon as our architecture evolves a little. This board, which existed on the example of the culture of platforms and microservices, will soon crawl upwards quite quickly. With the application of the PBC architecture, the Composable architecture allows, ah, why exactly will it crawl upwards? Precisely because your features will be independent, and solutions for them will be reassembled much faster, and most importantly, due to isolation, this will happen without failures. That is, any new services that, for example, will not fit into the system for some reasons, even cognitive ones, will not bring down the entire system. Well, that is, it didn't take root and it didn't take root, it will just fall off and that's it. The system will continue to work. The business will continue to make money even with partial failure. Here. And how to get there? Well, look, I sat here, thought, there is such an approximate plan, a migration plan. I'm not calling anyone to do it, yes, but I'm trying to do it in my company. That is, no code, analysis, event storming for the first 30 days. Ah, we are looking for boundaries, we are choosing victims for the pilot. We chose several. Then, for the next 60 days, we raise the platform on TVP, we write the first PVC. written. Ah, we use the Pattern Strand Fig. For something, we have already used it, to strangle the old monitoring and raise a new one on its basis. And then we scale, implement agents, configure governance, and so on. Here is our tech stack, which we have already tested in battle, tested on large projects, these are cubes, Kafka. Go Python. Well, Go is more, in fact, as a transport graph as a scheduler. Today the word flew out all evening. That is, the opposite of an orchestrator is a scheduler. Orchestrator is sequential control. A scheduler is one that asynchronously plans events and distributes them. Here, as a scheduler, we use a graph. Alin for the state machine. This is very important for understanding states, for contextual control, for context management, and Open Telemetry so that we can always understand and build those very sagas, about which a question was asked today. Here. Let's summarize in the form of a manifesto of simplicity. Said loudly, borrowed from Taleb. We define boundaries here by business, not by technology. This is a difference from microservices. A question was asked today, the answer is here. Next. Events are the main way of communication. This is another difference. We don't go into other people's databases synchronously. Composition is better than integration, that is, we assemble from blocks. And the goal of the architect here, above all, is a system that evolves, not degrades over time, accumulating huge technical debt from the very start. Here. Well, and now a couple more questions, yes, and we'll move on. There is only one problem with transitioning to PBC. It is very difficult to explain to companies that the platform implemented in them already has this technical debt, and it is very, very difficult to bypass customer requirements to use the platform. All true, right? There are such problems. Ah, but when, for example, you receive a bill for your infrastructure regularly, you can refer to it and say that let's at least start by looking, conducting an analysis, and how our infrastructure is actually used and whether it is used for its intended purpose, and how much we spend, yes, on PCO, and how much money we get back from these services? This will be the first step. That is, you can always start with this analysis. So, I am very confused by the large number of controversial slogans and conclusions, in which Russian company is this being implemented, will be implemented. I cannot actually name the companies because they are under NDA, but being, in fact, working in the company Integratori, we also use this approach for implementing the AI Enterprise approach. These are not small companies. Here. Ah, if you didn't come from the beginning of the seminar, perhaps you mentioned at the beginning where all this works. Well, in fact, in the near future, it will be implemented in many places, at least at our enterprise, we are currently implementing it in three companies. This approach. These companies, two Russian, one foreign, and each of them is multinational, meaning they are huge. This approach has been adopted by companies precisely because it allowed budgets to be formed based on business functions. Everything as we like. That is, not to do phasing and conduct demos, spending huge amounts of money, but to demonstrate precisely these highlighted blocks and then further implement these blocks. Or instead of the services used, use this fig pattern, or as new opportunities for business. Here. Unfortunately, yes, I cannot disclose the names of these companies yet. Well, let's move on. Orchestrators, in what sense? In linear control. If so, then yes, you didn't see. That's right. Well, we haven't finished yet, but you can already fill out the survey request. Was it interesting for you? If not, then it's also worth writing about, because feedback makes us stronger and allows us to make things more useful and interesting. Here. But while you're doing that, let's learn a little more about the course that starts very soon for architects. There we will analyze not just how to build systems in various variations, but also various process components related to building a sustainable enterprise, a sustainable architecture, processes related to managing this architecture, architecture as code, and many others. What is it? And I, and the architect, and who is this person? At the very beginning, when I said that there is a need to manage enterprises in a new way, to make them as efficient as possible, then the architect is the person who allows enterprises to use AI to make money, not to invest in some illusory future. That is, the AI architect is the one who will tell you in time what infrastructure you need, whether you should buy infrastructure at all at the start, for example, or just use APIs. Will suggest how to build the solution architecture itself. Will it be event-driven architecture or will it be a sequence of connections with orchestration, for example, will it be the implementation of ready-made blocks created on OpenС? and up to what stage it will exist, and how it will then evolve, and so on. That is, all these moments are related to architectural patterns that allow us to harness current solutions and will be present at the AI Architect course. That is, it starts on December 24th, and the duration will be 4 months. The course will be hardcore, because a lot will be related to practice, including coding. That is, there will be quite interesting patterns that will need to be solved both during the lesson and beyond. In general, the course is not about just playing with ChatGPT. The program is quite dense, so we cover everything from architecture planning to security and infrastructure. We will cover advanced patterns. Today we only discussed one. Well, and of course, there will be a project work. Our teachers are all practitioners, you see, right? That is, either managers or people working with ML systems. That is, and, accordingly, the lessons will be distributed in such a way that where it is necessary to work with a model, you will work with an ML engineer, with a practitioner, where it is necessary with architectural patterns, you will work with lead architects, with process managers, accordingly, you will work with networks, and so on. Here. All these people, including myself, solve these problems every day at work and try to make enterprises as profitable as possible. Not just profitable, but maximally profitable in the current market. How is the integration of solutions based on BBC with other information systems that are outside the responsibility of the current department's solution carried out? How does it look? These are traditional contracts, of course, at first. Ah, that is, the integration layer, look, there are several approaches, that is, either it is written from scratch, and in this case, of course, it is preferable to switch to a bus. Although, wait, it all depends on the size of the implementation. If the size is global and the contract is large, this is why I'm slightly shifting the focus, because we still have large enterprises, then the implementation is carried out through an event bus. Of course, it's convenient. And it repeats, replicates what we can build independent systems. The second method of implementation is simply transferring already developed solutions in two or even three ways. You can transfer them as components, you can transfer them as swarms, or you can transfer them with deployment in the customer's infrastructure using cubes, for example, or cut-down K3S cubes. Here. That is, there are many ways to interact with the customer. Starting from hardcore, when images were simply transferred on a flash drive or uploaded to DockerHub, ending with more convenient and advanced and trusting relationships in terms of a bus, for example, building a joint bus, interaction. Well, it differs little from traditional ones. Answering your question. Well, that's it. By the way, a couple of points, yes, related to networking, various materials, and so on, they will also remain with you forever. I already mentioned the start. Well, there are still a few questions. It's not about interaction with the customer, but with his other parts. But the customer's information system, look, as a rule, the customer orders individual modules, that is, he can, of course, order management modules for his information system, but, as a rule, the interaction itself is already structured in such a way that he tries to, well, based on my practice, he doesn't allow his system to be rebuilt entirely. This is critical for business, because business is actually tied to the system that currently exists, and he cannot demolish it at the moment. Here. And if he can, then the fig tree pattern is a must-have. We quietly sit and strangle its services, replacing them with ours. Here is the subject area of your projects, solutions, retail, investments, banks. Do I understand correctly that the manufacturing industry is not present? So, there is repair. We have, in fact, one of the large solutions is repair databases, which are related to the fact that many people who have expertise in building production processes are retiring, and, accordingly, their experience leaves with them. The task is to provide support with agents that will keep the enterprise afloat, even if these people suddenly leave. Here. And other supporting processes related to the fact that expertise is sometimes lost. For example, figuratively speaking, people create drawings of huge rockets, airplanes, and other things, right? To keep this expertise in proper order, it is also necessary to build a process. The agent here controls, and the quality processes of the drawings themselves, and then the process of delivery of the drawings themselves, the process of productivity, and so on. That is, well, we have two out of three projects, the ones you just named. The third project is more tied to the back office. These are among those where PBC is currently being implemented. We have many more projects, of course. No, in the area of your projects, there. No one will allow you to rebuild the system there. Uh-huh. Everything is so. Well, they will allow you to strangle the system, because everyone understands that the process is becoming inefficient. You cannot shut it down at the moment, because, well, you cannot completely cut off the money flow either, but you can strangle it to replace it with a more efficient one that will bring money in six months. Who will then support these agents? Well, we are usually honest with our customers, we come and say that, guys, by buying a new solution, you are also buying infrastructure. That is, figuratively speaking, you bought a car. If you don't drive it, don't be surprised that it won't drive. What knowledge is needed to take your course? You know, that is, at the moment, probably, first, the desire to take the course. Second, of course, some knowledge of any programming language, because there will be hardcore tasks related to coding. And either you can catch up in the process, that's also possible, but it will be much more comfortable if you have a foundation. And then I would like you to have deep analytical thinking, because any architecture comes from analytics, from reconciling two areas. That's why I talked about PBC today. Because it's precisely the kind of thing that keeps both business and technical components in balance. That is, here. Well, in general, all the main skills of design, and design starting from the base, you can learn during the course and then implement and get results. Why coding is also mentioned. Look, there is a pattern that explains this answer quite well. It's called C4: contexts, components, containers, and code. That is, these four levels are desirable, well, to master during the course, as they will be emphasized, so that the architecture is as consistent as possible. That is, let me explain, what is context - this is understanding the user, how much he needs this system. Next, what concerns containers, this is probably about how it will look for, perhaps, developers in context. Components are already for deployment, so that the system can be deployed. Well, and code, so that you understand what is happening in development, so that there are no big surprises for you, that your hop service has sharply increased in consumption, and you overlooked that the guys in the code have added a lot more to this service. Will there be a swarm of agents, like fighting a banking system? Well, a swarm of agents is one of the approaches, but it is quite attractive, quite maintainable due to atomicity. That is, if atomicity is properly maintained, then the probability of error decreases to the level of that very atomicity into which you have broken everything down. Therefore, it is convenient. In the near future, even more interesting approaches will appear. We will also look at them during our course as prospects, which already appeal not to atomicity and not to the all-encompassing model. But I'll leave that as a spoiler for now. So, if you have good knowledge of Java, great, object-oriented language will suit us. I generally love objects very much. Here, understanding of system design. No, believe me, it's not like that. That is, you are on an architecture course, and you need to understand high-level coding, that is, how consistently classes are designed, whether contracts are written correctly or not, and so on. That is, all these things are important to track. And it doesn't matter what language you look at it in. That it's written in Python, well, services are written in Python out of habit, this is also not a universal practice. That is, a lot is written, if interaction with hardware is in Rust. That is, this is a new, low-level language, on which most hardware interaction is written, because it allows for the most efficient implementation of memory algorithms. Well, answering your question, knowledge of any programming language will allow you to feel comfortable in this course. Well, that's it. Final summary. What should you take away today? We are moving from monolith to BBC. System complexity should be linear, not quadratic. And composable is an ideal environment for agent implementation. Also, a few books that I promised at the beginning. MVP is not equal to PVC. MVP is an approach to the product development stage, right? That is, it's just a stage, and PBC is a specific value that remains with you regardless of whether you have an MVP or not. That is, you can reuse it in any other product later because it is encapsulated and isolated. No, Python will not be very useful to you directly. That is, for example, you will need to design contracts. You can design them in any language. The most important thing is to adhere to design patterns. Or you will need to calculate the interaction between services, how they affect growth. Well, also, what difference does it make where you will, or you will need to implement architecture as code pattern. Whether you implement it in Java, Kotlin, Python, C++, C#. The main thing is to adhere to the pattern. That is, the AI architect is not about language, architecture as code is tied to managing expectations, so that all the patterns we have laid down are consistent. Not about sitting down and writing code. Well, thank you very much for your attention. Ah, yes, there is also a bonus. If you decide to sign up, Otus offers such bonuses. Join the community QR code. If you have any questions, write to me privately. My Telegram channel is called EI AR, or you can write directly to me via Minded with an @ sign. I'll write it now. Well, and come to study. It will definitely be interesting, because the team of guys who will teach you are great practitioners who have achieved success themselves and will help you achieve success. Thank you.