Transcription
[Music]
Friends, good afternoon. My name is Evgeny Vinogradov, and I am the director of the analytical solutions department at the company. Today, we have a rather interesting topic with the company's SMM architect: the enterprise architecture model on a graph. What is it, how does it come about, what moments arise during its construction, and various related questions that may come up.
Rom, hello!
Yes, hello, colleagues. Listen, tell us in a few words what prerequisites are necessary to engage in such activities, to be able to draw this model and use it. What practical applications can be found for complex structures and abstractions?
Well, to start, there needs to be a certain need to solve a specific task. The need arises with the increasing complexity of the organization. When you accumulate a large number of systems, business processes, and different infrastructures, you need to start managing them somehow. You do this to combat complexity. So, when you accumulate a sufficient amount of complexity that you want to deal with, then you approach the solution to this problem.
You need to have some development skills when such a need arises. At that point, let's say, what should you learn in advance? I would say to start working with standards because you need a methodological base. You should read and understand industrial standards for managing landscapes. Also, knowledge of some technical framework or programming language is necessary to implement solutions. It doesn't matter which one, but you should be able to work with it.
It's impossible to be fully prepared, but when the task arises, you can learn the missing parts. Yes, of course, the company always has specialists of various profiles.
Alright, then let's not drag this out. Let's move on to the topic of the report. The floor is yours.
Colleagues, I greet you once again. Today, I will talk about how we tackled the task of managing our IT landscape. We tried to build a model and explain what we did and why. I will also show some examples of what we achieved, including technical implementation.
So, let's get started. Why is all this necessary, and how did we approach it? As I mentioned, you approach the task of managing architecture when the system reaches a certain level of complexity. Here, I present the production process, which shows how many years we have been in production. We have accumulated enough complexity that we need to manage and structure.
The company is a product and technology company. We have three main products: an electronic wallet for individuals, payment acceptance for goods and services, and cash management services for enterprises, along with many different partner projects and integration solutions.
This is an important point that we will discuss multiple times. The company has been developing for quite some time, and you have accumulated a large number of systems. The main risk that arises is not the problems of system complexity. When everything becomes too much and too complex, you risk drowning in that complexity.
This is where a significant factor comes into play that can slow down the development of the company, the development of new projects, and so on. The main task of the architect, which we are discussing, is how to manage this complexity. It becomes critically important to possess knowledge—knowledge of the subject area, knowledge of the system.
One of the main tasks of the architect is to structure artifacts and manage information to document the dissemination of knowledge about the system and the subject areas it serves among a wide range of employees.
Of course, good architecture can limit the growth of system complexity, which also affects the cognitive load on teams. This means that the team can independently handle almost any task. Yes, the system architecture facilitates this.
The systems are decomposed, and a responsibility matrix for services and systems is established. Thanks to this, teams can work autonomously and can predictably implement new projects. In essence, this is our main goal: managing architecture.
In general, based on our experience over many years, we have identified approximately two classes of tasks. The first is how to design a solution when we are creating a new project or product, ensuring that everything is done correctly and well. The second class of tasks involves investigating what currently exists.
To create something new and do it well, you first need to understand what is already there. Designing a solution is always an analytical task. A person works on a specific task, and the solution always answers the questions of what needs to be done, how it should be done, and why it should be done that way.
In the second task, when we study what exists, the approach is somewhat different. We try to find information about how it is structured. We have a fairly broad class of questions, as different people seek answers to very different questions, and it is impossible to predict in advance what people will want to find.
Thus, we need to provide a flexible system for searching answers to various questions. The search always occurs across the entire system and multiple solutions and is rarely limited to a specific document.
I would like to emphasize again that when we design a solution, a person is always involved, and this requires immersion in a specific subject area. In this case, a simple model that describes some structure and scheme is usually insufficient because the main complexity lies in the subject area and requires immersion in it.
A natural language is always needed to describe the subject area and its tasks. In this class of tasks, the documentation approach works excellently because it generates documents that describe solutions and services, which effectively and accurately capture the subject area.
These documents are always grouped according to a certain hierarchy, usually by services and products. We will discuss this a bit later.
When we write documentation, we understand that it is always written by a specific person within a certain task. Since documentation is written by a person, there is always a risk that the actual state of production systems may not fully correspond to what is written in the documents.
In principle, there may be some errors, and there may be competing changes made by other teams. In well-structured processes, the document usually reflects how things actually work, but this risk cannot be completely eliminated.
So, we need to discuss something else today.
It is also quite challenging to establish some automated control systems for documentation. Documentation generally needs to be verified against reality. How do we do this? We will talk about that.
On the other hand, if we look at an analytical model, it is always a formal model of entities and attributes. Various metrics and control scripts can be implemented based on it. The model always reflects some strictly formalized facts about the system, but unfortunately, the model does not explain the reasons why things are the way they are; it simply declares the fact that it is structured this way.
But why? The answer lies in the documentation.
Of course, the model never describes the subject area, so again, the answer to why it is done this way will always need to be sought in the documentation.
Another problem with analytical models is collective work and change management. When many teams and people are involved, competitive changes can make it quite challenging to create a cohesive model. The complexity lies not only in the data but also in its visualization.
How to build visual diagrams? We will also discuss this a bit later.
Based on our experience, we have chosen the following approach: for solution design, when we create something new and discuss how to do it correctly, we apply modeling and assembling models from existing data.
We have a certain metamodel that defines the requirements for artifacts, documents, repositories, and model objects.
So, let's talk again about the model. What are the main tasks of the model? For us, it is a tool for exploring the system. The model contains data about how it is actually structured. We can explore the model to find answers to various questions.
Additionally, the model is a good tool for conducting various audits, including security audits, structural audits, and various analytical dependencies.
Based on the model, we can also verify documents, and when we edit documents, we can check that the documentation reflects what actually exists.
Now, let's discuss our metamodel to ensure we have a good repository of documents that works effectively and answers users' questions.
Of course, the question of modeling always begins with the metamodel. The metamodel is a system of requirements that describes the structure and relationships of the model.
The foundation of our metamodel is based on the following architecture management standards. For those interested, you can read about them.
Here are the principles we adopted. When choosing, we relied on a package of international standards, taking basic entities and their definitions from there, as well as the model of relationships.
It is very important that international standards are beneficial because they define a common language understood by a large number of people. If you invent something unique, unfortunately, you will have to invest time in training and explaining to experts what, how, and why.
However, if you use international standards, many of these questions disappear.
Moreover, since international standards define a certain set of typical entities, this greatly expands the possibilities for integration with various analytical tools. We can export the model to different tools and perform various analyses on them.
If you come up with something unique, it will be very difficult to do.
Another important point I want to highlight is that resources are always limited—both cognitive resources and team resources, as well as time.
The metamodels of industrial standards are complete but complex. You need to select a minimally sufficient set from the complete metamodel defined by the standard that will solve your specific task.
Thus, we limit the cognitive complexity of the model. The simpler it is, the easier it is to work with and maintain.
Of course, we all understand that no one has infinite resources. Many such initiatives fail due to the labor intensity of maintenance. You cannot create a model that is more complex than you can manage within regular work processes.
I would like to remind you of John Gall's law, which states that if you try to create a complex system from scratch, there is a high probability that you will fail.
Indeed, complex systems that work effectively always develop along the path from simple to complex. First, you create something simple, launch it into operation, and it proves its viability.
This idea is a working approach, and it truly works. Then, gradually, you start to increase functionality to add other missing features. This approach allows for a higher probability of success.
So, when leading here, it is very important to have a core entity. I showed you at the beginning of the presentation that we are a product and technology company, and our main set of products is as follows.
The approaches to enterprise modeling are different. We have chosen a service-based foundation. Our products and services are the main vertical decomposition approach in the structure and, in principle, in the decomposition of the system.
So, our service is the basic unit of decomposition. You could say it is the foundation of everything, both in the technological layer and the business layer.
We understand that a service is not an application, as is often thought. If we rely on the standard, a service is a unit of behavior. It is a service available to external consumers.
Thus, a service as a unit of basic behavior is a certain scenario. In our case, it could be a banking card, registering an electronic wallet, or issuing a banking card, which is then reflected in the technological entity and is part of the products.
We have a set of products as an aggregation of services at the core of the model. We have adapted it somewhat, but we still strive to adhere to the notation.
There are its technological applications, its business strategies, and an important decision we made is that the layer reflects the responsibility of the department that owns the objects in that layer.
I want to quote the standard scheme. The architecture standard defines layers: technological, application, business, and strategic.
Any model operates with entities, data, behavior, and elements of active structure.
In our technological layer, we have all the service infrastructure, which is the internal cloud of the department's operations.
The application layer consists of the applied systems where the implementation occurs, including our applications and the realization of business processes and services.
In the business layer, we have the structure of goods and services.
In the Strategy Motivation layers, we have key competencies and the target audience for these products.
Let's go through the layers in more detail. In the objects of the technological layer of our metamodel, the following entities are applied.
Once again, I want to emphasize that the technological object reflects the technological infrastructure managed by the operations department.
These are technological conditions and services. As a reference, we look at public clouds like Amazon and Google.
We have our infrastructure services and technological interfaces for connecting systems. For example, our system program ensures the operation of the infrastructure.
The communication channels of the systems include the entire transport cluster of equipment and the database.
All of this is managed by the operations department, which is the basic infrastructure on which the upper layers operate.
The application layer consists of the applications that our development team works on. These are applied services that provide a specific set of applications.
Each application is deployed to provide a particular service.
The business layer is probably the most complex and abstract. In our configuration, the business layer reflects the structure of the company's products and services.
We understand that products and services never exist in a vacuum; they are always targeted at someone. A product does not exist independently of a consumer segment.
Another important aspect of enterprise architecture is the key competencies of the company.
As a payment system, we know how to process payments, manage accounts, and open electronic wallets. These are our key competencies, and we can build products from them.
However, we do not have the competency to organize cultural or sports events. If we need that, we will have to engage an external organization to provide that service.
Thus, we understand that a product is always targeted at someone, and you cannot create a product in a vacuum; you need specific competencies for that.
Unfortunately, we have noticed that the current version of the standard does not provide a clear model of the relationships between business, strategy, and motivation. They exist in isolation from each other, and the standard currently does not provide a clear metamodel of the relationships between entities.
Therefore, we decided to simplify things and consolidate everything into a general business layer.
Now, let's take a look at the objects in the business layer. The key entity here is the business service, which is a service provided to end consumers.
I want to emphasize that this is a technological service. For example, if we consider a banking card, we have several types, but the consumer always sees the payment service of the banking card.
How it is implemented and by what means remains in the application layer.
There is also a channel for providing the service, whether it is an API, a website, or a contact center.
This is to provide value to a specific group of consumers. The consumer segments are also reflected in this model to show who the product is targeted at.
Another crucial entity is the competency of the company. The key competency of the company is what we can do. We can implement products and services only if we possess the corresponding competencies.
If we do not have those competencies, we must either develop them ourselves or acquire them externally.
Of course, products, services, and competencies do not exist in a vacuum, and there is always an organizational unit that owns them, which we also reflect in the model.
Now, let's look at a summary picture of the metamodel of our business and a bit of the application.
As I mentioned, the key abstractions are competencies, which indicate what the company can do. Competencies always belong to a specific organizational unit and allow you to implement a particular service reflected in a service channel.
The set of services aggregates, including in the business layer. The same application function can be applied across multiple products.
One of the tasks we solved was the classification of technical systems. A technical system must always belong to a specific subject area.
In our understanding, competency is also a subject area, and the subject area is always owned by a specific team.
From the architect's perspective, it is essential to ensure that systems grow predictably and to limit the load on teams.
We need to be able to draw clear boundaries between applications. It is important to avoid applications that cross competency boundaries.
If there is a shift or mixing of the responsibility matrix for services and business processes, it becomes challenging to identify a specific owner.
In such a structure, when you draw the boundaries of technical systems based on competencies and service sets, a stable team is always assigned to a specific application.
This creates a stable subject area, and competencies grow positively.
This also influences the structure of documentation. You have documents that outline how to do things correctly, how to design properly, and where to store documentary artifacts.
You will have a clear structure of documents organized by competencies. This is our approach to managing the repository of artifacts.
Now, let's move on to the theoretical part.
A typical problem that novice architects and analysts face is when you are asked to draw a general diagram that shows everything.
You might try to do that, but it is only possible when your system is very simple, say, with about ten components. Beyond that, the diagram becomes very complex and unreadable.
If you have thousands of entities, it becomes practically impossible to draw. You need to apply a different technical solution.
I want to emphasize that in the practice of managing architecture, there is a concept of perspective.
How do you want to look at the system? You can make a partial selection that answers a specific group of questions.
For example, security wants to see answers to its questions, business units want to see answers to their questions, developers have their own set of questions, and so on.
We thought that to solve this, a data graph structure could be used, where you can store an unlimited number of elements and try to build partial selections by querying the graph.
Modern graph databases also have a visualization layer. You can not only look at the data but also visualize it.
Let's quickly go through the main concepts of what a graph is. A graph is a data structure that always has nodes and connections. A node reflects a certain entity that can have specific attributes.
Nodes are always connected by various relationships, which can also have types and attributes.
You can add attributes to both nodes and connections.
Here is an example from our implementation. We look at a node in the graph and see that it has a specific type of application and a set of attributes.
You can make analytical queries in the graph database, applying conditions to both the types of elements and the contents of the attributes, thus making partial selections to find answers to different questions.
The graph database also provides a schema of the graph. If you draw an analogy with databases, you can always look at the schema to see what tables and indexes exist.
The graph database provides a tool where you can explore the graph objects, click on nodes and relationships, and see what attributes they have.
This is a discovery tool. If you do not know how the graph is structured, you can explore the schema and see what data is available for your queries.
Regarding the implementation, we have tried typical graph databases like Neo4j and ArangoDB. Our solution works with both, but we ultimately chose Neo4j for several reasons.
It is newer and works a bit faster. The visualization layer is well-designed, allowing you to build suitable diagrams.
The graph needs to be built from something. You need source data to create it. The tool for exploring systems should be built from actual data from production systems.
We need to reflect in the graph not what people input but what actually exists in the execution environments.
Our data sources for the graph include configuration management for infrastructure, which directly reflects the configuration of objects in the internal cloud, as well as application configurations.
All descriptor files and configuration files are formally annotated, making them easily parsed by scripts.
We also have other data sources, such as a corporate portal where teams, employees, roles, departments, and subject area directories are indexed.
While we have not completed everything yet, it will be done. The directory is also used, but it is still a temporary phenomenon.
I want to emphasize that technological maturity greatly aids in solving this task. If all your workflows are done manually, it will be challenging to implement such a system.
The approach to building an internal cloud and your systems on public clouds requires formalization, leading to formal descriptors of configurations.
This formal structure is managed as code, and everything can be parsed by scripts, indexed, and used to populate the database.
This allows you to achieve what we have accomplished.
Let's look at a few examples. First, let's briefly go through how to work with the database.
The graph database uses a query language called Cypher, which is somewhat of an adaptation of SQL for graphs. If you have worked with SQL and know how to query standard databases, you can quickly adapt to Cypher.
The basic syntax allows you to select a node, connect it to another node, and visualize it as a diagram.
This query will return the entire graph. If the graph is large, with tens of thousands of elements, the visualization will also be unreadable.
Therefore, you need to add some conditions to avoid exporting the entire graph and instead request something specific.
For example, we want to see which technological connection points are associated with a specific application. This query will show us that data.
If you need more complex conditions, you can construct a query with various conditions and combine them.
The graph is directed, meaning relationships can be directed or undirected. You can also specify the direction of relationships as a condition.
For example, you can ask who provides a function to whom.
Let's look at some examples of queries we have already implemented. Here, we want to see the dependencies of a specific application.
The upper example is a query for a specific application, while the lower example shows the interconnections between applications.
This is an example of integration, where we see which applications cross the network perimeter.
We verify our internal configurations to ensure that everything is set up correctly and that there are no unnecessary accesses.
We can also use this information in newly designed solutions.
Another typical question is about the connection addresses reflected in the application.
Given our business specifics as a payment system, we have many integrations with hundreds of partners and interfaces.
This question is relevant: what endpoints, addresses, and ports are associated with a specific application?
By connecting application configurations and infrastructure configurations, we can see this.
Let's look at a more complex query. I showed you a simple example that queries through one node, but you can extend the query to look through multiple nodes.
For example, we have a query for one of our applications that shows which interfaces it connects to.
This is a request from analysts to trace how data spreads through a broker. In synchronous integrations, everything is usually visible through access matrices, as one application explicitly calls another.
However, in orchestration and choreography, where events are involved, it becomes more complex.
You need to be able to find this information. This query will show which application creates data in Kafka and which applications read it.
We also initially thought that the business analytics system only consumes data, but it turned out that it also supplies data in some cases.
We can see which systems provide data and to whom they publish it.
As we approach the end, I want to emphasize that the main question is how to ensure the independence of teams and subject areas.
Each cluster of systems should develop as independently as possible and not depend on other systems, teams, or neighboring subject areas.
We can trace the connections between two subject areas and see the shared infrastructure object involved.
Now we know what that is, and we can see the connection between applications.
On the diagram, you will not see this connection in either subject area, but you will see that there is an implicit connection through a technological call.
Returning to the number of services and applications, our topic consists of products.
A product is a set of services. We can see what it consists of at the technological level, which applications implement these services, and which infrastructure objects are involved.
We can build a complete graph of the product from its technical components and connection points.
We can also make selections of these configurations.
Another topic we focus on is ensuring reliability. We have many products and a large number of applications with varying levels of criticality.
One of our typical queries is to trace which products and services are affected if a specific application fails.
We can also trace this through the graph. If one of the applications fails for some reason, we can see what it will impact.
Similarly, as we looked at the interdependencies of competencies, we can also examine the interdependencies of products.
We can identify where they share services, where they have common technological components, and where there may be overlaps in consumer classes.
We will see these connections as well.
We are nearing the end, and I want to say a few concluding words.
An important point to note is that there is no silver bullet. There are no universal solutions; they do not exist. Each organization follows its own path.
If you are looking for a solution that works equally well in all conditions and for any task, that is a utopia.
You are always solving a specific task, so keep that in mind. If you want to start something, always define the boundaries of the task: what and why you are doing it.
To build a model, there must be a preliminary discovery phase. You need to explore the business processes and artifacts of the organization.
You must gather this data from somewhere. If you fill it in abstractly, there is a high probability of errors.
To understand what artifacts the model can have, you need to look at the actual artifacts in the organization, the configurations in your cloud, the business processes, and the departments that produce working process artifacts.
Thus, it is essential to conduct a discovery phase to determine what you want to do and what task you are solving.
The synthesis of the metamodel should follow, defining what should be in your architecture repository or model.
Then, you select the appropriate tools.
I want to highlight two important points at the end. Limit the complexity of the model. Do not try to create a universal abstract solution that covers all cases; that is impossible.
Your main constraint will be your time because specific people will be doing this work. Besides creating the model, there are many other tasks, and time will be your primary limiting factor.
So, choose what is minimally sufficient for you.
Lastly, modern technologies significantly simplify the task because many objects have become formalized.
You have data sources from which you can gather configurations, index them with scripts, and so on.
In contrast, manual work, such as manual configurations and settings, is usually not feasible and will be very challenging to implement.
Modern stacks, including public and private clouds, and the whole Kubernetes story, play in your favor from an architectural perspective because they formalize entities and provide data sources that can be used relatively easily.
Thank you for your attention. I hope I haven't tired you too much. We have a few minutes left for questions, and it seems we are moving to the discussion zone.
Rom, thank you very much. I will probably reveal a small secret—not a secret, but I will say that I participated a bit in describing this scheme, and we had quite a few meetings to determine what exactly to write.
So, my personal wish for the last slide, which includes the architecture viewpoints you mentioned, is that it is extremely important to know for whom we are doing all this.
Without that, drawing diagrams becomes very challenging. I remember that things started to flow when we determined that the system analyst was one of the audiences for whom we were drawing the diagram.
It became clear what to reflect on the diagram because otherwise, it would have turned into a huge, complex mess.
Friends, we will now move to the discussion zone. We will need just a couple of minutes for that. Before we do, I want to remind you that there is a chat where you can ask questions.
Please write them down, and we will discuss them in the discussion zone. There are already some questions, but if you have anything else you want to clarify, ask, or tell the speaker that he is wrong, please use the chat for that.
We will read everything and try to discuss all the comments that appear there.
Additionally, once the stream ends, you will see a form to evaluate the report. We kindly ask you to assess it. A lot of effort and energy went into preparing it, and your feedback is crucial for us to understand what worked well and what areas need development.
Any feedback on the report is very welcome.
Moreover, for those participating in the discussion zone, we will have a gift for one of the questions chosen by the speaker. It will be a useful book on our topic related to analytics and similar subjects.
So, don't disconnect; we will see you in a couple of minutes in the discussion zone.