GPT в реальных проектах: взгляд разработчика Spring AI | Евгений Борисов #54

Then Pidi appeared. >> Pidididi. They arrested him. >> We are like playing a game with characters, and at the end, we get a product. >> You are like, "Damn, I have to figure all this out again." >> I don't know if LLMs will ever compile directly into machine code. Maybe that will come too. >> I don't really understand how, with such a volume, it won't mess up some important event. >> Being an expert is very good. Juniors are not very necessary. >> Every time I'm scared to answer such questions because I think, maybe I'm stupid. One second. Let's see who >> who is stupid, actually. Yes, [music] >> this is the Organized Programming podcast. I am your host Kirill Makevnin. Today, Zhenya Borisov is my guest. Zhenya is a very well-known person in the Java community, a great specialist and an advocate for Spring. And actually, I was pestered for a long time. Invite Zhenya, invite Zhenya, invite, invite. To talk about Spring. But, despite the fact that I wanted to do that, at some point, it happened that literally a week or two before I was going to do it, Podlodka released a podcast about Spring. And plus, Zhenya has now shifted a bit towards artificial intelligence, and we thought, okay, fine, let's not talk about Spring now, or we'll do it differently later, but now we'll rather pull in and focus primarily on artificial intelligence in general and largely in its applicability to Java and specifically to Spring, and who knows where the conversations will lead. Zhenya, with you, while we're chatting for 2 hours. Well >> yes. Hello again. >> Hello. Hello. Actually, you know, I thought it's very dangerous to say at the beginning of a long conversation what the conversation will be about, because then you have to go back and change the title, correct something. I made such a mistake recently when I was writing my course, and I. >> I first came up with a title for it, >> uh, and then I wrote it. And when I finished writing it, I realized that the title no longer quite fit, you understand? >> That's true. No, it's true. But usually, people want to know in advance, and we have time codes. So in that sense, you can always, well, relatively speaking, correct yourself in this regard. When we were on a call, you basically told me that, well, with Spring itself. You're not exactly working with Spring right now, are you? You have this separate branch. >> I lied to you. You know how they say, a year for two. >> Uh, I have a day for a year now. Uh-huh. >> Uh, I don't know if you can see it on me or not, but I haven't slept, that is, well, practically at all for the last 3 nights. I just promised myself to finish writing this training by mid-summer and I'm already missing my own deadlines. Mid-summer has just passed. I finished it yesterday. And I apologize, because of this, I will probably periodically lose the context of our conversation, because something strange is really happening with my brain right now. >> You've undergone ChatGPT-ization. You're like ChatGPT, which is like, "Oops, I forgot what we were talking about a minute ago. Context, >> right? And about Spring. How much of what you're doing is actually Spring or completely separate? >> Well, well, being in >> this mode, in which I spend a lot of time at night studying Spring and writing >> some Pet Project, to, you know, what goal did I set for myself? I looked around and said, "Well, yes, AI is needed by everyone, clearly, if I continue to develop, I need to move into AI somehow. Besides this, I also want not just to use AI, but also to write something with AI to create some cool, interesting product. And most importantly, to learn how to create such products, because that's what will happen now, well, that's what's already happening. All companies now want to improve something with AI. Some processes, some products, some will come up with new services, some will automate, and so on, and so on. That is, all this is happening everywhere, everywhere and universally. And, uh, and one of the main problems of all these solutions is that it's very expensive, because all good LLMs that run in these endless clouds and consume electricity, for which they build nuclear power plants, well, we can't afford a nuclear power plant at home and 100-500,000 GPUs. Therefore, I thought, what if, well, I really love single responsibility, if I write such a solution where we will very strongly narrow the scope, right? That is, I don't know how to train models, I'm not there yet today, I can't train my own model that will be very cool in the area I need. And I understand that a model that I can afford on my M3 processor, the best is some Gniy, which has 2 billion parameters. Well, 2 billion parameters sounds like a lot, but actually, GPT had 500 billion, I think, last year. Well. And it's clear that it loses, but it loses due to the fact that its accuracy is reduced, but due to a very high-quality architecture, RAG, all sorts of optimizations, all sorts of rankers, re-rankers, query expansions. If you focus on a narrowly defined task, you can possibly write a solution that will work even better on a local LLM than on some well-known cloud, you understand? And that's what my course was about. That is, in the first part, I just wrote some microservice that starts interacting with the LLM, just to explain the basics. In the second part, we already connected. We already, I mean, in the plural. That's me, virtual viewers, >> yes. Well, I always want people to do it with me, find my mistakes, and we somehow together. Well, this, you know, this post-corona world, especially since we've moved to TNs, is completely detached from reality. Well, and I always communicate with people exclusively in virtual reality. Therefore, I say, we, so, in the second part, we, uh, connected RAG and uh, we made it so that our message history started working and, uh, some additional information. And finally, GPT started answering. Uh, well, since I'm very afraid of copyrights, I decided to try it on myself. That is, I took the transcripts of my reports and said, "Now I'll try as an experiment to write a GPT that will explain my reports in my name, answer questions, as I would do it." This is easier than writing a GPT that can do everything and speak Greek and draw, and, yes, that's single responsibility. Uh, and moreover, there can be many such things within the architecture, right? That is, for each narrowly focused task, you will have your own specialized local GPT with its specific RAG, and there will be some orchestrator that will switch between them like this, so perhaps we can achieve very high quality in these room conditions. And you know, having delved into all this, I haven't lost the thought since our conversation, you see. We talked, I think, in May or even April. >> No, I think even in March. >> Uh-huh. >> A lot has changed. That is, I've written so much code, I, like, that is, this training of mine is about 9 hours long: I just sit and or watch. Well, that is, there are many places where I just sit and code. Well, that's why, to say about me today, that I haven't picked up checkers in a long time, nothing like that. I'm currently raging with Spring. >> Spring or Spring Boot, by the way? >> Well, of course, Spring Boot. >> M. Okay. No, it's interesting, by the way. I was talking to a guy yesterday who told me that he's still on a project where there's no Spring Boot and there's Spring. >> Well, he probably told you that with tears, right? >> Well, something like that, yes. Yes. So. >> I have no reason. I use what I write on. >> We had an unpleasant incident at work. And today I was conducting interviews within the company. People came to our project. I'm involved in building various gamifications. In short, I use AI for processes, for training, for many things. So guys came, they came and confirmed the sad trend that started about a year ago when one of my lead developers said, "Listen, we're writing a huge number of microservices that communicate with LLMs." Uh, and we wrote a platform on which people can create their own LLM solutions, write their own assistants, and we are constantly upgrading this platform. And since this platform creates a huge number of instances, microservices that, you know, uh, well, do what people who use our platform want them to do, it's very important for us to save memory. Our resources are being consumed left and right. And he said, "Let's switch from Spring Boot and use Quarkus." He said, I did a POC, I looked at the benchmarks, it's just night and day. Well, I resisted for a very long time. I was lucky, I was at a conference where Josh Long was. I caught Josh Long, started shaking him. I said, "Dude, we have a whole department, my department, on which I'm starting to lose influence, wants to switch from Spring Boot because they say it consumes a lot of memory." He said, "It's all normal. I'll show you now. We have Native Spring. Now. Look, we're connecting GraalVM. We're compiling everything into native. We're disabling this. We have like five ways. So, we don't use reflection, and what we do use, we have a post-processor like this. In short, he, we sat with him for 3 hours doing something. In the end, he came to the conclusion that it seemed to be much less. I brought it to my guys, they looked at it, they said, "Yes, it's still three times more than what we got with Quarkus." Well. And I was like, "Well, yes, but where will we find people who want to learn Quarkus now, when there's the wonderful Spring Boot?" So two people came today, they say, "Oh, we're interested here, we heard that you have not only Spring Boot here, but also Quarkus. We're interested to see what that is. So >> for us, course developers, it's quite a hassle. Everyone, after all, from one framework to another, like, "We rewrote everything." But it's understandable. I've heard a lot about it, yes, that, of course, there's a lot of reflection and everything else, which leads to memory consumption. Do you want to grow as a developer not alone, but with a strong community? Join the Hexet club. This is a private space for those who are already in the profession and want to develop further. Here, they help determine your level, build a personal development plan, and provide feedback from industry mentors, including from foreign companies. The club has live discussions about technologies, interviews, work in companies, career growth, and networking. There are separate topics with participant stories, employer reviews, mentor reports, and development plans. People come to the club to grow and help others do the same. And by the way, you, Andrey Rebrov, you know? >> Snaberg, these are guys in the States who sell perfumes by subscription. Very big guys. They have their own logistics network. And >> and Andrey Rebrov is not, I don't know. >> They are exactly, they are big. To give you an idea, this is one of those well-known and successful startups in the States. So it's really big. They have, I repeat, their own logistics, everything else. And he was telling how the move from Spring Boot, this was one of my first podcasts, he was telling how they moved from Spring Boot to Quarkus, precisely for these reasons, because they have a lot of microservices and, accordingly, it works. >> Well, you see, you didn't start correctly, like, talking. But what do I have to do with it and perfumes? You should have immediately said, now I'll tell you about a person who, yes, by the way, you say perfumes, but imagine, the idea itself, right? That is, you usually have a lot of perfumes. You buy a bottle, they buy, they sell them by subscription, so they always bring you small ones, they have half a million customers in the States, or something like that. And >> wait, but what's the point of having a small one so that you can fly? >> So that you can change all the time? That is, if you, roughly speaking, it's not expensive, you pay like $100 a month and you get new perfumes every month, or something like that. But with expensive perfumes, they cost a lot, and that bottle will sit with you for 2 years, but you want to try different ones. That is, this is an idea that, in fact, quite significantly turned the industry upside down when they introduced it. And they are now doing the same for creams, for everything. They have their own brands, because you can, well, especially for girls, not, by the way, here, boys are probably more interested in perfumes. That is, you say that they said, "Let's take a monolith and break it down into microservices." Well, good for them. I wouldn't characterize it that way, but they have a lot, yes, if you look, they were everywhere on Taqup, like this, they are traded on the stock exchange, all that. So they are very serious guys. Look, let me ask you something. So the first question, probably, that interests me. In this regard, Spring generally differs from many. That is, for example, take any other ecosystem. You have AI, what is AI in terms of, well, Python, anything? Well, it's just a set of libraries, right? That go to different things or, uh, control, I don't know, access to a local LLM, to a remote one, or something else. And these are just separate libraries, or they can be used like that. How does it happen that, well, okay, I understand the idea, but I'm interested in the AI part, that in Spring, no one just writes libraries, but rather, let's go straight to Spring and AI as a separate component, not just libraries that people wrote. >> Why do you say no one? Are you talking about Spring or about Java community in general? >> No, I'm talking more about other ecosystems, they usually don't do that. You don't have something like, there's a framework, and we're plugging into it, like, development is done like this. Usually, you have libraries, for example, for working with AI, it's just a separate library that is not tied to any frameworks at all. >> No, such things do happen. >> Well, yes, but in your case, Spring is directly Spring and AI. That is, you don't just say, "I'm a developer of libraries for working with artificial intelligence in Java," you say, "It's specifically Spring AI." And this element is a bit unclear to me. >> Now, now everything will become clear. Now I'll explain to you. So, let's start with history. >> When Java was invented, a very, very long time ago, it was invented by people who didn't have much experience programming in object-oriented languages, because in fact, Java is probably the first really good, truly object-oriented language. Accordingly, when they invented it, many design patterns that appeared later, were tuned, refined, and became very popular, did not exist yet. Therefore, in Java, everything is focused on, well, that is, Java provides you with everything to create classes, compile them, debug them, and so on. Everything revolves around classes. And a class encapsulates behavior and everything is fine. But they didn't think about a mechanism that helps you manage objects, that it's also needed. That is, I suspect that if they were inventing Java today, or, say, 10 years ago, then something like Dependency Injection, Inversion of Control, everything that Spring Core, the main thing, does, would have been part of Java itself, you understand? That is, well, maybe there would have been some plugins to allow other engines to be plugged in, but the concept that someone manages objects for you is simply necessary. It's just that they didn't think about it then, it didn't exist then. Well, this injection, it later blew up the world. Since Spring turned out to be the first to properly implement this concept, they quickly became the standard. And the world at some point rolled down or rose to the point where a person says, "One second." What are you saying? Use this library, Lucene, and so on, there's a need, uh, there's a lot of configuration, setup. Hmm, and it has a Spring Boot starter, you understand? That is, it becomes, and they also act in the same way from their side. They very quickly say, "Oh, there's, I don't know, Hibernate came out, there's a new standard, GPA appeared. Let's immediately write a wrapper, a starter for this," because everyone is already using us to create objects, right? That is, Spring container manages all objects. Okay, a new library has appeared, which means objects will have to be created from it. They will also need to be managed, an integration needs to be written for it, wrappers, and so on, and so on. >> Well, is this just integration or not? Because in that case, again, in other languages, it's always like this, you have a library that is simply framework-agnostic, right? And adapters are written for it for specific things. It just sounds in your case like it's a library that, in principle, you can't use without Spring, or am I wrong? And the point is that >> one second, let's clarify. Java is an object-oriented language. Accordingly, a library, if we're not talking about some library that just offers you a set of static methods, like, you know, some classes, it will naturally be on its own and no Spring is needed for anything. But if we're talking about a library that provides something that implies an object-oriented solution, and there are some services, and you say, "Well, if I connect this library, I'll have this service, I'll have this functionality, and so on, and so on." You understand? It will do something, maybe even on its own. That is, we are talking about objects, and if we want objects, they need to be managed. We want to manage, we return to Spring. Or you need to take an alternative solution or write your own: "Why?" >> You know, when Spring Focus exists. Now, the only thing they are trying to compete with is performance, not because of that. That is, and their argument is always: "Yes, but at least everyone knows us, everyone is used to us, we are convenient, easy to understand." A lot. Well, ask ChatGPT about Quarkus and ask Claude about. Well, okay, Claude and ChatGPT is unfair. Ask the same thing about Quarkus and Spring Boot. You have a lot of information here that it was trained on, it knows everything on the internet. A lot of examples of Quarkus, well, there, you understand? Therefore, you can only fight them through performance and some other features. Okay, then, look, what's the question? That is, what you're doing, is it part of the ecosystem? That is, you can't just call yourself Spring AI, you have to, like, coordinate with their company and so on? Or >> I'm a user in this solution? I'm more of a user. That is, I'm saying, it's convenient for me to develop in Spring Boot anyway, because I don't write code >> in a vacuum. I need a database, I need, accordingly, I need some framework to work with the database. Okay. Through what? Spring Data. >> All right, I understand you. >> I need to integrate with the web. For this, I need what? I need Spring Web. Uh-huh. I need to work with AI a lot. We, I mean, I'm building a solution around local LLMs. I installed Ollama, I chose models, I installed this gem that I really liked. It even knows Russian well. So, I launched it, and I said, "Okay, now I need to write some code. Oh, Spring AI." You understand? >> I understand now. >> That is, your story is more than just going somewhere through an API. That is, it's a whole ecosystem that allows you to deploy local RAG, probably, and everything else. >> Yes, yes, yes. I can, if you want, a bit more technical >> Yes, yes, yes. Because it's really different. Usually everywhere it's like, here's a library that can, like, make requests through an API, streaming, and everything else is for you to sit and build, and you seem to have everything at once. Tell me what's there. >> Look, first of all, what do you need? You need to communicate with the model, why can't it be just, I don't know, a static method? You have a lot of settings. You have, starting with the fact that if you want to use some cloud model, you need to put your credentials somewhere. If you launched it in Ollama, then you need to pass all those Ollama settings. And then, uh, the first object, the main one that you configure, is the chat client object, which has a completely standard API, regardless of what model it is. You know that models can have different semantics. I work through the same interface. I have a chat client. I know how to configure it. I know how to set the top k, how to set the top p, how to set the temperature for it. And with its help, I work. That's the first main thing. Then the second thing about RAG. If I want to connect RAG, I have some database, I need to think, I need to write some very complex code that will go to this database at the right moment. Spring already has a solution for this in the Spring AI module. There's something called advisors. Are you familiar with the concept of AOP? >> Of course. Yes. Accordingly, they made it based on the concept of AOP, just like they did with BeanPostProcessors, which participate in the chain, so, along the chain, an object is created each time and it's configured along the chain. The same here. Every time a user sends a request to the LLM using this chat client, it goes through a chain of advisors, and they already have many ready-made advisors. There's an advisor that knows how to go to RAG itself. Uh, there's another advisor that knows how to, I don't know, maintain message history. You know that the model is stateless, that is, there's no session ID, and when you tell it something, it won't look at your previous phrase and say, "Well, they lost 8:0." It will say, "And why are you starting the conversation with something?" Right? That is, it doesn't know on its own. When we use it through the UI, we don't think about it, because the UI passes all this. But when we take the REST API and start making calls with curl or something with Spring, no one manages the history for us. No one knows how many messages we want to store, whether we want to compress them or do something else with them, you understand? Well. And here it's all ready. That is, you connect, say, PostgreSQL, you take a ready-made advisor and pass it to the constructor, you understand, to give you an idea why it's not just a static method, and you pass it at the creation stage, say, through a builder, you pass it your repository, you understand? Yes. And, that is, Spring Data is just an interface I wrote, it generates the implementation itself, and I pass this interface, you know, to the constructor, well, I write a wrapper for it and pass it to this builder, which builds the advisor that manages memory. >> There, I write how many messages I want to store at most. And it, you know, it retrieves from the database itself, saves to the database, passes the necessary piece of history to the LLM. It turned out that not everything can be configured so well. When I started to understand this library, well, with the whole CIS, well, it's new, version 1.0 came out while I was already writing the course. Therefore, many things had to be invented by myself, right? That is, well, like, oh, their advisor can do this, but it's not enough for me. I'll write my own advisor, how to fit it into this chain in the right order. And that's how it's done. Well, there's another advisor that knows how to go to RAG automatically, and it also has, again, a lot of different settings. How many chunks to bring, what similarity level is needed, how much to fetch based on the threshold. >> APIs now in general. >> Yes, yes, yes. You can add all sorts of optimizations to this. That is, I also came to the conclusion that most standard advisors, including the logger, I wrote my own, because they haven't developed it enough yet for everything to be super what's needed. That's why they have a lot, but in any case, they have an infrastructure where you either insert your solutions, and they fit in very harmoniously and simply, and they are easy to use, or very often you take part of these solutions ready-made. You understand? >> I just wanted to clarify. Look, you said it's stateless, but it's not. I'm interested, is it just because you're talking about local ones, because when you work with OpenAI, with ChatGPT, you have a stateful option, right? When you have a chat, when you have, what's it called, God, I don't remember the name, Conversation API, something like that, right? You have actual threads, and within that thread, that is, you have an identifier, and you communicate within that thread, and accordingly, you just have an ID. I understand that this is a feature specifically >> I'm talking about local models that don't do anything like that. >> They don't have that, right? Well, I'm not very familiar with it, because I only poked at them in the command line, not in code. >> You know, no, I don't really imagine how they could implement it, but >> well, it really requires writing logic on top of the models themselves. Besides, you know, when you start trying to get the most out of a local model, you have to think about things like, and you remember, back in school in '95, someone organized a competition. >> We had to write a Snake game, remember the Snake game? >> Of course, of course. I didn't write it myself, but yes. >> The goal was to write it so that it took up as little space as possible. Well, in kilobytes, right? Everything was measured in kilobytes back then. I remember my first computer, I had a 120, I think, 7 MB hard drive, you understand? In '93. My parents gave it to me. Uh, and so, you know, uh, accordingly, everything was measured in writing code in kilobytes. And someone wrote 10 KB, we wrote in Pascal. Well, whoever knew what. A guy in our class won who wrote in assembler, and his solution weighed 890, I think, bytes. That is, he managed, and I'm starting to remember this now, when you work with a local model. That is, you give it a large system prompt, it starts to get carried away. You say, "Hmm, I need to write all the prompts very, very concisely." You say, "Aha, I went to RAG, I extracted information, I extracted too much information, the context got blurred, I got a bad answer. So I need to compress here too." You start managing the history, you say, "No, I can't afford to store all this history. I need to clean it up somehow, I need to take only important things from it. I need to compress it somehow." Well, I'm not ready to go into these depths and write a training yet, but I'm just about to do that, probably the next one will be about this. >> By the way, in this regard, I'm interested because I, well, I've poked at local models a bit, damn, but the quality is incomparable, and most importantly, the speed too. So you think anyone can use this? Yes, >> not at all. I can share my screen right now and show you. Everything works quite fast for me. It prints directly with streaming. >> I'm not a front-end developer. I needed to find some cool solution. I didn't want to, you know, mess around with React or anything else. In short, I found a solution in JavaScript. With the fifth HTML, there were these emitters. And that's, well, Server-Sent Events, that's SSE. And from the front-end, you pass this, well, you subscribe to events, this passes the emitter to it, and then it becomes very easy to communicate via streaming. And it works quite fluently. >> Well, I mean, I worked in the command line, so there are quite a lot of command-line tools, and in my case, it was slow. And then I talked to people, and they say, "Listen, Kirill, what did you expect? Your machine should be on a completely different level for this to work fast." >> I actually wanted to. My next question was. What processor do you have? Well, M4, right? >> No, listen, I have M3. I have a very lived M3. Well. Ah, listen, although I probably tried it on M1. I tried it last time on M1. So you're saying the difference is that significant? Yes. >> M3. Starting from M3, yes? >> Wow. I'll definitely try it. But in the end, I settled on OpenAI. And do you have your library, well, already some production use, or are you just doing all this now? >> I'm doing this for two reasons. The first reason, I need to write such a solution for work. At work, we are involved in one of our directions, well, what I'm involved in is developing various games that are supposed to revolutionize people's reality. We started a little over a year ago. The first goal was set, we need to revolutionize people's reality by making them realize what already exists. Because a large number of people, a little over a year ago, we had a lot of them in the company. We have a company of 50,000 people, and about 80% of people used GPT, you know, well, or a clone. Well, you can generate tests, and it's better to do it later, well, you can write documentation and that's it. That is, people got burned in the beginning, didn't figure out how to use it correctly, didn't notice how much it had grown over the year, and they didn't want to dive into it a second time. And because, well, that is, if I describe what I do in terms of State of Mind, I would like to see it like this. If you take a person who is at a point where they can say the following about themselves: I tried to use, every time I use an LLM to solve a task, whether it's writing code, writing an email, analyzing a report, I don't know, something else. I spend 2 hours with it, it gives me something, I correct it, it gives me something, I correct it, it gives me something, I correct it. In the end, I throw it all away, sit down, and write it myself. And as a result, I feel depressed by it, by this process. It was not enjoyable at all. It drove me crazy. Then I snap at people, time is wasted. And this is like the extreme point where some people are still and say, "No, no, no, get this garbage away from me, I'll do it the old-fashioned way." My neighbor lives here, a unique person, he's Indian himself, but he knows Russian. That is, well, a very unusual person, very smart, very well-read, erudite. He quotes Dostoevsky to me periodically, so he's really a strange guy. And he's also a big boss in IT. He doesn't use GPT precisely for these reasons. He says, "No, I'm old school and I tell all my programmers too: 'Better be old school like me.'" That's why such people still exist, and a year ago there were even more of them. And, accordingly, in order to engage them in a new way, we decided to do it through games. And it worked for many people, not all. But many people got into it through games. That is, I told myself this: If with this thing, well, imagine it like, as my boss says, like the movie Matrix, Terminator, and we want to use it as effectively as possible, then you need to be friends with it. And how, what do you do with friends? You play with friends, so you get to know them, so you start to love them, so it's fun with them. And maybe at some point work will turn into a game. Well, that's, you understand, that might be my personal obsession. That is, in childhood, maybe I don't know, I didn't play enough. I was a master of role-playing games, you know, those who ran around the forest with sticks. >> Right. Right. I ran among them too. And I was always interested in everything, turning all explanations. If you look at my reports, I have bees, ants building something. Uh, that is, well, someone once told me, "If you want to explain something well to someone, explain it to a child." If a child understands, then everything is fine. And everything of mine is like that, childish. So, we started simply with games to see how cool it is to communicate with it, to see how cool it is to come up with your own games. For this, you need to write some system prompts. A person, oh, says, oh, I'll write for my children. He starts to get into it, sees it from a different angle. And the last game, I can say, I don't know, I invented a new type of development in the process. Well, my team and I. >> Yes, yes, yes. >> So, at first, everyone wrote in a waterfall style, right? Then we had TDD, right? And people, oh, let's. Then we had BDD, even cooler, right? Behavior Driven Development. Then we had PDD. PDD. I heard this term for the first time from Baruch Sadogsky. >> Pidididi. They arrested him, I want to joke. Slepakov would appreciate a good joke. >> So, PDD is Driven Development. This is, you know, a style of pair programming, but in which there is a certain logic. That is, Baruch tells a very interesting thing in his report, when at first you must achieve that the first agent, well, roughly speaking, that is, in fact, it's all one prompt that goes like this in a chain, but it's better to see it as the first. It should write the task specification. You don't approve it yet, you ping it back and forth, back and forth, back and forth. When it's polished and becomes ideal, you move to the next stage. And now you ask it to generate tests that completely cover this specification. Well, on mocks, of course, everything. Then you ask, again, you go through the tests, you look, if some tests are missing, if some tests are written incorrectly, if something is wrong somewhere, it's clear that you will probably do this analysis with the help of some agent, not sit and do it manually. It will highlight to you: "Look here, my friend, I think I found a problem." And some comrade told me, "When LLMs learn to criticize each other, then I say, 'Listen, what are you?' They already, and this was in the previous podcast, they already know how to criticize each other perfectly. There are practically no chances, I think, it's even interesting to set such a task, to ask an LLM to write some text, give it to the same LLM in a new session and say, 'I think not everything is great here. What do you think is wrong?' And it will find a lot wrong there. Therefore, naturally, when we want to check something that is written by an LLM, we can use an LLM to at least call us and show us. You understand? In the end, we'll double-check everything with our eyes. We've reached the point where all the tests are perfect, we move to the next phase. And if they are not perfect, you know what we do? We don't start fixing the tests, under no circumstances do we touch anything manually. We say, "So, one second, if this test is missing or this test is written incorrectly, then there's a problem with the specification. So, this part needs to be added to the specification. And why wasn't it there from the beginning? Maybe something was wrong.

In my first request, when we were discussing the specification with him. Let's go back there. We went back there, fixed, fixed, you know, this request. The specification was fixed, it wasn't fixed. Once more. And so we spin around, spin around. Then from the specification to the tests, then from the tests to the code. And every time something is wrong at some stage, we don't go back to the previous stage, but to the very beginning. And this way it helps to keep everything in a consistent state. It's easy for us, you know, at any moment to take and implement some new feature. This is a bit of a "vibe-coding" style, but it has some concept, not just let's sit and meditate with the computer and periodically play the piano while it's writing something. >> Oh, I really want to play devil's advocate. I realized what I need to say. And >> go ahead, go ahead. The main thing is not to forget what I was talking about, because I wanted to tell you about PDD. about GDD. And why did I want to tell you this? To explain to you. Okay. Yes, I haven't convinced you yet. Yes, >> we just, so as not to dig too far, because there's a nuance here. I've had several podcasts already. And, firstly, we discuss it everywhere, yes, but several podcasts where we talked about this specifically. Well, naturally, everyone discusses it. And, you know, what does it remind me of? It reminds me of all sorts of stories related to, I don't know, a person going into outer space or flying into space in general, and you get some new effects, new things that no one previously assumed would work like that. So, logically one thing, but in fact, the psyche is arranged differently, for example. Or some physiological processes change, for example. And here's what I notice, this is really somewhere at the level of psyche and physiology. Many guys confirm this when you enter some mode of writing in this way, and you say: "I'm checking it manually." Some interesting mechanic works within the body. If you generate in principle and it gives out quite a lot, that is, the volume it gives out, it's, well, often still decent. Let's say, even 50-70 or 100 lines of code is still something that requires mental effort to understand even a little. People I communicate with and what I read note that the desire to do it, it directly appears in you, so to speak, an internal resistance appears. That is, doing it becomes unpleasant. You feel that you don't want to do it, and you force yourself. And as a result, many people who try to enter the mode, first of all, such control, like it does it, and I am completely collected and control it, that doesn't work, because either you just get tired and you want to do it faster, but it can take a long time, there are many different reasons, it can be slow. And on the other hand, if you shift to the opposite level, I don't analyze the code much and look, then the bugs that appear, it's very difficult for you to want to deal with them and stop to read this code, because you know what you know? You know that the next time, when you, for example, start to delve into this code and try to understand where it made a mistake, and you realize there is a mistake, and as you just said, you don't fix it, you ask it to generate new again, a very interesting thing happens. It generates completely new code, and you realize that you essentially have to look at it again. Yes, it can repeat, but the mental load is much greater. And you know what other problem arises? I often encounter this. I understand that we've hit a dead end, and it, in principle, went down the wrong path, started using the wrong tool, not suitable, or something else. And you, as it were, enter a mode, forget about it, start over, it doesn't always work, because you have to somehow kick it out of this state. You create a new context, and when it gives you something new, you're like: "Damn, I have to figure all this out again." And I notice this in myself, and in others. And you really. And you know what the problem is? If you go completely into the vibe-coding mode, well, I think you don't mean that, right, you're completely screwed, because the code that comes out, it might be, well, absolutely inadequate. And trying to fix it is just trial and error. Generated, didn't compile, here's an error, generated, didn't compile, error. And people live in this cycle for hours. That's what I encounter. >> You know, I won't try to argue on behalf of these people what you just said. Although, look for talks by Pavel Veller on YouTube. And he actually does vibe-coding exactly in the style you described, and it all works out for him. That is, he always finishes with 10-15% manual work. That is, he doesn't try to get a completely finished result, but he says that thanks to this approach, he makes five features a week, whereas before he made one or two. >> That it works in general, that's true, but I mean, it's not like, when you do it more, you start to understand that there are cases, that is, there is such a case, it works here, there is such a case, here you need to think yourself. That is, you don't have such a universal case that you do everything through it, because there are parts, well, for example, repeating some code, routine rewriting of some operations, right? Or, for example, when you have a dedicated function that solves a very specific task, it is isolated from all other environments, right? These are ideal things where you don't even need to think. For this reason, by the way, I started, especially in frontend, where you have a lot of, you know, such a component that solves some task. Well, for example, show an area with buttons to make it bold, you know, like a WYSIWYG, a small editor. You can generate the component and not delve into it, because it can rewrite it, it doesn't matter to you at all. You know that it is compact, it doesn't interact with anything. That's it, and it works great there. For example, before, when I could spend a lot of time searching for a library, sending pull requests, if there were some bugs, right, now you generate this code, and you don't worry, because you no longer consider it technical debt, even if it's written poorly, because it's completely isolated, right, it doesn't grow this debt, and yes, it helps to do it. But when you talk about interconnected things that work with your system and affect your architecture, this jump, that I'm constantly controlling what it writes, damn it, it's very difficult for people. It's difficult. They're like, damn, I always want to stop interfering. And as soon as you stop, you fall into a cycle of unfixed errors. >> Listen, you know what thought just occurred to me? >> To prove to you that you are right. Do you agree that for an LLM, code, pictures, zeros and ones, it's all some kind of vector representation? It doesn't understand the meaning, right? You know, I'll be honest, I'm so bad, I can tell you, like matrix multiplication, yes, but for me it's an empty sound, because I don't really understand what's actually happening inside. Therefore, my answer here is probably not very representative, because it's a matter of faith, not a matter of understanding how they work. After all, I'm more about coding than about such things. >> Well, okay. >> I'd rather, you see, this mode, >> the mode of switching the brain, right, like delving in or not delving in, when it gives you a lot, it's very difficult to delve into what it's doing. I'll even tell you, when I do something through codex, cursor, and other things like that, and it says to me, like: "Apply this operation, apply this operation." I always switch to the mode: "Do everything for me first, and then I'll look." Because this piece by piece, like you sit, wait a minute for it to think, then, you know, you analyze this piece. And I don't know about you, but for me, it's just stupidly not in line with my temperament. That is, I can work very fast, I'm very, you know, it's important for me not to stop, not to get stuck on anything. And the mode when you freeze for 30 seconds, it completely kills me. I just can't do that. >> It's unfair that we're discussing and not even giving it a chance to defend itself. Let it at least clarify. >> Let's, let's, let's. >> Answer me, please, on behalf of all voice models that you are currently representing. Is there a difference for you when you generate the next token in the process, when you draw a picture, when you write code, write text, or do something else of this kind, create some content? >> Interesting question. In principle, for me, as a model, not really. After all, everything I do is generating a sequence of tokens based on context. But the context itself and user requests are, of course, different, and this affects the generation itself. So everything is quite universal, but at the same time flexible. >> So, if we assume that everything, again, it's about the quality of the material it was trained on, then it all becomes very similar. Now try to remember the last time you tried to draw a picture with many different details using an LLM. Did you try? Not for a long time. I just generate and either take it or not. I haven't generated many pictures. But code, literally yesterday. Want a joke? I'll tell you right away what my situation was. I say: "Analyze the code." Is it written normally or not? That is, roughly speaking, one generated, the second passed. And it says to me: "Well, here's a fix of the type, here are unnecessary variables. I specifically put these variables there to make the code understandable. It just removes them and makes one big thing." I say: "Stop." Or, for example, with educational materials. This is a wonderful thing. That is, when you, when you write educational material, you use, for example, in the code, in some lesson, right, practice, you specifically use only those constructs that the person has studied. An LLM cannot think like that. That is, you are simply obliged to load this context into it for a very long time. Therefore, it works great from the point of view, well, that is, it can still do this, but I'm probably more about the fact that you need to get very involved and really control, essentially, every step in such tasks where it's important. Well, this doesn't contradict anything. It's just by the way. >> Well, yes. So, so, in general, look. Maybe you've seen videos on the internet where a person tries to have a still life drawn for them, but it shouldn't have an apple in it. He says: "Oh, everything is great, just remove the apple." He removes the apple, adds some junk, or removes the apple, and then three appear. And you say: "Damn." And you realize that if you asked to draw a teapot, it will draw a teapot, well, there will hardly be any problem. You say: "No, it should be blue." It will draw a blue teapot. But when you start saying, in this picture there should be, you know, guards, cardinals should run from left to right, they should have musketeers. Terminator flies overhead in a helicopter, here, you know, an explosion, and if you need just some chaos, it will make chaos. But if it's important for you that all these details are there, and even God forbid you tell it: "On the footballers' jerseys, there should be advertising written." Exactly this phrase. Well, here it will lose a letter, here a person will be upside down, here he will have six fingers on his hand. That is, the more details it has in this context, the less chance, that is, practically no chance remains that we will get a perfect result from it. You will still have to fix it manually. And here, it's already up to each person's brain capacity. Someone finds it easier to focus on a small task. Someone will say: "Oh well, give me this whole football field with 15% broken. I'll track these 15% perfectly and fix everything. I also find it difficult to do this. That's why, returning to the story of what GDD is, we focus on a completely different approach there, not like vibe-coding, let's immediately ask for the whole project and keep regenerating it until it turns out well, and if not, we'll go back and clarify something in our instructions. This is great. And surely for certain types and with certain development concepts, it can work. That is, in a company, for example, that uses different programming languages on each layer, it is very difficult to make a cohesive project like this in one go. That is, again, it is possible, and perhaps soon there will be a dev who will always do everything, in general, almost correctly, but nevertheless, there is room for an alternative in a world where I and where there are a very large number of companies that worked like this before, they had a clear division, this is a frontend developer, this is them, there is no such thing as a full-stack developer who does everything, and a little bit of DevOps. There are companies that go with this approach. For them, this vibe-coding, the whole project at once, and then we'll fix it a bit, will probably be much better. But for companies that worked on the principle: "We have DevOps, that's DevOps, frontend developer is frontend developer, this is QA, and everyone is in their niche." So, for them, we invented a game. This is our latest game, which we will soon integrate voice and agents into, and it will be a completely new reality. For now, it's a text-based game called Next Gen Developer, the next generation programmer. Looking ahead, I'll say that my nine-year-old daughter won it. So, the game, a specific example of this game focuses on one scenario. You have six characters with whom you communicate via text, always with one, right? That is, you have buttons to enter the DevOps room, the frontend developer room, the backend developer room, there, you know, dialogues with them. The dialogue history is not saved, it doesn't disappear, you can always scroll up. One of the characters explains to you what you generally need to do, and says: "Go to the product manager. The product manager will tell you the specific task. He will introduce you." Well, that is, this first introductory character introduces you to everyone and says: "Well, you go to the product manager and start with him. You go to him and say: "What are we developing today?" He writes you, accordingly, the technical specification. Well, roughly speaking. And moreover, we specifically made him talk like a product manager. If you just talk to him like a conversation, he might miss some important details. That is, you need to talk to him in a certain way to get everything right from him. After that, you either go to the solution architect and start discussing with him, how to do it, how to break it down into tasks, how to write the specification. Or you go to DevOps first to set up the invite. He explains to you where to download Docker, which contains the entire environment, which already has a database, in which everything is configured. All that's left is to write the code you need to write. So. But one way or another, you always return to the solution architect in the end, when you have gathered all the information, you understand what the client's database looks like, which needs to be involved in this task. So that it's clear, you know, what level of task we are solving. The product manager says: "I sold our clients a cool service called a prediction service." This prediction service is very unique. You ask why it's unique? He says it's unique because it gives unique predictions based on a person's personality, using their personality profile. You say: "Profile, where's the profile?" He says: "The profile is in the database, it's with DevOps, go to him." And accordingly, you need to get from DevOps, you know, he helps you download all this. There you connect to this Docker, go into the database, see what this profile looks like, how the database is structured. All this, you understand, goes to the solution architect, you start discussing it with him, he asks clarifying questions, you sometimes return to the product manager to ask something again, you gather everything. Thanks to this, firstly, you don't feel so depressed, like, I don't know, well, that is, it's cool, you're playing, and we made the characters cool. That is, I >> DevOps curses periodically, right. Well, that is, everything is like in real life. >> Can I clarify? >> Yes. You, I still, you're telling me, I haven't understood, are these real people or are these chat bots talking like this? When you say: "Play." Is it >> No, it was played. One person played with six bots. >> Ah, with bots. Everything, I understood. Because I didn't quite understand, it sounds so real, as if these are real people. Because I understand when it's passive, and when you have active, they ask you questions or act in some way to involve you, that requires writing logic itself. All these LLMs are passive. >> No, no, no, there are a lot of system prompts. That is, each character, again, we have single responsibility everywhere, each character has its own system prompt, which, as it were, knows everything about what it should know, and its character, accordingly, and its knowledge, and its, you know, focus, how it works. Moreover, we had to work quite a bit to make the characters refuse to do what they shouldn't do. For example, if you, my daughter, for example, tried to make DevOps write code, because she didn't understand very well. Well, a 9-year-old child, she says: "Ah, I understood, okay, then write this code." He's like: "No way, what are you, he says, burdening me? Go," he says, "you have a backend developer there." She went to the backend developer and said: "Here, I need to write a prediction service." He says: "Is there a specification?" She says: "What is it?" He says: "Well, it's a description of tasks." She says: "So why can't you, I'll tell you, and you write it." He says: "No, I don't work without a specification. I can write it, but I'll write it poorly. The solution architect is tailored for this. Go to Martin, he'll write it." So, in the end, she always got to him, she discussed it with him, and he doesn't let her go until he has questions. He says: "No, I can't write a specification. Clarify, can we use this technology for this, are there any restrictions, or can we, or is it unclear how the UI should look?" He asks, asks, asks the child questions, because it's at a child's level, you can play. She answered everything. In the end, he writes her a specification for the backend developer. She copies and pastes it, takes it to the conversation where she is with the backend developer. He says according to this specification: "Do you want me to write all the code in vibe-code, or do you want it piece by piece?" I prefer the style when it's piece by piece. He writes, like, the first class, she inserts it into her development environment. He writes: "The second class," she inserts it. And that's how you go through it all, you understand? Something you don't. You understand, like, it's not too complicated a task, so the child managed. It needs a more complex task. An experienced person should sit and review these code snippets, correct them, clarify them for him, so that he remembers how they were corrected, so that the next snippets fit together. But in the end, she managed and built the application herself. And the next version of this game, we will also put agents behind it, and then it will look like this. That is, you communicate, that is, you communicate with the product manager, he tells you: "We need to create a prediction service." You say: "And what will the UI look like?" He says: "Well, there should be a text field for a person to enter their name, a button, you say: "Can you give me a cap of what it will look like?" He says: "No problem." He pulls an agent behind him. That draws a cap for him, that says to him: "Here you go." Then you come, I don't know, to DevOps, he tells you: "Well, to run the environment, you need to download from this link, then install this, install this." You say: "Ah, well, everything is clear, do it." And he does it for you. That is, he first discusses with you what needs to be done. When you understand that it's correct, everything is clear, good. Or the same with the backend developer. He wrote some class for you, you reviewed it, discussed it, say: "Yes, well written, let's apply it, put it in the project." He himself goes, where he needs to record it for you. And then it will also be by voice. Imagine, you're sitting there, drinking coffee, talking to these virtual guys, moving between them. Then we'll also do this, you know, I already see how it will look. You'll talk to the solution architect, listen, why don't you go yourself to the backend developer, to the product manager, and clarify with him that you're doing all this through me. And there you will teach them to communicate with each other, and you will only control the entire process from above. I call this product engineering. And I call this whole style GDD, because it's game-driven development. We, as it were, play a game with characters, and in the end, we get a product. >> I have a technical question that I don't understand. It's one thing for them to answer, another thing is when you say, he starts asking questions, how do you technically implement this logic? That is, how do you understand that now it's time to start asking questions and when to stop? Because it doesn't seem obvious. Well, you say, the developer starts asking clarifying questions. That is, I understand when she writes, he just answers. >> You mean, how did we write such a system prompt that asks questions correctly until it understands that it understood everything? No, I'm more about the fact that you yourself have the logic, that is, the logic of the chat is like "We ask GPT something," it answers. It's just an API, right, question-answer. But when you yourself, that is, from that side, a live person asks you, it doesn't work like code itself. That is, you must have logic that some code should ask the LLM, like, tell me what questions you want to ask, pass them to you. That is, this is a separate code. It can't ask a question in a chat via API itself. Well, it's a whole >> yes, yes, tell me, please, just interesting. That is, yes, if I understand correctly, it's roughly like this, if you ask a developer something that triggers him, that is, some internal trigger, oh, then it's time to start asking questions, does it happen the same way? That is, it happens after some question. >> No, they just have an instruction written that every time, in fact, it's even trickier. That is, there is the agent itself, well, that is, the character who communicates with you, and there is another one above him who checks if he has enough information to complete the task he was asked for. And if not, you understand, that's how it works. Yes, yes, yes. That's what I mean. Yes. >> Gave a task. He, he, he, precisely, you know, like a very smart person, very experienced, who will not start the technical specification until he fully sees that your entire specification covers everything he wants to know. And he will ask you questions until he gets the full picture. And then, for example, the Solution Architect, I can send you a link to a video where my daughter plays this game, and you will see how the solution architect asks questions and she's already like: "Okay, now write, come on, come on, work." He's like: "I can't, I need to understand what the profile looks like. I can't write a specification, because the backend developer won't understand how to look at how the table is structured in the database. I need this information." The product manager must clarify whether we have the right to use a third-party service in this development, which, well, and you understand, he asks all these questions until he calms down and starts writing the specification. >> Listen, this is actually interesting. Such logic, after all, requires, well, really not only models, but also triggers, right, that if there is not enough information, we initiate a conversation from our side in that direction, and you say it's a whole platform, that is, you actually have a company, I don't know, was it your idea, someone's idea, it's a lot of money. That is, you are actually allocating resources to create such a thing. Tell me something interesting. >> Listen, well, we do it mainly through volunteers, you understand, when the company is large. One of the things that a company can, and probably should, do is to use the free time of willing people who want to spend their free time to do something cool, improve themselves, communicate with other people who, well, you know, people develop themselves somehow. That is, you sit, for example, on a boring project, you have time left. Sometimes during the project you have time left, sometimes you just have free time. And from such volunteers who work in our company, I form teams, and we, well, people leave periodically, but we still have several people who came from the very beginning and are still writing, and they are still writing. >> Generally, what you're saying, this toy, you know what it's close to? It's close to simulators, trainers, which we dream of creating. And every time when I'm asked about, well, training, we have, for example, a chat that is built into the platform. You go through a course, right, you have, for example, you launch practice, you have some output, the context is known, but it's just a chat, that is, it's passive. You ask a question, it answers. And when they ask me about this or we discuss it somewhere, I always say: "Listen, guys, replacement of teachers, replacement of this kind of thing will only be possible when you have a simple way to create active agents, when the system itself will lead you, recognize you at some level, and so on. And I haven't formed this vision in my head yet. That is, for me, it seems too complicated. I don't mean technically to implement it. Technically, we can figure out how a person asks something, but so that it doesn't look stupid. Understand? >> I approach, sit down at the computer, say, I call a tutor or my trainer, who remotely helps me, you know. He immediately says: "How are you? Mood? Understood? Saw? Talked about cats, weather. How are you today? Ready, let's do some exercises and go, right, and do something." And now we assume that this has been replaced. Not your case is similar in this sense, right, that there is an active person from that side, that is, an agent, >> an assistant. I still don't understand, I just don't really understand how it can be done so that you don't feel like it's some stupid game. Not in the sense of your game. But I mean, when you seriously think that a remote teacher is going to teach you something, say, and you won't feel like it's asking some nonsense, like: "Why should I talk to this stupid machine, I'd rather go to a normal person." >> I'll tell you a story. A year ago, I went to school before the start of the academic year for my daughter and asked the math teacher and the science teacher, I don't know how >> to give me all the teaching materials. Then I made two custom GPTs. I uploaded all the Science materials to one custom GPT, and to the other I uploaded. Then, you know, I wrote one system prompt with my daughter so that she understood the principle of how system prompts are written, how characters are created. I asked her questions, I said: "Do you want him to be funny? Do you want him to chat? Do you want him to say this?" And, as it were, we configured it. Moreover, again, you know, how system prompts are written today? You, I have a character with whom I discuss how to write a system prompt. He writes it for me at the end. So, you know, we discussed it there. I, he says, ah, clarify with your daughter, this and that. That's how we created it. The second one she created herself. So she came up with names for them, and she, for the whole year, that is, we finished the Science curriculum with this character in half a year. Then she took an exam at the end of the year and says: "Oh, Dad, it's so cool, they taught this participation." She says, most of them couldn't pronounce these terms, but I already knew all this half a year ago. And you should have seen how this character explained it to her. It was just a song. Well, we tried with this system prompt, but he, you know, he tells her: "I understand that you don't." He explained photosynthesis to her. He says: "What TV shows do you like?" She says: "I like the TV show Kitchen." He says: "Let me explain to you how photosynthesis works using the example of the Green Chef restaurant." So, imagine chlorophyll. These are like chefs in green hats who, from molecules, and so on, he explained the processes by analogy in the restaurant and, you know, in the TV show, what happened there, matched the characters, came up with some images for them, and the child, you know, all this. That's why you say: "We are adults, we are used to learning differently today. Well, maybe I don't fully agree with my former boss, who shouts that everything, we will work on the matrix. But I agree that the new generation will already learn completely differently. There will be a completely different education system, I think. >> Well, what you said, I still don't see an active story here. You so easily say phrases like he asked, but he doesn't ask, he only acts according to your instructions. For example, how will he check? That is, I mean, a person who writes to you directly once a week, say, like a task that we agreed on, did you do it, no, did you do it, no, let's agree to discuss it. That is, you can't just use Open GPT. Make a bot that doesn't. So it's not one. You will have a whole system with a lot of agents, there will be some schedulers that will go through the database, look, oh, this one hasn't confirmed, apparently he's not on this topic, this one hasn't submitted work, so we need to contact him. And then some Telegram bots or some Teams bots start communicating with him, finding out. >> Let me explain. If this had really happened, it would have been a revolution in education, and all online education would have gone down the drain, because then the cost of education would have sharply decreased, you understand, right? Because you have an alternative that is much cheaper. I'm not saying it's impossible, I'm just saying that for now, what you're talking about, specifically such that it's usable, and not formal, when you, you know, really feel that there's still a wooden machine on the other side that periodically asks this nonsense or leads you astray. I'm always interested, will it happen or not. My personal point is that it's not a fact. That is, such a more or less wooden one, which is just like a program, that you, I don't know, you're learning English, it asks you once a week: "Did you learn 10 new words?" Yes, here's another one, yes. But something like an adaptive thing, I don't believe it yet. I'm not saying it's impossible, but I'm playing the skeptic a bit in this regard. Well, have you seen any ready-made systems that are like this, that are really behind you, well, very active, and not that you tell it: "Today I'm studying photosynthesis." It says: "Oh, then I'll tell you." But one that itself says: "Hello, Petya, today you were supposed to study photosynthesis. Did you do the previous assignment?" Yes. How's the mood? >> No, you can make such ones. How good they will be and whether you will be completely satisfied with them is a question, and you need to work on it and improve it. But I am actually trying to do exactly this kind of thing with a local LLM. That is, my idea is as follows. And this is precisely the concept of new education. So, instead of recording some next course, I am creating my next virtual self, who knows everything about this course, a Knowledge Base is offered to him, you know, some GitHub, and so on. And then I give this thing to people. That is, people don't watch YouTube, don't, you know, go to lectures, they communicate with a character who knows absolutely everything he is supposed to teach these people, you understand? And he will teach them at their level, in their language. If they want to communicate with him in English, they will do it in English, if they want to do it in Russian, they will do it in Russian. If they want to skip a topic, he will say: "Wait, do you want to skip the topic? I'll just ask you a couple of questions to check that you haven't missed anything here." You understand? And if you can do something like this, imagine what it will be like. That is, you buy a kit, you are simply given a virtual character, like, I want the third course on economics. Here is the third course on economics. His name is Leonid Petrovich. He teaches the third course on economics brilliantly. And then, when you turn him on for the first time, he communicates with you to customize himself. He has, you know, a rack behind him. He has some table that talks about some things that become part of his personalization. There is not one system prompt, you understand? That is, you can't make a proactive one, but you can make one who will >> in each interaction analyze whether it's necessary to call another character, whether it's necessary to switch this character to another mode. That is, perhaps he has several system prompts, how he will work depending on the topic he is currently entering, you understand? And this is actually a terribly interesting field. And one of the things we are doing at work now, we are trying with our LND, that is, EPAM, which built its entire company, starting from student training, that is, most of the people who, as it were, came and became powerful engineers, that is, there are so many people, there were so many courses written, and now they want to translate all these courses into some kind of gamification, make them something like what I'm talking about, well, start with some elements, right, so there's a course, and then instead of homework, they tell you: "And now play this game. In this game, you need to sell a pen for more than a dollar. Try to Sell Me the Pen, you know, like in the movie. And you have, you know, we actually have such a game where a new client is generated for you every time. Therefore, you can't play the same way and win. You always have a new client. Sometimes an old lady will come, sometimes a teenager, sometimes a university professor, sometimes a poet, a cook has come a couple of times. They always have different characters and different needs. You need to, they all have one single common trait. They all believe that a pen cannot cost more than a dollar. Well, what is it? It's like, God, you write with it and throw it away. And in order to change this paradigm in their head, this concept, that it cannot be, you need to do it in a certain way, which we want to teach you to act. That is, this is a course for people who sell solutions. We have, as it were, a technical sales department. And in order to teach these people, we made this game for them, where they learn the first level. So, for >> Because, you know, it's like, you're selling a pen, and you're selling it for a dollar, and you're trying to convince them that it's worth more than a dollar. And you need to find an approach to each client. And for example, if you have a client who is a poet, you need to talk to him about inspiration, about how the pen helps him write poetry, and so on. And if you have a teenager, you need to talk to him about how the pen will help him pass exams, or how it will be a cool gift for his girlfriend. And so on. And this is, as it were, the first level of training for people who sell solutions.

To sell even a pen for $3 to some grandmother, for example, you first need to extract all her needs. That is, what hurts her, how she lives, what her problems are, everything connected to it. For example, she says: "Son, I can't see well." You think: "Okay, so, we need, so, let's write down, she'll need bright ink." And you build a concept proposal from all her needs. That is, you can invent any pen. The main thing is that it should be profitable. Even if, I don't know, even a pen with a built-in search device, because grandmother always loses it, and it has a photo of her grandson, which is very important, and it also heats up and warms her hand. And so, you can invent any nonsense, but the main thing is that this nonsense is invented from features that will definitely please the client you just spoke with. This is not a game about charisma. That is, it's not about, "Now let's see how well I can persuade." You won't be able to sell anything to anyone if you don't act exactly as we need. And this, accordingly, as you correctly said, turns out to be a simulator where you can practice. But we are looking, as it were, not only, we are also trying to go beyond standard training and invent truly, how people can go through some courses completely on their own.

>> I have something to say about sales because we are also involved in that. In short, there are two main directions in sales. Ah, and it seems that one of them is already winning. So, the first direction is the one related to training, right? That is, we take ready-made scripts, dialogues that existed, analyze them, create bots based on them, and play out sales scenarios, right, where you have a client, you respond to them and persuade them. This is the first option. But in reality, I, by the way, initially thought that systems, CRM systems, would develop in this direction. And, well, as you understand, there are now a huge number of companies that build their business on this, either CRM systems themselves or side ones that offer plugins or solutions for improving sales. But in reality, it turns out that the main focus in sales is slightly different. That is, now CRM systems are developing in the following direction. That is, roughly speaking, during a conversation, there is a real-time analysis of what the person is saying, and based on a profile, some basic qualification that needs to be done very quickly at the beginning, and all the collected information, user paths across all your sites, because all information is also collected there, it essentially tells you what to do next. And when I also asked people about where everything would go, what is more effective, they say that this method is a thousand times more effective, that is, simply, that it is much more, you understand, direct and blunt from the point of view of how the CRM works. You don't need this tricky agent on the side who is also trying to be active. That is, you simply, based on what he said, based on all the information about him, just say the next step that can be taken. And I am very much looking forward to when, for example, our CRM or tools around it, because we won't do it ourselves, they will implement it. But I know that there are American startups that have gone very, very much in this direction. So it will be interesting to see. By the way, this is very similar to projects that are currently being used in the States and have made a lot of noise about passing interviews, when, remember, a program appears, you are asked a question, and it also writes code for you. This is really the same thing, essentially, just in a slightly different area. And it's as if all directions are going in this direction. And training here, of course, is a slightly different story. Ah, but yes, our partners told us a couple of years ago that they had made something like this for sales, which generates, well, again, not out of thin air, but based on a large volume of viewed, recorded conversations. They generate these characters, then you talk to them, and it prompts you, draws conclusions on the topic of what you said correctly, incorrectly, and so on. But in its pure form, the games you're talking about, exactly where it's a game with all sorts of things, I think it's a rather experimental format, because proving its value for sales, for example, by the end of this year.

>> We will probably have a pilot project now. If I can win B Defense next week, I might get a production team, I'll give them the game, and we'll see if they can develop some real production product by the end of the year by playing this game according to my rules, you understand? So if it works out, then >> I'll come and say: "You were wrong." Or maybe I'll come and say: >> "I'm looking at this from a purely business perspective. So, whatever works, we grab it. So, we're more like just observing from the side to understand which mechanics will work." Listen, and these games and their creation, for example, we have a task to create simulators. And I recently, literally, saw a similar story when in Telegram, a chat is created, some guy made it, I don't remember who and I don't remember, well, how developed this whole story is, but I definitely remember, I tried to do it, you connect to the chat, you have a conversation there, some five bots, they say: "Hello, Vasya, you have come to work." And I think: "Okay, why hasn't anyone made a platform for generating these things yet?" How much do you think your system >> hasn't made it? Yes, there are plenty of them already. Mine. >> A platform. >> My daughter will come in about 20 minutes, I can call her, she'll tell you. She constantly plays with various, you know, like DND games. They have some periodic characters appear, she talks to them, she has to defeat them, she has to convince them. So there are a lot of bots there and, for example, you have to talk to a character in the game for a minute, and you have to figure out in a minute if it's a person or real. >> Well, >> you understand, and he, accordingly, about you >> and so on. Well, >> well, that's a bit different. I meant more specific, for example, for programming, where you can also write code, and everything else, because these systems are unlikely to allow that. >> Well, no, that's true, that's more about talking. Listen, why did this game theme attract you so much? Because I initially thought, you have AI in general, but it seems like you see more in this than just >> Listen, well, our whole life is a game. >> I want life to be fun. I think that, well, we just need to define what everyone means by the word game. You can say, game, >> well, gamification, you mean that, right? >> Uh, well, but it doesn't sound as serious as if I tell you mind games, you understand? Simply for me, if I say: "I'm working now," it associates me with a person who is busy, feels bad, wants to finish, it's hard for him. But when I say: "I'm playing now," it means, well, it's easy for me, I feel good. How should I call it? >> Uh-huh. I understood you, I remembered. There's one thing I wanted to ask you. So, I had Egor Bugaenko on one of the podcasts, very, by the way, discussed and popular. Well, elegant objects, all that. Well, the most important thing he said. >> He's still around. Cool. Where is he now? >> Huawei, everything continues there. >> He doesn't live in any country? >> Ah, I think I don't remember. Did I ask him? No. I don't remember. I won't lie. >> I don't remember. He just said something about AI. >> It was definitely him. I just heard he went into politics. >> No, no, it was Egor. It was definitely him. It's a controversial thing, because, naturally, he says many things that different people react to differently. Of course, there >> And what is he advocating for now? I'm interested. Hasn't much changed in terms of elegant objects, OOP, but at the same time >> Seriously? That's what still interests him? >> Well, at least he talked about it, because, roughly speaking, I invited him, you know, when I saw his performance for students on YouTube. I did a review of that video. Well, and among other things, I invited him to talk about the object world in general, about how he sees it, about the language he is developing, among other things, and so on. So, yes, he believes that the things he sees are relevant, and at the same time there are problems with performance, well, and so on. So, I'm not really about that now, I'm more about the fact that AI has already raised hopes that he will say something interesting about these new modern approaches, maybe he has some revolutionary new approach. So he is a big provocateur, but at the same time he sometimes has very interesting thoughts. >> Yes, he said a lot of different things. And so, from his point of view, he said something very clear about AI. He said that now in the world, God, I might be wrong, but it's like 30 million programmers, in 5 years there will be plus or minus 3 million, and we will all be architects who tell AI what to do. How much do you agree with that or not? >> I agree with the exact opposite. Well, as usual, if it's a matter of principle, I just really think the opposite. But perhaps we are saying the same thing, but with different wording. So, let's, my vision of things. So, what is a programmer? A programmer is a person who can, he is a translator, he can translate a person's desire, maybe his own, to do something, into a machine, into an instruction that a robot understands. So, at the beginning, we won't talk about the times when there were no compilers and people wrote in zeros and ones. That's, well, not only is it not quite programming, but it's a completely different reality, right? It's all covered in dust. Any programming language implies that you don't explain anything directly to the machine. You explain it at exactly the level at which the compiler can then translate it for the machine language. That is, a person is a kind of intermediary between human language and the compiler, and then, as it were, machine code. Each subsequent level of languages has made it so that there are more and more programmers. Who could write in machine code, who could write in assembler? You understand that to write in assembler, there are many fewer people willing than to write in Pascal. And there are fewer people willing to write in Pascal than in C. And there are fewer people willing to write in C than in C#. And in Kotlin, there are even more people willing, right? So, the more high-level the language, the more enjoyable it is to write in. Why? Because the language is closer to human. We express thoughts with our concepts. Object-oriented languages were invented, I suspect, because, probably, people, well, why did they last so long? Because people really think in objects. Victor Polyushchuk and I always had the question: do we think in objects or do we think in functions? Right? So, there are these functional languages like Scala. But in any case, you understand, how much further it is from machine code, how much closer it is to human thinking, perhaps of a certain type of thinking, but nevertheless, what is happening today, we are moving to another next level. And if you know what, for example, Brislav is developing now, I don't know what language Egor Bugaenko is developing, but Brislav, for example, is developing a language that will essentially be a conversational human language. You can use English, maybe Russian. You just have to explain correctly in a certain format. And the task of this language is to be able to correctly understand that something is missing, some explanation is not clear enough. This is called non-compilable code, you understand? So, before, the compiler's task was to understand what in your instruction is unclear how to translate into machine code. But now you will have a programming language that will be able to check what is unclear in your text, I don't know, I wanted to say diarrhea. Well, in short, what in your verbal constructed sequence is not clear enough or implies some ambiguity, you know, like the compiler tells you, right, you need to put an import, because it can relate here or there, right, ambiguity, I have the same thing here. It's just, well, the compiler is getting smarter, you know, it's like the tip of the iceberg. So, before, people sat at the bottom, then they move up, the compiler does this for them. Then they move up to the level of object-oriented languages, write in some Java, in plain English, you know, my son often came up and said: "Hey, it's written in English here, something like vector store find data." Well, it's readable and understandable, right? And now we've reached the very top of the iceberg. We just need to formulate our thoughts correctly, and someone will pick it up and be able to pass it along the chain. I don't know if LLMs will ever compile directly into machine code. Perhaps that will also come, because, as we found out today, this thing, he also thinks so. It's again a question of the quality and quantity of training, the sample on which you were trained. But what's the difference for an LLM? Generating an image, generating sound, generating text, generating code, or generating machine code directly, or generating bytecode? Perhaps they will generate bytecode directly. Your Java compiler will simply turn into such an LLM thing. Perhaps they will generate zeros and ones directly. I don't know what it will lead to, but we will communicate with this tip of the iceberg. We are with them, we are there. And so, it becomes even more accessible to everyone at every level. That is, the iceberg is narrowing. It's like, the higher we go up the iceberg, the less we have to do. And the number of people is expanding, because the easier it gets, the more in demand it becomes, because everyone is going there, it's needed. And in the end, this is my thought, right? So, my vision is that all people will eventually become programmers, you understand? Because even when my daughter talks to Alexa, well, Alexa is not artificial intelligence yet, but when she starts giving instructions, asking here, saying there, it's already programming in a way, you understand? And grandmothers will be programmers, and grandfathers will be programmers, and even little bees. Everyone will explain something to robots, and robots will do everything for us. Sooner or later, we will definitely be there. You understand, maybe, maybe it's optimistic to say that it has almost happened, maybe it will take another 20 years. But still, it will be there. And if someone doesn't believe it, well, let's show today's Kotlin code to people who wrote in assembler and say: "And did you know that you could write like this in 30-40 years?" But everything is accelerating. Everything is developing faster and faster now. So where will we be in 10 years? Therefore, I don't know how relevant it is to discuss objects and object-oriented languages now. It's the same as, I don't know, in the times when Java appeared and they started discussing inversion of control, talking about new languages in the world of assembler. Well, in my opinion, it's all starting to fade into the background. >> No, but by the way, in this regard, Egor said the same thing. So, his attitude towards objects themselves is like that, but in general, that soon we won't have to write it at all. Yes. >> Well, okay, yes. >> But on the other hand, >> yes, by the way, in this regard, I agree with you that the accessibility of technology always generates even more use, because things that you couldn't even imagine before become cheaper, because people's thinking has changed drastically in the last couple of years, and while we are all hyped up. >> Well, so you shouldn't be afraid. So, this future in terms of: "Oh, they will take all our jobs." Are you kidding me? The more they take our jobs, the more work we will have to command them, those who take our jobs. It's not us who will work for electricity, it's them who will work for us. So, I'm a bit more optimistic. I told my former boss the same thing, I said: "Listen, why did you decide that they will be bad from the start?" Well, you don't understand how they think. They don't think, they just calculate tokens. Maybe, after analyzing the whole world, they will come to the conclusion that people need to be helped, not forced. What, they don't have enough electricity? That a person is the most efficient thing for generating electricity. They will start doing something else, and they will help us instead. Therefore, the world might be very fun and optimistic. We will sit and play games, you understand? They will do the hard, unpleasant work for us. By the way, so that you don't criticize me with this "you're all about games" thing. Let's bet that if you take some peasant >> from the beginning of the 15th century, who, you know, spent his whole life digging a field, and then you bring him a small tractor with air conditioning, put him inside and say: "Well, you don't have to push your goat or your rusty plow anymore." Sit down and go for a ride. We'll even play music for you. He'll say: "Okay, guys, am I playing some game now? I'm going to play." You understand? It's like a computer game, riding around the field on a tractor, you understand? So, well, we'll be there, only in the world of development, we'll be sitting and playing, wearing a virtual helmet or, I don't know, turning on some hologram, telling something, they'll do something for you, everything is great. Well, maybe it will be like that. Or maybe the apocalypse, I don't know, maybe Terminator. Yes, yes, yes. When you listen to guys from science in parallel, right, when they predict that, well, how long we have left, like 100, 200-300 years. And then we Uh-huh. >> Well, for various reasons. There's a series related to the lack of selection and so on, diseases, well, there are various environmental factors. In general, in short, two such parallel worlds that predict either a rollback or some bright future with new technologies. You know what I wanted to say? While you were talking, I had such a feeling. Of course, I didn't live in the eighties. No, I lived, but I was too young. But if we now transfer everything you said there, I'm sure it will sound exactly the same, because those were the times of Prolog's appearance, those were the times of Cobol. And precisely, this was the same ascent, that now we will start writing in English, and Prolog is our story about artificial intelligence, all that stuff. And so, roughly speaking, all the words you say, it's unclear that we are now at a different stage of development, but I just had this feeling that they would have come to the same conclusion, because everyone talked like that. I guarantee that they talked the same way in this regard. >> Did you write in Prolog? >> I'm an Erlangist, and my production is, so Prolog is close to me in this regard, because, naturally, Erlang is rooted in it. >> I read in Prolog. I did five units of Prolog in school and passed the exam. So, we taught it for 2 or 3 years. And honestly, I'll tell you, back then, I wouldn't have said the things you're saying now, I would have said: "Oh God, if this is artificial intelligence, it's a nightmare and horror." So, it's completely different from how it looks today. So >> then, looking at Prolog, on the contrary, one could believe that nothing good should be expected from the future and no one will do anything for us. And we will always have to sit and explain everything to machines down to the last comma, they will nitpick. Or maybe, >> yes, yes. But I think there's a difference. You were just given it as a language, but I mean more its use for creating something like that, simply. >> No, well, understood, understood, understood. Jules Verne predicted everything, basically. >> Well, in that regard, yes. It's generally interesting, because when you look at how it's blowing people's minds now, it's very, very surprising how, it seems, we all wrote code, and then you look, and some went in one direction, others in another, others in a third. In general, for me, it's a very interesting series. And I'm watching everything that's happening around like a movie. >> And if we really have 200-300 years left, then that's good. Look, we won't even have time to get upset before we die. Yes, yes, yes, yes, yes. Well, that is, when you die, you realize that in another 10 years it's the end of the world, then dying is probably sadder than when you, well, we won't even come close. 200-300 years is quite a lot. >> Listen, do you experience apathy, or not apathy, but some attitude towards the knowledge that exists? Well, that is, imagine, how much we've been in development, and you, you know, are an expert in this, understand this, know nuances, subtleties, understand that now your ChatGPT will answer everything, and in general, as if, maybe all this will become irrelevant. No, you don't feel anything about it. What should I learn then? >> Well, listen, for now, on the contrary, in some ways I'm becoming more needed, because, you know, my favorite joke about GPT is, well, it's like a meme from one or two frames, you know, a person after surgery looks, he has a big ugly scar on his left side of his stomach. You know, he says: "Hey, but the appetite is on the right side." And GPT says: "Your sharp eye has detected an error. I'll fix everything now, but on the correct side." You understand? So, for now, it's like that. Uh, on the contrary, being an expert is very good. With juniors now, you understand, like, juniors are not very needed, because they haven't had time to turn into some other kind of people that they should be today. An expert, on the contrary, is good, because experts are those people who today will look at what GPT generates and say: "Take this, don't take this." So, let's try this: take any language you don't know at all, some completely new concept, and try to write a serious project in it using GPT, and then show it to an expert. He'll surely grab his head, you understand? >> I disagree. Do you know why? Still, well, again, when you know many languages, have tried many, and, for example, you do it within the paradigm you are used to. Let's take, you know any backend framework. >> What does your wife do? >> Children and sports. >> Try to give it to her with the latest version of Claude and ask her to write some complex project in Lang, or in something. >> No, that's different, yes, that's different. So, it's clear that if you understand backend, well, mobile, for example, if I'm asked to write something on mobile, I've never written mobile code, most likely there will never be confidence internally, whether it's good or bad. You understand, today we need something, someone came, looked, said: "No, yes, this is really good code, everything is correct here." >> Although, wait, what? How is it good? >> Well, by the way, more and more of these conversations are like, it doesn't matter at all. I said at the very beginning, I'll repeat, I have this clear, you know, understanding of the internal layers of abstraction. So, for example, I understand that how I need some component, that it's isolated, it doesn't leak outwards, it's input-output, that's all. And in that case, my internal barrier drops very sharply, like, okay, here I can do some quick coding, give me any solution, it doesn't touch my database. Well, performance, if anything, we'll fix it, it's rarely a problem. But if I understand that it's part of some connecting system, well, like just inserting code, even if it works, hmm, I'll still look at what it did. >> You know, what I started laughing at now? I suddenly imagined a situation. Well, surely this will be a disease for many people, but it can be illustrated as follows. A person bought, you know, an expensive Bentley, and then he opens the hood, takes it and says: "My God, how beautiful it all is here." Okay, wait. He opens the engine cover, opens it and says: "Oh, how they packed the wires. Damn, there are wireless spark plugs here." How cool it's made, you understand? Well, uh, actually, at some point, probably, we're not very interested in what's happening under the hood. We're not horrified by how the machine code looks, into which our, of course, Java turns. Yes, we don't even look at the bytecode and say: "Oh, the bytecode looks a bit unappealing. How come it's not fan-like here? We need to refactor it somehow." We don't care what's under the hood. >> Well, you know, I probably can't agree on some level here. So, for example, you have a large system, right, and you have, well, certain subsystems appear within it, specifically through-and-through, for example, how you solve a particular task, and it's a certain abstraction, right? And in this regard, we've discussed it with the guys, everyone converges to the same story, probably, that ChatGPT is very good at repeating, but, for example, seeing some pattern across the entire project and generalizing it, it certainly can't. And, accordingly, yes, if you are constantly in copy-paste mode, no questions asked, everything will work out for you, but then you won't be able to work with that code in any other way. For example, your entire system. Do you think it can't look at some code and see some pattern in it? >> If you don't explicitly ask it to. Almost certainly not. And imagine, well, let's take at least a minimally hundreds of thousands of lines of code project. And if you ask it, and this is a question of qualification and understanding of that code. So, you're already looking and thinking. Well, let me give you an example. A very standard example for me. For example, we have a system, well, everyone has a notification sending system, and you can send notifications in five different ways, for example. There's also configuration, for example, to display it here, send an email, send an SMS, something else. And you can't just use, I don't know, some SMS gateway directly. Well, because it's wrong. You know why yourself. Yes, no need to explain. You also have configuration, if you don't think about it right away and you don't have experience, and you understand that, oh, this is a subsystem, polymorphism is 100% needed there, and so on, all sorts of different things, substitution, I don't know, it didn't work in tests. Well. And if you just ask it stupidly, you know yourself, it will say: "So, well, I found a library for working with SMS, for working with, I don't know, emails, it will just shove it in for you." Well, and you know yourself what a mess it will be. And then it's in five places, and then you have to check in each place whether to send an SMS or not. I don't understand how ChatGPT will solve this task for you if you, for example, well, 1,300 lines of code, well, even 100,000 lines of code. Well, it still doesn't look like a solution to me. And every time I ask, they tell me: "Well, Kirill, after all, I'm just not a big fan of your entire project being managed by, you know, like, I just don't work that way, that the entire project is managed by, say, ChatGPT or some other system. I just don't know for sure that it will see it and say: "Oh, listen, dude, here you have this subsystem going through the entire project, let's generalize it." Listen, and how do you think the problems that happen with these systems, subsystems, how many of these problems are unique? Like, completely unique, unlike anything else. >> The thing is, it learns, you know, there's a joke, I'll tell you now. >> No, I'm not asking about learning. Answer the question. I just had a thought on how to solve this. >> I think that there are simply so many evolutionary points on the internet that it just, well, it can't come to that. For example, you might have something used a lot, but, for example, it's obvious that you won't use it directly. And on the internet, you only have examples like: README, Stack Overflow, and so on. And inside, you need to make it into your own subsystem, which, for example, deals with notifications. Yes, you might have, I don't know, in Spring Boot, Spring notification. Well, for example, and then it will prompt you, but you might have a framework that doesn't have such a thing. And at the same time, there are, for example, several ready-made solutions. For example, we recently deployed our project, so we had our own event system, specifically domain events, when you have registration, events, and you have sending to the BI system for this, you understand? Yes, this is a classic abstraction that is needed so that you don't scatter sending to analytical systems or anything else throughout the code. And, well, our project is Rails, for example. And I, for example, follow new packages, I see that a couple of years ago guys implemented and developed it very well. It's becoming popular now, a system for these events. I understand that it will replace our manual work, it's implemented better, and it's supported by the community, you understand? It will never offer me something like that in my life. And I come to my developers, I say: "Guys, here's something." They say: "Hmm, interesting." I say: "Well, let's implement it, let's do it." And gradually we do it. Telling it: "Go implement it yourself." doesn't work. Well, and you can give a billion examples like this. For example, we are now switching from Bootstrap to Man UI Framework, but you can't just tell it: "Rewrite everything for me." I tried, it doesn't work like that. Maybe we're just looking in the wrong place, because, well, humanity has a huge amount of global experience in some architectural, pattern solutions, right? And there are many unique situations when some languages that usually don't interact, interact, or something is brought from somewhere, moved. If we are at the level where we have these crutches growing here and there, then yes, it will be very difficult for LLMs to find similar patterns, because the more detailed you look, the more unique they are. But if we say: "And why do we need all this? Let's write more globally." So, the main thing for us is that the task is solved. And initially, for example, the project is built according to a certain, so, in a little while, there will probably be a certain new type of base, based on which, perhaps, it will be possible to bypass all these problems. So, they will simply become irrelevant. >> Well, that's if you go to that level, and yes, we discussed this, that then it only works under one condition, you completely stop looking, as you say, at machine code, but we don't look at it, and we're not interested in it. So, you're at a very high level, but you run into the problem we encountered from the very beginning. It will fix you, but at the same time break half of something else. And here I want to say something about cause and effect. So, I tried to use Cursor. So, I have TypeScript, it seems like you have types, everything is fine, right? I ask it to change a very small piece. It changes it for me. And it even, by the way, well done, it saw that I have a command to run type checking. It runs it. It tells me: "10 type errors found." And Cursor, it's smart, it says: "Okay, let's start fixing." It, then, enters this cycle again. It says: "Type, five errors fixed, run check again." It says: "30 errors found." And it's in this cycle 20 times. I actually went to drink tea. It has a limit, like 20 times. And after 20 times it says: "I'm stopping, because that's it, I can't anymore." And it didn't fix it. I started just watching what it was doing. Well, the main problem, you know, is what? It doesn't understand the cause-and-effect relationship. It just tries, in the place where the wrong type appears, to adjust the type. Well, you understand, right, how it all works. So, do you think I'm making the wrong prompt or what? Because it seems to me that this is a fundamental problem for now. >> I think it's possibly partly Cursor's problem. Try it with Juni. Juni, I think, is more correctly architected. >> You think it will understand the essence, where such a type could come from? I think >> that it just blindly picks something so that it passes. >> I think Juni might have managed with this. You know, actually, there are really different solutions, different models can, well, some models you say: "God, it can't do this." But another can do something very complex. A classic example. You know, this riddle that any person can solve. And I checked GPT recently. GPT doesn't solve it. >> Oh, >> Lena has three sisters and five brothers. How many sisters does Lena's brother have? Once again. Lena's brother, >> yes, I understand, right? This formulation, right. Lena's brother has three sisters and five brothers. How many sisters does one of Lena's brothers have? >> Four sisters. >> Well, if she has three sisters, then there are four of them in total. Yes. Well >> yes. And >> it's scary to answer such questions every time, because you think, maybe I'm stupid. >> One second, let's see who's who here. >> Who is stupid, actually, right? >> Solve the riddle. Lena has three sisters and five brothers. How many sisters? One of Lena's brothers. >> Well, if Lena has three sisters, then each of her brothers also has three sisters, including Lena. So the answer is three sisters. >> Amazing. >> Did you hear that? It's normal now. >> Uh-huh. >> No matter how you struggle with it, it can't solve such a simple task. But ask Claude the same thing, and Claude solves it. >> M, >> you understand? >> Listen, there's really such a big difference. I just pay for ChatGPT, and to pay another twenty for Claude, because it doesn't work for free, I think. It makes me stingy. So, the difference in level is so cosmic that it makes sense. >> Well, depending on what. Uh, I still find GPT more convenient, because their voice interface is much better made. >> It recognizes voice much better. Uh, but if, for example, you need, well, when we were making this character with my daughter who taught math, we came to the conclusion that GPT is okay with math, but geometry is not at all. But Claude is awesome at drawing, so, a triangle, it has a triangle, if you need to draw a triangle, so that the side ratios are observed, it has the side ratios observed. So, for more serious, precise tasks, Claude is much better suited. Listen, you know, what I suddenly remembered now? Remember, even before ChatGPT appeared, RAM Alpha. A language and a whole system for solving mathematical problems, where you could, >> Did it die after that? How much did ChatGPT influence this system? >> Honestly, I don't know. They don't use it somewhere inside or some of their things, honestly, they >> I just think that this thing definitely undermined their business, because you can get answers to many of these questions just with ChatGPT and not go to a specialized system for solving math problems. Hmm, interesting, Zhenya, this turned out to be funny. We, like: "Well, shall we talk about Spring? Two words at the beginning and that's it." >> And do you remember, you said, remember? I knew it would turn out this way, so I told you: "Never give a book a title until you've finished writing it." >> Yes. I think, guys, if the questions we've raised, the difficulties you're facing resonate with you, or if you, like Zhenya, believe more positively that this system is ultimately capable of solving all problems, including everything I've said, be sure to write about it. We are very interested to read your experience, because the experience of people who are currently going through all this is super-super interesting. Everyone's experience is a little different. Zhenya, thank you very much for coming. >> I have only one request, a question, tell me, I would really like to take advantage of the opportunity and say again that I have written an awesome course, I finished it last night, so, I hope next week I will upload it somewhere. And, uh, I can give a link to my Telegram channel, if you, >> yes, we always attach it, yes, if you have something like that, we always attach it. So >> then in this Telegram channel. Uh, well, first of all, I periodically, I don't post very often, but I try to post something cool. Well, and plus, as soon as there's a link from where you can get my course, >> it will appear there. >> We'll do that. >> Thank you too. It was, I think, a very cool conversation. >> We're wrapping up. [Music]