Transcription
To those who were at the conference, as Kolya said, part of the slides and part of the content will already be familiar. This is a more expanded version of that same report. And, actually, there will be a bit more about cases, there will be time for questions. So let's start. And today we will talk about how the world has changed from the perspective of an AI systems developer. This is more of a technical report, but it will also be useful for founders and everyone who is building or interested in building systems. A couple of words about me. I am the co-founder of Death Stark and Spread Simple. We build AI-first products. I have been in development for 12 years, even more if you count commercial development experience, for 3 years. I am involved in AI systems. My team and I deliver production systems for enterprise and startups. And, perhaps, the most interesting thing is that a lot of projects pass through me, I have a lot of exposure, and I have had the opportunity to see many of them from the inside, including large projects, in addition to those that we develop ourselves. I write about AI engineering. There are links here. So, agents, a rather obvious trend. In fact, quite a lot of people have talked about this. I will only say that this trend will intensify. I will not dwell here on the fact that this trend is real and how it affects business and development. I will only say that according to the major analytical agency Gartner, this trend will only intensify. And I believe this forecast. And how has agent infrastructure changed over the year? On this slide, I wanted to show how agent infrastructure has been strengthened specifically for developers. There were quite a few events. I have included only the big three: OpenAI, Google, and Anthropic. And even these are not all releases, but key ones. And note that what we are used to appeared very recently. The Response API, it seems, has been with us forever, but it only appeared in March 2025, that is, less than a year ago. MCP was a little over a year ago. Codex SDK is practically recent, yes, after, significantly later, Anthropic's agent SDK, which we will talk about today. And all this is to say that the Completion API has become much more agent-like, and the API is becoming more and more agent-like. Here is a screenshot from OpenAI documentation, how they position the completion, or rather, the response API compared to the completion API. Many of you know completion? That is, essentially, it's what we are used to. You send an array of messages, you get a result, right, that is, completion. And unexpectedly, I think for OpenAI, this became the standard for the entire industry. Almost all providers picked it up very early. And even those who do not support it directly, for example, Google, offer a compatible API. Not to mention that practically all open-source solutions and Chinese LLMs support completion APIs. In this regard, it is a great standard, it gives us interoperability, models can be interchanged. And everything was great, right, and continues to be. But as we will see further, there are a lot of new features. For example, grounding, right? That is, when we see in feature releases that the model has gained the ability to search for something, a content interpreter has appeared, context caching, editing, computer use, deep search, and so on. And how much of this is about the model's weight itself, or how much relates to the model itself, right? Practically nothing. These are all infrastructure features. Next, I analyzed a feature table, a rather large table on several slides, you can take it later. And why is this table interesting? The structure is as follows. There is the feature name, links to the documentation of the providers of this big three, and standard alternatives. Of course, very conditionally. Some alternatives are broader, some are narrower, some are not drop-in replacements at all, right, but nevertheless. And the first category is retrieval from the web, essentially it's RAG with web access. That is, with a modern API, which is agent-like, in my opinion, with the big three, you also get the ability to do RAG without any external tools. And this is also helped by search in external sources, that is, web search, web fetch, that is, models, models in quotes, can fetch data, that is, read it, see it, right, and even run entire deep searches. Well, this is not new, but perhaps many did not know that deep search is available as an API directly within Google, Google API. That is, it is not just a product feature, it is a whole API. And a rather unique feature for Google specifically is map routing. That is, it is possible to directly analyze various geo-data, for example, solve the traveling salesman problem and see where the nearest places are, and so on, and so on. All this is available, I repeat, through the API. And here are some alternatives, all these links are clickable, you can see how to do it yourself. Next, documents. Here is a category of features related to documents. For example, the file upload API allows you to upload files to the servers of these providers and do something with them, for example, retrieve from them or reuse a file multiple times, for example, PDF analysis, and so on. That is, to create your own small project, which you might have encountered in ChatGPT or in CODA. And quite advanced document understanding. Again, these are not features of the model itself. Pay attention. It's as if the providers are trying to blur the line between the model and the infrastructure. Although AI engineers, I think, understand that for the most part, this is some kind of, well, very, roughly speaking, an add-on around the models, not the features of these models themselves. Memory state and state, a lot has appeared around state management. For example, OpenAI has conversation state directly in the response API, so you don't have to store the state yourself, you can store messages directly on their infrastructure. You can do advanced context management, for example, relatively recent context editing. By the way, my company started with this release of Context Editing from Anthropic, which essentially we used to do ourselves, that is, we did a kind of clever surgical runtime update of the context, and Anthropic in this case released this feature, which does it much better. And this is a built-in feature in the API. And, of course, all this leads to the fact that you can often save a lot of time if you know the API of a particular provider well, and you can avoid certain risks of vendor lock-in if, again, you know certain practices, which I will discuss later. There is, for example, a memory tool, that is, these are already agent-like features, more and more agent-like features are built directly into the APIs of these providers. I won't even mention prompt chaining, which is, in fact, an infrastructure feature. A huge number of releases related to infrastructure, to tools, have been made in the last quarter, in the second half of last year. Especially related to Anthropic, that is, search, programmatic tools, streaming, a huge number of specialized tools. For example, even OpenAI has a shell tool, which appeared earlier, and special tools for working with code. All this saves a lot of time when used correctly. Programmatic Tools are something in between code execution and regular usage. The infrastructure executes several tools on the provider's servers, aggregates the results, and returns the result itself to the model's context. That is, a huge amount of this mass usage of prompts does not fall into the context window, which is super efficient, but it has its own price, of course, at least the price in the form of cost. Stack-based execution is a super important part that obviously does not relate to the model, but relates to agent infrastructure, but is often presented as a model feature, even in official press releases. This is code execution. And you may have heard that, for example, Opus can work for 5 hours in the background. Come on, this is not Opus as a model, this is the numerous infrastructural add-ons that maintain this agent cycle and constantly perform these completions, right, to give the desired result in the desired direction. And as we see, there are alternatives. That is, one of the reasons why I added this column and why I think this table can be useful in many places is that you can often make a choice for yourself, whether to use a built-in feature. Of course, you will save a lot of time and often get the best result you can get with a limited team. Or still use an alternative. I will talk about this at the very end. How to develop such an intuition, what to use when. Valera will talk a lot about when there are non-functional requirements, right, when, for example, due to legislation or company security policy, you cannot afford it, or simply the founder's paranoia. But that's a separate story. It's not always, let's say, a technical solution. It's a policy, right, that is issued. I'm talking about when you have freedom of choice. So, code execution has also become an API feature relatively recently, that is, the APIs of the big three. All of them allow you to execute model code directly on their infrastructure and use it as part of your workflow, right, as part of your workflow. At OpenAI, it's called interpreter, at Anthropic, code execution, and so on. Alternatives like Python, Docker, of course, have not gone anywhere. Well, computer use can also be done independently, you can use built-in features. In my tests, built-in features almost always significantly outperform all attempts to do it yourself, except for rare exceptions when there is some domain-specific thing, and there you can indeed get better metrics, but this may be a temporary phenomenon. Well, and of course, Skills is a relatively recent release that is being actively adapted, right, a separate standard appeared at the end of the year, Agent Skills, and other frameworks and providers are actively adapting it. Why did I mention it here? Because, in fact, Skills can be used as an API. That is, a skill that you have packaged in your code, in code, you can package it in an archive and use it through the Anthropic API for your scenarios when all you have is a request that you can make to their server. Well, and finally, output, right, output and operations. Well, there is the possibility of batch processing, there is the possibility to configure safety settings, right, it's a whole thing to do safety guards, good guards, and fine-tune them. Google offers this as a built-in feature and says it's a model feature. Google even offers observability, that is, a kind of replacement for tools like Fuse, and so on. And you can see each request in detail in Google Cloud. And much, much more. So the table would have grown even larger. I included the most important things. I think it's clear here, models are catching up, the infrastructure gap is growing. We often see benchmarks where, for example, JLM or DeepSC or from DeepSC, we haven't heard anything for a long time, and Quen, achieve almost parity with the big three. But we don't see this in practice. So why? Precisely because infrastructure is super important, and that's why you choose a stack, not a model. And often this synergy that you can get by using a set of infrastructure tools that comes with the model, you get some unattainable performance. At least, that's the status quo as I see it today. And here's an analogy that I chose, where all this is moving, as it seems to me, right, again, email used to be, well, email is a very old standard, as you know, even older than the WWW. And its own SMTP, its own server used to be the norm. Now imagine that we ourselves deploy our own SMTP for corporate mail, even though there is open-source software for this, and not much GPU is needed at all, generally server resources. It's super difficult because Gmail is a whole ecosystem. Besides the protocol itself, there are a huge number of infrastructural and product features that are hard to imagine without it. So, chat completion was a standard. What now? It's funny that when this report was first presented in a shortened version, there was nothing like it, and I even predicted, or rather, requested some alternative to chat completions. And what do we have? In fact, OpenAI released Open Responses just a few weeks ago, which is great, I think, and shows how justified this trend feeling was. And this is a very, well, a wonderful step, in my opinion, but it's still questionable whether it will work, whether it will take off, because there is a lot of skepticism in the technical, professional community that Google will adapt this standard as they did with the Completion API. Nevertheless, many open-source and not only libraries and providers have already announced support, for example, VCEL, VLM, OAM, and so on. I dug into this standard. In fact, it doesn't cover everything that OpenAI currently offers, but perhaps that's just for now. So, the second part of the report will be more about cases, and at the end, I will talk about the decision-making framework. So, actually, yes, first of all, it must be said that RAG has not gone anywhere, it has just changed. And, if before very often on Telegram channels, on blogs on Substack, on Reddit and everywhere, there were many questions like, which embedding model to choose, which embedding model, very much about retrieval, about these pipelines, about graph RAG or regular RAG, hybrid RAG. And, actually, now it's already the 25th, 26th, right, what source to use. And we understand that retrieval is just one of the agent's tools. Whether it's in the form of embeddings, in the form of file search, or SQL queries, is a secondary matter, or rather, depends on the context. And the agent decides whether it needs retrieval or not. Before moving on to the case itself, I wanted to say a couple of things about where the roots of such solutions come from, the prerequisites, right? So, coding is the leader in AI adoption according to many, and research, for example, Anthropic proves this. Their economic index shows that 44% of API usage is development, much more than all other categories, each of them. Code has shown phenomenal success and continues to be heard even more this year, as if, and not without reason. In general, coding practices are the cutting edge of AI, in my opinion. And here is the most mature tooling and best practices. And sometimes bash is all an agent needs, right? Why? Because LLMs are trained on code. Bash is a rather, well, old, let's say, way to tell a computer what you want from it. And, therefore, you get quite accurate results when you need them, right? Grabfind is super fast and super accurate. When you don't need it, you can use more fuzzy algorithms, including similarity search. This approach itself does not prohibit it. Minimal context, because files are loaded on demand, so you don't need to think about how to re-rank and so on. And, here it's not even about files replacing databases, but it makes you rethink when a database is redundant, and when files are all you need, right? Well, and regarding debugging, again, these are regular command-line tools, and they are available both for humans and for agents. The agent SDK philosophy is that essentially, we are talking about the Anthropic agent SDK, but similar practices are already offered by many other providers. The philosophy is to give the agent a computer and use the same tools that programmers use in their work. File system, bash, file editing, API calls, LLM, and so on. That is, essentially, these are wrappers, but this is also tooling, right, and the typical cycle is to gather context, perform an action or a set of actions, sometimes check the work in parallel. Here, reflection is a super important part, right, on what has been done, and repeat as many times as necessary until the result is achieved. I'll say right away that I'm talking a lot about code, about code agent SDK, and so on, but this doesn't mean that I'm saying, use only this stack and that's it. There are excellent alternatives from the other big three, including, for example, OpenAI Codex SDK, Gemini CLI, you can make them work similarly, although, as far as I remember, they don't have an SDK around it. There's OpenHands, it's been developing for quite a while, there's OpenCode, a very good open alternative, and many don't know it, it also has its own SDK, which is generally not bad, only it works a little differently, right, they deploy, start a server, you access it via REST API, but that's not important, sometimes it's even more convenient. I'm talking about the approach in general. And the architecture of a File First agent, as I call it, is roughly like this, right, we have data, we bring this data as close as possible to the agent, to its environment, to its place where it executes code, because this achieves minimal latency and maximum ability to use bash tools. There is a runtime. This can be a local machine, it can be Docker, it can be a container, sandbox, and so on, and so on. The main thing is that there is a runtime. And there is the possibility to use these coding agents for non-coding, well, and perhaps for coding, depending on your use case. On top of this, there is an interface. The interface can be LLM, it can be chat, it can be API. Here again, it depends on various requirements and on how, well, again, how it will be convenient for the user, or how it will be faster to implement. Closer to the case. This is a European company, it is involved in marketing. They came with a request that they tried RAG and it didn't work. And, moreover, they put a lot of effort into it. They used a lot of techniques. I looked at what they tried to do, and it certainly cannot be called a naive superficial approach. They used hybrid RAG. They probably only stopped short of ontology and graph databases. They used almost everything they could find at the time. So, query expansion, I won't even mention it, a lot of techniques. They got hallucinations. The main case. The main case is that they want to search, that is, they need knowledge management and a good understanding of the knowledge stored in Google Drive in a huge amount. And there is everything they need to know about projects. These are scenarios, various information about client positioning, information from call transcripts, everything, everything, everything, practically everything related to the project itself. I'm not talking about documents like contracts they signed, although that's also possible, but it's not about that, but about what is useful knowledge for projects. They have stored it in Google Drive historically for a very long time. And they need RAG for it. In order to, again, do their work, create new creatives, prepare for calls, and set tasks, and so on. In general, the idea to replace RAG with agent SDK came directly during a call, because they had already started adapting code-code, and I was actively involved in it at the time. And we did a pilot, and it went very well for them. It worked and works perfectly. And periodically, small adjustments to prompts are needed, adding skills and tools. I'll talk about this a bit later. But in general, the speed of implementation and the quality of the result were much higher than expected. Key components in this specific case, file synchronization, occurred through Airbyte. And Airbyte is essentially an ETL for cloud storage. In this case, Google Drive, but you can use Yandex Disk, Dropbox, and many others, there are many providers. Search was used, for example, using the bash command `tree`, you could quickly get key files and directories, especially directories. Using the `grep` command, you can search, including advanced search, both by content and by binary files, and using certain command-line tools, you can perform fuzzy search. And, by the way, semantic search can also be improved here. It's just not needed in this case, it works quite well. Just like code-code works without embeddings in its coding agent. And certain tooling was built around reading, because it's one thing to read text files, right, and, for example, txt files, transcripts, and quite another thing to read PDFs, images, and so on. PDFs and images are built into code-code, it reads them perfectly, with certain limitations, for example, PDFs up to 30 MB only. But other formats are read perfectly by tools like pandoc or custom scripts. Some of these scripts were specifically adapted to their specific reports. That is, we know the structure of a particular Excel file, and we simply speed up the search and reading of such a file. That is, the tool is adapted to their specific knowledge base. And on top of all this, agent SDK, virtualization through, that is, sandboxing through Docker, and LLM, I'll talk about that too. So, in this case, what advantage did it provide? No re-indexing. This was one of the biggest pain points for this client. And, in fact, we have implemented similar solutions for other clients as well. Every time you change your approach to RAG, you have to change, you found out that the chunk size should be different, you re-index, you found out that a different embedding model works better, again re-indexing. You just added the idea of summarization, re-indexing is normal, only it's data engineering, work that not everyone can afford, let's say, and, well, it's just resource-intensive. The answers are quite accurate, because if the agent doubts its cycle, it will recheck. And if we think it doesn't doubt enough, we will force it through prompting, through skills, and so on, to check its work better. The agent sees the entire document context, often loading it entirely in the process. And the domain expert iterates himself, adding skills, LLM tools, prompt engineering, and so on. It's funny that we, this was probably one of the interesting experiences when we worked directly with the domain expert in interactive sessions, that is, we launched, saw what was not working, immediately changed the settings of the coding agent, for example, the system prompt of code-code or a command or a skill, and so on, and immediately saw the results. That is, in such a one-hour, two-hour session, the performance could be significantly increased. Because the system itself is higher-level, that is, you focus on instructions and domain knowledge, rather than on some engineering gears that need to be reassembled every time in more rigid workflows. Agents need an environment, right, as we know, they need a runtime to execute code. The execution environment is quite, well, it's a new type of infrastructure, relatively new, but you constantly need to think about it, because if AI writes code, then how it executes it and where it executes it is a very important question. First and foremost, a security question. Specifically in this case, the architecture, this is the file structure, it was difficult to extract something super specific. I tried to extract as much as possible without violating the NDA. This is the file structure, where you can see that we have, the worker that is mounted into the container, and there is certain code that launches this, including LLM and task runner, right. The logical diagram looks like this. We have the external world, we have Airbyte synchronization, outside of Docker we get a folder that is always up-to-date. Synchronization runs regularly via cron, and we mount only what is needed into Docker, with the correct permissions. For example, in this use case, we were not allowed to change, that is, file changes were not required, so we immediately gave it access to this folder, because if, well, this is simply a policy of minimal permissions. Inside, there is a task manager that launches, actually, code agent SDK instances. And it is allowed to write its own scripts, optimize. Very often, you can see in the process some meticulous request that requires data aggregation. For example, gather information about
Everyone, uh, marketing companies over the last year and do some specific analytics, right? So, RAG is something that, well, hardly knows how to do. Code-code is directed on the fly. It simply writes a script, launches RAG, uh, rechecks its work, and returns some metric that was requested. All of this happens in the background. It can even reuse this script later. I'm not even mentioning that through skills and through the same scripting, but a more deterministic one, when, uh, we have pre-written the necessary scripts, it's possible to achieve a more optimal execution speed and greater reliability. So, uh, yes, why did I show this here? Because isolation is a super important concept. Don't just create agents on a local machine that execute code like you do, and always think about security. And here, in particular, there's even Network Isolation, meaning it's not allowed, uh, it's allowed to visit certain domains, for example, to download certain dependencies for itself and interact with. In short, there's a whitelist of domains, everything else is forbidden. When does File First apply? When does it not apply? Actually, this is a very conditional table, but I think it will be useful to talk about it. When you have hundreds, thousands of files, actually, in that project, well, I find it difficult to say now because the number of files is constantly growing, well, tens of thousands of files. And it handles it perfectly. Including because most questions relate to a specific client, and they have a specific folder, and the agent simply won't grep in the wrong folders, it will look where it needs to. Uh, files are updated frequently. Uh, this, this applies, this is a green fact. This means you don't need to re-index everything constantly, but you need simplicity and speed. There are no resources for this engineering pipeline, because with good skill and good resources, RAG can yield colossal results. But often, in my experience, teams run into data engineering problems. That is, something works perfectly for hundreds of files, but then scales poorly for thousands. That is, they don't understand how to build durable background processes, these workflows, essentially, how to do ETL correctly. This is the domain of data engineering, and it goes beyond the scope, well, it overlaps with engineering, but many lack the qualifications for it. Uh, complex tasks, advanced reasoning, it's a perfect fit, like the example with Pandas, and so on. And there's no fixed workflow, flexibility is needed for our case, entirely. When it doesn't apply, when, well, simply a huge number of files on a scale, I doubt that, for example, LinkedIn, to answer a client's question, if when they were creating their, well, some kind of, uh, support agent, that they would create a similar system, which is in isolation, in a sandbox, for each client, to answer, to do all these extra actions. So, this is more and more about internal corporate scenarios. Semantic search is also needed, also not a blocker. This can be bypassed, in principle, because through CLI, right, and not through CLI, you can do MCP, you can, you have a huge number of ways, including using embeddings in such an agentic approach. When there's critical latency, right, because it's clear that this optimized, directed graph will, of course, be maximally optimized for speed. Critical token cost, also not very suitable. A lot of users, as I said, it's hard to scale. Although a lot of users, the concept is flexible. For some, uh, 10,000 users is not a problem if there's proper orchestration. Well, and there's a clear workflow, meaning we have cases where we simply have a directed graph, we know exactly what should follow what, some kind of. If there are any conditionals, they are quite transparent, understandable, not needed. We don't aim to complicate it through the approach. Simply because, well, it's more reliable. So. Uh, yes, there's still a little time, I'll tell you about MCP. This deviates a bit from the topic of agents, but I think it's interesting because it's related to this case. MCP, when did they appear? Yes, and to this day, local MCPs are something completely geeky for most business users. When you do it yourself, like, nothing complicated. JSON config and that's it. When you try to explain to a business user how to add MCP to their, uh, environment, for example, a deployed code-code, it's rocket science for them. So, well, it's understandable. Open JSON config, paste config strings there, configure environment variables, restart the application, maybe install some dependencies. So, this is not how it should, this is not the UX we are looking for for models. We made MCP for this case. And it works, it worked, and it works perfectly. MCP, many of you know, and Valera also wrote about it, essentially, it's the correct MCP, as it should be. It's an HTTP-based MCP with OAuth 2 authorization. Firstly, it's secure, and secondly, it's super convenient. We did this through OAuth Kit and Fast API. Funnily enough, to connect it, we did it at an early stage, and we rewrote the project from TypeScript to Python simply because Python has Fast API and it has the necessary connectors. And in general, Fast API is so good that we even changed the programming language for this project from TypeScript to Python. It wasn't too painful, and it was at an early stage. Uh, yes, no configs. I recorded a short video to show you how it works. Uh, so, now it's the beginning. So, we simply add, we click, go to connections. This is on the example of Code, but others work similarly. Uh, an agent is added, well, in this case, an MCP, and authorization happens through OAuth, and that's it. So, this agent is available in our list of connections, and now we can make requests with this MCP just like with a regular one. What's here? What advice can I give? OAuth, if it's not already implemented and basically not there. In this case, there was a situation where this project didn't have OAuth. It's very convenient to do it through various tools and OAuth Kit. There's Keycloak and so on, there are standard solutions. The key here is, of course, the ability, the ability to, well, just a moment, okay, I won't go into it, I can't remember the term now. Ah, I remembered. DCR, dynamic client registration. Roughly speaking, OAuth Kit is solved in four lines of config. And it turned out to be so fast and convenient, and we get top-tier security. These are not some tokens that don't update. It's very reliable and very convenient. So, I don't see any reason not to use it in similar scenarios. This GMCP can be connected, and teams connect it to Code, and to ChatGPT, and so on. So, there's a little time left. I'll talk about the decision framework. Uh, okay, okay, okay, build versus buy. We'll skip this. When to build, when to buy. Uh, let's go back a bit to the beginning. And we understand, right, that we have a huge number of features, and there will be more. This is the catch, that we see a clear trend that providers will increasingly add features that are absolutely unique to their platform. And moreover, they work maximally, uh, coolly specifically within their ecosystem. So, Code + Code will always work better than Code + GLM, for example, or at least in the near future. When to build, when it's your core IP, your competitive advantage, and your unique data, you should build, use this right-hand column, the table, and figure out how to do it yourself, how it could be replaced with your own controlled solution, and if necessary, with another solution. Buying is easier when it's an infrastructure task. For this company, for a marketing agency, it's not their main business. That is, knowledge management, even advanced, even AI knowledge management and some AI scenarios are their secondary business. They are not going to resell it. It's like SaaS, it's their corporate automation. And plus, many things are better to buy, because you won't build such search as Google's yourself, no matter how much you want. Or sandboxing, now, I think it's cheaper to buy than, although this is also a question, actually, it depends on non-functional requirements. So, okay. Here I've explained, actually, I'm out of time, so I'll wrap up and hand over to Kolya. Yes, thank you very much. As always, it's interesting to hear you. Let's throw in some questions now. Some that I saw, some that came to my mind, two of those that were already asked. The first question is, how much does the solution on Cloud Engine SDK cost in terms of tokens? Because Anthropic is quite an expensive pleasure. I can say that for this client, it was in the range of 10 cents to 70 cents per question. Even though some questions that were like 70 cents, the agent dug into them for a very long time. It wrote a script, it launched it, it evaluated it, checked it. Well, meaning it could work for 10 minutes, and that's normal for such a task. Also, specifically for this client, what's characteristic is that their hundred-dollar subscription worked. It was enough for all their tasks and is enough, meaning they don't spend these tokens via API. >> Damn. I didn't have access to their RAG expenses, so to speak, but he complained that it wasn't cheap either. Tokens weren't free there either. >> It's not cool that it falls into the B2C subscription for business, that's definitely cool. Okay. A question about the File First approach. How problematic is it when there are multiple languages? >> Uh, do you mean the language of the documents? I assume. >> Yes. Yes. I think that the language of the documents and when the language of the document differs from the language the user is working in, it's not a problem at all. Well, I assume there are certain nuances, of course, in this specific case, it was Dutch, which is very similar to English in its kinship. If they are from the same subgroup, it seems, uh, and they constantly switch from English to Dutch. And there's no such problem. I think the problem with languages can only be if it's an unpopular language. Russian definitely falls into popular languages, because, as far as I remember from statistics, the top looks like this: English is first, then German and Russian share second and third place, it seems so. So, Russian is definitely in the top three most common languages on the internet by content. >> Listen, can you explain to me on your fingers how this works in the case of Russian? If, for example, our documents are in English, and the user asks a question in Russian. And there are also documents in Russian. How does the agent understand, uh, what to grep there? Should it grep Russian words or English words? >> The beauty or convenience of this approach is that you have levers of influence on it. That is, you can prompt it in such a way that, for example, it considers the folder prefix. Or you can tell it in advance, meaning consider, how to query, or you can even ask it to query in multiple languages at once. That is, we tuned this approach to querying, how it made requests, specifically through prompt engineering, and immediately checked. That is, you see the result, you have a very fast feedback loop. You give it an instruction, see how it works, draw conclusions, and either skip benchmarks. And often, the main thing is to make it understand what your preferences are and what languages the database has. >> Agreed. I'm actually using similar approaches on a couple of client projects right now. Somewhere custom agents, somewhere, uh, cloud, OpenAI. And it's surprising how easily changes are made in the system. That is, you can completely change the system flow with just a couple of changes. Yes, it won't be rigidly fixed, but it will still work stably because the agent always has this recovery system if something goes wrong. And here's another question. Someone wrote that bash is trash. The chances of the agent getting stuck are 99%. Can you explain how this works, why the agent doesn't get stuck, and if it does get stuck, how it recovers? >> Well, here, of course, we'll need to see if there are any life hacks, maybe how to make the agent's work more stable? >> Uh, I can say that I haven't encountered this. And here it would be better to listen to the arguments of the one for whom bash is trash. And I don't understand how it can be trash. If it works perfectly in coding, and even in such scenarios, it gets stuck. Again, Code-Code has a timeout instruction. If it hangs, it will try another approach. And all of this is finely tuned through cookies, if this is a regular problem for you. Uh, with timeouts for MCP. That's real trash. That's a separate issue, how we made MCP think longer than the MCP client makes it think. For example, Code has 30 seconds, and this is an unchangeable value. So, we had to resort to trickery to make it work longer. But with bash, there are no problems in practice. At least, we haven't encountered them. Okay. I also had a question about isolation. Actually, I see a lot of misunderstanding about how to do sandboxing. So, maybe you can share some life hacks here too? I saw you mentioned Docker, but maybe there's a comparison of Docker with other approaches. And why Docker was chosen, maybe some heuristics on how and in what cases to choose something other than Docker. >> Yes, isolation is indeed an interesting point, because the range of solutions is quite large, meaning you can get very involved, meaning there's more native isolation, a finer part, for example, the same cgroups in Linux. There are specialized solutions like Daytona, and so on. There's Docker itself, which in this case we chose for its simplicity and better developer experience, because it was convenient to enter the container at the right moment, see what was happening there, and there was no need to spin up a new container for every little thing. This is one company, they have a shared container that works on different tasks, they don't have billions of parallel requests. So, there's no such problem. So, I would choose this heuristic. If you highlight, if you have a need for, so the first question to ask is: is it a monolithic or microservice architecture? That is, if you have, as here, for example, one agency and everything revolves around it, then it's one solution. So, Docker is quite suitable here, and it might even be a single one that doesn't shut down. If you have a huge number of clients, meaning a multi-tenant project, and you need to constantly start and stop a huge number of containers, then there are a huge number of solutions, from Code Flare, from Daytona, and so on, they provide sandboxing as a service, meaning they provide a high-level API, and they allow you to launch these sandboxes on demand as much as needed. They execute, they start, they shut down, and so on. Regarding sandboxing, I'll also say that Docker alone, meaning there are a huge number of tools. That is, you have a large number of network isolation options, how to mount volumes or how to work with temporary files, right. Docker itself provides a huge number of tools. And I'll also say that there's a cool technique, it's proxying network requests. That is, you can route all your requests through a proxy, for example, Nginx proxy. And you, meaning for the agent, right, and even inside the container, it won't be obvious that there's some environment variable being substituted. That is, it will be a regular request, and you intercept this request from the outside and inject, for example, an environment variable there. This is a very cool technique, we sometimes use it. It's a bit more complicated to implement, but it's top-notch for security. That is, in principle, even if Code-Code goes into another folder or something, it has no .env, meaning, in principle, all it has is that it thinks it's making a regular request without a header, and the header is added from outside the container. This is a very powerful technique. >> Damn, I highly recommend everyone who is concerned about security and plans to use something similar to re-listen to this part. I really like what you said. Let's move on to the last question and then to Valera. I'll combine several into one. The first was about how testing is structured. Were there any question-answer pairs, or was there an LMS judge, how is such a system tested in general, when it's not a classic RAG, but something more flexible and agentic? And also, this prompt engineering, right, which is based on reviewing the results, was it manual or was it already an agentic evolutionary self-improvement when the prompt changes based on some logs, like in the recent ERC3? >> Uh, with reflection, right? >> Yes, yes. >> Uh, >> I understand. So, in order. First question about tests and benchmarks. We made a very simple benchmark. At the first stage, it was, essentially, with checkbox-like boolean checks. Found, not found. That is, we asked a specific question, and the client prepared 50 questions, and we checked with regex if what he was looking for was found. They were all very detailed questions, meaning non-obvious ones. And most of them, uh, well, these types of questions failed in RAG one way or another, at least due to limitations. This is a fairly primitive approach. We have now connected, well, we connected almost immediately, rating, uh, and are collecting logs to do the next round and benchmark based on, well, what their agency actually uses, meaning real data. In general, benchmarking, well, evaluating agent systems is very complicated, it's more complex, many moving parts. Evaluating it by internal parts doesn't make much sense, because that's the point, you give the agent freedom. Uh, externally, well, there are three, three, three directions, right, traditionally. deterministic evaluation, LM as a judge, and human as a judge. We used deterministic and we, well, RAG-based, and we used human as a judge, meaning the agency's employees themselves actively tested it and said where things were wrong. So, in general, I'm very skeptical about the evaluation framework. It's a separate topic, probably, but, well, it's a separate topic. Uh, second, about prompts, about prompts. The first version of the prompt was written by Code-Code based on input, including call transcripts, and studying, not without access to the dataset on which we tested, but with instructions on what scenarios might occur. So. Uh, then we iterated together with the client in such interactive sessions. >> Super, thank you, Fat. There were very interesting questions, interesting answers. Let's move on to Valera. Thank you all. Bye. M.