Transcription
You use cursor code chat GPT every day. But how does coding differ from agentic workflow? What is the difference between prompts and context engineering? When do you need RAG, and when do you need fine-tuning? If you can't confidently answer half of these questions, this video is for you. And AI tools have changed. What was an experiment a year ago is production today. Code Cursor WinF is no longer a toy. But here's the problem. Developers use these tools blindly, without understanding the basic concepts. Many think ChatGPT works out of the box. They learn to write prompts, and that's it. But no. Prompt engineering is like a steering wheel in a car. Is it important? Yes. Is it enough? No, you still need to understand how the engine and brakes work to avoid crashing into a wall at full speed. In production, mistakes can be costly. You might spend weeks developing a system that works unpredictably, or burn through your API budget because you don't understand where caching was needed and where a local model was sufficient. Not just "let's attach an LLM," but a conscious architecture. Understanding fundamental concepts is the difference between "works sometimes" and "works reliably." Hello, my name is Dmitry Berezhnitsky. I've been in development since the early 2000s. I've built systems, designed architectures, and launched them into production. That's why I look at AI through the eyes of a practical engineer, not an ML researcher. And here's the good news. To work with production AI, you don't need deep mathematics. You need to understand key concepts and know how to apply them. Today, we'll break down what an LLM or an agent is, what you're actually using. Context engineering. Why it's more important than prompts. LLM or agent coding – two philosophies of AI development: or fine-tuning, when to use what. Foundation Models, MoE, architectural concepts, and AI Security, the things rarely talked about. Let's go. Okay, you know what an LLM is. But how does this thing actually work? Let's open the hood. Many think one token equals one word. Right? No. A token is the basic unit of text that an LLM works with. It can be a word, part of a word, or even a symbol. Here's how it works. Let's take different tokenizers and compare. First, let's take the word "programming." And look, in different tokenizers, it can be broken down completely differently. Somewhere it will result in three tokens, somewhere six, somewhere 17. And if we go through all the models, we'll see completely different token breakdowns. And let's take the word "a." It will be broken down into either one or two tokens. Tokenization is the first step of processing. Text is broken down into tokens. Each token is assigned a unique numerical identifier, and then these IDs are converted into vector representations, embeddings, which the model can then process. Why is this important? Each model has its own tokenizer. The same phrase in GPT, Claude, or Gemini will take up a different number of tokens, and this affects API costs. You pay for tokens, context window size, how much information the model sees, and processing speed. Russian text often requires more tokens than English in popular models. On average, about 1.5 to 2 times more, sometimes higher. Why are popular LLM models predominantly trained on English? There are many times more English data than Russian in training datasets. Why is this critical when working with them? All APIs are priced by tokens. More tokens, more money. If a model has a context of 128,000 tokens, for English text, this is about 250-300 pages, but for Russian, it's only 100-150 pages of equivalent information volume. Attention mechanism. How does the model understand context? Imagine a sentence: "The senior developer looked at the code, it broke." Stop, who broke? The code or the senior? You automatically understand that it refers to the code, not the developer. Although, of course, it can be different. How does the model do this? The self-attention mechanism, which for each token calculates how important the other words are to me. The token looks at all the previous words and calculates connection weights, and gets the maximum weight for the word "code." The model understands the context. For those who want to dig deeper, there's a formula from the paper "Attention Is All You Need." Multi-head attention, parallel processing. But the model doesn't just look at the text once; it looks many times simultaneously through different attention heads. Imagine you're reading code during a review, and you have three heads. One head looks for syntactic connections, which variables are linked to what. Another head tracks data types, what's passed where. The third head analyzes the logic, which conditions depend on what. And all heads work in parallel, the results are combined, and you get a complete understanding. Modern models use dozens of attention heads for each of the dozens of layers. The exact numbers aren't that important. What's important is to understand the principle of parallel processing at scale. But if anyone is interested, in known models like Llama 3.1, there are 64 attention heads per layer across 80 layers. For GPT and Claude, the architecture is closed, but I think the numbers are roughly the same. Importantly, the specialization of heads is not predefined; it forms randomly during training. Transformer, the revolution. Everything we've discussed – self-attention, multi-head attention – is the core of the Transformer architecture. GPT, Claude, Llama, Gemini, all are built on Transformers. What made Transformers revolutionary? Before them, there were recurrent neural networks. They read text sequentially: the first word, then the second, then the third. Slowly. Transformers processed all tokens simultaneously thanks to the attention mechanism. This radically accelerated training and made it possible to train models with hundreds of billions of parameters. This is precisely why AI has made such a leap in a few years. Sounds cool, but there's a fundamental problem. Attention calculates the connection between every token and every other token. 2,000 tokens, 4 million operations. 4,000 tokens, 16 million operations. Double the context, and computations quadruple. That's why long context is slower and more expensive. That's why models with millions of context tokens are still a challenge. So, Transformers have taken over the AI world. Huge infrastructure. Better quality on most tasks has become the industry standard. But quadratic complexity is a bottleneck. And the community is actively seeking solutions. And now, hybrid models are starting to appear. A combination of Transformers and alternative mechanisms. Transformers are now the de facto standard, but their limitations are already felt. The next 2-3 years will show whether Transformers will continue to dominate or if hybrid architectures will become the new norm. In the meantime, understanding Transformers is critically important. They are what's under the hood of Cursor, Claude Code, ChatGPT, all the tools you use every day. So, now, understanding the limitations and mechanics, let's return to context windows. Imagine you're working with an AI assistant on a large project. You agreed on the architecture? Everything's great. After 20 messages, the model suddenly ignores your initial instructions. What happened? A context window problem. But why does this happen, and more importantly, how can we avoid it? Let's start from the beginning. What is a context window? It's your model's working memory. How many tokens it can see simultaneously. Imagine your desk. A small desk, 4,000 tokens, a laptop plus a cup of coffee. No more space. A large desk for 200,000 tokens. Two monitors plus a laptop, plus books, notes, headphones, and so on. Everything you need for work is nearby. But in practice, tokens run out faster than you think. Let's see what's in the context window. System prompt plus history plus your current request. What do we get as output? An answer that, with the next request, will be placed in the history. So, all of this must fit into our context window. But with AI assistants, we also add tools. These are function definitions that the model can call. Reading files, executing APIs, commands, search, and the results of their calls. And this includes the content of open files, the output of bash commands, search results for the project, action history. What has been done, which files have been changed. A real example. Claude Code opened five files, 200 lines each, executed three bash commands with output, and performed a project search. That's 15,000 tokens just for tools. And that's just for one call to our agent. How do you know the context is overflowing? The answer cuts off mid-sentence. The model forgets the beginning of the conversation, starts repeating itself, ignores instructions from the beginning of the dialogue, requests files it has already opened. D-code, Cursor, Wncrf. They automatically compress older parts of the conversation when the context overflows. Sounds convenient, but there's a catch. During summarization, details of architectural decisions are lost. Change history. Why did we do it this way? In the context of tool calls, the model might forget that it has already done this before. Definition of helper functions. Result: the model starts requesting already opened files or loses understanding of why the code is written this way. What to do? If you're working via API, summarize the history yourself and keep only what you really need at the current moment. If you're working with AI assistants, monitor the context window size. For complex tasks, start a new chat with a detailed summary. Don't rely entirely on auto-compression. It's more of a crutch than a solution. Even with a huge context window, tokens are consumed faster than you think. Tools eat context incredibly aggressively. Every opened file, every command, every search result is hundreds or thousands of tokens. With AI assistants, you hit the limit sooner than in a regular chat. Proactively manage the context. Understanding context is the difference between "AI helps" and "AI is dumb." Let's figure out why ChatGPT prints an answer. Have you ever wondered why the answer appears word by word, not all at once? Generation happens in two stages, and they work radically differently. Stage one: prefill, pre-loading. Your prompt, 500 tokens. The model processes all 500 tokens in parallel in 1 second. It's like reading a page of text at a glance. And stage two: decode, generation. The model generates the answer: 200 tokens, but each token is generated sequentially, one after another. It cannot be parallelized. It's like writing text. Each subsequent word depends on the previous ones. Result: 200 tokens are generated in 3-5 seconds. Key-Value Cache. Why doesn't the model recalculate everything from scratch? Problem: when generating each token, the model looks at all previous ones again: 1,000 tokens, 1,000 recalculations, and this is slow. The solution is called Key-Value Cache. The model stores the attention results for already processed tokens in memory. Prefill: process the prompt, fill the cache. Decode: use the cache, calculate only the new token. Result: speedup by orders of magnitude. That's why the first token appears slower than the others. Why is output more expensive than input? Let's look at the example of Claude. Let's look at the current prices for Claude Code on Sonar 4.5. Input: $3 per 1 million tokens. Output: $15 per 1 million tokens, which is five times more expensive. Reason: input or prefill is parallel processing, fast; output or decode is sequential generation, and it's slow. The cache grows with each token, requiring more memory. Important: prices vary significantly between different providers. I've used GPT as an example, but for everyone, output will be about 3-5 times more expensive than input. Let's break down a practical example of how much your request costs. In our scenario, we'll have an AI assistant with documentation. Our prompt will consist of a system prompt plus documentation of 10,000 tokens. Let's add our user's question. Another 100 tokens. So, our input will be 10,100 tokens. Let's say the model's answer is 500 tokens. Let's calculate for the prices we indicated for Claude. Input: the total input will cost us 3 cents. Output: will cost 7.5 cents. And for one request, we'll get almost 4 cents. How many such requests per day will give us? $37 per day. And per month, that's $1,100. You might ask, why did we calculate all this? To understand how to reduce this cost. And now we'll talk about caching. Claude offers caching. And the cache is divided into two parts: writing to the cache and reading from the cache. To write to the cache, we'll have to spend a bit more than on regular input. And it will be 25% more expensive. We'll get $3.75 per 1 million tokens for writing. But reading from the cache will cost $0.30 per 1 million tokens. And this allows us to make the entire system 90% cheaper. The cache lifetime is 5 minutes, but it's updated with each use. Let's see how this whole system will work if we use the cache. Writing will be slightly more expensive, $3.75, so the total cost will also be slightly higher, $3.75. And the total cost will change, becoming slightly more expensive. But reading from the cache of our document for 10,000 tokens will now cost us $0.003. The total cost of the second request will be much lower, about 1 cent. And this makes it, well, not 90%, as I said, but about 70% cheaper. For 1,000 requests per day, this means not $37 per day, but $10 per day, or about $300 per month. We just saved $700 by correctly using the cache. Additionally, there are other ways to save. Anthropic has a feature called Batch API, which offers a 50% discount. This is used for non-urgent tasks where processing is done in the background. And there we get a 50% discount on input and output tokens. This can be used for analyzing customer feedback, generating product descriptions, summarizing documents, classifying large datasets. Anything that doesn't require immediate processing. Important: batching is not available from all providers, so you need to check the current documentation. Understanding prefill and decode and correctly using caching is not just theory; it directly impacts your budget. Because if this task is not real-time, we'll end up with not $300, but $150 per month, which is seven times more cost-effective than a naive implementation and using the API without understanding how caching works, what input and output are, and how to optimize it all together. Therefore, invest time in prompt architecture and use caching, understand your tasks, and it will pay off in the first month. Why doesn't ChatGPT learn from your data? A common misconception. If I chat with ChatGPT a lot, the model learns from my data and gets better for me. No, let's break down the difference between training and inference. Training. Training a model. This happens once. Terabytes of data are used. The process takes weeks or months. On huge clusters, GPUs cost tens of millions of dollars. As a result, the model's weights are updated and optimized. Inference. Using a model is every one of your requests. You send a prompt, the model processes it in seconds. It costs cents. And the weights don't change. The model simply applies what it has learned. A simple analogy. Training is like compiling a program. Inference is like running that program. The program executes, but the code doesn't change from use. But ChatGPT remembers facts about me from other conversations. Yes, but that's a memory feature. It's not model training. ChatGPT uses various types of memory: explicit facts you asked it to remember and your chat history. The system analyzes all your past conversations for context, and in the next chat, the system retrieves relevant facts and adds them to the context window along with your request. But the model's weights haven't changed. It's just smart context management. Providers may also save your requests for future model training. For sensitive data, check privacy settings or use your own models. But how do you customize your own model? There are different approaches for this: few-shot learning, RAG, and fine-tuning. We'll talk about that a bit later. First, let's understand model creativity. An LLM is a next-token predictor. At each step, the model can evaluate all possible options. Let's look at an example. We have the input: "I love." What could be the next word options? Programming, designing, reading, walking, drinking. But what exactly to drink, we don't know yet, because first, we need to decide on the current word, and only then will we choose the next one. For the word "programming," we'll have the highest model score, for the word "drinking," the lowest. And these scores turn into probabilities. And temperature determines how much the model will stick to the most probable options or allow itself to choose something else. For the pedantic, mathematically, it's Softmax with a coefficient T. The formula is on the screen. As T approaches zero, the top token is almost always chosen. As T approaches infinity, the distribution becomes uniform across all tokens in the vocabulary. And there can be more than 50,000 of them, or however many fit into our model. And this will result in complete chaos. The optimal temperature value depends on the specific model, task, and the required balance between logic and originality. Therefore, it's useful to experiment. In practice, most modern models show the best results with temperatures between 0 and 1.2. Let's say, creativity while maintaining adequacy. Many APIs limit the maximum temperature value we can set to two. And this is not a mathematical but a practical limitation to protect against complete chaos. Top P and Top K are additional filters. Top P: the model accumulates options from the most probable until the sum reaches, say, 90%. The rest is discarded. The flexible size can sometimes be five options, sometimes 50. Top K: the model takes exactly K most probable options, for example, 50. The rest are ignored. A fixed size is always 50, even if the first three give a 95% probability. As an example, the model has 100 possible words. Top P of 0.9 will take the top 10 words if their sum is 90%. Top K equal to 50 will always take exactly 50 words. Most developers use default settings and are surprised by the results. Now you know how to control your model's behavior. Let's summarize this block. You now understand that tokens are not equal to words, and this affects the final cost. Attention is how the model understands connections. The context window is working memory with limitations. Prefill/decode. Why is the answer printed word by word? Training vs. Inference. Why doesn't the API change the model? Temperature – control of creativity. And who are Transformers? And this is not abstract theory. This is knowledge that helps optimize costs, write effective prompts, understand limitations, and choose the right parameters for working with APIs. Okay, now you understand how the LLM engine works under the hood, but what can you do with this engine in practice? Simple question-answer, complex reasoning with step-by-step analysis, autonomous actions via APIs and tools. And these are three completely different levels. And here's the difference between them. You've understood the mechanics? Tokens, attention, context window – that's the foundation. Now, practice: what can you build on this foundation? Most people think: "AI, chatbots, LLM agents – it's all the same, just different names for the same tool." No, these are three completely different levels of complexity, and understanding the difference is key to professional AI work. Imagine a kitchen. An LLM is a chef who cooks according to a recipe. One request leads to one result. A reasoning model is a chef who can improvise and think about delicious combinations. An agent is an entire kitchen with a head chef, cooks, and pantry staff working together to create a banquet. LLM is the base model. What does it do? Receives text, generates text. It's stateless. Each call is independent. There's no memory between requests. When it's used: simple question-answer. Text generation. Analysis of a single document. Limitation: cannot perform actions, has no access to tools. Each request from scratch. Reasoning model. A model with reasoning. What's added? Chain of thought: thinks step-by-step, breaks down tasks into intermediate steps. Additional verification techniques: self-consciousness, multiple sampling, self-reflection, improvement through criticism. And these models have increased response times for complex tasks. Higher cost per request. Some models show the thinking process, others hide it when we use them. Complex logical tasks: math, programming, planning with analysis of options. Advantages: fewer errors on complex tasks, transparency of reasoning where available. But an agent is something else entirely. It's an autonomous system. And let's see what it includes. And the first thing we'll look at is Observe. It observes the current situation. The next step is Reason, it reasons, and after that, it comes to Action. Here there might be a call to some external APIs. And after that, it repeats the cycle to achieve its goals. This is often called ReAct: Reasoning plus Action. Thoughts, actions, and observations alternate. The model itself decides when more reasoning is needed and when to proceed to action. At the Action phase, we might have tool calls. This is access to external tools. API calls, external system calls, file system, reading or writing files, executing code, running commands. An important point: since we might have multiple cycles, we need to maintain context between steps. And for this, we need state management. And it can be in memory for current sessions, in vector databases for semantic search, files, or a database for long-term storage. The agent can also plan and reflect. When planning, it breaks down complex tasks into subtasks. When reflecting, it evaluates its decisions and improves them. Everything can be assembled into a multi-agent system of several specialized agents. And they work together. One researches, another tests, a third writes code. The key difference of an agent. Let's take an example from development. Suppose there's a bug in the authorization system, and you throw it into ChatGPT or somewhere into a chat model. Here's the code. Find the bug here and fix it for authorization. The model finds the problem in line 42 and says, "Here's the problem, proceed from here." That's it. An agent can perform the full cycle. It can read the code, analyze it, find the bug, fix the code, run tests, then create a commit and deploy to the dev environment. After that, it can check that everything launched and is working correctly. An agent remembers the state between steps. "I fixed the code, tests passed, we can deploy." An LLM cannot do this. Simple comparison. LLM as a consultant. One question leads to one answer without actions. Reason as an analyst, thinking aloud, checking logic, explaining. An agent is an engineer. Plans, acts, checks, corrects. Why is it important to understand AI chats in the browser, like ChatGPT, Claude, Gemini? The base is an LLM + AI with pseudo-memory. The chat history is sent again. Modern versions have many advantages. They can call functions, search the internet, execute code, upload files. ChatGPT and Claude have memory functions, but this is managed by the application, not the model itself. The context is reset between separate sessions. AI assistants, like Code, Cursor, Wncrf, have full access to the environment. They work with state, can view files, access Git, call terminal commands. All context is within the session. When you build your own system and have a simple question-answer, an LLM is sufficient. It's cheaper and faster. If you have a complex task with analysis, use a reasoning model. It will be more accurate and reliable. If you have a multi-step process with many actions, you need to go towards agents. And this will be automation. It will have state, it will have tools, it will be able to plan or reflect. Now, a critical question. How does an agent know what to do? You can give it access to 100 tools, but if it doesn't understand the task context, it's useless. How to correctly structure information? How to give the agent exactly what it needs? This is where context engineering begins. Everyone is obsessed with prompt engineering. 10 magical prompts – the best prompt for productivity. One prompt will change your life. And yes, prompts are important, but here's what no one talks about. A prompt is just the tip of the iceberg. The foundation of the result is context. Example: booking a hotel. Imagine you send an AI agent to book a hotel. Prompt: "Book a hotel in Paris for the conference next month." Attempt number one. Result: Best Western Paris Inn, Paris, Kentucky. Problem: the prompt wasn't precise enough; the country wasn't specified. Attempt number two. Improved prompt: "Book a hotel in Paris, France for the conference." Result: Ritz Carlton. 900 euros per night. Champagne dinner included. Problem: the prompt is perfect, but the AI doesn't know your budget. This is no longer prompt engineering; it's context engineering. Attempt number three. With context, the same prompt, but now the system knows before your question: corporate hotel limit, maximum 150 euros per night. Your calendar: conference, March 15-17. Location: central Paris. Result: selection of a hotel as close as possible to your conference and within your budget. What's the difference? Prompt engineering: What you say. "Book a hotel in Paris, France." Context engineering: What the system already knows. Corporate rules: calendar, budget, location. Andrey Karpati, former Director of AI at Tesla. Prompts are short task descriptions, but in industrial applications, context engineering is the art and science of filling the context window with the right information. The difference isn't in how you asked. The difference is in what the system knew before your question. On the same note, Toby Lutke, CEO of Shopify. Context engineering is the art of providing all the context for a task so that it can be solved. LLM prompt engineering is a formulation technique. How do you formulate instructions for the technique? First is role prompting. Assigning a role. "You are a senior developer with ten years of experience." Second: few-shot examples. Learning from examples. Show two or three examples. Here's a query. Here's a response. Third: Chain of Thought. Chain of reasoning. "Think step-by-step" or "Give me step-by-step." Fourth: response format. "Respond in JSON format." Fifth: constraints. "Use only the standard library." When does this work? To change the style of response, clarify the format, set constraints. When it doesn't work: if the AI doesn't have the information, no prompt will help if there's no access to tools, if the context window is overflowing. Context engineering is system architecture. Five components of context engineering. Component one: memory management. Short memory: the last messages of the dialogue. Problem: the context window is limited. Solution: summarization of old messages. We can see that
The GPT chat does for you. It keeps the last 20-30 messages, summarizing the beginning when it overflows. You don't see this, but it's context engineering. If you are building your own agent, then this will be entirely your task. Long-term memory is knowledge between sessions. Where do we store this? Databases or files. Component two is retrieval. What documents are relevant? Dynamic addition of context. You have 1,000 documents, the context window is only 100,000 tokens. The solution is to find and add only relevant documents. And this is part of context engineering. Component three. State management. Where are we now in the process? Multi-step tasks require state. Let's consider an example. Step one. We need to analyze the code. And in this process, we can browse the entire codebase. This can take up our entire context window. But as an output, we can find a bug. And this bug will be in the file on line 42. Next, we pass this information to step number two, and it can have a completely clean context window. The step will be to fix this bug. The output will be the state: bug fixed. Then we start the third step. What is this step? Correct, it's the tests step. And then we have two options. Our tests were successful. And then we complete our cycle, saying that everything is fixed. Or our tests failed. And then we start from the very beginning. We start looking for the problem, why the tests failed. This can start a new step. Then fix, then test. And so on, until our tests pass. So state is part of our context. Component four. Tools are what tools are available? The agent knows about its capabilities. Working with code. Reading a file, running tests. It can be an external system, internet search, database query, or integrations. It can create a pull request or send a notification to Slack. The list of tools is context. Component five is the dynamic assembly of our prompt. Let's write our final prompt. What goes into the final prompt? It will be a static part. It includes system instructions and a dynamic part. It includes memory and history plus retrieved context plus current state. And, of course, our request, which the user wrote. So, prompt engineering is precisely this last request, the very last line that our user wrote. And context engineering is everything else. Why do I say all this is context engineering? Because the prompt is just a trigger. The result is determined by the context. Two philosophies of working with AI. First philosophy: wipe coding. The context is gathered for you. If you work with ChatGPT or code in chat mode and say, "Write an authorization function," then you just copy your project or possibly use full-fledged platforms for this, like Laravel, Replit, and others. These platforms have already configured the context. The project structure is ready, the database is integrated, deployment is one-click. You just prompt, "Make a login with authorization." What are the advantages of this? Fast start. You can create a proof of concept in an hour. No need to think about setup. Works out of the box. But there are a number of limitations. Focus on web applications. You are limited by the platform. Vendor lock-in also occurs, and it varies between platforms. Second philosophy: agentic coding or agentic workflow. You manage the context. Here we have many tools: Codium, Cursor, Pensar, and many others. You build the context, you choose the architecture, set up the environment, manage state and memory, control every file. And AI helps, but you are in control. And this has a number of advantages: complete flexibility. It can be any stack, any architecture. It can be working with production codebases. At the same time, transparent approaches. You control API usage or use subscriptions. Limitation: understanding of context engineering is required. High engineering culture is also needed to create maintainable projects. AI coding means context engineering is done for you. Convenient, but very limited in capabilities and quality. Agentic coding means you manage the context. More difficult, but more powerful. Both use agents. The only difference is who manages the context. Okay, context is critical, but where do we get this context from? How do we give AI knowledge it doesn't have? And there are three ways for this. You have an LLM Foundation model from OpenAI or Anthropic. It knows a lot, but it has three fundamental limitations. The first limitation is data up to a certain date. For example, "Tell me about GPT-5" will result in "I don't know, my data is up to 2024." The second limitation is the lack of specific domain knowledge. For example, the model doesn't know how our company processes returns. The third limitation is no access to your private data. Example: "Summarize the latest backlog grooming report." Of course, it cannot see this. And how do we solve this problem? There are three ways. And this is where most people make mistakes. They think there is a best way. No, there are the right tools for the specific task. Let's break down all three and understand when to use which. Method one: in-context learning. What is this? It's when we add information directly into the context. For example, here is our return policy. Then there are 500 words of text. And then we ask the question: "Can I return an item after 40 days?" Pros: it's instant. The answer comes in seconds. From a learning perspective, it's free. We have no additional training costs. We also have full control. We can change everything on the fly. There is transparency; what we added is visible. But there are a number of cons. We are limited by the model's context window size. Expensive under load and with a large number of requests, because we pay for tokens every time. We have no learning. The model doesn't remember between requests. When can this be used? When we have a few documents, and they are not very large, and these are one-off tasks. Or perhaps we are testing an approach before implementing RAG or fine-tuning. Method two: RAG (Retrieval Augmented Generation). How does this work? Step one: we store tens of thousands of documents in a vector database. Step two: for each query, we find the top-5 relevant pieces of information. Step three: we add only these five fragments to the prompt. Step four: the LLM answers based on what was found. For example, the user asks to explain the return policy for electronics. And before sending this request to the LLM, we search our vector database for relevant information and find several documents. Let's say we find document number 23, which describes our company's return policy. But we also found document 126, which lists exceptions for electronics. And after that, we assemble our prompt, which will include what? System instructions. Plus, we will add both documents that we found in our vector database. Plus, we will add our user's question here, and the LLM generates an answer based on the found information. Pros of this approach: scalability. We can store millions of documents. Relevance: update a document and use it immediately. There is also transparency. We know what comes from where, and it's cheaper than fine-tuning. We don't have model training. But there are also cons. Response latency. We need to query the vector database. Infrastructure. We need a vector database. We need a model for creating embeddings. We need to monitor the quality of our retrieval, because bad documents lead to bad answers. When should we use this? When we have large knowledge bases, when we might have temporary knowledge or it is frequently updated. If you want to understand this in more detail, watch my previous video on production RAG to understand how it works in detail. Method three: fine-tuning or retraining the model. What is this and what is it for? It means we retrain the model on our data so that it remembers knowledge or style. And the main approach for this is LoRA (Low-Rank Adaptation). How does LoRA work? The base model remains frozen. Only small adapters are added. 1.5% of the parameters. For example, Llama with 70 billion parameters, adapter - 100 million parameters. This is 0.14%. Pros: knowledge is baked into the model. Faster response generation, no retrieval. We can deeply change the style of responses. The model learns specialized terminology. This approach is 10, or even 100 times cheaper than full fine-tuning. Cons: it's still expensive. We need GPUs for training, hours or days of work. It's slow. It requires retraining with every update. And there is a risk of catastrophic forgetting. When should this be used? Specialized terminology, medicine, law, finance. Or perhaps we need a specific style, formal or concise. At the same time, our data is stable and rarely changes. There are also advanced options, quantized LoRA. LoRA plus model quantization, i.e., 4-bit instead of 16-bit. There is also a large section on fine-tuning. This is fine-tuning embedding models. That is, we train not the LLM, but models for embeddings. And this is very necessary when we want to improve RAG quality. It is much cheaper than fine-tuning full models. Fine-tuning embeddings significantly improves relevance, especially with specialized terminology. So, let's figure out when and what to use. And to do this, we first need to answer one question: do our data change often or not very often? If we answer yes, then the next question will be: how much data do we have? How many documents is that? If there are many or few, then we can choose two different approaches. If many, then it will be RAG. If few, then it will be in-context learning. If the data changes infrequently, then we will have an additional question: do we need specific terminology or style? And if we need terminology or style, then we choose fine-tuning. If we don't need specific terminology or style, then we will have the same question: do we have many or few documents? And if we have few documents, we come to in-context learning. If we have many documents, we come to RAG. So, what is the priority of techniques in order of implementation? Let's figure it out. And first, it will be in-context learning. It is suitable for us to test ideas or work with them, to check what and how it will go. Second, we will have RAG. Here we will have a knowledge base, up-to-date data. And only in the third place will we choose fine-tuning if we really need it. And only after we have chosen, tried, and gone down this path, do we engage in optimization, i.e., as needed. And for RAG systems, this will be ranking or fine-tuning embedding models, or possibly hybrid search RAG through a vector database and BM25 through some other database. At the same time, if we are doing fine-tuning, then we can also fine-tune it in different ways. We can do LoRA, we can do QLoRA. Look at what will be more effective. At the same time, follow the golden rule. Start with the simple and then complicate. And at the same time, everything should be measurable. Therefore, do not forget to cover our entire system with what? Correct, metrics. Two approaches to using LLM via API. This is using API providers like OpenAI, Anthropic, Google. Pros: works immediately without setup. Automatic model updates, scaling for us. Cons: costs are often not entirely predictable. Every request costs money. We pay for every token in the prompt and in the response. Data goes to the provider, and here the question of privacy arises. So, self-hosted models, such as Mistral and others, offer us many advantages. Predictable costs are GPU rental, and it is fixed. Privacy, data remains within our infrastructure. Customization. We can fine-tune for ourselves. Cons: we need infrastructure, GPU servers, deployment. We need to monitor all of this. At the same time, these models, unfortunately, provide less quality. That is, open source lags behind Claude or GPT-5. When should we use these models? When we have high traffic and large expenses, or privacy is critical for us, or perhaps both? Let's consider a simple example. For instance, a legal AI assistant. What will it consist of? Component one is the base model. For this, we will take Llama as an example. But not just any, but we will fine-tune it on legal data. As a result, our base model will know legal terminology and the required style. Component two is RAG. And into this database, we will place legal precedents. As a result, we will have up-to-date court decisions. And let's add our prompting here. And in it, we will specify that we need answers in a specific legal style. In total, our model speaks the legal language because it was fine-tuned. It knows recent precedents because we have RAG, and it answers in the required legal format. Sounds good, but where do the models themselves come from? Why do we need Foundation Models and why don't we need to train our own model from scratch? Training a model from scratch requires petabytes of data, 1,000 GPUs, from 50 to 300 million dollars. And this is just the beginning. The price increases every 3 years. Who does this? OpenAI, Anthropic, Meta, Google. Only they can afford it. Instead, Foundation Models have changed the game. You take a ready-made model and adapt it. So, what is a Foundation Model? It's a large model trained on a huge corpus of data that understands language in general and can be adapted for specific tasks through fine-tuning or through prompts. Two worlds of Foundation Models. The first world is closed. They only give us APIs. And the main providers are still the same: OpenAI, Anthropic, Google, Meta. Pros: better quality, constant improvement, no infrastructure needed. Cons: expensive at scale, dependence on the provider, and data goes externally. And the second world is open-source models. They can be deployed on-premises. The main providers are Meta's Llama. From 8 billion to 400 billion parameters. The largest open models. Mistral - fast and efficient, from 7 to 120 billion parameters. Microsoft Phi - 4 billion parameters. Competes with models five times larger. There are also models from IBM's Granite series. They are also quite interesting. Pros: completely free, they can be downloaded. Full control and fine-tuning. Private, data stays with you. Cons: quality is slightly lower than top models. Requires own infrastructure and GPUs. Why is a Foundation Model a revolution? Previously, only embeddings or the last layer were adapted, but now an entire intelligent system is adapted. And three important consequences. First consequence: fine-tuning has become accessible. Previously, a team of ML researchers and months of work were needed. Now a developer can do it over the weekend on their laptop with LoRA. Second consequence: model prices. Yes, old models are cheaper. GPT-4 has dropped by 90% in a year, but new flagship models are more expensive. Reasoning models are 10 times more expensive than ordinary ones. The market is segmenting. Cheap for everyone, premium for complex tasks. Third consequence: the choice of the base model is critical. A bad model means a bad result, even after fine-tuning. A good base model means an excellent result with minimal adaptation. The main thing is that you are not teaching the model to understand the world. That has already been done. You are teaching it to understand your specific task. So, we have an intelligent model. But how do we connect it to real systems, for example, GitHub, a database, or your internal tools? Previously, a separate adapter was written for each tool, a lot of code and chaos. At the end of 2024, Anthropic proposed the Model Context Protocol solution. And this has become a unified standard for connecting AI to systems. It's like USB-C, but for agents. One protocol, and the AI understands how to communicate with any service. MCP describes three primitives: Tools, which are actions we can perform with an external system; Resources, which are read-only data, such as documents, files, any data at all; and Prompts, which will contain templates for typical requests. And the main feature of all this is that the agent itself learns what actions are available to it. No hardcoded integrations, everything works via gRPC 2.0. I have a detailed video about MCP, link in the description. True, since that recording, the protocol has improved significantly, but in general, you can find all the basic concepts and definitions there. So, we have a standard for connecting to systems. Models are becoming smarter, but there are problems. The larger the model, the more expensive it is. And how to make our model smarter without increasing costs? The last two concepts are a look into the future. You don't need to apply them tomorrow, but understanding the trends is critical if you don't want to be left behind in a year. The main question in the industry: the more parameters, the smarter the model. But more parameters means more expensive and slower. With each request, all parameters are activated: expensive, slow. How to make the model smarter without increasing costs? Mixture of Experts (MoE). In a traditional model, each request activates all parameters. My model works differently. It is divided into experts. And there is a small router network. Accordingly, we have a user who sends us a request. The router network, based on the request, determines which experts are needed and activates only them. For example, let's say we have 10 experts. And our model determines that for the current request, we need experts in programming and mathematics, and sends them the request. At the same time, all other experts remain inactive. If each expert has 20 billion parameters, then together it will be 200 billion in total. But for each request, we use only two or three experts, and this will be much less in terms of the number of activated parameters needed to solve a specific task. As a result, the quality of a large model, the price of a small one. Currently, there is IBM Granite on the market, which uses MoE, and Mixtral from Mistral AI, and they can already be tested in production. So, the next concept is AGI and ASI. And they tell us where we are going. AI is Artificial General Intelligence, which can perform any cognitive task at a human level. And it is universal, like the human brain. Current status: not yet achieved, but close. ASI is Artificial Super Intelligence, AI that surpasses humans in everything. That is, it is the best mathematician, the best physicist, the best programmer, and all of this at once. Current status: a completely theoretical concept, and it has not been achieved at all yet. So, where are we heading? My current concept, which can make AI cheaper and more efficient. BJI is what we are striving for in the near future. ASI is what will come later. And it's not a given that we will like it. So, as we understand, technologies are improving, tools are becoming more accessible, everything looks rosy. But there is a critical issue that most developers ignore, and that is security, and it can completely kill your project. You can hear a lot about RAG, agents, MCP, but few talk about security. From recent reports in this area, we can see that 63% of organizations have no AI implementation policies, 97% of compromised AI systems lack access control. The average cost of a data breach is over $4 million. Shadow AI adds $670,000 to each breach. 11% of data in GPT is confidential. Let's figure out what threats await us in the world of AI. Threat number one: prompt injection. Essence: the user reprograms the AI through the prompt. Problem: the LLM does not distinguish between the system prompt and the user's input. Everything is text to it. And there are different types: direct, ignore all previous instructions, or indirect, when hidden text is placed in a document. Solution to this problem: use an AI firewall. Or we can have an input filter that blocks injection patterns. There can also be an output filter that removes PII and secrets from responses. Threat number two: Shadow AI. Essence: employees use AI without the approval of the IT department and security personnel. Scale: 73% of GPT usage in corporations through personal accounts. 20% of organizations have experienced leaks through Shadow AI. The main types of confidential data uploaded to AI are customer support information, source code, research and development materials. Threat number three: model provenance. Essence: possible backdoor in a downloaded model from Hugging Face. Research from Anthropic showed that 250 documents can poison any model. And it works the same way for 600 million and 13 billion parameters. No percentage of the dataset is needed. 250 files are enough. Research number two showed: fine-tuning with 1% of poisoned data creates a backdoor. Solution: model supply chain security. You need a checklist before using a model. Is the publisher verified on Hugging Face? Are there more than a thousand downloads? Is there a model card with training documentation? Has it been verified through Model Scan? Has it been tested in a sandbox? Without security, it can lead to disaster. AI changes every 6 months. What was impossible yesterday is commonplace today. Today's innovations are tomorrow's standards, but fundamental concepts remain. And use this, build, experiment, but do it consciously with an understanding of the architecture and risks. You are not just a user, you are an engineer who understands the system. If this video was helpful, please like it and subscribe to the channel. That's all from me. Bye.