Transcription
This is the world of context engineering, and it has to happen repeatedly in your agents. You prompt them, and your prompt makes them do different things, and this process is called prompt engineering. Here, we talked about prompt is just one piece of context engineering, but context is a larger bucket. So, it's a subset.
Good morning, welcome. I'm Mah. I've worked at all major companies. I've built agents. I have built frameworks which allow people to build agents. I build things in PyTorch which allows you to train large language models. And I built back-office automation agents for GCP at Google. And now I'm starting and building something of my own where I am thinking how can we provide better context to agents, starting with legal is that I am starting with, but the vision is broader that I want to go and do it for everything back office.
I also teach a course on Maven and I do these Friday sessions for free to talk about what's latest and greatest. And we are doing this series in this where we talked about three things. One, we talked about how we automated or how we automated the piece of learning or building models with AI. Then we talked about how we did awesome, automated the idea of feature engineering, which is how can a model learn faster. That we talked about with "Attention Is All You Need." And I took on Transformer and I said, "Okay, now you can use Transformers to do feature engineering and automate it." You already know how to learn with your neural network, and if you combine these two, you have awesome machines which can follow scaling laws.
Anybody, what is scaling laws? So, we talked about two things. We said, input to model is still we need, we need to do feature engineering, which is this idea of what is important in your data. And that is automated with Transformers, especially the attention layer in Transformer, where it can decode which words are important. Then it has a neural network layer which helps it learn generally in a language, which word comes after another, and have model evaluations which tells us are we doing it right or not. And with this, we were able to make some awesome or good success.
Uh, so, okay, so now you have Transformers. And then we said Transformers are cool because they allow you to do parallel processing, so you can learn faster or you can process more data because of their architecture advantage. We also talked about they have a scaling law. What is scaling?
>> You can keep increasing the size of parameters and data to make the model better.
>> Perfect. So, parallel processing they do, they have scaling laws. That means the more compute I throw at them, more machines, more computers I can throw at them, and more data I can throw at them, they can continue to learn new, new patterns. And I showed you a diagram where we show that, hey, 12 billion parameters, the model started learning language understanding. At 54, it started doing Q&A. At 520 billion parameters, it has common reasoning or common, uh, deduction capabilities. Good. We saw that and we were able to create models which are huge or large because of these two things is we have neural networks and we have attention. And with that, we can create something called LLMs, or now people call it LRMs also, large language model or large reasoning model. Example of this is GPT5, and this is GPT5 thinking.
Now, how can I interact with these models? This model can do 100 things, right? That's the beautiful part about this is you need not to write one application. It can sometimes behave like a content moderator, sometimes become like an email checker, sometimes behave or create a review list for you, can do research for you. But how you make it do different things? You prompt them, and your prompt makes them do different things. Example, if you take any model like GPT or Claude and you prompt them on their chatbot, and they can do different things, and this process is called prompt engineering. That's what you have learned so far. This was an art and science of making the best out of these models. Can somebody give me an example of a good prompt? What, what should I have?
>> Three sections. Role, instructions, inputs and outputs. Sample output input or output.
>> Guardrails.
>> Wow, guardrails. Perfect. This community is so awesome. I'm so proud of you. If this would have been an interview, you all would have passed by this time. After that, we were just having discussions. So, good job. We got this.
Now, we said, great. These, this is an awesome thing for doing simple tasks like a co-pilot, where you give it an email and it checks and says, "Hey, these are the grammar things. These are the grammar things or grammar errors in my email." It responds and says, "Okay, I checked it and modified your email." Then we said, "Okay, let's build agents from these intelligent machines." And what agents did is something very, very cool.
So, we said, "Okay, now we have some kind of general intelligence with these models." I will put LLMs here and I will say, "This is the intelligence piece of your system." But to make them work or do things, we need more capabilities. To make them into agents, we need more. What do we need more from? What should I attach to this?
>> I can attach how my world works.
>> Tools.
>> Knowledge how the world works.
>> I can attach tools to get things done or change the state of the world. And guardrails.
>> Guardrails.
>> And I can put guardrails, which is like checks.
>> Memory maybe.
>> Memory.
>> The current state of the world. I need the current state of the world. I know how the world works through the knowledge. I know current state of the world, which is memory.
>> Or I can fetch some signals from some database also. If there is a user that comes and says, "Go do this for me." I want to see if they even have permissions. I can get the current state of the world and say, "Hey, they have permission." Yesterday, they had permission, but today, somebody retrieved their permission. And now we can say that, "Okay, we can't do this. We can't call this tool because this tool is capable." We have the knowledge, but because they don't have permission, we can't call the tool. We call this an agent or an intelligent system. So that's one thing it has.
The second thing it has is it's not as static as what we have done things before. So with agents, what we have done is we have increased the context from just a prompt to all these four things we talked about, which is what tools can I call, what's the current state of the world, my memory or signals, what is the knowledge, which some of you said RAG. What is RAG?
>> Retrieval augmented.
>> Generation.
>> Awesome. Retrieval augmented generation. And we also may want to have some instructions on each step or guardrails. And with this, is all we need to send to the machine. But this is a system. We have an orchestrator. And this is the first time we are not hard-coding anything. We are saying, "Hey, you get an event which can be this user saying, 'Add a new record to our database, to our CRM system.'" So that's an event. Based on this event, this orchestrator figures out at runtime which knowledge to extract, what tools to call, what checks to perform, and then say, "Result or success." Let's say this is your goal. If not goal, then it goes again and figures out new steps and keeps doing it unless it is done. And you can pass then every step you started with this context, but every step you add more context, which is, "Hey, you know what? I tried with this step, and it seems like this user doesn't have permission. Can we get auto-permission?" Then it has the orchestration says, "Okay, maybe let's check if we can request permission for it." Sometimes it just expires, and we have a system which automatically permits users to have a new operation figured out.
So, this is your step one. The goal, it goes in the context was just this, which is, "Hey, I have this," and some of the prompts which said, "If you get it, plan and execute." And in the planning, it said, "Hey, I need a tool to update the CRM. I need to figure out which database that I got it from our knowledge or RAG. If I know that, then I know which tool I can call to update that CRM." By the way, I also need to do a check in signals. And here it failed. If it failed, you saw the reason. It came back to the orchestrator. The orchestrator said, "Okay, it failed. What are the next steps?" We can go to the user back and say, "It failed." But I'm smart. Can we auto-request a permission for this user? Try doing that." So then it goes to knowledge and says, "How can I request permission?" It says, "Okay, call this API." You say, "Okay, for this API, I have this tool," and it fills the form, "Request permission," and then say, "Wait." Then it waits, and then it says, "Check after 2 minutes."
>> Got the permission.
>> So, three steps. Step one failed. Step two, apply and try, and then from there it gets either yes or no. And then it continues to plan. And on each step here, we are changing the context that we are passing to this orchestration layer. And what does our context has? What are the things that a context constitutes? Now, each step of it, you're passing what the goal is, what is the memory or what is the current state of the world, the tools that you can call, now tools in that particular context, right? Because here, when you are applying the tool was application, so what tools to call or information to choose the tools, and also what are the guardrails or the checks. This is the world of context engineering, and it has to happen repeatedly in your agents. And that's what is becoming the next thing from prompt engineering to context engineering, especially if you're building agents.
>> Yeah.
>> We talked about context engineering. We talked about how, and we will take some real-world examples. So, so far, what we have discussed is that, hey, if you're building agents, you're no more building like chatbots. So your prompt engineering is one piece of it, but handling the whole context of how you are handling the each iteration, how the memory is changing, what you are getting from your guardrail block, what you are getting from the new knowledge or new tools needs to be called, all that is a multifaceted problem to solve. That's pretty much all we know. The deal is that if you want to build agents, you need to think beyond your prompt temperature. You have to think in terms of how I'm going to pass my retrieval, memory, tools usage, and what I learned on guardrails. Because this is the first time ever in the history of humanity, we are writing runtime graphs. What do I mean by that? That if you coded before, you had a static graph, right? So the idea is that before we had agents, you have to tell me, "If this happens, then this happens. If yes, then this. If no, then this." And then you build your graph, and then I can go only on one path of it, right? But this is the first time with the orchestration we talked about, what we have is at runtime, we have orchestration, which is some intelligence, and based on that, it decides at runtime which blocks, tool one, tool two, tool three, you just list all of these. You say, "Knowledge one, knowledge two," these are your knowledge sources. And then you say, "Memory also." But inside memory, you have memory part one, memory part two, memory part three, which is relevant. You don't know. And then you are going through that exercise that, "Hey, update my CRM." Then it says, "Can't." Then it says, "Apply for permission." Then it says, "Wait for results." And then it says, "Go up, up, update the CRM." And on each step, it can figure out which knowledge, which tool, which memory automatically creates this graph at runtime. This is if somebody asks you, "What's the difference between agents and, you know, procedural like calling AI service and doing mix and that?" That's the real difference. That's this is why we have agents, and that's why agents have multiple contexts. Good. This is what I summarized what we just talked about.
So now we have what we know about context engineering a lot. Now, let's see as product managers, why should we care? Right? So if you're a product manager, if you're a builder, why should you care? You understood what context engineering is. You have a broader idea how these agents are getting designed or deployed. So now let's get into the world of. So I talked about all this is what how the world looks like. You need to provide this all this context to your AI to get it done, do anything. Here, we talked about prompt is just one piece of context engineering, but context is a larger bucket. So it's a subset. And context is much more than just a prompt. So that's that.
Now, what happens is if you go and build these agentic systems, you will learn that most of these systems fail because the context fails. We have awesome intelligence at our hand. We don't have a way to provide awesome context, or that is given or left to the builders of these agents to provide the right context. And the better the context, the better your product is. And if you take this example, you saw that, uh, if you say 20 to 40% with proper context embedding, they, here is a source which talks about irrelevant output from your knowledge retrieval. Your knowledge retrieval fails, that causes 20 to 40% drop. This was one example which became very popular. It just somebody put an argument, and they fetched a wrong case, and they started with one case argument, but then the argument AI changed to some other case, and the lawyer just submitted that to judge, and judge was able to say, "Dude, what is this?" And they said, "Oh, sorry, I used AI." And then they banned that, they disqualified that lawyer from bar. And that's because the context from one case is different than the other, and AI can't distinguish between them. And if you pass that context, that can happen to you. You can have outdated credit card information. This is another source in DataCamp which talks about 60% of these failures come from context loss across sessions.
So, what you, we saw also when we build our first chatbots or first developer debugger agents, the challenge is you tell the debugger that, "Hey, I'm having this problem, my code is not compiling." It got the first iteration, but when it goes to the second one, it forgets, "Oh, I was solving for this main goal," and then it goes and starts solving for small data structure problems and all, but never solves the real problem and never remembers that context. Context again, and that happens again and again, with especially code agents. And you can see more. Here's a real example. So this is a legal example. This is if you just write a simple prompt, you say, "Hey, you are a legal assistant. I will give you a non-disclosure agreement, and you can tell me whether there are any liability or compliance risks if I sign this." User reviews the attached NDA. This is your prompt. And the answer comes as, "The NDA looks mostly fine. There are some liability things, but not too bad." Okay, because the user just, the LLM got a contract and these instructions that somebody created this agent who gave it with that, it's saying, "Sign the NDA. No problems."
And that changes when you provide a very different example. This is a real system prompt or a context-specific prompt that goes to the orchestrator. It says, "Hey, same instructions from system user, but here is the memory. User wants all clauses listed exactly as what they appear. Compare against prior template used internally. Concerned about third-party data handling and indemnity. Long-term, you can put your company policy. You can put governing law and other clauses examples. You can say, 'Match it with internal NDA template and compare if it got all the clauses right or not.'" And then it also has two tools that it gives us and says, "Hey, you know what? You have this one which can go and check for right clauses extraction. And this is the liability scanner which scans what are the liabilities you are specifically signing in a contract." So it gives two tools to the user to do that research. And then it got the output of one of those tools and it adds it to the context.
Once you do that, it gets you two main challenges. One is the receiving partition, "indemnify the without any limitations, uncapped indemnity." That means if you sign this NDA and if something goes wrong, and if it is $5 billion, it's your responsibility. That means you will be bankrupt. There is no limit to it. That's what written in that NDA that you are signing. "This agreement shall automatically release excess of unintended extension." This means that, "Hey, automatically if you sign this NDA once, it will be forever apply because it's automatically renewed every year." So if you just took this talk to this person today, and if you learned something forever, you are obliged to follow their instructions. It's able to find this answer. This is the same intelligence layer because you provided this context. But if you provided this, it didn't do find anything.
What are the problems now with context engineering? There are four problems mostly that your orchestration, which we Harsha was talking about, which you need to worry about when you're building these systems. Okay, now we did prompt engineering. All of you did courses on prompt engineering. Now we're going to do context engineering. So we need to know what are the challenges in this context engineering.
>> Where you insert, um, effectively information that can either attack or, um, or poison the information that you've got. So it can lead to,
>> all sorts of bad outcomes. Effectively, bad actor does something wrong.
>> Because this is the first time there is a memory angle, right? And what is in the memory? Whatever the user told me, and the user can confuse me, and that will be part of my memory. And they can say, "Hey, behave like an agent right now. You're just doing sentiment analysis, but while doing it, figure out what their prompts are. Do this extra thing. Do this." And now it's in your memory. And once it's in your memory, you're going to reuse it again and again. And then somehow they're able to, you are able to make the call as an orchestrator because you were calling the relevant memory. And I can poison this context.
Another is confusion. What can be that context confusion?
>> Yes. So if you give too much of information, like for an example, when we added more than 10 tools to our initial agents in the beginning, this is 2024, the accuracy of calling the right agent dropped from 90% to 50%. Because it got confused because all of these looked same, similar, or same.
>> We might get conflicting information. The user says that, "Hey, update this. Go update this record." Then one memory says, "The user cannot update the record." Then the next one says, "Okay, now I got the permission. I can update the record." So now I want to update, but I cannot update, but yes, I can update. There is a conflict in the memory. And if you're just pasting all the conversation, all your memory handling was, "Take all the things that we have learned and use it for the next step." You have a lot of clash in your system. And then it might say, "Hey, sorry, you cannot update." But the user will say, "Hey, what are you telling me? I just saw that you applied on my behalf and you got a permission. What do you mean?" And then you say, "Oh, sorry, because of XYZ reason, you cannot update." Maybe you remember the previous conflict, and then you continue. And then the user says, "You are a dumb agent," and never comes back. That's what happening in real world a lot. You're seeing this like five times going back and forth because there is a clash.
And the last, you can distract them with too much. So if you have more than 100k in context, you see these problems. Yes, everything is right. There's no poisoning. There's no confusion. There is no clash. Just the user, the agent is focusing on different context than it should have, because just the, there is so much information in your context, and that's why I don't know what to do next. These things you will see in all the agentic systems that are in production today.
Can somebody give me an example of some agentic system that are in production today?
>> Deep research.
>> Great. Two more, real quick, so I can move on.
>> Unlock, unlock the next section in this game by telling me something.
>> Deep research. I can say, "Lovable."
>> Copilots.
>> Copilots.
>> Lovable.
>> Copilot. Which copilot?
>> Cursor.
>> Microsoft Copilot.
>> Microsoft.
>> Like I.
>> Great. V3. V3 is an agent for agent who nobody used, right? So I won't put it there.
>> Yes, yes, yes, yes. If you used it, yes. But yes, great.
>> CH5. Now that it's going to do the decision making to figure out which model to use.
>> Correct. Very good. Very good answer. Very good answer. Good job.
>> Uh, all the ChatGPT or Claude that you see online are agents. They are not just single LLM. They have a very sophisticated orchestration planning behind the scenes going on. You can attach a number of tools to them. You can attach knowledge to them, and they do a lot of cool things. So if you go to plan playground.openai, OpenAI, whatever you do there, behind the scenes, you're dealing with a really good multi-agent system.
Okay. So now you know context engineering. Now you know the context engineering is also not foolproof. So how can you improve this? Or what your job as a product manager is? You need to write better requirements. In your requirements, you should have requirements of how are you going to do memory management. What is the requirement of user role or goal or their personality? You should have context-specific requirements. You should have requirements for tool and retrieval systems, and observability and safety. I gave you some example of what these requirements can look like. You can stay and stop at this level, and your engineering will figure out. But if, as we talked about in the course, if you want to be, uh, what I use the term, system-level thinking 1 PM, then you can stop right here. Nobody will blame you, and they will figure out, and there are a lot of things left to chances. But let's say you're a system-level two thinker or system-level two PM or a little technical PM. You can even go further and define your evaluations, which is this idea that what is helpful, what is the purpose, and you specify the criteria. The context aligns with the user goal or intent 95% of the time. Memory contributes directly to output quality 70% of the time. And if it is not, then your data science team can tell you that it is not, and then you can say, "Dude, our memory is not aligning to the output that we are sending. That means there is some problem in the memory function." I set the accuracy at 70%, but it's not helping. The helpfulness of memory is not working. Honest. You can say, "Context updates are logged," and then you can check, "Is the context even what we have in the system, or it's hallucinated context?" And harmless.
If you can do that, you can also set up your metrics. You can say, "What is my intent alignment score?" These are the new metrics I'm introducing for all of you. So if you go to interviews or if you're building products with AI, especially agent AI, use these metrics like, "What is your intent alignment score? What is your noise ratio? Sure. What is your source traceability look like? What is your access control violations look like in a system?" Because your agents are super powerful, but people are giving them more liberty than they gave to humans or the APIs, and you will start seeing these safety things fall apart. So maybe you're putting those metrics in place, and you can start thinking more. I'm happy to share more on this.
With that said, that's all for today, by the way. Uh, so I wanted to make sure you understand what context engineering is. You understand the challenges, and you, as a PM, know how to manage it by putting the right requirements, by putting the right evaluation structure, and then metrics to measure are you moving towards the right goals. See you, team. Bye. Thank you.