Transcription
After building over 200 nocode AI automations, I've come to realize that context engineering is the most important thing in determining the quality and consistency of the AI systems. So today I'm going to be going over the six most important context engineering lessons that I've learned, breaking it all down and applying it to NAN AI agents. So I don't want to waste any time here. There are timestamps down below if you want to jump around, but let's get right into it.
All right, just to start off here with module one, intro to context engineering if it's a term that you've never heard before. So context engineering is the art of feeding your AI agent the right information that it needs to complete tasks effectively. Unlike prompt engineering which focuses on crafting the perfect single instruction, context engineering is about building systems that can dynamically provide relevant information to the agent. So in a nutshell, the agent should be able to receive some sort of message that triggers it. It should understand, I have all of these different tools. I have all of these different sources of data or information that I can look at. Which ones do I actually use to get information to help me answer or do the right task?
Now, the problem is most AI agents today that aren't using a lot of context engineering, it's like having a conversation with someone who forgets everything right after you've said it. So, the solution is after the agent reads through its system prompt, it should understand, I have these different tools that I can use to get more context and this is going to transform me from a simple question and answer tool to an actual assistant that can remember things and take action intelligently.
And I just wanted to say that this isn't a new concept. It's been around. I think the term context engineering is a little bit more on the newer side, but it's always been around. These are videos I've made multiple multiple months ago talking about the foundation of AI systems and automations is data and context. And over here was a completely separate video where I talked about context is absolutely everything. The systems are only going to work as well as the data and context you feed in. Data in, data out, garbage in, garbage out. And especially if you're working on systems for a client or something like that, you're really going to have to leverage their subject matter expertise to train the systems to behave the way an employee in their business would.
Also, you may have noticed that kind of behind me, my background is very bare. This is probably the last video you'll see from me in this apartment as I'm transitioning to my new apartment, which is pretty exciting, but sad day. Anyways, if it's still not really clicking and you want a bit of a deeper dive on some of the more technicalities of context engineering and where the term came from, then definitely check out Cole Meen's video he did covering this topic. I'll link that in the description below. But in our video, we're going to be focusing more on actually bringing this into Nen and on no code solutions.
So there's really these six main components of context engineering. So let me quickly break it down. So just to go through the chronological flow of an AI agent, the first aspect is the user input. This is basically the dynamic request that we're asking the agent to do every single time we trigger it. So it receives the input and then the second piece of context is the system prompt which has been around for a long time. Prompt engineering. This is where it will read through its instructions to understand okay what tools do I have? What do I need to do? What is the user asking of me? The next piece is memory where it can go check what are my past conversations I've had with this person. Is there anything that we've already talked about that can help me do my job better? Then we have retrieve knowledge which can kind of go handinhand with tools but I just broke them up for now but you know subbase right here it is a tool and then so is send a message in Gmail but the idea is that it can use its tools in order to access different knowledge that it has. So that could be a vector database it could be an API call to search the web it could be looking up something in a CRM. That's all an element of getting more context to take action more effectively. And then the final piece would be if we have like a structured output parser where we're going to tell the agent this is how we need you to output information.
So hopefully this diagram looks pretty familiar to you if you've built a few agents in NN already. And that's sort of the breakdown of different ways that we can give the agent more context. That doesn't mean that every agent or every system we build has to have all six of these, but these are the different things you can kind of tweak and play around with. And this is a really cool analogy I heard which is kind of like the difference between prompt engineering and context engineering. Prompt engineering is like studying for an exam, you know, the week before. But context engineering is showing up to the exam with a cheat sheet that you can look at every single time you're faced with a problem you don't know. For the best results on that exam, you're going to want to have both good studying and a good cheat sheet. But ultimately, if you only had to have one, I'd probably rather have a cheat sheet.
All right, so moving on to module two. This is about memory systems in AI agents. So there's kind of three categories of memory that we have in Naden AI agents. The first one is working memory, which is just basically the agent's processing things. It's using its system prompt and it's using its chat model in between actions to figure out what did I just do, what do I still need to do. And this is on an execution basis. We have short-term memory, which is kind of like just conversation history, a brief context window of what has been said to whoever is currently interacting with the system. And then we have long-term memory, which is a bit more of that persistent knowledge that can survive across sessions. So when we have all elements of proper memory, our agent can be able to remember user preferences, remember previous conversations, and also maintain context across multiple sessions.
So when it comes to short-term memory and we're talking about a context window, we have the ability to choose what that context window length is. So in this case, what we have is a conversation between Nate and an AI agent. Because we gave it short-term memory, it's able to make sense as an actual conversation. I said, "Hey, my name's Nate." It said, "Hey, how can I assist you?" I said, "I have a dog named Workflow." It said, "That's a great name. How old is workflow? What kind of dog?" I said, "He's a golden retriever. What should we do this weekend?" And it answers because it remembers that we have a golden retriever. It answers all this kind of stuff because it knows our retriever is named workflow. And these would be three separate interactions that would be stored as a pair. So if our context window was two, it would only remember the most recent two interactions that we've actually had. And we have the ability to change that context window length if we actually come to the memory system that we have set up whether it is simple memory in naden or postgress memory or z memory whatever we want to set up. So right here you can see context window length we chose three but just keep in mind something we're also going to cover later in this video is the longer your context window the more tokens that will be processed. So it will be more expensive but your agent will retain more context and more conversation history.
And the next aspect of this short-term memory is a session ID. So, we're able to set a session ID, which is pretty cool because our agent can have unique conversations with person A, person B, person C, and keep them all separate in its memory. So, we could link something up as a session ID like an email address. So, every time an agent receives an email, it's going to look at the email address and say, "Okay, let me go see conversations I've had with this email address, and then I'll be able to use that conversation history to answer the question." Same thing with like a phone number. if it is a text SMS bot or something like that or employee ID if it's like an internal Slack agent whatever the case is the session ID can basically link back to one person and then when we come to think about long-term memory this is persistent memory that survives across sessions and in N&N it can be stored in multiple different ways we'd have a user graph with something like Zep which I have a video on and the benefit of a user graph is that you can see there is a user in the middle which in this case is John Doe but then it has different relationships and it understands preferences about the user. Things like what do they like, where do they live, what do they do on a daily basis, and the relationships is what makes this user graph powerful rather than just a bunch of random facts that may not link together. You can also really simply just store memories in a Google doc, and then you can tell your agent, this is a Google doc with long-term memory about user A. If you want to look up information, use that tool. It could also be a vector store, which would be more of a chunkedbased retrieval. Or it could also be something like a CRM where when a request comes in, you're going to look up that lead or that client in the CRM, find out information about them, and then use that to tailor your response back or the actions you need to take.
All right, moving on to module three, which is about using tool calling for rag. It's kind of a good segue from that end of module two because we were talking about using tools to access something like a Google doc memory or superbase or HubSpot. And tool calling can also be referred to as function calling, especially if you're looking at like some documentation about an AI model. But what it allows us to do is it allows our agent to interact with these external systems, send off a request to receive data back, perform actions beyond just generating text. It's like giving your agent hands and feet in the digital world. So, you know, chatbt on its own, if it didn't have any tools, all you could do is kind of talk to it and have a conversation. But if you wanted it to send an email for you or talk to your nan workflow or whatever it is, you'd have to give it access to that tool or that function.
And then rag retrie augmented generation is a technique where AI agents retrieve relevant external documents or data at query time and uses it to respond more accurately. So the most simple analogy I like to put it in terms of is let's say I asked you what's the capital of California. If you didn't know the capital, you would probably look it up, which would be your aspect of rag, looking up information, retrieving information. So you can generate a more accurate answer. It does commonly get associated with vector search rag or you know chunk based rag which is why a lot of people think of a rag agent and they immediately think of a vector database. So of course that's the first example which we have a vector database rag down here. This first flow is basically we're putting a Google doc from our drive into a superbase vector store. And then up here we have an AI agent and you can see that under the tool section it's connected to a tool called superbase vector store. So now when we ask our AI agent a question, it's probably not going to know the answer because it's pretty specific to the Google doc, but it knows that it has a tool called Subbase Vector Store with information that it can use to answer the user's question more accurately.
But like I said, it doesn't just have to be a vector database. Here's an AI agent that has three different tools. Perplexity for research, Tavly for research, and open weather map to get weather information. So obviously if we ask this AI agent, what's the weather like in Chicago? It's not going to know. But it knows I have a tool called open weather map which I can go ahead and hit which will let me find the weather. Same thing for research. The agent will have information about whatever it was actually trained on using its AI model, but it won't know, you know, an article that came out yesterday about some crazy event. It would have to look up the research on the web using one of its two tools. And then there's also using tools to get information from your internal systems. So something like HubSpot getting contact data or you know project data air table same thing or Google sheets and those are all still aspects of rag because the data that exists here it was not trained on in the AI model and it has to reach out externally to get it and then of course where it gets cool is when you have all sorts of different tools and the agent realizes okay here's what this tool does here's what this tool does here's what this tool has access to now which one do I use and that's where you do need a good prompt engineering strategy in order to make sure your agent understands that.
So, just to put that into perspective real quick with a use case that doesn't have any vector search, let's say I'm talking to my ultimate assistant and I say, "Can you send an email to Dexter Morgan with a blog post about Nvidia's recent earnings report?" So, what would happen here is the first piece of context is the Telegram trigger that sets off the workflow. So, we grab a message from Telegram and then the agent basically will look at its system prompt which is inside the ultimate assistant. It will read the system prompt. It will then use its chat model in order to start to process what do I do next. From there, it decides, okay, the first thing I need to do is actually create a blog post about Nvidia's recent earnings report. So, I'm going to hit my content creator agent, which is Rag, because I need a blog post. So, the content creator agent makes that blog post, and then we come back to the ultimate assistant, who then has to think about what's next. Then it realizes, okay, I'm going to send this as an email to Dexter Morgan, but I don't have Dexter Morgan's email. So, I'm going to hit my contact agent for more rag. The contact agent then comes back with the email address. The ultimate assistant says, "Okay, now I'm good to go. I can actually go ahead and send that email off because I have the two things I needed, which were the blog post and Dexter Morgan's contact information."
Moving on to module four, I did want to have a quick section about chunkbased retrieval because it is so important, but it's something that has a lot of complexities that are important to think about. So chunkbased retrieval, which is usually a vector database with embeddings and stuff, is a technique where large documents are broken down into manageable pieces that can be searched and retrieved more effectively. Now, this is important because AI agents with different AI models have different context windows and sometimes those are limited. So you couldn't drop a 65page PDF at your agent and expect it to be able to see all of it and read all of it and process all of it at one time. But if we chunked them up and we put them into a vector database, the agent would be able to really quickly and easily search through them semantically and pull back the chunks that it needs. The issue here is that with chunkbased retrieval, we're losing the relationships and the context of the entire document as a whole. So, if I dropped in a 65page PDF, it got chunked up and put into a database and I said, "Hey, Mr. AI agent, can you summarize that whole PDF?" It would not do a very good job. There are some things we can do though to make our chunkbased retrieval a little more accurate and customize it to our use cases. So, let me dive into that real quick.
Before we get there, I just wanted to explain visually how this works. So, this is our 65page PDF. We chunk it up into a bunch of pieces. We run it through an embedded model to get turned into a numerical representation of that chunk. And then the chunks are placed in the multi-dimensional space based on the meaning. So let's say this chunk right here was talking about trees, it would get put up here. This chunk was talking about dogs, it would get put over here. And then this chunk was talking about fruit, it gets put here. So in the future, if we put in more chunks and we put in a whole PDF about dogs, they'll all be near this other chunk that means dogs. And we can use things like metadata, which is just data about data, to make the context of the chunks more rich. So in this example, we're putting in transcripts of a YouTube video. So we have YouTube video A, YouTube video B, and YouTube video C. If we put all the chunks in the vector database, we wouldn't actually know which chunk came from which original video. But if we add metadata like okay this chunk the title of the video was this the URL of the video was this and the timestamp of this specific chunk in the overall larger video is this and then when our agent actually pulls back the chunks it knows exactly what video the URL and exactly what time in that video it came from. So that's a way we can really enrich our data and our retrieval and our answer generation with our AI agents using something like metadata.
There's also other things you can do with chunkbased retrieval like reranking or scoring because when you pull back the most relevant chunks, they'll usually be a relevant score. So you can have a workflow where you're taking back a bunch of chunks and then you're only going to keep ones with a relevant score above like 75 or something like that. Or you could use a reranker. So I'll show a real quick visualization of what that means. So let's say we ask a question about what do I do if my ball goes out of bounds. That question gets embedded and put into the vector database. We'll grab the nearest vectors and pull them back. When we pull those back, they get turned back into their actual text representation and then we feed all those into the rag agent who will then answer. But if we use a re-ranker, what we can do is rather than pulling back just three chunks, we could pull back 10 and then we could feed them all into a reranker. It will automatically assess which ones are the most relevant and then keep the top three and then feed those into the agent and arguably you'll get more accurate results.
Anyways, here's the actual system I used for that YouTube transcript rag agent. And I just wanted to show you guys this is kind of the pipeline that we have where we're getting the transcript, we're getting the timestamps, and everything we need and feeding that into a vector database. And what we can do is inside right here, this default data loader. That's where we can load in the metadata we want like the video title, the timestamp, and the video URL.
All right, moving on to module five, we have summarization techniques. Summarization is the process of condensing large amounts of information into a concise relevant summary that can be efficiently processed by the AI model. Kind of similar to the theory of chunking because you know we don't want to feed so much information into an AI model. This is really important not only when we're thinking about the context window but also from a cost perspective because the more characters the AI model receives the more tokens it has to process which means it's going to be more expensive for you. So the idea here is if we can take a big chunk of text that has good information that we need but just summarize it in the key points, it will still help the agent create a better answer and it will optimize our cost.
So one cool example of that is a video I did recently with Zep memory which is like a user relational graph for memory. And what you can see is if we use Zep memory natively like this, it was pulling in so much context and a lot of it wasn't even that relevant and we were wasting so many tokens every single time. So what I did is I pulled in context window and user graph separately through an an HTTP request and I was able to control keeping the relevant ones only keeping a few and feeding those into the AI agent and it processed way less tokens and it cut the cost significantly.
Another cool way you could think about summarization is, you know, it doesn't have to be vector database, but in this case, let's say it is. What you're doing over here is you're giving the agent access to the vector database and everything it pulls back gets processed by the AI model, which could be tons of tokens and it could get expensive. But over here on the right is a cool thing that you can do where you're going to do rag in a subworkflow. So same theory the agent will hit it superbase tool to get information but rather than just quering the tool and getting the raw output it will be quering a subworkflow that we built which then queries the tool feeds it into a summarization chain makes it more concise and then feeds that summary back to the main agent and it's basically keeping all the same important information and doing its job for cheaper.
And then finally which I think is probably the most important module is about mindset. So these are the five things I wanted to touch on that I found myself talking a lot about in my communities and stuff like that when I was answering questions about context engineering. The first one is to begin with the end in mind. If you have a highle idea of the system you want to build, you typically know what the agent will be doing and what type of queries it will be receiving. And if you can define the exact queries your agent will send and the document types it needs. This will help you decide whether it needs to fetch full files or if relevant chunk semantic search is sufficient or not. And when you understand the type of knowledge bases the agent will actually need to use, that's going to help you set up your pipeline in the beginning, which is the foundation of everything of how am I actually going to get this data into something that my agent can talk to. Do I need to give it the full doc? Do I need to chunk it up? What do I need to do here? So before you build that database, that source of truth, think about the way the agent is going to use it. A lot of times it's not just going to be one database. Maybe it's multiple vector stores or maybe it's multiple files and folders.
The second point would be about designing your data pipeline. These data pipelines are so important. You have to think about all the sources that you're going to have. Are these going to be static or dynamic? Do they update frequently? Do I delete them frequently? How do I set up an automation to keep all of this relevant? If your agent's looking at a scraped website, how often does the website get updated? How often do we need to re-trigger the automation to pull in the website? Same thing with your database folders, your vector databases, you need to think about your refresh frequency and what happens in the case of deletions. And that leads really well into the next point, which is ensuring data accuracy. The whole reason of setting up context engineering and our rag systems is so the agent uses its tools, it pulls back relevant, up-to-date, accurate information and then answers. So if your knowledge bases are out ofd or inaccurate and the agent pulls it back, it's going to also answer inaccurately. But really the data pipeline is so so important. It is basically the foundation of standardizing your data, preparing it to come in and then uploading it to the right spot. And this is where you have a lot of control because what you want to do is have predictable inputs, predictable standardization, and then predictable loading into a database.
Point number four is to optimize context windows. Load only the most relevant information to control costs and prevent overload. If you're taking a history exam and you need to look up information about World War I, you would have your cheat sheet with clear labels and you would literally just go straight to the little section about World War I. There's no reason you would start from the beginning of your whole cheat sheet, read all of it through all the way down, and then finally get to what you actually need. So, if you can set up systems to make sure that your agents only looking and making requests for the most relevant information based on that user input, then quicker executions, more highquality results, more consistent results, and also most importantly, cheaper results.
And then number five, which I think is really important, is to embrace AI specialization. ever I have an AI agent or an AI model in some aspect of an automation, I try to think to myself, how can I make this AI just do one job really well? It's cool that you can have AI agents do tons of different jobs and all handle it in one like some sort of super agent. But that's kind of the whole reason why with the Ultimate Assistant, rather than giving it all the Gmail tools, all the calendar tools, all the contact tools, we give it fewer tools and then all it has to do is take care of basically delegating the query to the right spot. If you have a process of like four major steps where your agent could do research, write the report, you know, create a message, whatever it is, it would probably be more consistent for you to have one agent take care of each of those steps in the process. Think of it like an assembly line. Everyone does one thing and then they take that input, they do it, and then they pass it off to the next step. This makes it way more efficient. This makes everyone really good and really quick at their own job. And it also helps with your prompting because then you're able to get in there and really specifically prompt this one agent to do its one job. And then the cool thing there is you can have each AI step use a different AI model if you want. We all know different models are good at different things and sometimes it can be tough to see which model should I use for which use case. But if you have everything split up, you can go ahead and test a different model for every single different step in the process.
So that was all I had for you guys today. I didn't want to overwhelm you with too much information, but hopefully it was helpful. If you want to access the link to this Excal so you can take a look on your own time, then you can access it in my free school community. The link for that will be down in the description. All you have to do is join the community and then search for the title of the video up at the top or you can go to YouTube resources, find the post associated with this video and let's say it was this video right here. You would see basically a link somewhere in this post that would take you to the Excalibraw.
And if you found this video interesting and you're looking to deeper dive into some of those aspects of context engineering that I touched on, then definitely check out my paid community. Link for that is also down in the description. We've got a great community of members. Everyone is building every day with NADN sharing their challenges, sharing their wins, sharing their projects. And we've got a classroom section with two full courses. Agent Zero is the foundations for AI automation. So if you're a beginner, a great place to get some structured guidance. And then 10 hours to 10 seconds where you learn how to identify, design, and build time-saving automations. So, that's going to do it for this one. I would love to see you guys in those communities, but hope you enjoyed the video. If you did or if you learned something new, please give it a like. It definitely helps me out a ton. And as always, I really appreciate you guys making it to the end of the video. I'll see you on the next one. Thanks guys.