Transcription
Prompt engineering was always a little bit ridiculous, but context is king. You've got a window of at most a million tokens to tell your LLM everything it needs to know about your business and what you want your agent to accomplish. If you think about what has to fit in there, that's just not enough space to tackle complex tasks in the real world. If this is gonna work, we need to understand the building blocks of context in modern agentic applications and have a framework to help us design context with precision. Let me show you what I mean.
Let's remember what an LLM even is. It's a stateless box that takes tokens as input. You just think of them as little pieces of words and gives us, we'll say, likely tokens as output. It's important to kind of bring back to this core thing. It's very smart, does a lot of really cool things. These likely tokens are extremely useful that come out, but it doesn't remember anything. We give it a bunch of text on the input and text comes as the output for the text mode. And when I said, you know, you have about a million, that's for some models, for some it might be a hundred thousand, something like that. And that could correspond to maybe, you know, 60,000 words or 60 to 700,000 words, somewhere in that range.
The real problem with the LLM is that it only knows what it's read on the internet and it can't do anything. It just gives us tokens on the output. And we want to write agentic applications that take action and that know about what's going on in the business. So for our agents, we'll have some kind of input. That's probably a prompt from a human at some point. This agent now here is going to make LLM calls and probably make them iteratively. It'll probably call an LLM more than once. And it has the capability of calling tools as well. Those will let it actually take action out in the world. And both the LLM call and the tool call can be iterative. This agent can have a workflow that it kind of works out over time. That can be running for a long time, potentially depending on the agent. And that introduces some real problems with context that we need to solve.
All right, let's look now at kind of a schema for thinking about context. What goes in the context window? These tokens, this big prompt, this string that we're submitting to our LLM. There are six things in here that I want to lay out. Let's first say, we'll say here, this is what I'm going to call user message. All right, now it's okay to think of the chatbot interface, the typical, you know, grok, clod, chat TPT kind of thing. You type a thing and it says stuff back. And the you type a thing, that stuff that you type, that's the user message. And that's probably going to be the initial input to our agentic service here, some kind of user message.
That is not all. There is always a system prompt. In the chatbot applications, you have some capability of customizing that system prompt. There's always system prompt, you're not going to see, but you can always tune that up a little bit. And that's going to tell things like, what's the personality of the LLM? Like, what do you want it to be? Do you want it to be a harsh and demanding coach? Do you want it to be sort of a gentle and nurturing counselor? You know, to kind of tell it what you want. Give it some guardrails. Like, don't give me bomb making instructions or be specific and detailed and cite your sources in your output. These are all sorts of things that might go in a system prompt. So it's just sort of up there. You don't have to say all that stuff over and over again. It's always present.
We also have tools that we might have access to. We said the whole point here is that the agent has access to tools and in its terminal step, when it's done with its work, that's a tool call. That's causing an effect out in the world. So as a part of this context, this text we're passing into the LLM, we have a description of the tools that are available to us.
We also have resources. Now this, oh, this is a complex topic all on its own. Because the LLM only knows what it reads on the internet, we have to tell it things that are specific to our business. Like, hopefully all your Google Docs are not shared publicly. All of your Wiki articles and internal support tickets and all that kind of stuff. You know, there really is private data inside your enterprise that is never going to be a part of any LLM's training data set. That's the resources. So we have to somehow do a good job of picking out which resources belong inside this context window into the submission to the LLM. Those are things that are going to be relevant to the user message enough, but not too much. We'll talk more about that in a bit.
You've also got the assistant messages. And assistant is, remember, think of the chat model, right, you type stuff, it says stuff back. It is your assistant. So you give it a user message, and then it says something back. That is the assistant message. So as we iterate through, and this is the case when you're using a chatbot application, you have actually a history of user messages and a history of assistant messages that are accumulating inside the context window. So the longer the agent iterates, the more space this stuff takes up.
Finally, you've got tool calls and responses, call and response tools or calls and results. This is, as an agent is doing its work, it may call tools as intermediate steps, not to terminate execution and have its full result, but to go do other things to help build up context to get to where it can be done with its work. So you may have a history of tool calls and responses also as a part of this. So these three components, the history of tool calls, user messages, and assistant messages, these are going to grow as execution unfolds. Resources, I'm drawing it as the same size box as all the rest of them. It might not be the same size. That might be a substantial number of tokens there that are taken up by resources. System prompt, that's probably reasonable, right? Tools, that's probably going to be limited. But these other things can be a lot of space.
You see, when you start to look at what really goes into context, you begin to understand even the big models with a million tokens, which again is not all of them. That doesn't feel very Ruby anymore when you're thinking about building complex agentic applications. And it gets a little worse than that. It's not always the case that more is better. If you're using a model that has, say, a 100,000 token context window, it's a pretty consistent result at this point that maxing that out doesn't give you your best results. Somewhere in the 60 to 70% range, and again, let the research unfold on this and let that number stabilize, but kind of somewhere in the 60 to 70% of the capacity of the context window is probably where you get your best results. So more isn't always better. We actually do have to have tools for engineering what goes in here. And let's look at what that set of tools is. There's four things I want to cover.
First, there is the system prompt itself. Now, writing a good prompt is important. We need to get this right. This is a thing that is kind of a one-off for our agent and you want to nail it. And there is a little bit of a Goldilocks thing going on here. We could, in this ditch over here, be too vague. We could say, I don't know, do a good job and go get all the stuff you need to give me a good answer. That's kind of stupid, right? That's not really going to work. Over here, we could be too prescriptive. And for example, if you find yourself in your system prompt defining kind of if-then logic and saying, make sure you do this and if this happens, you want to let the LLM figure that out. That's what it's going to be good at. So don't do that, don't do that. You want to be in the middle where you define outcomes, you define maybe broad approaches to things. And I realize this is itself somewhat vague, but it's a balance that you're going to have to learn how to strike. And that is sort of the art of prompt engineering is making sure that you're not too prescriptive, not too vague. So get the system prompt right, engineer that thing. That's something you actually have to pay attention to.
What's next? Tools. Pretty simple here. You just want to make sure your tool descriptions are specific and detailed. You never want them to be too long. More is not always better, but you want them to really nail down what does this tool do? You want the LLM to be able to, as it were, understand what that thing is for. So you want to be specific and you want to include a schema. We have to know what goes into that tool, what comes out of that tool. That's important because the LLM will need to know that it has all the right inputs to be able to conclude that it can call the tool and it's going to need to know what it gets out of the tool because that may be an input to a next step of say a resource it needs to go get or something like that. So full schemas, specific, precise tool descriptions.
Okay, what's next? We've got data retrieval. And this really goes right here to resources. This is what we're talking about. Now, the initial approach when folks just started trying to use LLMs in an enterprise context was the retrieval augmented generation approach or RAG, where we've got a vector database, we index kind of any document we think we might need to have involved in a lookup. And now when I've got a user message, I'll take that user message and use it in the vector database to go retrieve any documents I think are related, put them into the context and away I go, right? And that's good for kind of an information search, chatbot kind of thing, it worked. And that may well still be a part of your stack and I'm not here to tell you that you're old fashioned or behind the times or anything, but RAG by itself is not very precise context engineering on a go forward basis. There's more we can do to economically use the context window that we've got.
So now we've got MCP, the model context protocol and its disclosure of resources that can be queried. Those resources like tools are described with text and maybe with parameters that can be involved in a query if they're queryable resources. And so those descriptions can go into the context window of one call and a subsequent call, if you ask the LLM, can say, hey, you know what? I need you to go query this resource. That can even be, there's this little arrow here where this agent is making multiple iterative calls. One of those calls could be simply to ask based on this user message, which resources should I go retrieve? And the LLM will tell you based on its conclusion, you go get those resources. Now you've got a big string. You can dump that into the resources section of your next context window for your next call as the agent moves along.
There are other things we can do to make economical use of this. Like say, if you've got a user record that could be kind of long, maybe just give it a user ID. And if there's a resource or a tool or something that clearly can turn that user ID into a full record on a subsequent call, you can get the model to ask you. You can prompt it to ask you if it needs more detail on that user, call this tool or ask for this resource. So there are all kinds of things we can do here to just be economical with our data that we have in here. Rather than anything that might sound like it's related, we can just sort of describe what's available and use prompting to have the model tell us what we need to go get and include.
Finally, we've got the long horizon thinking. And this is for an agent that we know is gonna be really iterating for a long time. We suspect that it might need to make a bunch of calls. And these things that grow as the agent's runtime increases, those are gonna be our enemy. And so there are some things that we can do to help keep those small, or at least within the context window of any given call.
First is compaction. One thing LLMs are incredible at is summarizing text. If I've got some resource I've retrieved on a previous call and it's 50,000 tokens or something, a really big thing, I don't wanna have all of that in my next call. So I could use a single LLM call with my own prompt to say, please summarize this in 500 words. And now I've got that compacted summary that goes in as a resource. That can often give you pretty good results.
Next is memory. Memory is literally just, it's gonna be a little key value store hanging off to the side here. And that will let me at one point in my agent's execution, take some resource, some intermediate result, maybe an assistant message that itself could be somewhat lengthy. Maybe it's some kind of structured data. Maybe it's like a bunch of JSON or CSV or who knows what's going on with this crazy agent you're writing. And we can put that in here under a certain key and on a subsequent step when we need it, retrieve it out of that memory. So it's conceptually like what happens with retrieving resources from an MCP server, but it really is memory that's local to our agent and that we get to use to store that stuff. You don't have much space here. You might need to put certain things over here where this space is relatively cheap.
Finally, we have, I'm just gonna say additional agents. Composition. I can take my one big agent. And I think this is a thing that is just likely to happen as you're building an agentic application. Tell me if this has ever happened when you're building a microservice and it starts to get a little complex and you realize it's a little bit of its own little mini monolith and you have to split it into pieces. This is what it's like to build software. When you're writing an agent, it's probably gonna end up decomposing into multiple agents. Let's say one of the resources you've got is your documentation site or your wiki or something like that. And you've got some kind of complex search logic that's doing a lot of iteration. Man, that's just its own agent. So split that off. And now this agent can ask that agent, I've got this text. I'd like to know what documents from your system look relevant. And now rather than just a text search on that document store, you can have an agent that does some kind of thoughtful sorting through things and prioritizing and back and forth with an LLM to do a good job of giving you some concise summary of what comes from that store. So be ready to take little chunks of agentic functionality in your agent that really could live on their own and make them their own thing.
So there you go. We've got a schema for context, the user message, system prompt, whatever resources we can cobble together, listing of the tools, the assistant messages we get from our back and forth with the agent, and of course a record of the tools that we've called. That's a lot. That's all got to fit into the LLM's context window and those aren't getting a lot bigger. So we have these tools, engineer your system prompt, describe your tools well, retrieve data intelligently, use this resources section wisely, and for long horizon iterative execution, you've got these tools available to you.
Context engineering is the present and future of agentic AI development. I encourage you to put these tools into practice. And as always, you're building something with this, let me know in the comments. And if this is work you're doing, especially if you work with streaming data, the company I work for Confluent has got some really cool stuff to help with this. We've got a new thing called Confluent Intelligence that helps solve some of these problems, particularly for agents that work with streaming data. So if that's interesting, check it out. There's a link and a promo code in the description that'll let you get started trying that for free. Check it out.