How to apply context engineering

Hey, this is Lance from Langchain. You might have heard a lot about this term condux engineering recently. I want to talk about what it is, why it's emerged. I want to implement from scratch six different conduct engineering approaches using Langraph.

So Drew Brutnik has a great blog post on this topic. This is one called "How to Fix Your Context." It outlines a few different ways that applications can fail when context grows. Context poisoning: A hallucination makes its way into context and it becomes repeatedly referenced. Distraction: Context grows so long that the model overfocuses on it. Confusion: Irrelevant information in the context can affect the model's behavior and clash: Contradictory information in the context window. He has a whole blog post just on these failure modes with a bunch of examples from the literature, which I encourage you to look at.

This post here, "How to Fix Your Context," covers six different ways you can manage these problems. And I've summarized those here. There's offloading, RAG, tool loadout, context pruning, summarization, and quarantine. I'm going to show to implement all these right now.

The video description has a link to this repo, "How to Fix Your Context." It has a set of notebooks on each of those six methods. And each notebook talks about the particular approach and shows a simple code example for how to implement it, all using Langraph.

Now, let me talk about RAG first. RAG is the act of selectively adding relevant information to help an LM generate a better response. RAG is, of course, not a new theme. It's been discussed very extensively over the last several years within the context of AI applications. You can ensure only relevant information to your task at hand enters the context window of the LLM. And some of the most interesting examples of production RAG systems are, for example, like Windsurf or Cursor. Baroon from Windsurf captured a few interesting approaches that they used in production. But I do want to show a simple example of implementing a RAG agent and how it works.

So let's say I want to build an agent that can selectively retrieve information across a few different blog posts from Lillian Wang, who's a researcher previously at OpenAI and now at Thinking Machines Lab. I can just load the pages. I'm going to split them into chunks. And the reason we do this is because typically in RAG systems, we want to chunk our context into blocks and retrieve blocks of context based on semantic similarity. Load those blocks into the context window of the LLM. So I'll split them. I'm going to create a vector store right here. This is going to just live in memory. And this by default will retrieve four different semantically relevant documents to my question. I can ask a question: "Types reward hacking." This will return to me a list. We can see that right here. Cool.

And now I can just turn that retriever into a tool that an agent can use. This is really simple. All that's happening here is I'm giving it a tool name. I give it a tool description. So now I have a retrieval tool. It works just like before. I can pass in a query and I get relevant documents out. My retriever by default returns four documents. This tool just concatenates them all into a single string.

Now to build an agent, all I need to do is take that tool and bind it to an LLM. I'll go ahead and use Claude 4 here. This "bind_tools" is a method within Langchain that allows you to just pass a list of tools. Now I want to build an agent. What is an agent? It's just an LLM calling tools in a loop until a termination condition. It's very simple. So I have my LLM with bound tools. I'm going to run that until it decides not to call a tool anymore, and then we exit.

Now in Langraph, I implement that really as just a simple graph with two different nodes. One node will be the LLM call. One node will be the tool execution itself. In Langraph, you also define a state object. The state object in this case can simply be a message list. And in every node, I'll just append to that list. So I'll start, I'll pass a user message to indicate the research topic. The LLM will then make a tool call. It'll add that tool call to the message list. The tool node will receive that, ask you the tool that the LLM requests, append a tool message to the list that gets sent back to the LLM. This repeats until the LLM says no more tool calls.

So in Langraph, each of those nodes are just Python functions. This can be anything you want inside this function. So you can declare in detail any logic you want. These functions take in the state, in this case a message list, and write to state with an update. In this case, I update the message list. The pre-built state object in Langraph, it's just a dictionary with a single key "messages," but it has a nice property that will automatically append messages for you. When you do this return from any node in Langraph, it'll just append this message from the LLM call to your list in state.

The tool node looks at the most recent message, which will be the last thing that the LLM node appended. It'll look at all the tool calls and it'll just invoke them. We saw that above. Remember, we can just call "tool.invoke" to run the tool itself. Package that observation into a tool message. Write that back out to the message list. And we have one piece of conditional logic that basically says, "Hey, look at the last message. If it was a tool call, route to the tool node, otherwise end." So, this is our termination condition.

We can ask a question: "What are the types of reward hacking discussed in the blogs?" Here's our user message. This goes to the LLM call node. The LLM makes a tool call. Great. Retrieve blog posts. And you can see this is the tool observation. So this is a concatenation of four relevant chunks in our vector store. That goes back to the LLM call node. The LLM sees that and says, "Okay, I want to make a second tool call to search for a little bit more." We get a whole bunch more feedback. It goes back to the LLM node, which then says, "Okay, I've learned enough," and it goes ahead and provides a summary, and then we terminate because no tool call was made. The model responded directly with an answer.

Now I want to call out something important here. Let's look at the trace. I'm in Langsmith. You can see here 25,000 tokens. What happens is we made a tool call to retrieve documents. The document retrieval is pretty token-heavy. You have around 7 to 8,000 tokens that are coming back, and those observations just get appended to your message list. So at every turn of the agent, that message list is growing, and this is exactly one of the key motivations for context engineering. In many different agents I've built, I've seen this problem of token accumulation due to many observations getting appended to this message history.

So now I want to talk about some kind of intuitive ways to deal with this. One thing we see that's quite common is context pruning. This is the act of removing irrelevant or otherwise unneeded information from the context, and it can really help with the problem of context distraction. As context gets larger, the model can overfocus on it and fail to perform novel actions. Also, I want to highlight that Chroma put out a very nice report recently showing LLM performance degrades in some model-specific and surprising ways as the overall context grows. And this is particularly a problem for agents for exactly the reasons we just saw.

So, let's just apply some pruning. The setup here is identical. We have a retriever tool just like before. Only difference is this: I'm going to apply pruning in that tool node. And to do that, I'm just going to use an LLM that's prompted to basically remove irrelevant information relative to my initial request. This is the prompt in the tool node with pruning. All I'm really going to do is I get that tool observation. I fetch the initial request provided by the user. I initialize a model. In this case, I'll use GPT-4 Turbo, but you could test different models for this pruning step. And I just run that on the raw tool output. Get pruned context back. Write that pruned context to my message. Everything else is identical to what we saw before.

We run our agent. Same query as before, but in this case, you can see the tool output is much more condensed because it's run through an LLM, which does a pruning of everything that's irrelevant to my initial question. And so, we can really see that the tool observations here are much more compact than we saw in the original case. And this is one very simple way to significantly context bloat.

Now, let me show another technique that's very similar to this. And I want to talk about some risks and concerns with both of these methods. So, another idea is just applying summarization to those tool observations. A little bit different than pruning, but very related. Summarization is the act of boiling it down into a condensed sum. We'll set up our tool just like we had before. Use Claude 4 just like we did previously. Only difference is I'm going to use in this case a summarization prompt. You can tune these prompts as you see fit for your application.

There's one subtle difference. With pruning, you're just getting rid of irrelevant information. With summarization, you're condensing the entire context down into a more compressed summary. Summarization is a bit more useful in cases where the context is broadly relevant, but it may be redundant. So, you want to just compress it into a more compact form, but retain all the relevant information. With pruning, it's more like there's some bit that's relevant and some other part that's explicitly irrelevant, and you're trying to strip away the irrelevant bit.

This is set up just like before. I now have a tool node with summarization. I apply summarization right here. Again, I'll use a fairly small model to do that. Just like before, I pass in the summarization prompt, the raw observation, and I get a summary of that observation, and I write that summary to my tool message and return that to the LLM. Looks just like before. User request in, makes a tool call. You can see now the tool output is like a nice summary of everything that was in that big context blob that was initially getting back.

Now I want to call out something very important here. Cognition and Manis have both very nice blog posts talking about context engineering. Both warn that summarization and or pruning need to be carefully done because they both risk information loss. So, Cognition mentions that they use a fine-tuned model for this within Devon to really ensure that key events or information is retained when performing summarization. Manis actually discourages summarization and mentions they tend to use a technique called context offloading, which I'm going to show next. Clearly a useful practice, but it needs to be carefully done to ensure you don't lose information.

And that's a nice segue into this fourth technique of context offloading. This is just the act of storing information outside the LLM's context, potentially via tool call to store and manage that information. Now, we see this technique used all over the place. Anthropic used it in its most recent multi-agent researcher. The researcher creates a research plan, saves it to a file, so it's available. And they mention they do this because the research process might exceed the context window of 200,000 tokens, but they want to ensure that the plan is recoverable because they need it, for example, in the final writing phase. So by saving it to a file outside the context window, you can preserve it.

Very intuitive thing to do. Manis does this as well. They have an interesting and different take on it, though. They always create this plan-to-do.md for any Manis task. Manis tasks typically have 50 different tool calls. So they're very token-heavy, and they will continually rewrite this plan over the course of agent execution. They mention that this is a form of recitation. So, it really encourages the agent to think about its to-dos and kind of recontextualize where it is in the process by rewriting that to-do.md file. So I thought that was an interesting point about how rewriting the to-do list iteratively as you're performing the task might help to steer the agent and keep it on track.

Now, I do want to highlight there's different places to offload context too. In Langraph, there's actually a very convenient, easy way to do this, which is that Langraph has a state object. This state object persists throughout the lifetime of our agent. It's a very nice place just to offload information to and keep it outside the context window of our LLM.

So, I'm going to define a custom state object called "scratchpad_state." It's going to extend our prior "messages_state." So, this is going to have a new key, "scratchpad," as well as the "messages" key. I'm going to define two tools that will write and read to and from the scratchpad. And I'm going to have a search tool here. I'll use Claude 4. And this is the important point. I'm going to have my agent kick off a research process, read from the scratchpad to check if there's any available information, create a plan, write the plan to the scratchpad, perform search, and iteratively update the scratchpad as it goes. Simple toy example of what Manis is doing with their kind of iterative rewriting of their to-do list as it proceeds.

So, just like before, I have an LLM call node. I have a tool node. Now the only thing I do here is in the tool node, I look at the tool call name. If it's "write_to_scratchpad," I just take the observation and I just write it to state. Return a tool message that says, "Hey, I wrote to the scratchpad," and that state update is performed right here, where you can see I write notes to the "scratchpad" key in our state. Same idea when we're doing reading. We just go ahead and read from state. Add that to the tool message, and then we can also run Tavly search.

So our agent looks just like before. Let me kick off the request to compare the funding rounds of two different fusion energy companies, Commonwealth Fusion and Helion. Here's the request. Again, our agent reads from the scratchpad. Great. There's nothing there yet. It writes a plan, performs a few tool calls. We can see that here's our research plan. That's great. It even says "starting research." Does some searches to the scratchpad as it goes. And you can see towards the end it has a final comparison written to the scratchpad, and it produces a final answer. We can look at the final scratchpad. We can see it was writing to this over the course of its research, taking notes, and it pulls that all together into a really nice summary. So this really matches what we do as humans. We often take notes while we're doing research, and then we collate those notes into a final report and report that out. This is exactly what's happening here.

Now, in that particular case, we're just saving that scratchpad to a state object. That state object lives throughout the lifetime of our agent, and that's often fine, but sometimes we want to persist information across different runs of our agent. Now, in Langraph, each run is what we call a thread. And threads are analogous to if you open up a chat in ChatGPT, that's analogous to like a Langraph thread. If you open up a second chat, that's like a new thread. Okay. Now, ChatGPT has this memory feature. We can save things from one chat, and they're saved to this memory system, which is accessible across all future chats. Same idea here. Langraph has a long-term memory store that you can write to and then read from in different threads.

So this is useful if we want to persist things across many sessions or interactions with our agent. So we can grab this memory store like this. In this case, it's just an in-memory key-value store. We can write to it with "store.put," supply a key, supply value as a dictionary, and supply namespace as a tuple. Great. We can get from it just by fetching from the namespace and key. Nice and easy. Now I do want to highlight in Langraph deployments, this store can be backed by Redis or Postgres, but for quick testing here locally, it's just an in-memory key-value store.

So now all I need to do is update my tool node to read and write from the store rather than from state. Again, we just use "store.put," "store.get." Very minor change relative to what we saw before. Cool. Graph looks identical to what we saw.

Now, here's a key point. I'm going to kick off one thread. This is like one interaction with the agent. And this interaction is a set of messages that's all stored to this one thread, which I'm going to name "one" right here. And I'm going to kick off some research based upon Commonwealth Fusion. Cool. Just like before, nothing in the scratchpad. That's fine. We write to the scratchpad, make a plan, do research, and finish up. Here we have a nice overview of Commonwealth Fusion. Great.

Now, here's the beauty of the store. I'm going to create a new thread. This represents a new interaction with our agent. Of course, this is a toy example, a notebook session. The store is living in memory, but in the case of a production agent, this could be an independent session with a new user. The point is the store can be accessed in all the subsequent sessions. So, I'm going to ask a question: "How does Helion relate to Commonwealth Fusion?" Now it's going to read from the scratchpad to see, "Hey, is anything available to me?" And boom, you can see here's that summary of Comm Fusion we just did, available in the scratchpad. Great. I have some information about Comm Fusion already. I'll create a plan. I'll go ahead and dig into Helion. Does a bunch of that work, and so we get this comparison of Helion versus Commonwealth that utilizes the scratchpad we wrote in the first thread. Simple example of writing to a long-term store versus writing to state. Both are achieving the same thing: agent can offload context to either the store or state and reuse it as needed as it performs tasks. Very simple, powerful idea.

Now, let me show another useful idea. It's a bit related to the RAG theme we talked about previously. Tool loadout: Actively selecting relevant tools based upon your task. Now, this really helps with the problem of context confusion. Often times we see agents perform poorly when they have many tools that have overlapping definitions. So, it can be hard for them to figure out which tool to actually use. And so really simple and effective idea is to do semantic retrieval based upon tool descriptions based upon your task. And so you basically retrieve only relevant tools and bind those on the fly.

And let's basically create a list of tools based on all mathematical functions in Python's math module. Here's just some code to do that. I'm going to create a little tool registry. I'm going to embed the tool descriptions. You can see I'm doing that right here. And I'm going to save this in that Langraph in-memory store. So this is available to me in any Langraph node. I have a list of math functions with descriptions embedded. So that's great.

Now all I need to do is in my LLM node, I can do a "store.search" across the tools namespace based upon the query. And the query is just that first request that the user asked. So this will return only the relevant tools based upon the query determined by semantic similarity. And I bind those relevant tools to the LLM on the fly right here in the LLM call node. I go ahead and run right here. And the tool node will just go ahead and invoke the tools. Agent layout is identical to what we saw. The only difference is we're actively selecting tools in that LLM call node based upon the request.

Try this out. Use tools to calculate the arc cosine of 0.5, and it calls this arc cosine tool correctly. That's great. Nothing too interesting there. But what's neat is if you go to the trace and you look at that initial LLM call, you can see right here, one, two, three, four, five. Only these relevant tools have been bound. So it's not binding all the functions of the math library. It's only binding these five, which were selected based on semantic similarity to my request.

Now let me talk about this final technique of context quarantine. This is a big topic right now. Quarantine is isolating context in different LLMs, often in sub-agents. It can help with context clash or distraction because you're basically quarantining different topics into their own context windows as opposed to making one single agent grapple with potentially conflicting and highly different subtopics. You see this a lot in research. Anthropic's multi-agent researcher makes a good case for this, and they show that isolated context windows outperform single agents by a large margin. Overall, the system can utilize more tokens because each sub-agent has, in the case of Claude, a 200,000 token context window. So your effective context window size for the system could be much larger.

Now let me show this in action. Then I want to call out some concerns about it and when to use it. I'm going to use Langraph's multi-agent supervisor architecture. This is a pretty simple layout where you have a supervisor agent that can offload research tasks to sub-agents, and those sub-agents all get circulated back to that supervisor to decide what to do next. This is an extremely simple example. I'm going to define a math and a search sub-agent. You can see this search sub-agent is actually just returning some dummy results. I'm going to use this "create_react_agent" abstraction. This is just wrapping what we've been doing previously in every other notebook just to make the code a little bit more compact. But you already know how to build an agent from raw nodes and edges from everything we've seen previously.

I'm going to create a math agent. I'm going to create a research agent. The math agent has math tools. The research agent has search. The supervisor then can delegate to either one of those agents. I'll use Claude 4 as a supervisor. I give it the two agents. I give the supervisor prompt. And we can see this in practice. I'll ask about the combined headcount of FAANG companies. We delegate the task to the research expert. Research expert does the web search with our mock search tool, transfers back to the supervisor. Supervisor looks at that and says, "Okay, cool. I'm going to go to the math expert." Math expert does the math on it, goes back to supervisor. Supervisor provides an answer.

This is a toy example, but it just shows you the flow for how a multi-agent system can work with delegation. If you want to see a more interesting example of this, check out our open research repo. This is an open-source implementation of Anthropic's multi-agent system. It performs quite well on the Deep Research benchmark. Our results are going to be uploaded soon, but I believe it's top four or five. But the point is it gives you a nice reference for how to implement a multi-agent system for a popular task, research.

Now, that all said, I want to call out something that's important with thinking about multi-agent systems. So, Cognition laid this out nicely in a blog post that is titled, "Don't Build Multi-Agent Systems." They argue that multi-agent systems can be risky. And the reason is pretty simple: if I have multi-agents performing subtasks in parallel, they're making decisions independently, and there's risk of contradictions or conflicts that will affect the overall composite system.

Now, I've seen this myself in an earlier version of this open Deep Research repo. I took the report writing and put it into each sub-agent. So, each sub-agent wrote its own section. When I combined those sections together into the final report, the reports are disjoint because they're being written independently. Classic example of this. So a good mitigation and Walden of Cognition kind of highlights it here in this nice post from Jason, is that if you must use multi-agents, constrain the information gathering rather than decision-making to minimize conflicts. There's a simple way to say that is you make sure the tasks that are being paralleled across multi-agents are not too tightly coupled. And research is just information gathering. We do the writing in one shot after the fact. So even if there's conflicts, we let the writer deconflict everything and ensure that the final working whole is coherent. So I think it's a way to resolve the two views between Anthropic's take on "use multi-agents" and Cognition's take "not to use them," but in cases where the risk of conflict across a multi-agent system is lower because a task is less tightly coupled, it can be something like research or information gathering, not like subcomponents of an application that have to work together in a very explicit way.

So that gives you an overview of these six techniques. Again, I thought Drew's blog was really nice. Laid these out really cleanly. You can implement all these pretty easily, as I showed in Langraph. This repo is open source. You can poke around and look at the different notebooks. They're all toy implementations. So, I encourage you to take them and improve on them and adapt them for your particular use case. But these are six methods that we've seen commonly used across many examples of different agents shared by many different companies. I try to highlight some of the risks, in particular with any kind of compression like summarization or pruning, you need to be careful about information loss. With quarantine, you need to be careful about coordination across sub-agents. But with that, I'll let you look at the repo and hopefully this is useful. Thanks.