Transcription
Memory is perhaps the biggest unsolved problem in AI, and it is one of the only problems in AI that is getting worse, not better. As we get better and better and better at intelligence, we get worse at memory, relatively speaking. In fact, there's a name for it in the model maker community. It's called the memory wall. We are not improving the hardware chip capabilities of our memory systems nearly as fast as we are improving the ability of those chips to infer or compute words or do LLM inference. That generates a growing gap between our intelligence capabilities and our memory capabilities.
Don't worry, we won't stay at the hardware level for long. I want to go through with you the core issues that we see as builders, as users of AI, as designers of AI systems. What is the root of the memory problems we experience? If we're at a systems design level, if we're at a usage level, if if we are even using ChatGPT, why are memory problems so sticky and hard to untangle? Why have we not seen better solutions in the market? I think there are good reasons for that. And then once we go through those root causes, how can we start to think about solving them? How can we think about solving them as users? How can we think about solving them as builders?
So, I'm going to go through five root causes and then we're going to flip the script and I'm going to go through eight principles for building a solution because I want you to walk away from this and I want you to feel empowered to actually design better memory systems. I don't want you to wait around for someone in Silicon Valley to make a pitch and get funded for this. You can design your own solution here.
So the key thing to keep in mind through this whole conversation is that AI systems are stateless by design, but useful intelligence requires state. So every conversation is stateless, meaning it starts from zero. The model has parametric knowledge, which the weights we talk about in a model, right? But it doesn't have episodic memory. It does not remember what happened to you. And I'm sorry, but the 10 or 11 sentences or the the very lossy memory that ChatGPT has right now, or the ability to search conversations that Claude has right now, is not good enough for that. You have to reconstruct your context every single time. This is not a bug, actually. It is an intentional architecture. It is a design for statelessness because the model makers want the model to be maximally useful at solving the next problem, the problem in front of you. And they cannot presume that state matters. It doesn't always matter.
So the promise of memory features is that vendors are going to be able to magically solve this by making the system stateful in ways that are useful to you. But this creates a whole host of new problems because statefulness is not the same for all of us. What should it remember? Is it passive accumulation? Is it active curation? How long should it remember? Is it persistent forever? Is it stale ever? Does it drop off after 30 days? When do you retrieve it? Do you retrieve it when it's relevant, sort of like Claude does? Do you retrieve it all the time and potentially it's noisy in the context window? How do you update it? This is one of the biggest problems with LLMs. People tell me they'll put their wiki into a retrieval augmented generation system, and I'm like, when was the last time you updated your wiki? If it's not updated, how do you overwrite it? How do you append data to it? How do you change data? These are not implementation details. They are fundamental questions about what memory is and its purpose when we do work.
Memory matters because we humans are able to quickly and fluidly negotiate between stateless brainstorming, things that are like wild and we don't need to use a lot of our past memory, and very stateful work. LLMs are not good at that. Loading that context is very, very hard right now.
So why is this so persistent? We've talked a little bit about how the promise is hard to fulfill, but what are some of the root causes that make it hard for vendors to do this?
Number one, the relevance problem is one of the gnarliest unsolved problems out there. What's relevant actually changes based on the task that you're doing. Are you planning? Are you executing? The phase of your work. Are you just exploring? Are you refining your work? The scope you're in, right? Is it a personal or is it a project? I know someone who is in the healthcare industry. And they have to be very careful because if they were to ever ask for health advice, then the memory retrieval within ChatGPT would pull up work stuff, and they are afraid in the same context if they pull up a work thing that their personal health data will leak in because it will all look like health data. So the scope matters. What has changed since the last time you talked? The state delta is what we would call that. If you come back and you say this is a new version, does it really understand that's a new version or not? Semantic similarity, which is what a retrieval augmented generation depends on, is just a proxy. It is a proxy for relevance. It is not a true solution. Finding similar documents works until you need to find the document where we decided X, and that's very specific. Or ignore everything about client A right now, but pay attention to clients B, C, and D. Or please only pay attention to what we've decided since October 12th. These are all things that we humans can understand and execute on when we go and manually retrieve information. But the AI using semantic search, it's just not the right tool for that job. There's no general algorithm for relevance. There's no magic relevance solve that the AI can depend on. You need to use human judgment about task context. And that means requiring very complicated architectures to accomplish a specific memory task, not just better embeddings in a RAG memory system. And that, by the way, is one of the big reasons why these like one-stop shop vendors often struggle with real implementations.
Number two, the persistence precision trade-off is a massive issue with memory systems. If you store everything, retrieval becomes very noisy and it becomes very expensive. You jam up your context window. If you store selectively, you're going to lose information that you need later. If you let the system decide what to keep, it optimizes often for something that you didn't ask it to. Maybe it optimizes for recency. Maybe it optimizes for frequency. Maybe it optimizes for statistical saliency versus actual importance. And if you wonder what statistical saliency is, have you ever tried having an argument with ChatGPT or Claude or Gemini about the fact that it's emphasizing the wrong thing in something it's writing? That is saliency. That's a saliency defect. Human memory is actually, funnily enough, very good at this through the technology of forgetting. We use incredibly lossy compression with emotional and importance weighting. And so we've actually done studies on human memory. And it turns out that you can, with practice, get better and better and better at recalling specific things. But if you choose not to recall something that happened to you, you're just going to lose it. And what's interesting is it seems to be a database keys issue for us. I realize I am like some, someone in the comments is going to be a neuroscientist and just rightly take me to town. But my understanding of the reading is that you have to be able to remember the equivalent of a database key to retrieve the memory. And if you can do that, the memory becomes accessible again. But your short-term memory, so to speak, in humans is very lossy. And so you lose the database keys if you can't persist them with intent, if you don't intend to remember them. And that is why fundamentally your childhood memories can be very accessible. But what happened last Thursday? You're sitting there and you're like, did we eat out or did we not eat out? Which which day did we go to the movies? Right? It's not because you have a profound issue with memory. It's because your brain is desperately compressing information to make it useful to you and has dumped out those database keys. And when you go to the effort of remembering, you're literally retrieving the database keys to get the memory back. Forgetting is a useful technology for us. That's the point of that. AI systems don't have any of that. They either accumulate or they purge, but they do not decay. And what I'm talking about when I'm like, did I go to the movie? Oh, yeah. It was the movie. Who was that character? Oh, now I have I'm recovering the key and I'm able to get it back. The memory has decayed into a lossy approximation in the memory key, but I can recover it if I put effort into it. We have nothing like that in AI. That is a uniquely human technology. And it's funny, but we have to think about forgetting as a technology when we talk about memory.
Number three, the single context window assumption. Vendors often try to solve memory by making context windows bigger. But volume is not the issue. The structure is the problem. A million token context window is not a usable million token context window if it's full of unsorted context. That is worse than a tightly curated 10,000. The model has to still find what matters, parse the relevance, ignore the noise. You have not solved the problem by expanding the context window. You have simply made your problem more expensive. Sometimes substantially more expensive. I know people who make calls and they don't budget the calls and they're like, "Why is my API bill high?" I'm like, your API bill is high 'cause you're stuffing the context window and you're just kind of trying to throw queries against it. It does not work well and it also is very expensive. The real solution requires multiple context streams with different life cycles and retrieval patterns. It is hard. You have to design it. It breaks the mental model of just talk to the AI. That is why there is no one-size-fits-all solution.
Issue number four is the portability problem. Every single vendor builds proprietary memory layers because they think in their pitch deck that memory is a moat. I get it. It makes sense on a pitch deck. ChatGPT memory, Claude recall, Cursor memory banks. These are not inherently interoperable. Users will invest time building up memory in a given system. And the model makers like that because it makes the switching cost real, and you can't port what ChatGPT knows about me to Claude, and your memory is locked in, and so on. The problem here is a problem of the commons. This behavior set from vendors and model makers and tool builders encourages users to leave memory to the tool rather than encouraging them to build a proper context library. And I get it from a product design perspective because like, how many users are going to really build a product context library? But if we reframe it and we say portability is a first-class problem, users should be inherently able to be multimodal. I think that's more relevant. And maybe from a consumer standpoint, you don't care because you have 800 million users in ChatGPT. It dwarfs everything else, etc. One, that's not entirely true because Gemini has, I think, closing in on half a million or half a billion now. But the other reason is that from a business perspective, you have to be multimodal. It is it is a liability to be single model. And so if you're building business memory systems, you must solve the portability problem. And the issue is any given vendor is not incentivized to make that truly portable either. They want to make that proprietary to them. And then you have the same bottleneck, but now you're on a vendor who may not be as well funded as the model maker. And so it becomes a house of cards.
Number five, the passive accumulation fallacy. Most memory features assume you just use your AI normally and it will figure out what to remember. That is the default mental model of users. And so that's the assumption that memory features build around. But this fails because the system cannot distinguish a preference from a fact. It cannot easily tell project-specific from evergreen context. I've often seen that mixed up. It doesn't automatically know when old information is stale. If you've ever wondered why ChatGPT or Claude or Perplexity comes back and talks about old AI models as if they are active today, that is the same issue. They can't tell when old information is stale, and it optimizes for continuity. It does not optimize for correctness. This is the "keep the conversation going" issue. Useful memory fundamentally requires active curation. You have to decide what to keep, what to update, and what to discard. And that is work. And so vendors promise passive solutions because active curation, they are told, does not scale as a product. I think we have to start by framing that problem better because it turns out passive accumulation doesn't solve for it either. And this is still a big enough problem that it costs us billions of dollars at the enterprise level, and it's extremely frustrating for users both personally and professionally. The answer cannot be there is no answer or we'll fake the answer.
Finally, number six on the root cause side, then we're going to get to solve. It'll it'll feel better. Memory is actually multiple problems. And that's part of why it's so hard. I hope you're getting that idea, right? When people say AI memory, what they really mean is any number of preferences, how I like things done. That could be a key value that's persistent. They could mean facts. What's true about particular things or entities that can be structured, it might need updates. They might mean knowledge, right? Domain expertise. And that can be parametric, right? That can be embedded in weights, but it might not be, right? And then what do you do? It can be episodic. So it could be conversational, temporal, ephemeral knowledge. It can also be procedural. Have we solved this before? Right? If episodic memory is what we've discussed in the past, procedural memory is how we solve this problem in the past. And those are also different things. And so you have exemplars there, you have successes and fails in procedural memory. Every single memory type needs different system design to handle storage, retrieval, and update patterns. And if you feel like you're getting a headache here, you're not alone. This is why we don't have a good solve. And this is why I want to lay out in the next section principles for solve. But it starts with being honest about the problem. Treating this problem as one problem guarantees you are going to solve none of the real problems well. And that is why we have memory as a persistent issue, in fact, a growingly worse issue in the AI community. Vendors fundamentally are treating this as a solve for infrastructure and not a solve for architecture. And so bigger windows and better embeddings and cross-chat search scale, but they don't solve structurally. And users keep expecting passive solutions because they're frankly sold passive solutions. There's an expectations issue here. "Just remember what matters" is not something that you can expect to work. But we're told that it will work. So if memory requires architecture and users want magic, the gap between what's promised, what's delivered, and what's needed has never been bigger. We have a memory wall of our own beyond the chip level in how we design our systems. And it won't get solved if we solve the wrong.
So let's say you've gone through all of this and you want to solve memory correctly. I am going to give you principles that work whether you are using the chat and a sort of power user at home and you want to build something yourself because this absolutely works for that, or whether you are designing larger systems because it turns out that the principles for memory are fractal because the problem is fractal. We have the same kinds of memory issues when we are power users individually in a chat as we do when we are designing agentic systems.
So the principles that work. Number one, there's going to be eight of these. Settle in. It's going to be fun. Memory is an architecture. Memory is an architecture. It is not a feature. You cannot wait for vendors to solve this. I think you get this idea. We won't spend too long here. Every tool will have memory capabilities, but if you leave it to tools, they will solve different slices. You need principles that work across all of them. And you need to architect memory as a standalone that works across your whole toolset.
Principle two, you should separate by life cycle, not by convenience. So as an example, you need to separate personal preferences, which can be permanent, from project facts, which can be temporary, and those should be separated from session state, which can be ephemeral or conversation state. Mixing different life cycle states, mixing permanent with temporary with ephemeral, it just breaks memory. The discipline lies in keeping these apart cleanly. And again, this works if you're in chat. It works if you're designing agentic systems. If you have a permanent personal preference, it is possible. It is as simple as a very disciplined system chat update where you go into the sort of system rules and the system prompt for ChatGPT and you say, "This is what you need to know about me. These are my personal preferences." And model makers are starting to make that more exposed because they want that. But they don't tell you how to use that properly. And when I observe how people actually use that "tell me about yourself," it is absolutely a mix of personal preferences and ephemeral stuff and project facts because no one has taught them to use it better. And if you're designing agentic systems, it gets more complex, but it's the same separation of concerns. You have to separate out what are the permanent facts in the situation here. What are project-specific facts, and what is session state?
Principle number three, you need to match storage to query pattern. So that means you're going to need multiple stores because different questions require different retrieval. Now, in the chat situation that I gave you, ChatGPT can retrieve the memory if it's a system prompt kind of a thing, and it just calls it into the context window, and it's super simple, and you'd never think of it as memory for most people, but that's what it is. If you're designing an agentic system, it is understanding the difference between, for example, what is my style, which could be a key-value because it's a written style of some sort. What is the client ID, which should be structured data or relational data. What similar work have we done, which could be semantic or vector storage data. And what did we do last time, which should be event logs. Those are four different types of data, right? You have key-value data, structured data, semantic data, event logs. Trying to do all of these in one storage pattern is going to fail. And that is why when people say, "We have our data lake and it's going to be a RAG," I'm like, why? Why is it going to be a RAG? Have you heard the word RAG repeated a hundred times like a magic spell for memory? It does not work that way. You need to match storage to the query pattern. Otherwise, you just have a very expensive data dump.
Principle number four, mode-aware context beats volume hands down. And so more context is not better context. Planning conversations need breadth, like they need to have space for alternatives. They need to have space for comparables. Brainstorming conversations are similar to planning conversations. You need to be able to range. Execution conversations, execution workflows in agentic situations, they need precision. They need precise constraints. Retrieval strategy needs to match your task type. You cannot just sit there and think to yourself, okay, I'm going to have a brainstorming conversation, and it's going to be incredibly precise, and just hope that it works. This is why I talk about prompting so much. Effectively, what prompting is doing is giving context that is mode-aware to an AI so that it can be in the right mode. And that's super effective for chat users. But guess what? If you're designing agentic systems, it is your responsibility to architect mode awareness into the system so that it is aware that this is an execution environment and that precision matters and that it is audited and eval'd on precision.
Principle number five, you need to build portable as a first-class object. You need to build portable and not platform-dependent. Your memory layer needs to survive vendor changes. It needs to survive tool changes. It needs to survive model changes. If ChatGPT changes their pricing, if Claude adds a feature, your context library should be retrievable regardless. And that is something that almost nobody can say right now. And the people who are doing it tend to be designing very large-scale agentic AI systems at the enterprise level. But this is a lesson that we all need to take with us. I think it is a best practice. It is sort of like keeping a go bag next to the door in case you need to get out in case of, I don't know, something happens to your house. You need to have something that is portable that carries relevant memory that you can use to have productive conversations with another AI. I fully admit there is not an out-of-the-box solution for this. There are people who are power users who configure Obsidian to do this right, as a note-taking app, and they tie it into AI, and it becomes a portable, platform-independent way of handling this. There are people who use Notion for this. The thing that is a common trait is that they are obsessed with making sure the memory is configured correctly for them, and the AI has to come in and be queried correctly or called correctly to engage with a piece of the memory that matters. Whether that is a key-value piece like "what's my style" or a semantic search like "what similar work have we done together." A good data structure accounts for that.
Principle number six, compression is curation. Do not upload 40 pages hoping the AI extracts what matters. I see people do this when they overload the context window and they ask for an analysis of a report. You need to do the compression work. You need to either in a separate LLM call or in your own work, write the brief, identify the key facts that matter, and state the constraints. This is where judgment lives. And if you don't delegate it, you will be happier with the precision and context awareness of the response. Memory is bound up in how we humans touch the work. There are ways to use AI to amplify and expand your judgment. You can use a precise prompt to extract information in a structured way from 40 pages of data, and then in a separate sort of piece of work, figure out what to do with that data. But it remains on you to make sure that the facts are correct, that the constraints are real, and that the precision work you're asking AI to do with that data is the correct precision work. The judgment in compression is human judgment. It may be human judgment that you amplify with AI, but it remains human judgment.
Principle number seven, retrieval needs verification. So semantic search will recall well but fail on specifics, right? It will recall topics and themes. Well, you need to pair fuzzy retrieval techniques like RAG search with exact verification where facts must be correct. You should have a two-stage retrieval call path, right? Recall candidates and then verify against some kind of ground truth. This is especially important in situations where you have policy or you have financial facts or legal facts that you need to validate. Something like this is exactly why there was a very prominent fine leveled against a major consultant firm in the last two weeks. I think the fine came to close to half a million dollars because they could not verify facts around court cases in a document that they prepared, and they hallucinated them and they didn't catch them. Retrieval failed. Retrieval failed. And because the LLM is designed to keep the conversation going, it just inserted something plausible and nobody caught it. You need to be able to verify retrieval against ground truth. Now, if it's a small task, that might be the human at the other end of the chat, right? It just is a step that needs doing. If it's a large agentic system, it is the exact same fractal principle, but you need to do it in an automatic way using an AI agent for evals.
Principle number eight, memory compounds through structure. So random accumulation actually does not compound. It just creates noise. Just adding stuff doesn't compound. If if we just added memories randomly the way we experience them in life and we had no lossiness, no forgetting ability, we would not be able to function as people. Forgetting is a technology for us. In the same way that forgetting is a technology for us, structured memory is a technology for LLM systems. So evergreen context goes one place, version prompts go another place, tagged exemplars go another place. And at a small scale, yes, you can do this. People are doing this with Obsidian, with Notion, with other systems as individuals. And yes, you can scale this as a business. Same same principle. You let each interaction build without degradation if you have structured memory. Otherwise, you just have random accumulation. Otherwise, you have the pile of transcripts you never got to, and you're like, well, this is data. We're logging it. It's probably good. It just it's going to be random accumulation. It creates noise. You're not going to have structured memory.
These are the principles that work. They work whether you are a power user with ChatGPT or a developer building agentic systems. Frankly, they are guideposts for you if you are evaluating vendors in the memory space. These are tool-agnostic principles. They're designed to scale with complexity and they're designed to give you keys that solve the memory problem because they make consistent context persist reliably without the brittleness that we see with current AI systems.
And so my challenge to you as we wrap up this video, we've gone through root causes. We've gone through why memory is a hard problem. We've gone through eight principles for how to solve for this memory issue. Please take memory seriously. The reason it matters now is because if you solve memory now, you have an agentic AI edge. These systems are going to get cheaper and more powerful, but you can't assume they're magically going to solve for memory. As I said at the beginning, there's a chip-level issue here. It is a hard, hard problem. If they don't magically solve for it, if you take responsibility for memory and build it yourself in the way that works for you, you are starting the timer earlier than everybody else around you on getting memory that is functional across a long-term engagement with AI. Because you have to start to think, we're in year two of the AI revolution. Wouldn't it be great to have memory that goes back to the year two when you are working with AI systems in 10 years, in 15 years, in 20 years? Everybody else is going to have memory that started much later, and they're going to lose that discipline, that acceleration, that ability to manage deep work over time that AI is going to be capable of with proper memory structures. So there is a moment here for you to think about and put in place a memory structure that works. Don't lose the opportunity. This is a this is a complex one, but it's on you and me and all of us together to build memory systems that handle our own needs, whether that's personal needs or professional needs. I know you can do it. Drop in the comments how you're doing it because I think we should all crowdsource.