Transcription
Today I want to talk very briefly about the memory problem for large language models. I believe this is going to be one of the dominant issues we need to discuss in 2025.
At the end of the day, large language models are becoming very intelligent, but they still have atrociously bad memory. 100,000, 200,000 token memory. It's like having a PhD in your pocket that forgets a conversation from 10 minutes ago. It's not working well.
And the problem is, if you do the math on the cost of memory, there is no easy solve given our current solution architectures. I, I did the math. I wrote a substack on this. It would take over half a trillion dollars to solve this problem, just at the current daily active user count for ChatGPT, which is roughly 125 million, and that's growing all the time. The problem is getting worse all the time.
And that's assuming you don't want human-level memory, which lasts years. You would be happy with long-term memory that's several months. And I don't even know if we would be happy with that, to be honest with you. But that would still be vastly better than just a few minutes of memory, which is really what we have today. If the chat is going well, I burn through chats with Claude in about 10 minutes. Claude's great. Well, Claude lasts, and then we're done.
And I think that one of the things that we need to ask ourselves is, if you have this much intelligence, but it has this much short-term memory, what kinds of problems are useful to solve with that kind of intelligence? And what kinds of problems are we inherently limited from solving because memory is an issue? Even 0.1 Pro, 200,000 tokens, you have very limited memory.
Now, people are doing incredible things with it. So I'm not saying that you can't find really cool problems to solve. I absolutely you can. But it is making me wonder, is memory the next breakthrough that we need to be looking for? And if it is, I don't see anything on the horizon that help helps us to solve that right now. And I think that's probably worth talking about a lot more than we currently do.
Especially if we're using LLMs all the time and they have very short-term memories, is that going to affect the way we remember things? Does that change the and shape the way we remember things? There are anecdotal stories coming out now of the way people are changing their vocabulary, changing their thinking because of the way they interact with LLMs, especially early on at formative stages in education.
If that continues and we are used to working with these thinking partners that are very smart but have very, very limited short-term memory, like it has read everything in the world, but it cannot remember your conversation from 20 minutes ago, does that shape the way we remember things too? Maybe for good, maybe for ill. Maybe we're the ones that have to get better at remembering because our thinking partner can't. I don't know.
But to me, it's one of the most interesting dynamics in large language models right now, and I think it deserves to be talked about about more than it is. So I wrote a substack on that if you're interested. Um, otherwise, enjoy the YouTube.