Transcription
Imagine if Claude Code remembered everything and got smarter every time you spoke to it. In this video, I'm going to show you Andrej Karpathy's Obsidian rag system that gives Claude a self-updating memory so the Claude code never forgets, gets smarter in every conversation, actually knows who you are, and how to fix its massive limitation that 99% of people aren't even talking about. So, by the end of this video, you'll have a Claude code memory system that will work for the long term. And if you don't know who I am, my name is Jack Roberts. I built and sold my last tech startup with over 60,000 customers. And now I run several AI automation businesses and teach thousands of people stuff like this every day. So if you haven't already, grab that coffee and let's dive straight in.
Andrej Karpathy, famous founder of OpenAI. So Andrej Karpathy just gone viral by talking about Claude Code plus Obsidian to create what we call an Obsidian memory system. Now, the way that it works effectively is it will have a lot of files that Claude will create, or it'll download, and Claude will organize and create the system for you. And the idea is that it's very easy for Claude to edit these if you want to update the information, and you can even view them visually in Obsidian. It is essentially like your personal Wikipedia. So anything you want to remember, we can actually just do by having this kind of index that Claude can cycle through based on whatever information you want to. Meaning that you don't lose important context whenever you're using Claude.
Now, this went viral with 18 million views talking about Claude Code and Obsidian because it solves a real problem, and that is context loss. In other words, the fact that Claude has amnesia for all its wonderful perks. Sometimes it just forgets it's a model and it will hallucinate. We expect at least 1%, sometimes up to 10% of the time, depending on how long the conversations happen for, and it can be confidently incorrect. And the problem is, we don't know when it is confidently incorrect and when it's actually just being accurate. So with this system, Claude can read and write its own memory, and it's supplementary to and supports alongside Claude's basic context. It's like this adds a fourth memory system, which I'll explain in this video, and it gets better crucially the more that you use it because we index more the right information and we build it out over time.
However, duh, 90% of the coverage that I've seen on this, I would describe realistically as node porn. And it does actually, the system have one huge, actually several huge limitations, but don't worry, I got you. I'm going to explain what those limitations are, which I haven't seen anyone else talk about in this kind of hype cycle, and how you can actually fix it so you get a very long-term memory system that will work in any situation and give you the accurate data. So, let's begin with the source of this, actually download it, show you a real use case of how this will work in your new system.
So, Andrej Karpathy basically explained this in about four or five different points. Effectively, it's using a large language model to build personal knowledge bases for different topics of interest. The best way to visualize this is like you are essentially building a personal Wikipedia. When you're talking about an Obsidian memory system, it's your own personal wiki. So data ingestion happens at point of Obsidian web clipping, or we can give it data. There is a best way to do this, I think, and a not-great way to do this, which I'll show you. But effectively, we create this wonderful raw directory, and the LLM will incrementally compile a wiki, which is just a collection of MD files in a directory structure. That's it. And then it has a directory, and you ask it questions, it goes through and it finds the relevant things, clicks on them, and then good things happen.
So, in terms of IDE, this is basically what are you looking at to do this? Now, Obsidian itself is also a visual layer, but you could use this with any project you're doing. Could be Anti-gravity, could be Cursor, you name it. And he's put some additional things in here. He's got this concept of LinkedIn to-do health checks, but I'll come on to that later in the video when we go through everything. So, it's actually super duper, and I say duper because it is super duper easy to set up. So, what we're going to do first of all is download Obsidian to sharpen our thinking. Cool story. These guys, not many people built a very successful company. We're going to go ahead and download it on the computer. I don't have it on this one yet. So, we're going to go through the entire process cleanly together. I also want to touch on this nerd view as well. I think it's interesting. I know a lot of people are like, "How can I visualize it?" In reality, I haven't seen anyone make any informed decisions off it. I think it's just something that looks cool, but it's not really necessarily going to help you do anything better. Although the relations between certain things on your memory system are really important, which we can touch on later.
So, we download to applications. Now, we're just going to open this guy up. Obsidian, just think of it as a way to visualize files. That is literally it. It will show you in your visualized files. So, you can create a new vault, open folders, vault, whatever you want to. I'm going to do a quick start and show what it looks like. We come in. If you click on this one here, this is a graph view, which will show you basically the relation between all of the key things in your database. But firstly, let's go to dark mode, which you can do by clicking on Obsidian and then File and Settings. This exists on Appearance, and then you can flip between light mode and dark mode here. Dark mode is usually preferred. So now we are ready to rock and roll.
Now, we just need to add some important information. So the way we do this actually is we're going to head back over to Andrej Karpathy's tweet. Now, if I come over here, for example, one of the things he did is built this little GitHub, which is like really simple. It's very vague, and on this LLM wiki, he basically explains a core premise a bit. The idea is that instead of just retrieving raw documents at query time, the LLM incrementally builds and maintains a persistent wiki, a structured, intellectual collection of markdown files that sits between you and the raw sources. And you know what, guys, all we're going to do literally is come down. We're going to do a classic Control A and C. So we're going to come down and grab all this stuff here. Beautiful. And then we're going to head over to our environment of choice. And we're going to be using Anti-gravity to do this. Of course, this could be done in a terminal. It could be done in a Cursor. Doesn't really matter. I'm just using this as an environment. Let me open up a brand new folder. So, I've called this one Obsidian. Now, I'm going to open up a terminal. I'm going to run Claude Code just because I prefer that environment. I'm going to come up and let's run Claude.
And one little hack, by the way, which I highly recommend you do, is you can download these images locally. So, if you go to Obsidian Settings, File and Links, set attachment folder path to a fixed directory, e.g., "raw assets." Then in Settings, you can hotkeys and click "Download to find attachments for the current file," which means that when you grab information from a page, it will save it there, which is going to be super duper handy. So, come back over, then I'm going to give it the final prompt, which is: "Hey there, I would like to create a structure in line with the below. Please ask me any questions if you wish to, but just follow this along and create it for me, please." Beautiful.
And before we actually run it, it's best to understand exactly what the capacity Obsidian system actually is. So, most RAG is what we call reservation. In other words, that we're constantly going and getting the same information every time we ask the question. But with the LLM wiki, the way it's going to work is it will compound over time. So, it's the same question every time you start from scratch with traditional RAG. With this system, it gets smarter with every source that you add. So, traditional RAG, this is how it works: Query, chunks, answer. Claude finds chunks, stitches an answer, and then forgets it. Tomorrow, does the same process. But with this system, the way that it works is one new source will update 10 to 15 different wiki pages. The knowledge stays, and every future query reads from it. So, it's self-building in that sense, which is cool. So, this has what we call the compounding effect. So, one source, 15 pages are updated. You drop a new article into your folder. Claude reads it. It finds every page in the wiki it relates to. Say, for example, you and I want to grow on LinkedIn, and we say, "Hey, this is a great article on growing LinkedIn." When we add that to the system, this will then update every page that it's linked to, which is really cool. It flags anything. This one's really cool, that contradicts us. So, any contradictions are getting ironed out in the process. Stuff that it knew. One source in 15 files updated in a single pass, which is why it's so super cool. You're building a personal Wikipedia. And it's only good if you can update it, right? So, three layers, no database. Everything lives in three places: a folder for the sources you feed into, a folder for the wiki that Claude writes, and a single file that tells Claude the rules. The rules of the game. So, we've got raw sources, the wiki, and a schema. You can see what that looks like. So, the sources might be articles, PDFs, transcripts, web clips, you name it. We've got summaries for the wiki. Claude writes it, and you kind of read it. And then the rule book, which is your conventions, your workflows, and it turns Claude into a maintainer of your wiki.
Now, the way it works is you can ingest. So, we're going to drop in things. These are the three kind of operations you're going to be able to query. So, one is like asking it questions, and then this process called linting that Karpathy mentioned, which effectively is every couple of weeks, find contradictions, orphan pages, stale claims, and the kind of maintenance that humans would naturally forget in the whole process. Awesome. So, these two files do a lot of heavy lifting: the index.md and the log.md. Effectively, though, you're going to create the LLM does a bookkeeping, and the wiki will compound in value.
So, what kind of things could we build with this? Well, we could do a book companion. We could do a research deep dive, a team or business wiki, and a personal life wiki. How does it actually work? Four rules, really simple. I'm going to pick one domain. So, not everything, like one book, one project, one research question. It works better when you have it as a set project. So, one project for one thing. So, growing on LinkedIn, or I don't know, managing this merger and acquisition, that kind of thing. Think of them as individual mini Wikipedia bases that you can call upon and question for different topics. You're going to invest in the first section. So, Claude's going to ask you questions, answer them properly. This is where your rule book gets written. And then you're going to run two screens: so, Claude Code on one side, and then your Obsidian graph view on the other. If you want to take a look at it, you can see the nodes lighting up as Claude does various different things. And then every couple of weeks, you're going to go ahead and purge it. So, that first session, you're going to paste that prompt I gave you. Claude's going to ask you questions, "Hey Jack, what's this working for?" And then you're going to co-write the rules with Claude in that process. Then, every session after that, the loop's going to begin again. You're going to drop a new source. Claude's going to integrate it, update the 10 to 15 related pages, and then you can see it expand and blow up in Obsidian. You create it, it maintains, and the wiki compounds.
So, let's head back over now to Anti-gravity and see how Claude is getting on with this. So, it's got a few questions. Now, it needs to know what the domain is about. We have everything now. And again, Claude.md, it's got all the different files, the systems. It's really cool, really handy. So, I think the first one we should be, should be a personal life assistant that we're going to use to leverage to make big decisions about our life. And we can just feed it that information. So, let's come down and give Claude that information. So, tell me the first domain, drop any file you want to into "raw" or "ingest," and do a real first run. Want me to get in it to get the version ready to rock and roll? So, I'm going to say something like, "Hey, dude. What I would like to do is do a file basically just on making better decisions and running my personal life. For example, there's an article I read on high agency, and I would like it to be, we're going to call this one a general productivity life decision-making wiki. I'm going to drop in information about me, who I am, what I'm about, just relevant stuff, and then also things and mechanisms of thinking that I find that are valuable that I want you to integrate for different things. What I'm first of all going to do is just drop in a URL to you, and I want you to go ahead and scrape everything in there, images and also text. But I'm obviously in the future going to use web clippers for that. Let me know when you're ready for the first URLs." Awesome.
So, I'm going to send this one off, and I'll show you that hack in just a second, but this is an example of an article in high agency. It is a fantastic read. My buddy Nick shared this with me. Really, really highly recommend it. I'm going to do is grab this link here. Come back over. I'm going to drop it in right there. It's ready for it. LifeOS sounds like a great name. Let's start building out that personal wiki. Awesome. I'm going to drop that in there as well. And then I could, I'll probably just get Claude as well to say, "Oh, and by the way, just add in all the information you know about me to this life wiki." Personal folders. If you go to my business folder, has lots of information about me, business information, add all of that in there as well. Wonderful thing if you've got code on your laptop and different folders that cover this. It can just go and grab that data.
Now, the hack, by the way, and something that Karpathy mentions is good, is that Opus itself has a web clipper. So, what you're going to do for that is click on these dots at the top right. Come down to Extensions. And in fact, you're looking for the web store itself. And basically, we're going to search for Obsidian, and this is the Obsidian web clipper. Like that. Going to come down, add this to Brave. Uh, there we go. And just add that as an extension. And then, personally, what I like to do, if you come in these dots, come down over to Extensions, click on Manage Extensions, find your Obsidian web clipper, click on Details, and just make sure you pin this to the toolbar like so. And now you can use that web clipper when you want to. So, for instance, if I was here, if I click on the little Obsidian, you can see I can now actually add this to Obsidian, and you can basically specify everything that you want to.
Now, one thing to call out with the Obsidian web clipper is if you click on the Obsidian logo here, for example, and then you click on this, this will then be the settings. So, effectively, what we can do is add all of the vaults here that we want to save to. So, I might have a personal operating system, I might have a YouTube growth, LinkedIn growth, whatever it is. And what you can do is create a brand new template, give what you want to. So, I could say "personal operating system," whatever it is. You go through this whole thing, you can save it, and you can, this will then actually add it directly to your actual folder itself. You just name the location of the folder path here, and then that's how you dynamically add everything over there.
So, now if I come back over to Anti-gravity, and now code's done everything for me. I've got all the raw files. You can see on the left-hand side now it's not only basically grabbed the thing that I wanted, but it's also added loads of specifics. So, if I click on assets, for example, it's got images. Boom. Same here. These are all from the article. So, I've got all these downloaded, all saved. Very, very cool. And in fact, if I come over now to Obsidian, which I have right there, you can see I've got the high agency George Mac. Now I can vectorize everything that I want to. And if I want to see the whole node view, all I would simply do is come over here, check this out, open a graph, and I can actually do anything I want to. So, with this in mind, I can now physically start asking it a question. And bear in mind, for me, this has got everything about me, decisions, mental models, people. You literally know, okay, these are mental models that it's kind of adapted for it to go ahead and use and do various things. It's got decisions here, which are going to be all the decisions that I make about me. Loads of really cool things that we can then piece together. And the more we use it, the better it gets.
Now, I've come back over here. Now, you can at this point build a search engine where you can ask it questions, make it a beautiful interface, do whatever you want to. But honestly, I don't think it's worth your time to get started. You have a search engine. His name is Claude. He lives here. So you can have your master folder open up in your environment with all of your subfolders within it and just chat to it here. And then once you exceed to a certain level and you want to add different bells and whistles, we can go ahead and we can do that. And you get to a point where with something like this, which is what I built for my community. If I ask, for example, "What is Anti-gravity? How do I set up Gravity Claw?" This is all, this is all trained on every YouTube video I've ever had, or my school community, everything, you name it. So you can start building fancy stuff like this for yourself with your personal operating system. But I just wouldn't do it off the bat, just to save me your time. You can just ask it to do that. You can give it API keys. You can do anything that you want to.
So, what we can do now is just test that it works. So, say, "Hey, based on everything in this folder, I am considering how I can be more effective at my job, how I can help my community members out and my clients, and to do that, I would love to know strategies of how to create more time in a day. Um, let me know your thoughts, and in addition to that, any of the files that you referenced on the left-hand side in making that decision." So, I'm just asking it a question based on its knowledge, based on what's cool. The more information that we add to this, the better and richer that it physically gets. And if I come out and I say something like, "Hey, this thing isn't true anymore," it will start to iron out those contradictions. And you can see it's got a list of files referenced down there. So, now you start to build up this wiki and just follow that process for different areas.
So, this system is awesome, but it has limitations. Limitations I haven't seen anyone else talk about. And if you don't understand where this falls down, you're not going to get the best out of this memory system. So, let's go through the Obsidian RAG limitation that I've not seen anyone really cover, and how you actually build this into a system that is more robust and gives you better long-term memory. I've thought very long and hard about this, and I've put all the kind of data and research into it. So, first of all, if you're just using Obsidian RAG with Claude Code, it breaks in five places, or you need to be aware of a stress test. Number one is your Claude.md grows over time. This is also true for your index file. So, if you think that every single thing you have in your index is like 75 tokens at 10 files, 750 at 10,000 files, you can see how your token window, just for looking through the index page, grows exponentially, and so does your token cost for using this system. So, there's a tax you must pay when going through that that I think is just important to be aware of. There's no semantic search. It doesn't do it semantically. So, it goes by the topic of the thing rather than the semantics around it. Summaries can go stale. We call back drift. Your million token window can fill up a bit quicker this way. And it's not really built for very large data sets. This is built for more small to, I'd say, early medium-sized data sets. Said another way, it can fill magical 100, but eat you alive at 10,000. Well, Jack, I don't think I have 10,000 files. Cool. Are you sure? I mean, with these things, if you're going to invest in a memory system, it's really important that you have something that can scale with you over time, and it's just worth bearing in mind on what your tokens per query might look like. These are just figurative examples, but you can see the kind of difference. The more files you add, the more expensive it can actually get. Obsidian scales linearly, Pinecone stays flat. So, these are rough order of magnitude estimates for this capacity pattern, not measured benchmarks. And the shape of the cube is what matters. So, it's just worth bearing in mind and calling out, just so you understand what's happening.
So, here's an interesting one. If I get had YouTube transcripts, this is something that is better off, in my view, actually living in a Pinecone database. So, a good way to think about it is that Obsidian is for the stuff that changes, and Pinecone is for the stuff that doesn't. You actually don't have to use Obsidian if you don't want to. I think one of the core benefits is the fact that you can clip and bring it in. But I'll explain all that in a second. So, let's say, for example, we have 184 YouTube transcripts. Giving that to Obsidian, I think, is not a good idea. That's way better off on Pinecone. I'll explain why that's the case. So, think about this: if you're putting it through Obsidian and using Claude Opus 4.6, Claude has to read, even Apple's getting involved now, the entirety of the context window first of all to understand it. Then it has to type back out to the various different sections and summaries. Pinecone, for example, runs a script. It runs a query where Claude's not even involved, and the embedding cost is like 100 times cheaper, based on the embedding model that you're using. So, your tokens at point of ingestion is way lower. So, it's saving you a lot to put it into Pinecone rather than Obsidian in this case. So, two systems, different physics.
So, Obsidian RAG, and this is just a great way to physically understand what happens when you're physically using it. And this might sound technical. This is so helpful to understand because when you understand your systems work, you can build anything. So, when we're using Obsidian, the session begins. Claude loads the Claude automatically, and that's the schema rule book for how everything works. You're going to ask it a question. Claude reads the index.md. Okay. And it's got a catalog of every page that you've got, an online summary. And it just says, "Look, well, Jack asked me about the cloak of invisibility." Well, um, you know, Harry Potter probably sounds like a great one to read. So does this. So does this. So does this. And it goes on and so on and so forth. Claude uses the retool specific files. The pages cross-reference with other video Obsidian-style wiki links. And then Claude reasons over what is loaded and writes back updated notes, adding new links, and appending to your log. So, Obsidian lets Claude reason over the structure of how you've set this up. It's a graph. It can answer, you know, "Which hypothesis did I revise because of customer 5?" Pinecone doesn't do that.
By contrast, Pinecone ingestion is one-time. Vectors are stored in Pinecone's index. You attach the exact original text. You can ask a question. Pinecone, you know, Claude just sends it over there. Pinecone runs an approximate, what we call nearest neighbor search, which finds the top K. Don't worry what that means. It's not, it's just like a technical gobble for like, "What's most similar?" very, very quickly, and it returns the exact text. We always discuss this as a librarian, and rather than pulling the entire text Harry Potter, it just grabs the specific paragraphs that you need to actually understand it. So, Pinecone gives Claude perfect recall of massive archives, exact text, flat cost, but it's similarity only, no graph, no reasoning over the relationships.
So, think of it like this: Claude the MD is who I am. Pinecone is what I've said. Obsidian is how I think. Now, there are three layers to this, not two, because you'll have your core.md file in your projects based on how you're physically using this, unless you're just querying it like a search engine. Now, what are they best for? Basically, core.md is your identity, voice rules, you know, read first and instructions. Obsidian RAG, active projects, decision logs, idea gardens, anywhere that structure matters. And then Pinecone is going to be transcripts, research archives, books, anywhere exact recall physically matters. Said another way, the magic word for Obsidian is reasoning, not memory. Claude.md is you, your identity. Pinecone remembers you. It's how you recall. And Obsidian thinks with you, reasons over structure.
So, if you imagine, for example, we're going to work, one of the best ways to think about this is Claude is like your identity badge. You're called MDM. It's your name. You roll your rules, read before anything else, and it never changes midday. Your workshop is where you actively build, where you're doing hard work. You know, notes are linked by string. Reasoning happens there. And then warehouse is Pinecone in this example, where we put all our big stuff, all our big files, all our video transcripts, all our archive of every email we sent, we throw and let that live inside the warehouse.
So, with this in mind, what do you actually do? What is the complete structure to run this sustainably so that you have a system that never ever forgets? Well, the way that you understand this relationally, okay, you have your identity in your Claude.md files. If you're using the Claude app, that's the "About Me" section. Again, you can have that based on projects. Obsidian can have all the little mini things, the little projects that you're thinking about. That is how you think. Pinecone is your full archive of all your history. And then NotebookLM is something we bring in to do deep dive and research. So, for example, if we're in a Claude project and we're learning about the manufacturing process of white bottles, I could say, "Hey, spin up a notebook with 200 resources explaining everything." Then we can bring that back over to our laptop and even throw that into our long-term memory if we ever want to recall it in anything that we're physically doing. And understanding how these four connect relationally is really important.
So, for example, if I was taking my entire YouTube transcripts, I'd store that in Pinecone, my warehouse, my long-term storage, the place where the information is never ever going to change. And the key thing to understand here is that you can connect Pinecone so easily now to Claude. It is ridiculous to the point where you can just ask it a question, and Claude will reach over to Pinecone, find the answer, and bring it back. And for every project that I've physically done, and I can even show you this now, I showed you earlier in my AI with Jack dashboard. Like I find stuff like this where I can ask questions, and I have processes here that will automatically on a daily basis scope the transcript for everything inside my YouTube channel. And it also works inside my community, right when people are saying, "Jack, I need to find something like, I don't know, let's say, pricing AI services," whatever the thing is. This scrapes everything daily, and this all lives in Pinecone. And it's great. And you can click on this, come and check out the community. And again, it's like there's a lot of different stuff that exists within the community that you can come and grab. And all that is linked in Pinecone. So, I find it on a day-to-day basis wonderful. But the thing about Obsidian and the Obsidian Index system is it brings a completely different value set. You just need to understand what you're using it for and where you realistically want to introduce Pinecone to support that. And so, without a long-term memory system, we're leaving value on the table. Which is why the next thing that we need to do is learn how to integrate Pinecone into our system so we can secure information forever, which we're going to do by watching this video right.