Transcription
Hey, I'm Dave. Welcome to my shop. Today's episode is one of the most requested I've ever done: how to add your own knowledge files and documents to a large language model, both local and online. I'll explain the difference between retraining a model, using retrieval augmented generation, or RAG, and providing documents for the context window. I'll show you how to upload your own documents, both to ChatGPT and to a local Ollama model running under Open WebUI. Once you insert your documents into the model, they become part of its knowledge base for answering your questions.
Now, before we dive into retraining, RAG, and context documents, however, I want to give you a little demo of a modestly sized model running locally on the dual Nvidia RTX 6000 8 setup. That's because in my last episode, I saddled the big workstation with a massive model that the other machines couldn't even hope to run: 405 billion parameters loaded into its 512 GB of RAM. But the protests in the comments were many. Folks wanted to see this beast of a machine chewing on a regularly sized model to see how it would perform. So, let's take a quick look at just how fast it can run a 1 billion parameter model.
So, to run the smaller model, I'll take the listing, and we'll see that we do, in fact, have Llama 3.2, the 1 billion parameter model. And so I'll run that with the verbose flag. And as soon as the model comes up and is ready, we'll ask it to tell us a story, and we'll see how many tokens per second it can actually generate. Well, it's cranking through pretty quick here, 345 tokens per second. Let's get a longer story to see if it can sustain that. It scrolls by very quickly, and when it is done, how many tokens per second do we get? 324. So, it's able to easily sustain over 300 tokens per second generating on the dual RTX 6000 8 machine. I would mention that it's a Threadripper, but it's using almost no CPU during that generation.
For comparison, we'll launch the 70 billion parameter model, which is predictably about 70 times as big as the 1 billion parameter. We'll ask it to tell me a story, and we'll see how many tokens per second it can generate once it's done generating. Now, I'll let it run for a little bit here so you can see the natural output speed, which is still very usable. If you weren't generating tons of content, I think it's actually quite fast, but it's certainly slower than the 1 billion. Let's fast forward to the end and see just how many tokens per second this one is generating. And at about 20 tokens per second, it's quite a ways off of the 300 pace, but it's still pretty useful, and it's more than usable. So, it's still fast enough for most purposes.
Now, let's get to our main topic of adding information to existing large language models. There are three primary ways which you can do it: retraining a model, using retrieval augmented generation, or RAG, and uploading documents directly into the context window.
First, let's consider retraining a model. Think of the model as a bit like being a student who has already learned a lot, but now you want to teach them something new or correct their understanding. You go back to the basics with them, bringing in new books, updated lessons, and putting them through another round of study sessions. They don't forget that much of what they already know, but they use your training to fine-tune and add to their knowledge. The process is thorough, but it's a lot of computational work, takes a lot of time, and once they've learned it, it's permanent. Every time you use the model from then on, it will have that updated knowledge embedded deep within it, ready to apply in all relevant situations. But retraining a model requires a lot of resources: more data, lots of computing power, and time. It's like sending your student back to school for a while to learn and improve on what they already know.
Now, let's compare that to retrieval augmented generation, or RAG. Here, instead of retraining the model, it's as if your student doesn't have all the information they need at their fingertips, but knows exactly where to look. And when asked a question, they quickly consult a library of books, pulling out the most relevant sources, and then give you a response that combines what they knew before and what they've just looked up. This process is faster than retraining because it doesn't involve the deep, permanent learning of new material. Instead, it allows the model to retrieve information dynamically from a database or document pool, crafting its answer based on up-to-date sources. It's a much more agile process, and it's great when you need the model to adapt to ever-changing or large sets of data without retraining it every time.
And finally, there's uploading documents to the context window. Imagine you're in a one-on-one conversation with the model, and you hand it some notes. The model can reference these notes while talking to you, but it won't internalize them in the way it would with actual retraining. It's like a student who gets to peek at a cheat sheet during an exam. They can look at your document and use it to answer your questions, but once the exam is over, or your session ends, they'll forget the information. This method is the quickest way to provide specific, immediate knowledge, but the information only lasts as long as that specific session or conversation. When you're done, the model won't retain the uploaded document unless you upload it again next time.
So, in summary, retraining builds long-term, permanent knowledge within the model. RAG fetches relevant knowledge dynamically without needing to retrace. And uploading documents into the context window is like giving the model a temporary cheat sheet for quick reference. Now, each has its own strengths depending on how permanent or flexible you need the model's knowledge to be.
Before we get into these approaches, let's look at why we won't be doing the first method: a formal retraining of a model. And the first reason is openness, meaning that you have no access to the ChatGPT model itself. So, there's no way you can directly modify or retrain it. With something that's more open, like Llama 3.2, the models are generally designed to be open in the sense that you can access the model weights and modify them, but certain conditions still apply. You need to access the model weights, the core data that defines the model's knowledge. Depending on licensing and availability, you may or may not be allowed to use those weights freely for commercial purposes or large-scale projects. But let's say none of those are serious roadblocks.
But the next problem is hardware and software. Retraining a model takes almost as much in terms of skill and resources as training that model did in the first place. Fine-tuning an LLM like Llama 3.1, even on a smaller data set, still requires some serious hardware. That means you're looking at leasing or buying data center GPUs like the Nvidia A100, and depending on the model size, it could require an enormous amount of RAM to work on it. The other hurdle at this point is that it requires significant programming, even if it is mostly just Python. But you wind up needing to write code using PyTorch or TensorFlow or something similar to run your fine-tuning tasks. And when combined with all the hardware required, this likely rules out full retraining for most people. So, we'll focus on RAG and context, both of which are fully doable without custom coding.
Let's start with the easiest of these mechanisms: uploading additional documents into the model's context window. Our documents become the cheat sheets that we referred to earlier, allowing the model to reference them and incorporate them into its answers. With ChatGPT, you've likely noticed and even used the upload button at some point. When you upload a file to ChatGPT, it becomes part of its current context window.
Okay, let's ask it about something that it may have knowledge on from its general knowledge base, but I want the specific order in which to push the buttons on the 1134 to boot it up. Let's see if it knows. So, that's sort of a general approach for starting a PDP-11. Um, it does not take into account the owner's manual of the 1134, which tells you how to actually use the boot switch to boot to a ROM location, which will bootstrap the machine. So, it doesn't seem to have that context.
So, in our next step, let's give it that context. This is the simplest of cases beyond just dragging a document into the actual browser window. But we'll click on the paperclip to attach a document. We'll say, "Upload from the computer," and I will pick, let's see, we'll pick the PDP-1134 manual here. Now that it has that document and it's uploaded, we can ask it questions about the document. "Read the attached document and tell me the order to push the button to start the system." And now, thanks in part to the technical documents that we uploaded at the beginning of the session, it has the information that it actually needs to give us the correct answer to use the boot and switch, which will go through the M931 bootstrap, which will boot the system from ROM.
You could do this each and every time that you wanted ChatGPT to have access to this additional knowledge, but it gets a little cumbersome, which is especially true if you have multiple documents to do each time. Let's look at a better approach: creating our own custom GPT with our context documents fully baked in.
Okay, here we are at ChatGPT, but we don't want just regular ChatGPT. Not even 0.1 preview, not even GPT-4o with Canvas. We don't want any of those things. We're going to create our own ChatGPT. Let's do that by going to Explore GPTs. Then, in the top right, you'll see Create. And then we give a name and a purpose to our custom GPT: PDP-11 Expert. And it is an expert at PDP-11 stuff. And what it's going to do, it's going to answer questions for us about that stuff. Now, this next point is where we can upload our files to form part of its knowledge. So, we click Upload Files, and I've got a folder called PDP where I have a bunch of documents that are already OCR and their PDFs, and it can handle that format. So, we'll just upload them directly. It'll take a few against uploading the files, but it goes surprisingly quick. And now that they're all uploaded, soon as our button becomes available here, we'll click Create. And I'll say, "Anybody with the link can access it." PDP-11 Expert is a name, and we'll save it. Next, we can click on View GPT to actually use it.
Next, I'll ask it a very specific question that will require it to do some research. We'll try that. It's searching its knowledge base. Let's hope it finds something. You kind of have to actually read the document, but it actually does have that information. So, let's see what it comes up with. 8067. That's the correct part number, and that is how you wire-wrap the boards. That would have been really handy when I did this about two weeks ago. I mean, why read the manual when you can just ask ChatGPT?
In a previous episode, I showed you how to run Ollama and Open WebUI locally on your machine. So, let's take a look at the steps needed to provide context documents to Ollama.
So, here we are in the local machine and running Ollama and Open WebUI. I've clicked Upload File by clicking on Plus, and I've selected the PDP-1134 User Manual, just as I did last time with ChatGPT. Now, I'll ask it a different but equally specific question: "What are the power requirements for the PDP-1134?" And it cranks out an answer really quickly, and it is, in fact, the correct answer. And at the bottom of our answer, you will see that it is referenced the context document. Even better, we can click on that link and bring it up and actually see the part of the document that is making reference to. I find that incredibly handy when you're looking up a citation of where it got the information that you're asking it about.
The size of the context window itself can also be quite constraining. Think of numbers like 4,000 tokens for a model like Llama 3. If the documents you were trying to provide exceed the size of that context window, clearly it's not going to work as well. For example, I have a set of a half dozen PDP-11 reference documents that I'd like to incorporate into my searches, and the context window is generally not large enough for that.
Our step then is with retrieval augmented generation, or RAG, which is a system designed to dynamically retrieve information from an external knowledge base or database. So, when the user asks a question, the system searches through a large repository of documents or data and pulls out the most relevant pieces. This information is then combined with the model's internal knowledge to produce a more accurate and contextually informed response. The key strength of RAG is its ability to efficiently handle large amounts of data, pulling only what's necessary at the moment of the query. This allows for a more scalable approach, especially when dealing with complex, evolving data sets. The retrieval process ensures that only the most pertinent information is used, which makes RAG particularly useful for situations where accuracy and specificity are crucial.
In contrast, uploading documents into the context window is a more static approach. When documents are uploaded, their content is directly inserted to the model's input window, giving the model access to that information for generating responses. It's essentially like you had just typed the documents into the query window. While it's a straightforward way of providing additional information to the model, it can be inefficient for large or numerous documents. The model has to work with everything that's provided in the input, regardless of whether all of it's relevant to the specific question being asked or not.
So, RAG's ability to dynamically retrieve information makes it a more efficient and scalable system. It can handle large data sets without overloading the model's memory or processing capacity. The retrieval process allows the system to be more selective, bringing in only the most relevant pieces of information in response to your query. This also makes it more adaptable in situations where the information being accessed frequently changes, as RAG can always pull the latest updates from the source without needing any manual adjustments.
Now, the PDP-11 isn't changing much these days, but to maintain accuracy on more current things, the user would need to regularly upload newer, revised documents, which adds a layer of manual upkeep. Ultimately, RAG offers a more dynamic and scalable approach, especially well-suited for handling large and evolving knowledge bases, while uploading documents into the context window is more suited for smaller, more static data sets. RAG ensures that the model's responses are not only accurate but also timely, reflecting the most up-to-date information available. Especially if you're pulling the data from a database directly, then you're getting the very latest snapshots of the database. If you're not doing that, even with documents, if they're updated frequently, at least you're getting the most frequent documents. Conversely, uploading the documents provides a simpler, though more limited, method of augmenting the model's knowledge.
So, let's take a look at setting up RAG on our own system. Ollama. One note about the configuration: I'm going to run the Open WebUI locally this time, rather than in the Docker container, simply by enlisting in it from GitHub and then launching the start script from the folder called backend. I'm going to do it that way because it makes the documents folder easily available in my local file system. If you're running it in a Docker container, it'll still work, but you'll need to use the Docker copy command to copy your files into the container's version of the data document folder.
To make our documents available to models within the Ollama Open WebUI system, we need to go to the admin settings. So, we'll go to the admin panel, we go to settings, and we'll go to Documents. Now, I've copied the documents that I'm interested in into the actual folder underneath the Open WebUI folder that I enlisted in, in order to put the documents there, but it won't automatically find them until you come in here and scan. So, let's click on scan and see what happens. We monitor the other window to see its progress. Now, you can see it's obviously pulling the deck manuals and so on. I can see stuff about cylinders and hard drives and so on. So, it's now parsing this data, processing it, and it is now ready. So, I can click save, which is slightly obscured by a little dialogue box that I can't get rid of for some reason, but I'll click save, and those documents are now available to be used in creating custom models.
Now that we have our documents registered with the Open WebUI interface, we can create a new model, a model which incorporates the knowledge encoded in our documents. When we subsequently query the model for answers, it will use retrieval augmented generation to incorporate that knowledge into those answers.
So, in the Open WebUI interface, we want to go to Workspace. Once we're in Workspace, we want to create a model. We'll give it a name, we'll select a base model to work with, and I'm actually going to go with the smaller but efficient Llama 3.2 because I want it to be more dependent on my data and less on its own data that it brings to the table. Whoops, I got those backwards. Let me swap those. We'll put that there, and we'll type PDP-11 Expert in the description. And now it's a simple matter of telling the model which documents to reference. The easiest way is to pick all documents, and it will be able to reference everything in your data/documents folder. However, that will be slower. So, in this case, I'm just going to give it one or two documents that apply directly to my actual machine, and we'll see if it can then incorporate that document knowledge into its answers using RAG.
Now, back at the main chat window to interact with our model, we'll pick it from the list. It's now available as the PDP Expert. Let's try a really specific question and see how it does. And you can see in the left-hand window, it is found and hit our data and referred to it to produce its answer. Ah, and because I probably didn't give it the information that it needed on the memory board, the closest it could find was the RQ DX3 controller. So, it's going to answer its questions with pretty much that as its main knowledge about the PDP-11. So, let's ask it something that it might know. Okay, that's a better answer. And let's see. So, it's got the right jumper, and it's found the right point in the manual. I don't know if the manual has actual more detail than that or not, but it did find the right reference.
So, as we can see, it is, in fact, hitting all the RAG data, which you can also see scrolling by in the left-hand window as we do a query. And again, it's able to synthesize and recite the facts that it learned from looking at the RAG data and give me little look at approximately correct answers. And so, by creating a custom GPT with your documents embedded in it, it will use RAG to update and do retrieval augmented generation as it answers your questions.
If you found today's little sampler of AI augmentation to be any combination of informative or entertaining, please remember that I'm mostly in this for the subs and likes. So, I'd be honored if you'd consider subscribing to my channel and leaving a like on the video before you go today. And if you're already subscribed, thank you. And with that, you're now an expert on context windows and retrieval augmented generation. Collect your official certificate on the way out the door.
Be sure to check out the second channel, Dave's Attic, where you'll find our weekly podcast where we answer viewer questions on episodes like this live on the air. Well, not really live, it's recorded a day in advance, so almost live on the air. The podcast goes live every Friday afternoon, and they're called Shop Talk. So, take a look, maybe watch a back episode or two, see how it goes. Thanks, and I'll see you next time.