Transcription
Okay. Can a coding agent solve the biggest problem of retrieval augmented generation? This is what I'm trying to explore with my new agentic file search system. The idea is very simple. Instead of using embedding based semantic retrieval, can simple search tools get you the same level of accuracy? Because today's coding agents have to do retrieval of code snippets, but they use very simple tools like file search operations and code execution.
Now, I built a very similar system which can do retrieval using simple file-based operations without any semantic-based retrieval component. However, embedding-based retrieval has its own place. The question is, why can't we just combine them together? And that's exactly what we're going to be doing in this video.
So, this is the third video of a series on agentic file search. The project is open source and already has close to 500 GitHub stars. I will appreciate it if you also give it a star if you like the project. I will highly recommend watching the other two videos to get a better understanding of how this whole system is set up. But on a high level, there are three different phases.
The first phase is what I'm calling parallel scan. It simply reads initial parts of every document and, based on the user query, tries to identify which documents are most relevant. Then, it does a deep dive on the candidate documents. So, let's say you get from 100 documents to 10 documents using the initial scan. Then we do a deep dive by sending every document to a large language model.
But here is the best part, which the normal RAG system is not able to accomplish. In most of the cases, this step is not going to be enough. It will probably miss relevant documents. That's where phase three of backtracking comes into play. So, the system is smart enough that if it finds that there is a reference to another document, it can go back and read that document, even if it was missed during the initial two phases.
Now, this is critical for any real-world application, especially if you are looking at legal documents such as patents. They reference other patents or diagrams that might be in another document. Now, with this system, you can actually provide your domain knowledge to the agent, and it will be able to backtrack and cross-reference other documents. So, it becomes very helpful in the real world.
However, there's a major drawback, and the problem is that since it has to read every document, it's extremely slow. So, you can't really use it for real-time applications. Now, some of you asked me, why can't we just combine this with normal traditional retrieval augmented generation as a filtering step? And that's exactly what we're going to be looking at in this video.
Now, this is going to use a dual-path search pipeline, and I'm going to explain how this works. So, in its existing setup, it doesn't use any pre-processing whatsoever, except that it does document parsing through docling to enable every document is in markdown. But now, this new setup is going to introduce the concept of smart chunking. So, essentially, we divide every document into subdocuments using chunking. We compute embeddings. Right now, I'm using Gemini for that. Along with that, it also extracts metadata from every document.
Now, the metadata can be user-defined, or the system can itself automatically extract that metadata for you. For metadata extraction, we're using LangExtract, which is a Gemini-powered information extraction library. I previously covered this in one of my videos. I highly recommend watching that. So, in the end, everything is written into DuckDB.
Now, the corpus that you get has four different tables: the first one is the original documents along with the metadata, the chunks that were created, embeddings, and then this schema. These are definitely the metadata. These are basically the metadata fields, either defined by the system or user-provided. Metadata is either user-defined or automatically extracted by the large language model, and this is critical for real-world applications.
So, let's say if you have invoice data, you want the LLM to extract certain information for you. You can create that as a metadata field, and then you can use that for filtering of your documents before you feed it into the normal agentic search system.
Now, here's how the system works at query time. So, when the user query comes in, we run the same query through multiple different passes based on the mode. We're going to talk about those in a minute. So, you have the semantic search path, you have the metadata-based filtering path. It basically is going to filter documents, not chunks, but actual documents, and then we deduplicate them and feed those into our agentic file search. So, again, you are reducing the search space that the agent has to work with, which will give you a pretty good boost in speed.
Now, here's the beautiful part about this setup. You don't really have to worry about the accuracy of retrieval in this first step because the agent is going to take care of most of that.
Okay. So, the system has four different modes of operation. The first one is pure agentic search. This is what I have covered in my previous videos. Essentially, it's just using the agent along with these simple tools that are available to do exploration and search. By the way, the idea came from harness engineering, which is a new emerging field. And the idea here is that you want to give your agent generalized tools, not task-specific tools, but generalized tools that it can leverage to solve problems rather than having rigid workflows. If you're interested in the topic, I am putting together a video on that. So, make sure to subscribe to the channel.
The second mode is where you use semantic search as a pre-filtering. We do semantic search on chunks' levels, but the agentic file search step is going to receive the documents. Now, these can be enabled or disabled with simple flags that you provide to the agent when you're running it in the CLI.
The third mode is that you can just enable metadata-based filtering of certain documents before you pass it on to the agent. And the last mode is that you can enable everything. So, it will do filtering or the initial retrieval based on the metadata that you provide, semantic search, and then we'll pass on the candidate documents to your agent.
Now, at the moment, everything is powered by the Gemini model because they have the longest context window and they're pretty good at tackling the RAG stack problem. But if you want to use another provider, you can do that.
Okay, so let me walk you through the installation process, which is pretty quick. You just need to clone the repo and run the dependencies by running this command. Now, if you want to use an API for proprietary models, I highly recommend using the Gemini model. You will need to provide your Gemini API key. There is also a version of this on a separate branch that you can run with a local model. However, since the agent has to work with a number of different tools and it's a multi-step process, I have found that smaller open-weight models are not good at it. So, I would recommend something like a 32B model. I created a video on that. I will put it in the video description if you are interested.
Now, in order to run it, there are two different modes. One is the CLI. You just point it to the folder and you can ask questions. And the second one is a web UI, which is a web interface that you can use to retrieve information. So, let me walk you through the web interface of how the retrieval looks like.
Okay. So, here's how the interface looks like. Now, first, you'll need to select a folder. In my case, I have selected a folder that has 11 different documents. I am doing this because I just want to show you a quick demo of how this thing works. Now, when you are indexing your documents, you're going to be presented with this screen for the first time. Once you index them, they are going to persist in your file system. So, you don't have to run this again.
So, here are a few options. If you want to use the semantic search, you will need to generate your embeddings. So, you'll need to click on this. Now, you can define a custom schema, and this is a document-level schema. This is extremely helpful if you know the nature of your documents. Again, let's say if you're working with invoice data, you might want to provide who the payee is, what are the invoice amount, dates, addresses. That can be extracted by LangExtract for you. So, you can just define the schema over here, or you can use the auto-discover, which basically is going to read a bunch of different documents, then based on that, it will try to identify what type of documents are present and what type of information this system can extract.
So, for example, the documents that I'm providing are legal documents related to an acquisition. So, you can see that it saw that, okay, the document talks about acquisition, so it tried to extract data fields or metadata fields that it thinks are going to be relevant based on the information that is present. Right? So, you could run this and then change it, modify it based on your own needs. And once this is done, just start indexing.
Now, during the indexing process, it's going to do document parsing to standardize everything. Then, it's going to extract the metadata. It runs a standardized process on the metadata as well. So, let's say there might be some documents which might have like a financial impact. Other documents might use another word for it. Right? So, it runs these normalization processes to make sure that the metadata extracted doesn't have duplicates in the system and you're using normalized fields.
Okay. So, once everything is done, you're going to have an indexed document, and now you can start your retrieval here. Now, I can enable or disable different components. So, for example, I can enable both the semantic metadata and the agentic pipeline. And let's say if I ask something like, "Summarize the risk assessment PDF file." Right now, the system is smart enough to automatically identify what exactly it's supposed to do because it is an agentic system. If you use a simple RAG setup, it will probably not be able to summarize an entire PDF file for you. Right?
But in this case, if you look here, so it says, "Starting phase one with semantic search for quickly retrieving the index content and metadata for this PDF file." Right? It does file-name-level semantic search as well. So, this is the PDF file that it got, and then it says, "Okay, now I am retrieving the full document to extract the prioritized task factors, mitigation strategy, and financial impact summaries." Right? Then, it automatically goes and retrieves the whole document, sends it to the Gemini API, and we get a full assessment of what exactly is in that document with proper references to different sections in the document, which is pretty powerful.
This is a very powerful design pattern, and I'll highly recommend checking this out if you are interested in retrieval augmented generation. Now, keep in mind, this is not a production-ready system. I have quite a few other ideas that I'm playing around with. But if you need help with a retrieval augmented generation system for your own applications, you can reach out to me. I'll be happy to help. Details are going to be in the video description.
Anyways, I hope you found this video useful. Thanks for watching, and as always, see you in the next one.