Transcription
Building an effective RAG agent is fundamentally an information retrieval challenge. You need to retrieve the right information based off a user's query to generate the correct response. And if you don't, it's game over.
As a lot of knowledge bases are messy and unstructured, you essentially need a search engine to retrieve the right information. Semantic search or vector search is the trendy approach for retrieval. However, it's just not good enough for certain types of queries. Your RAG search engine needs to be built on a solid text-based search foundation to handle cases where semantic search fails. This is essentially hybrid search, a topic I've already covered on this channel.
But there's lots of different ways you can configure hybrid search to work. And in this video, I'll be taking things to a whole new level by allowing the AI agent to adjust the retrieval weights depending on the question that has been asked. That way, the agent can prioritize different types of retrieval for particular types of questions. And I also add in pattern matching or fuzzy matching as a fallback in cases where dense embeddings and sparse embeddings fail. This is great for searching for IDs or codes that are buried in blocks of text.
Today I'll explain and demonstrate all of this in N8N, dense embeddings, sparse embeddings, pattern matching in the context of a RAG agent hooked up to Superbase and Pinecone. And if you like this level of detail, then please give the video a like below and subscribe to our channel. It really helps us out. Okay, let's get into it.
In a recent video, we loaded in thousands of product manuals into a Superbase vector store. This was 10 gigs of data spread across 215,000 chunks. So, I wanted to see how accurate the retrieval was in the system. One of the tests I did was I picked out an arbitrary PDF. This was for a Whirlpool refrigerator. And I asked some questions to see could I get relevant responses back. And I really wasn't happy with the results. And I had a feeling that I wasn't able to find this specific manual that contained the answers to my questions.
So, what I did was I just checked to see could the AI agent retrieve chunks that contained this product code because if it could find this, then it could load up the entire file or a summary of the file to help answer the question. So, I dropped this into the chat and I said, "Tell me about this specific product code just to see could it find it." And this was using hybrid search, by the way. And as you can see below, it says it couldn't find any specific information about a product or model with that code.
Now, this was confusing because I know that information is in the vector store. So, if I come into the documents table and if I use an ILIKE operator and a wildcard search and I'll just drop in that product code. And if I click apply filter, you can see that there are four results. So if I'm able to find this via a simple filter in Superbase yet the agent isn't able to find it, then we have a big problem.
So I carried out an extensive deep dive to figure out why the agent wasn't able to return these chunks. And based on what I learned I was able to build this dynamic hybrid search system which is now successfully able to retrieve the right chunks to formulate the correct response. And there's actually a number of causes that are worth discussing.
And the first one is messy unstructured data. There are numerous published papers that report 80 to 90% of all data in enterprise is in unstructured formats similar to the PDFs that I was importing here. They could be Word documents, emails, meeting transcripts, voice notes. And this unstructured text can be really difficult for a RAG system to properly process. So if we take this example here of this 36-page PDF, this translated to 200 chunks in the vector store of which two of them actually contained the product code. Now there are two manuals that actually contain this product code. That's why we have four results here. But the issue here is the actual extraction of text.
So if we look in here, firstly this is a multilingual document that contains both English and French, which is a different problem. But you can see that the product code isn't actually extracted independently. The code runs into the word "instructions" and the word "you". And on the right hand side here when the keywords are extracted from this text for full text search you can see that it's not extracted cleanly. And the reason for this is that I use the extract from PDF node in N8N which simply just extracts machine-readable text from a document. So you can see that this is machine-readable, you can select it, but the actual layout is all messed up. So when I select this text and go up, it's not actually selecting "followed" which you think it would, but it's picking up text here. And this is what I mean by messy text.
Now, this could be improved with OCR. Instead of using the native text extractor, so with a system like Mistral OCR, when we're ingesting the documents, we could send them into that service and it will retrieve much cleaner text back, which can then be chunked and embedded into the vector store. And we do recommend that you OCR all PDFs because it actually maintains layout and hierarchical structure like headings and bold and tables. But at the same time, OCR is not perfect. It can mess things up, too. Particularly if it's a messy scanned document where it's not cleanly extracting out text.
So, another solution for this messy data problem is to actually pre-process the data when you're ingesting it into your system. And this could be done through human annotations or it could be AI-powered. So, you could send the document or the chunks into an LLM and get it to extract out relevant information. And that information could then be prepended to the chunk. So it could be a chunk description or it could be added as metadata to the chunk as well. So that's one approach you can take which is to clean up your messy data when you're importing it into your RAG system.
But you will never get your data fully 100% clean which is why you need to look at other retrieval mechanisms when dense embeddings and sparse retrieval doesn't work. You need a fallback search mechanism. And that's exactly what we built here with our dynamic hybrid search, where as a tool for the agent, the agent can define what weightings to give to dense embeddings, sparse embeddings, or pattern matching as a fallback for situations like this.
So let's explain these terms. What are dense embeddings? What is sparse retrieval? What is pattern matching? If you'd like to get a head start with dynamic hybrid search for your RAG agent, all of these RAG workflows, including our state-of-the-art RAG system, is available in our community, the AI Automators. Link in the description below.
So, first up, dense retrieval or dense embeddings. This is your typical semantic search. It's an AI agent that has a vector store as a tool. For example, you're using an embedding model like OpenAI's text-embedding-3-small or 3-large. And how it works is when you ingest documents, when you import documents into your vector store, the original text of the documents is transformed into dense vector embeddings. So here for example we have a sentence from this product manual which includes information about the water valves and the ice maker of a refrigerator but it also includes a part number which is important. So this chunk is then transformed into this dense vector embedding. This is a representation of this text and it's spread across 1536 dimensions or numbers if you're using OpenAI's text-embedding-3-small model for example. So the number of dimensions changes depending on the embedding model that you use.
And you can kind of think of these vectors as numbers that are plotted in a high-dimensional space. So here we have three dimensions. And you can see all of these various numbers are essentially these numbers here. Now in reality there's actually 1,536 dimensions which is a bit hard to visualize but you get the idea here with 3D. And the important thing is these numbers capture the semantic meaning of the text. So if your paragraph was about cars for example, then that word "car" would be plotted close to the word "automobile" or "vehicle". So if someone was searching for "automobiles", it would return information about cars even though it's a completely different word.
So onto the query side of it then, if someone was to ask the AI agent why the ice maker was not working, in this case that query would also be transformed into these vectors or these numbers and then it would be sent into the vector store using an algorithm like cosine similarity to figure out what numbers are closest to the query numbers because they would be the ones that have the closest semantic meaning. And as a result you get, let's say, 10 results that are closest to the query vectors that you sent in and you know, those are the closest matches.
So, these dense embeddings are brilliant at understanding the meaning and the context of the information that's ingested and of the questions that are being asked. And they can find conceptually similar content across different phrasings and noisy queries like the car, automobile, vehicle example that I gave. They're also able to handle synonyms and related terms. And there's lots of different models that can support multilingual embeddings as well. Where they fail though is they lack transparency. They're a bit of a black box. You don't know why they're actually returning what they're returning. And they don't work well at all when it comes to exact codes and identifiers like the part number that I showed because it's all down to the training data. If that part number was everywhere in the training data, then it might actually understand how it all fits in. But it's more likely that it hasn't come across that part number before, and it just doesn't really know how it contextually fits in with everything else. And the other thing is it can't guarantee exact matches because it's going to find conceptually relevant and similar ideas in the data set and it'll return those as much as the query terms that you sent in.
So there's lots of different examples of dense embedding models. I talked to OpenAI's versions but Cohere also have ones. Gemini, GINA, BERT is an industry standard one. Colbert is a different version of an embedding model that uses late interaction. But if you want to read more on this, there's the MTEB leaderboard which I'll link in the description below. And this contains all of the industry standard embedding models that you can pick and choose from.
So back to the original query, "Tell me about the part number." You can kind of now understand why the dense embedding models didn't return the right chunks because they're not really trained on arbitrary part numbers that are random letters and numbers and symbols. What did come as a surprise though is the sparse retrieval or the lexical retrieval that's built into the hybrid system that I used also didn't return the accurate chunk. So let's have a quick look at this then.
What is lexical retrieval? This goes by lots of names. It could be full-text search, sparse embeddings, sparse representation, lexical search. But what it all boils down to is a tokenization approach. So back to our ingestion, we have our original sentence of text. And for this full-text lexical search to work, it needs to tokenize this paragraph or this sentence. So it works its way through an analyzer and a tokenizer. And depending on the system that you use, it could lowercase the text. It could stem the text to root words. It could remove stop words. It could remove punctuation. So even though this is the text, you end up with this type of index where the keywords are included. In some cases, they're stemmed. So if you look at "dispenser" is in the original text, however, that stemmed to "dispense". And that's a good thing because it means that it could match on "dispensing" or "dispensary" or "dispenser". It's the same with "controls" is stemmed to "control". But the problem here is that the part number is actually split apart. And this is the reason why even though lexical search gives you better exact matches than dense embeddings, it is lossy. It is transforming the text and it might mess it up in this case.
So as we import this PDF and this sentence here, we now have an index of the various words. And then there's two different types of implementations of lexical retrieval. One uses an inverted index and another uses sparse vectors. And I can give you an example of it here in Superbase. So we have our 215,000 chunks. You can see the content of the chunk. You can see the metadata associated with the chunk. And here you can see the vector embedding, the dense embedding. This is the 1536 dimensions that I talked about. So the sparse representation then of this content is in the full-text search field here. This is using TS vectors. So Superbase is built on Postgres. So this was the text that was indexed and then these are the various keywords and actually their positions of the keyword in the text that have been pulled out. So when someone carries out a full-text search, they're searching across the keywords that were extracted. But similar to my problem with the product code, you can see here that it's pulled out this bizarre piece of text because spaces weren't included in the extracted output when we were ingesting the file. So it's a little bit of a blunt instrument. It's looking for white spaces. It's looking for punctuations. It's looking for full stops. And if it doesn't find them, then it just creates strange keywords like that. And Superbase uses TS vector and TS rank. So that is an inverted index implementation. And as you saw, it's pulling out the terms and the positions of those terms in the chunk. And then all of that is saved into an inverted index for querying later on.
There is a different approach to lexical retrieval, which is sparse vector embeddings. So that same piece of text that I showed you here that could be sent into a sparse embedding model to get term IDs and weights. So to give you an example of that, if we go to Pinecone, I have a sparse index set up which uses Pinecone's sparse-english-v0 model. And I've ingested this PDF which has given me 283 records. And if we look at one of these records, we can see that we're getting these indices and values. So these indices are essentially the IDs of the terms within an overall vocabulary. That's why it's the Pinecone sparse English model. So this model has been trained on English literature and it has extracted out all of the various terms within the vocabulary and has mapped it in this model and it also understands the weights of words within different contexts. So here the word "water" is mapped to the ID 4522 and it carries the weight 0.78 within the context of this sentence. So it's a more sophisticated implementation of the inverted index because it is using a sparse embedding model but it's the same concept as the inverted index version and you can even use the same algorithm. So you could use the BM25 scoring mechanism here with an inverted index implementation and you can use BM25 as a vector using the sparse vector implementation.
So these term IDs and weights are then plotted again in a vector store across multiple dimensions. And then when it comes to querying, if it's an inverted index implementation, you have your query "ice maker not working" that is then stemmed and lowercased and punctuation removed, tokenized and a lookup happens of those tokens against the inverted index and the results of that lookup are then scored or ranked using a particular algorithm of which then you get the most relevant results back. So that's how BM25 works. That's how TS rank works with the Postgres implementation of TS vector. And for a sparse vector implementation, it's the same idea. You have your query, it's tokenized, and then using the likes of product or cosine similarity, it figures out what are the closest vectors, and then the results are sent back. So the actual scoring within the sparse vector implementation happens before they're ingested into the vector store because that's what determines the distance. Whereas in the inverted index, the scoring happens after the candidates are retrieved.
So lexical retrieval or sparse retrieval is the industry standard, particularly BM25. It's high precision, it's fast, it's good for exact matches, it's explainable and transparent. So it is the core of a lot of search engines. And then even with these learned sparse models, you have some semantic expansion. The idea of "car" being expanded to "vehicle" and "automobile" applies here as well. And you can fine-tune these models depending on a particular domain. Where they fall down though is, okay, they're not as semantic as dense, but that's fine if you're going to run them side by side in a hybrid system, but they're not ideal for multilingual text, and that's a big problem. In my example here, this user manual is both in French and English. So, the tokenization is going to be all over the place. A lot of these tokenization algorithms use white space to split apart words. But in Chinese or Japanese, they don't use white space in between words. So the sparse model needs to be language-specific because you have all of these kind of linguistic processing rules that create the tokens. It's not great for misspellings and typos either because you saw the tokens that were extracted. You need exact matches on those tokens for it to return. And with the sparse models, there is a dependency on training data as well. And so for a lot of those reasons, it's not as good as pattern matching for exact matches of codes. You saw in the example above, it's split apart the code based off a separator.
So some of the examples here for classic inverted index implementations, you have BM25 and TF, which are essentially term frequency scoring mechanisms. There's Postgres TS vector and TS rank, which I've gone through in a previous video and is what we've implemented here. You then also have learned sparse embeddings. So, SPLADE is one of the main ones. The Pinecone sparse English example I showed you is actually built on the DeeperImpact model as well. And this is what lexical retrieval looks like in N8N. So, I have two different agents here. One demonstrates full-text search with TS vector. So, that's your inverted index. And here we have a sparse embedding search using Pinecone's sparse model. So, here I've asked, "How do I turn on the ice maker?" And that's gone to the full-text search in Superbase and it's returned various chunks that are quite relevant. So you can see we're getting information about ice dispensers, ice makers across all of the various chunks. We're returning 10 here. So this is quite relevant and arguably better than semantic search actually. And we're getting a decent response. Now I haven't given a product code. So it's generalizing based off the chunks that were returned, different styles, important notes, etc. So a decent response. Of course, if you provided a product code, it would be a little bit better.
So what's happening behind the scenes here? This is a tool call. It's hitting my full-text search edge function on Superbase, which is this one here. And all this is doing is it's just relaying the payload to the database function. So I have a search documents FTS database function. And it's quite a simple function. It's essentially just taking what it gets from the request and using the TS rank function, it's passing the query text and then it's outputting the response ordered by the relevant score. So, pretty straightforward. You can take a screenshot of that if you want. And if you've seen my other hybrid search video, you'll know that the actual FTS column in the documents table is actually set to a default value of `to_tsvector`. So when this row is populated with content, it's auto-generating the actual tokens. So you don't need to purposefully populate this field. It's all auto-generated. So that works pretty well as you can see.
Let's try that then with the sparse embedding search on Pinecone. Now I don't have 215,000 chunks in Pinecone. I just loaded up the one document, but just to give you an idea of how it works. So we have our AI agent. The AI agent has a tool which is the sparse embedding search tool. So this is essentially just calling a sub-workflow and the sub-workflow is the same workflow. It's calling itself. It's calling this node here. And then when it hits this sub-workflow, we set the variable which is the Pinecone index. We generate a sparse embedding using the embed endpoint of Pinecone's API. And you can see here that we're using the sparse-english-v0 model. The input type is query, which is important. Truncate is set to end and we're passing the query text as set by the agent. That returns the sparse vector for the query which then hits the Pinecone query endpoint and we pass our sparse query and it compares it against everything that's in the data set and returns the closest matches based off product, I believe.
So let's give it a go. Let's ask the same question and then there is the generated response. Again, we're getting two different styles. Now there is only one document actually loaded up here. You can see on the right hand side it has returned the various chunks where there is a match, a lexical match of the tokens. And if we jump into the sub-execution you can see that we got the query "how to turn on the ice maker". And then for generating the sparse embedding for the query, these are the indices and the values. The indices are essentially the terms within the vocabulary that is sent into the query endpoint and we get back our matches which is then relayed to the agent. And this is the setup on Pinecone. It's just a simple sparse index using that model. And I've just ingested all of the data for this document into this index. If you're looking for more information on this, I have a full breakdown of Pinecone ingestion for sparse embeddings in our hybrid search video which I'll provide a link in the description below.
So now that you have a handle on what dense retrieval and sparse retrieval looks like, let's now move on to pattern-based retrieval. And no one is talking about this when it comes to RAG agents and there really should be. So let's have a quick look at how it works.
So on ingestion again, we have our document we're bringing in. This is the sentence of text and instead of splitting it into tokens, we create what's called an n-gram index. Essentially, we're creating an array of overlapping character fragments. So this is trigram. So it's three characters. So "t h e" as you can see there and they overlap. So you can see we have "the water inlet". So the first one is "the", the second one is "he ", the third one is "e w", the fourth one is " wa" and it goes on. And all of these fragments are then stored in an n-gram or a trigram index as you can see here. Now these can also be stored as vectors as well. I'm not going to get into that in this video, but the beauty of this approach is when you send in a query, which is what I did here. So, I'll add a filter content ILIKE is the operator, which is a wildcard search, and I'll enter my wildcards, which are the percentage signs, and then I'll just paste in the code and click apply filter. You can see that that was lightning fast, number one, and it's able to output the chunks that actually contain that string or that substring. So, that's essentially what happened there.
And what happens behind the scenes is the query itself is also split into character fragments. A lookup happens of the trigram index. Lots of results are returned and they're ranked based off similarity. So here for example we have an exact match on this term which is what we're getting back here. But let's say one of the documents contained "103" for example, then they would also be returned but with a lower score because not all of the characters would have aligned to the query itself.
So now if I run this query with this AI agent, so you can see we've got a response from the hybrid search and on the right hand side here, let me open it up. You can see that we're now getting back the chunks that actually have this information. So if I click on search, if I drop in the code, you can see it's there in that chunk. It's there in that chunk. It's there in that chunk. And that's the exact one I showed earlier. It has "instructions", the substring, and "you" after it. And you can see the AI agent was able to generate an accurate answer now. So key information about the product. It's a refrigerator with the user manual. So this is epic retrieval where dense embeddings and sparse embeddings failed.
And the beauty of it is that you can use the reasoning of an LLM to actually decide which search mechanism to prioritize based off the type of query it's sending in. So if I click on hybrid search on the left hand side here you can see that it sent in the query which was the code and it weighted the pattern matching 70%, sparse matching 20%, and dense matching 10%. Because the LLM knows this is a product code that's more likely going to show up in a wildcard search than in a dense embedding.
So where this type of pattern-based retrieval shines, it is ultra high precision because if you can get the query right, it will find the substring within a large quantity of text. So it's ideal for handling the likes of IDs and codes and tokens which may be broken apart in a tokenization process for sparse embeddings and dense embeddings just would have no chance with in the first place. It's ideal for fuzzy matching. So things like typos, partial codes, because the way the character fragments are generated, you might get a partial match on a word which may then be returned by the trigram index and if it's scored high enough in the similarity score, it'll be returned to the agent. It supports regex. So here we're just using basic wildcard retrieval, but you could get quite sophisticated with it. And it's language-agnostic, so it's just based off what text is actually in the chunk in the first place.
Where it fails though, there is no semantics or meaning. It's basic search and that similarity score is pretty rudimentary. So there's no relevance score really. It's just how many of the character fragments match the query. It also produces pretty large indexes because you can see for each piece of content there's a lot of duplication of character fragments. So your database size can bloat quite large but it is pretty fast. I ran this wildcard search without the trigram index and it took around 2 seconds. I ran it with the trigram index. It brought it down to around 100 milliseconds which is perfectly fine for an AI agent.
So examples of this type of pattern-based retrieval using n-grams or trigram indexes, um, wildcard search, ILIKE search, regex, edit distance which is the Levenshtein algorithm which is ideal for fuzzy matching as well. As I mentioned in the intro, I baked in this pattern matching into our new hybrid search function on Superbase. So what this looks like, similar to the Pinecone example I showed you, we're calling a sub-workflow which is hybrid search. And here we're generating a dense embedding for the query. And we're sending that dense embedding into the hybrid search function along with the query text itself as well as the weights for dense vectors, for sparse keywords, and for pattern matching or ILIKE. So you can see that these are dynamically populated based off the workflow parameters and it's the AI agent that actually populates these. But all of this hits the Superbase dynamic hybrid search edge function. And similar to the sparse edge function, it's not doing anything other than passing the payload into our database function, which is our hybrid search with weights database function. This is where the magic happens really.
And there's quite a long database function here. There's a number of arguments where you can change the weights of the various retrieval mechanisms. But essentially, we're declaring all the variables for the function. We have our filter logic for metadata filters which is important if we wanted to create a subset of vectors before carrying out search. And then we have our weighted queries. So for vector search it's going to the vector column, the embedding column. For keyword search it's going to the FTS column using TS rank. And for the ILIKE pattern match search it's going to the content field and it's passing in the query surrounded by percentage marks for a wildcard search. And then it uses reciprocal rank fusion to fuse the result sets together based off the weightings that have been passed in by the agent to provide the output chunks. So there was a bit of back and forth with Claude to generate that. Um, but it works pretty well. And as you can see if I ask the question, "How do I turn on the ice maker?" We are getting a decent answer back again. It's based off a number of different models though.
So then back to our original question that we asked, "Tell me about the product code." Click on enter. That goes to the hybrid search system. It's passed the code along with a heavy weight for pattern matching. And there we have a full breakdown of that product based off what's in the actual chunks. So the natural follow-on then from a system like this is to add an additional tool to actually load up that full document or summary of that document and that's going to be a topic for my next video.
If you'd like to get access to these hybrid search workflows along with our state-of-the-art RAG agent and ingestion pipelines, then check out the link in the description to our community, the AI Automators. Don't forget to give the video a like below and subscribe to our channel for more content like this. Thanks so much for watching and I'll see you in the next.