Transcription
Friends, hello everyone. Well, how did you celebrate the holidays? Ready to learn new things? Have you ever noticed that your agent doesn't find data very well at all? Well, that is, you ask for something specific, and the chunks with the information you need either end up in the last places of the selection, or don't end up there at all. Well, for example, you asked something: "Tell me about the limitation of liability in the document from paragraph 12:3". If there are many places in this document that talk about limitation of liability, then paragraph 123 might be completely overlooked or have low priority. This happens because vector search works based on similarity search. And in this query, it's more concerned with the phrase "limitation of liability" than with the specific paragraph. A reasonable question: is it possible to build a Sherlock Holmes who understands exact queries literally to the last word, and also understands general semantic questions? Well, of course, it's possible to build all of this. And all of this is solved with the help of, as you probably already know, hybrid search. And today, by the end of this video, we will build an agent that works based on hybrid search and finds your data with incredible accuracy. But I have three basic conditions for you. First, a like, then a subscription down there, and definitely some comment, so that as many people as possible see our conversation. Let's start building. The coolest thing about this situation is that, in principle, we won't need to learn any new technologies, because, for example, hybrid search is available in Superbase, and we use Superbase constantly. And as it says here, we combine keyword search and semantic search or vector search in one function. And moreover, there is a detailed instruction here. Here, of course, I assume that you already have Supase installed. If not installed, then we have done it a thousand times. Watch this video, there is a complete detailed instruction. Set up your Super Suab Base and come back here. And all of this is built on a concept called Reciprocal Ranked Fusion or combining inverse ranks. We won't delve deeply into this, but generally speaking, the system will search for similarity ratings for your query, both by keywords and by semantic search. The final rank will be built on the basis of combining the results of these two searches, which increases the accuracy of your search and takes it somewhere to the heavens. And the sweetest part is that we will need literally a few movements. Create a table, create indexes for it, and the search function itself. This is what we usually do when we create tables for vector search. Remember? Let's not drag it out. Let's go straight to Superbase. I'm opening my local Base. Here I'm looking for the SQL Editor. I'm returning to the Supabase documentation. And they tell us: "Create a new table documents". I'm taking this SQL code. I'm pasting it. Let's create a table named Documents Six. Just because I already have many similar tables. And here, pay attention, the vector size is 512. If you are using embeddings from Open AI, then you need to specify 1,536 here. And execute the query. Done. We get an error that the vector extensions type does not exist. We simply remove the word extensions here, leaving only vector, and click run again. Everything, our table is created. Let's go straight to the table Editor, let's check. We see that here is our table, which has four important fields right away. ID, the content itself. Now there is an FTS field of type TSvector and our usual embeddings. It seems that everything is going according to plan so far, but right away, so as not to get up twice, let me click edit table, because we forgot to add metadata, and we most often use metadata. Here we add a column, we call it metadata. And its type must be Jason B. Click save. We check that our column has appeared. Great, this is what we needed. Here we return to the SQL Editor, then to the documentation, and we see that now we need to create two indexes for this table. I'm taking the SQL, pasting it here. Our table is called document 6. Here we pass, pass, click run. Everything, success. How do we check this? We go here to database, then look for indexes, and we see our indexes for the document 6 table. Wonderful. Let's go back to the documentation. And now they tell us: "Create a hybrid search function". Let's take this function, return to our Supase, open the SQL Editor again, paste everything. And let's examine it carefully, carefully. First, I will name it hybrid search six, so as not to get confused later. Vector dimension 1,536. We remove the word extensions, we won't need it. And everywhere I see the word documents, I change it to domain six. Here and here and here. Click execute. Great. And we have created a hybrid search function called Hybrid Search 6. Returning to the Superbase documentation. And then it is explained how it can be used. It says, if you want, just execute an SQL query. Well, it's better to create a function yourself. That is, this function is written in JavaScript or TypeScript. And essentially, what will it do? It will first turn our query into an embedding, then implement the search function, both semantic and keyword-based, and will return us the results. But this, to be honest, is a bit of a hassle. I suggest a cooler option. Let's go straight to N C. So, here is my NV Cman. I click create workflow here. I'll add a chat right away, because I will be communicating with my hybrid search through the chat. And let's look again at the hybrid search in the Superbase documentation. I can clearly see that there are two key things here. First, we can pass the direct search string, by which keyword search will be performed. And then I see that we can pass a vector, that is, the vector of our query. And we pass this combined story to our hybrid search function. And Supabase does everything for us. Well, rather, the function itself. But can we do all of this manually? Yes, of course, we can. We can do almost anything. Let's go to NVC man. I want the user's query directly. Here I enter the word hello. I want to vectorize this hello directly, and then substitute it into our SQL query, which will go to Suay, as if it sounds logical. How can we vectorize something manually? We know that almost every neural network provider has its own neural networks for embeddings, for vectorization. Let's try to search directly for Open AI. Let's look inside to see what's there. There is communication with the model, there is image analysis, there is audio generation, files. But, to be honest, I don't see any direct call to embedding models here, but this shouldn't stop us, because there is always a universal node that has saved us a thousand times, called HTTP request. I'm adding this node, and we are now going to send requests for vectorization completely manually. This will be a POST request, of course. Where are we sending? I just looked that we need to send API openi.com embedings. We need authentication, so we will use such a magical thing as predefined credentials. I choose this type. If you have already used credentials from any services in other nodes, then you can always access them through this functionality. Here I found Open AI, and it says: "These credentials are already contained in your NV CM: "Do you want me to use them?" Yes, I want." Because essentially it will substitute these credentials into the request header, and I don't need to specify anything manually here. But I want to send a single body. Here I choose Jon. And again, I just looked in the documentation, the request for vectorization is sent very simply. We specify the model, we need text beding 3 small, we use it constantly. And then the text itself, the string itself, which we will vectorize. And we have it here in the chat input. I just took it and transferred it here. And essentially, nothing stops us from vectorizing our word hello. Let's execute this step. Oh, wow. Open Ai returned embeddings for our word hello. That is, this is a vector for our phrase. Do you feel it now, how the magic unfolds? What N8 CNN usually hides under its hood within the nodes, we are now doing it manually. Essentially, the same thing happens when you simply add the SUP Base Vector Store node and put chunks into it. The same thing happens. Each chunk goes through the vectorization process, and we get our vectors. Now let's return to the Subbas BASE documentation. And it says to communicate with our hybrid search like this. So let's take this SNIP. Let's return to VMAN. And how can we send SQL queries to our Superabase? Well, in fact, to Postgress, which is the basis of Superbase. So let's look for the Postgress node, add it. And here there is a function to execute an SQL query. Click on it. There is a field for the SQL query. I'm pasting it, but let's look at it more closely now. I will change the credentials to my local ones. So, the function itself is called Hybrid Search 6 in our case. We named it that ourselves. Then the query itself. Why do we need Italian sauce? We don't need it. But the query is initially in our chat input. So I'm taking it all, putting it in. And then it says: "Give me the vectors". And we just generated the vectors, so let's remove these brackets. We see that our vectors are in the beding field. Let's put them here. Let's look at the bottom, how it all renders. Look, our vector is here in the array field. This is definitely not suitable for us, because Postgress will spit it out, because it only needs a vector. It doesn't need any word array. Therefore, we use the bulletproof argument of kings. We write jon to stringify here and take our entire variable in brackets. Let's look at how our vector is rendered now. Great. Now the format is top. Well, let's execute. Let's see how it will work. So, the same error. Let's remove the word extensions. And of course, the dimension is 1,536. We forgot to change it. Now let's execute again. There is no data, but this is because there is nothing in the table itself. Let's put something in there. Let's look for the Superabase Vector Store node. Click here to add documents to the Vector Store. Specify the correct table here. Yes, we just created documen. And in the options, we must specify the search function. And it is called Hybrid Search 6. This function is used not only to retrieve data from the vector store but also for correct writing, because there is a lot of magic happening inside. Then the model for embeddings is the same, only now through the official node. And the biggest question is: where do we get the documents from? You get them from where they are stored, and I will get them from my Obsidian. We recently learned how to get files from Obsidian and vectorize them using N8N. Here they are all in a bunch, so I will use my old function that keeps my Obsidian up to date and vectorizes all notes. By the way, a great topic, watch this video. As soon as we add something to Obsidian, all this knowledge will be vectorized and will be available to our agent as our second brain. But that's not the topic of discussion now. I just click execute workflow. It does everything for me. Therefore, we will simply return to Superbase and see what is in our documents table. Pay attention, the beginning is very similar. Here is just the content of my notes, and then in the FTS field there are these beautiful values. They look very strange, but this is the whole point of keyword search. This process is called stemming, that is, essentially, the system simplifies our text, breaks it down into words, and assigns special ranks based on how important that word is in the text. It's a cool process. The system looks at how many times a word appears in the text, its position, closer to the beginning or closer to the end, its density in this text, and how rare it is in general. Then, in the table, there are our usual embeddings, vectors, and metadata, which we added in advance as a column to our table. By the way, friends, if you are worried about this automation, how everything happens here, then at the end of each video I click the download button and download the JSON of this automation and save it to my free Telegram group. You go, open this group, find the video relevant to you, which you like. Under this video, look for the json file. Click to download it, upload it back to your N8N using the same principle, and continue experimenting. And I put all complex automations into my pro group. The link to it will also be in the description. There is a lot about Nan, automations, artificial intelligence. Definitely check it out down below. Well, let's test it. I will write something very simple so that we can check the effectiveness of keyword search. I will write only one word, let it be the word "study". I click Enter, I see how my word is vectorized now. And then our SQL query returns relevant chunks. And look, right by one query, at the very top, in the first place, is a chunk about studying with a personal tutor. And you can see how full-text search works, how the ranks of each word are set. And, generally speaking, this is a relevant entry that I had in mind, which we found right the first time with one word. Now let's try to perform a vector search separately for the same word, without using word search. Here is my old agent, which looks at my same table, but uses not the hybrid search function, but the vector search function. So, I write the word "study", send it here. Let's see how the agent works. It calls our vector store and look at what chunk is the most relevant according to vector search. Instant translation of speech in the ear is the end of language learning as a necessity. And this is the very basic difference that we talked about at the very beginning. When I said the word "study", the vector search tries to organize the search by meaning. And since it found a chunk about learning, that is, by semantic load, it is really about studying, but the chunk itself was irrelevant for my specific case. That is why hybrid search should be implemented, which will combine the results of vector search and keyword search into a single structure and give your agent the combined results. Well, friends, we will finish here. You can get this automation that we just built in the free Telegram group. If you are interested in automation that takes files directly from your Obsidian, vectorizes them, and puts them into your vector database for hybrid search, then get it in our pro group. The link to both resources is in the description. That's all for now. Bye.