Transcription
So far, we talked about points, what they are made of, and how Quadrant compares them using distance metrics like cosine similarity, Euclidean distance, and dot product for the approximate nearest neighbor search. But of course, none of this matters until we give Quadrant something meaningful to compare. And this brings us to the beginning of our pipeline.
Our pipeline starts with the data and how we want to represent the data that we want to search. In practice, this means that we need to start thinking about the structure of what is going to be stored and how we are going to store it. Most real-world data is messy, and documents are long, and descriptions are unstructured. We need a way to break down this data into manageable and semantically relevant chunks for our vector database.
Storing an entire document as a single vector might seem easier, but it's much less precise. Imagine storing the entire Quadrant collection configuration guide into a single embedding. That article includes a lot of things, such as HNSW tuning, and quantization, and charting, and so on, concepts that we're going to see later on. But if a user asks, for example, "What does the MP parameter do?", then the retriever will give back the entire document instead of the specific question that the user is asking. Of course, the answer is there, but it's buried on top of a lot of unrelated context. And if you use a large language model to process it, you're going to end up wasting a lot of tokens on unrelated stuff to give back to the user.
By splitting the guide into smaller chunks, Quadrant can retrieve the specific section that talks about the M parameter and give back to the language model or retrieve back to the user only that section. And this is why chunking makes all the difference. Instead of treating the document as this whole entire monolithic block, it breaks things up and makes processing much easier. Paragraphs, headings, subsections, you embed them all separately, and you now have a single embedding for each one that can capture more context about what that is referring to. And you can also add metadata for each chunk. So, for example, things like section title, or number of the page, or the source URL from where that data came from, and the entire text itself. You can store all that along with the chunk embedding.
But of course, the shape of your chunk matters. It's going to define what your embeddings can capture, what's going to be retrieved, and what your language model can reason over. Let's take a look at a few strategies that we can use to split our documents into chunks and the trade-offs of each one of them.
One of the most common methods that you can use is fixed-size chunking. You basically define a number of tokens you want to have per chunk, for example, 200, and maybe a little bit of an overlap so you don't lose too much context. This is usually good enough when your documents don't have a consistent enough formatting, but you might split things mid-sentence or mid-thought, which brings us to the next method.
Sentence-based chunking breaks the documents into sentences using a tokenizer, which is trained to recognize punctuation patterns and sentence structure. Then you group these sentences into a chunk that stays under a specific word limit, let's say 150 words per chunk here. Each chunk is more likely to capture a full thought. But because sentence sizes vary, you're also going to end up with irregular lengths for your chunks. And this is the trade-off of this method.
The next method here is paragraph-based chunking. Paragraph-based chunking leads into a structure that usually already exists, especially in articles, or books, or blogs, or even emails. Since most writers usually write paragraphs around topics, if we break our text into paragraphs, then we usually get more semantically meaningful chunks by default. But paragraphs are unpredictable. One can be a single line, and another one can be an entire page. So, in those cases, you might need to layer in token limits or fallback splitting. Still, when the structure is clean and predictable, this method works extremely well.
Another method that you can choose is sliding window chunking. And this method solves a very common problem in fixed-length chunking, which is context lost at the boundaries. Instead of splitting without overlap, you define a window size and a stride. Each chunk overlaps with the previous one by a defined number of tokens. And this helps maintain continuity between chunks. Sliding window is especially useful when you are more concerned on not missing anything rather than optimizing for speed of storage. You usually get higher recall and more chances to match with queries that have a more subtle context. The cost here is redundancy because it's going to take more storage and compute to process those embeddings. But sometimes it's worth the trade-off.
So far, all methods that we talked about assume that data follows a specific structure, either words, sentences, or paragraphs. But some data just doesn't come that way. Maybe you scraped some data from the web, or maybe it's some messy data from a CMS that you need to work with. In those cases, the structure isn't there or is not consistent enough for you to trust. And this becomes a problem because most LLM pipelines require text chunks under a specific token limit.
Recursive splitting uses a fallback hierarchy of separators. It tries to split on large blocks first, but if that's too long, then it falls back to shorter ones like sentences or lines. And then, as a last resort, it goes to words or even characters. And each level of recursion preserves as much semantic coherence as possible. This kind of logic is already built into tools like LangChain or LlamaIndex, which both include recursive splitters. But you can also try to build something similar yourself using just Python and basic text parsing.
Everything up to this point was about structure. But structure is not the same thing as meaning. Just because two sentences or paragraphs appear next to each other doesn't mean they are relating to similar things. Semantic chunking uses embeddings to find meaning shifts. You detect where topics or semantic coherence change, and you break there. The trade-off here is, of course, cost, because you're embedding the full document upfront just to decide where to split it. This is the slower and heavier method for you to choose, but it is also the most powerful because it is the one that's most likely to guarantee that each chunk will come with a coherent idea. This is also available in LlamaIndex with the semantic splitter node parser and also in LangChain with the semantic chunker. You won't always need something as powerful as semantic chunking, but it is very useful when you're looking for something that must be as accurate as possible and you don't care too much about performance or costs.
But chunks themselves are just fragments of text. They don't tell you much about where they came from, what their topic, main topic is about, or how you can best retrieve that part of the text. This is where we want to start thinking about metadata and which types of metadata do we want to add to our chunks.
As we saw before in Quadrant, your metadata for your points live in the payloads. You can use it to store anything that you need to organize or identify your chunks. For example, source document titles, or URL from the page where it belongs, or tags, or even the number of the chunk. Those are all things that are helpful for you to store within your chunk, as well as the text that is contained within that chunk itself. That means that for your entire dataset, if you want to filter for things like article name, or if you want to filter by tags, that will be something that you can do, and it will make your life much easier as you try to find more specific data within your vector database.
If you index thousands of chunks from different documents, but you only want to retrieve the top candidates from each document, then you can tell Quadrant to group by document ID. And each group will give you the most relevant results within that scope. And of course, when Quadrant retrieves the points, the payload comes with them. It means you can display the actual full original content of that chunk, the section it came from, or the link to the original source without actually having to do any more additional lookup after the search.
Remember, Quadrant does not make any assumptions about what your data looks like, or even its structure, or anything like that. It basically, at the most fundamental level, compares vectors and gives back the closest ones. But the overall semantics, context, structure of how you want that to be delivered to you, that's you're going to have to be the one to decide and structure the way that makes the most sense for the performance that you want to achieve.
In the next video, we're going to build a simple movie recommendation system with the ideas that we just covered. We're going to take a dataset of films, embed their descriptions, chunking when we need to, and adding payloads that are going to facilitate our filtering. We're also going to set it all up into Quadrant and run real queries to see how it performs. So, let's do it.