Transcription
In this video, I show you how to parse complex documents with five different tools to create usable data for AI agents. To reflect real world complexity, I chose documents like NVIDIA's financial report, medical records with MRI scan, and research papers. For the tools, Andrew Ng, agentic extraction, Docling, Llama Parse, and more. I also built a RAG with ChromaDB and Langchain to compare the actual quality of extraction. I'm Maryam Miradi, PhD with twenty plus years in AI, teaching thousands to build AI agents in ninety plus countries.
Files files. So the first one is the hundred thirty pages NVIDIA document, the finance document. So for our code, we just need a bunch of API keys, the llama cloud API, and also the Vision Agent API, which is for Andrew Ng agentic extraction. And then we can go ahead and load the page just before extracting that. Let me show you.
This is the NVIDIA one hundred and thirty pages PDF. It has a table of content. It has complicated graphs and it has lots of tables, as you can see, financial tables. But also in the first page, things like logo can be quite difficult for extraction. So why are we interested in these PDFs? Because we see very often that the agent confederatory tells you, "Hey, upper revenue is forty seven billion," when the actual number is actually ninety four billion. So it's swapped two rows silently, and that's in sixty eight percent of extraction. Error in financial documents is hallucinated numerical values. So if we look at the data in enterprise, about ten percent is structured and ninety percent is the so-called dark data. They are in PDFs and scanned documents. So in other words, we have sixty eight percent, at least here, problems. So that's why we want a better extraction for such a big PDF in our…
Back to our code, let's take a look at all of our extracts. They are all here in the extract.py. Five methods from Landing AI, from Andrew Ng, DPT, Docling, Llama Parse, unstructured IO, and PDFPlumber. Parse, unstructured IO, and PDFPlumber. The first one is the DPT of Andrew Ng, pre-trained transformer. The first one is the DPT of Andrew Ng, pre-trained transformer. The only thing that you need to do is just… import Parse… and then parse the The only thing that you need to do is just… import Parse… and then parse the The only thing that you need to do is just… import Parse… and then parse the The only thing that you need to do is just… import Parse… and then parse the PDF path… and get the result in markdown. PDF path… and get the result in markdown. PDF path… and get the result in markdown.
The next one is Docling, which is also extremely famous from IBM. The next one is Docling, which is also extremely famous from IBM. The way it works for Docling is document converter, then you convert it, and then you export it to Markdown. Just a little bit of pass, a bit of formatting. Next one is Llama Parse. We get Llama, we say our result type is markdown, load the data, and get the result in text. Then, unstructured IO. Again, the PDF path, we get the partition PDF from Unstructured, partition PDF, use it, find the elements, and then join all of the elements. Last one is PDF Plumber. This is like the classical read page by page, appended, extract table, and then put everything together at the end by a text port. Then we've got extract all of that in a dictionary and loop over it and extract all of that. And then at the end, we can print all of the extracted methods.
So, the most popular one is actually the most recent one, the Andrew Ng's DPT Document Pre-trained Transformer. The way it works is the model breaks tables into structures first, so cells and the position of that, then extracts data from isolated sections and can work in parallel. And as you guys have seen, it's just three lines of code. And they care about the structure first, so you will just do that multi-level header. They would look at the hierarchy, then they will decompose everything, break it into small pieces, as I said, and create that parallel extraction. And most importantly, it has also visual grounding because of the bounding boxes around each of the pixels, and they just say that's the place they will get the attention. So let's take a look at the extraction and see if we can see this element back or…
So if we look at DBT, the first fifteen hundred characters, the first thing that we notice is it has identified the logo. And further in the extraction, you will see that none of the methods is identifying the logo of Nvidia. If you remember on the page, we had the NVIDIA logo. When we are running the extracted, the next method is Docling, also really famous. We barely have a project this famous. So in about one year, forty thousand plus stars. What IBM says is that Docling can do parsing of multiple documents, any document you can see, advanced PDFs, unified representation of formats, and a list of all of the advantages. So, let's see how is the extraction going at our side.
And as you can see, this is just a little bit less character. By the way, I made one module of my paid course, AI Agents Mastery, free. By the way, I made one module of my paid course, AI Agents Mastery, free. It's thirty minutes, zero to hero deep dive, and you will learn how to build agents. And you also. It's thirty minutes, zero to hero deep dive, and you will learn how to build agents. And you also get a free guide of building AI agents. And if you want, you can also join my five-in-one AI Agent Mastery to build real-world AI agents with nine different projects. All of the links are in the description. Mastery to build real-world AI agents with nine different projects. All of the links are in the description. Extraction going at our side. And as you can see, this is just a little bit less character. Extraction going at our side. And as you can see, this is just a little bit less character. Extraction going at our side. And as you can see, this is just a little bit less character. Five hundred ninety five thousand in comparison to DPT. Let's take a look at the output. It looks a little bit more messy. Do you remember we had the table of content… This one, I Little bit more messy. Do you remember we had the table of content… This one, I Said that's gonna be a problem. You can see this is really a problem. This is the table of content for DBT. Said that's gonna be a problem. You can see this is really a problem. This is the table of content for DBT. As you can see, the items are just neatly… This is like a way better table of. As you can see, the items are just neatly… This is like a way better table of content than the one that Docling extracted.
The next method is Llama Parser. And it didn't extract it well. Llama Parser. And it didn't extract it well. It gave a very fast timeout. So you immediately can see it has a little bit of problems with very big PDFs. We can look at the extraction… But unfortunately, it's not complete. Very big PDFs. We can look at the extraction… But unfortunately, it's not complete.
Next one is Unstructured. It just doesn't use so much of separators, it just eliminates them inside. And then this is where it makes a complete soup out of that table of contents. And then this is where it makes a complete soup out of that table of contents.
The last one is PDF Plumber. This is such a lightweight library. It just gives you results in so many situations. It has a decent amount of stars and has been written by Jeremy Singer Vine. Let's take a look at extraction. Table of content. Also less of the separators. As you can see, although it's very lightweight, really beautifully put that table of content… … And also extract all of the links very well. . Just keep this one in your toolkit. . Just keep this one in your toolkit.
So the next thing that you could do is to put this PDF in a RAG and ask all of the questions. So the next thing that you could do is to put this PDF in a RAG and ask all of the questions from it and see if it's working. From it and see if it's working. But before that, let's just try actually to extract other. But before that, let's just try actually to extract other documents that I promised, like the medical lab or MRI. The paper itself is about one hundred forty eight pages, but I took one complex page, which is this one, and I wanted to see how good or bad the extraction is done with this page. We just use the same code, extract with all of the methods. Then I noticed something about DPT, it shows that API is valid, but I don't have any credit anymore. And then I noticed that only by just using this one document, I've used eleven hundred forty nine credits. It's a lot of it. And these are actually the subscription plans. So DBT, although sounds like a very good option, seems to be insanely also expensive. One document, one trial, about ten dollars. For the rest of documents, I just pass DBT.
As you can see, Docling is not extracting anything. LlamaParse, looking at this page full of information, is doing a really great job, and it's just extracting that. So you can see, just giving me that it was an overview of domain expert agent. These complicated one-pagers can be really good for Llama Index. Unstruct, actually Fairlink and PDF Plumber as well. Unstruct, actually Fairlink and PDF Plumber as well.
Then we go to a medical lab report. It's this data that you have different, and I have just downloaded one of them because they are real-world documents. Let's take a look at the extractions. So Docling is extracting a lot of information. It does not showing it in a very good way, I should say. It's just a bit of soup. It's extracting the information, but it doesn't keep logic. Llama Parse just making very beautifully a table. So Llama Parse is really good in this one-pager complicated stuff. Unstructured and PDF Plumber both fail. Then I've got ID card. Both fail. Then I've got ID card. I got it from here, but I'm not quite sure if it's legal to show it. They have all of these test images. This don't show the the ID, but I show you the extraction. So Docling didn't come with anything useful. As you can see, it was a German ID, and you can see that the names and birthday is just extracted. This is public data from Kaggle, but this don't show the ID itself. Unstructured, nothing, and PDF Plumber also nothing.
Then let's go to MRI of a brain also. And PDF Plumber also nothing. Then let's go to MRI of a brain also from Kaggle picked data of this MRI brain. From Kaggle picked data of this MRI brain. And you can see in MRI scan, you have always, like, a small text. I wanted to see how all of these are reacting to an image with barely some text. It's from I wanted to see how all of these are reacting to an image with barely some text. It's from this MRI images from the brain tumor… Let's take a look at the. This MRI images from the brain tumor… Let's take a look at the extractions… Barely anything from Docling… The Llama Parse extractions… Barely anything from Docling… The Llama Parse extractions… Barely anything from Docling… The Llama Parse showing a part of it, Unstructured nothing, and PDF Plumber nothing. So on images they are showing a part of it, Unstructured nothing, and PDF Plumber nothing. So on images they are really not that great on extracting the text. I think when it comes to those kind of images, you better use vision models. Vision models, they have longer latency. They are expensive. It takes longer to just extract information. But at least for now, we don't see that much of information coming back from the PDF extractors.
So imagine that we wanted to create a RAG for each method to ask actually questions. So imagine that we wanted to create a RAG for each method to ask actually questions. You need to know what are the steps of the RAG. I have already explained that in another video in my channel. But very shortly, if you want to build a financial advisor with RAG, you just parse my channel. But very shortly, if you want to build a financial advisor with RAG, you just parse your PDFs and then get those in a text splitter, then you change them to number with embeddings, and then you put them in a database because then you can just do a similarity search. Most of the time you use ChromaDB for a database. And finally, you can just do multi-query retriever. And then you've got the financial advisor. That information can become your audio, you can use ElevenLabs and create this very interactive advisor. Audio, you can use ElevenLabs and create this very interactive advisor.
So let me just show you how to implement what I just explained with Langchain and ChromaDB. So here are all of the libraries. We've got a Chroma, the character splitter. We'll start ChromaDB. So here are all of the libraries. We've got a Chroma, the character splitter. We'll start with creating a vector store. Got our splitter, give the chunk size, a splitter text, the OpenAI Embedding as Embedding. And then we can here retrieve the information as a retriever from the vector database anytime we ask a question. And this is a prompt, the chat prompt template from the template. And finally, we use a chain from the Langchain, from the template. And finally, we use a chain from the Langchain, which does prompt, LLM, and the STR output parser. Which does prompt, LLM, and the STR output parser. And this is the way actually we built a RAG. And then we can also compare the RAG performance for different vector databases. We can add that. Different vector databases. We can add that. And we can also score the result by showing, for example, the ground truth, meaning an And we can also score the result by showing, for example, the ground truth, meaning an example and the answer and then just compare it. Example and the answer and then just compare it. And then we have displaying a result in the table. So it's pretty straightforward. So we can build the RAG, then I start asking questions. I started with a number of questions. A lot of them getting wrong answers, then I started with just one question. A lot of "not found". Except for the PDF Plumber, it came with the exact numbers that I wanted. And this is where it's not anymore about your extraction, but also your retrieval plus your reasoning. So be very careful when you're extracting, then you put it inside a RAG because you split. So be very careful when you're extracting, then you put it inside a RAG because you split it, and any split can just also cause that disjoint of information. And if you just make the chunks too big, it's too much of information. Too little, it's just too fragmented. And I tried different embeddings and different size of chunking. I made chunk size a bit bigger and the number of k higher, and then DBT started to win. When the other ones still are not giving any information, except the PDF Plumber. Docling, Llama Parse, and Unstructured didn't give any good information. Even I improved my embedding and chunking, and PDF Plumber in general stayed a stable thing, and DPT is an expensive plumber in general stayed a stable thing, and DPT is an expensive alternative. And when it came to one-page information, then it was Llama Parse and alternative. And when it came to one-page information, then it was Llama Parse and Docling that gave a lot of good results. Docling that gave a lot of good results. Docling that gave a lot of good results. In none of my tests, Unstructured came out as something special. Then I thought, let me just improve a little bit also on Llama Parse. So just improved the number of. I made verbose is true because it gives it gives a timeout. I made verbose is true because it gives it gives a timeout. So then the problem was resolved for Llama Parse. But then I had to, but then I noticed that I had to use another embedding which was not working very well for DBT or PDF Plumber. So it's really important to work with your use case. There is So it's really important to work with your use case. There is no such thing that Docling is amazing, DBT is amazing. None of them are just the magic box. Each of them are good with a set of parameters. You have to really play around with those parameters. So just out of the box, I notice none of them is working. As an AI engineer, data scientist, AI professional, need to still set the parameters for your problem. Otherwise, you would not get any good result. So that's the comparison and the way I noticed, especially when you are working combined with RAG, remember that you do not have only the problem which is extraction, but also retrieval plus reasoning, and those three problems get combined, and then you need to resolve those problems separately. PDFs are just one type of context that you can give to your agents. And we have other types of context like prompts and tools. And that's the context engineering. I have these seven methods of. And that's the context engineering. I have these seven methods of. And that's the context engineering. I have these seven methods of context engineering for production AI agents. Context engineering for production AI agents. And you can watch that video, and I see you there… And you can watch that video, and I see you there… And you can watch that video, and I see you there…