Transcription
How to competently and maximally clearly convert various files from various formats, such as PDF, doc, HTML, audio files, and much more, maximally clearly preserving all the structure, all the tables, all the columns, all the paragraphs, all the sections, say, into a Markdown file, in order to work with it more conveniently later, to bring everything into one, so to speak, structure, or to use the given Markdown file subsequently, to build code of that kind, which will be maximally clear. And all this can be done with the help of Docling. This is an open-source library from IBM Research. And today I will show you what it can do and how it works. So stay tuned, it will be very interesting. Look, I have such a research paper with Deep Neural Network for determining grasp points on rigs, that is, on solid objects. In general, there is such a research. It is in PDF format. I want to preserve the entire structure, all these tables, all these formulas, preserving everything completely. The main table for me with these results, to translate all this into Markdown format, so that I can work with it. And in the future, I want to put this Markdown file into a RAG. That is, first I will need to cut it into chunks. With the help of an embedding model, I will convert all this into vector embeddings and put it all into a vector database and subsequently build an agentic RAG. In general, such a research. Let me show you how this can be done with the help of Docling. I will show you how all this can work, and then we will move on to the theory and consider in more detail what Docling is. For this, I have already written a script with the help of Docling, which transforms the file from PDF format, respectively, into a Markdown file. Let's just run it. I will show you what this Docling is capable of at all. This is a very interesting thing with many neural networks that run locally on your hardware. So, the script in my case is called conv_pdf.py. Let's just run it, and I will show you how, accordingly, Docling works and how well it transforms PDF files into Markdown files. Well, in principle, and any other audio, HTML, and all that. It converts excellently. So, we already have a ready Markdown file. And let's look at this Markdown file in more detail. I will say right away, of course, this tool does not work 100% perfectly, but if we compare it with other tools, such as PDF, for example, then in the case if we consider more complex PDF files and other files with tables and the like, the main table interests me very much, I think many too, with the preservation of structure, with paragraphs, sections, then, of course, this Docling works better than its, so to speak, competitors, at least because there are neural networks here that greatly help in this matter. As we can see, everything has been converted quite well into a Markdown file. Everything is clear, everything is there. Of course, not 100%, you should understand that it's not exactly 100% good, but it really works very, very well. Let's just find a table, I'll show you. Bam. Look, a table. Remember this table. We have methods, input, grasp accuracy, speed here, and we can compare it with the table that is, accordingly, in PDF format. Remember the numbers 98 and 263. Below 98.263. And the upper first row is 97.7 and speed, respectively, 9.9. Everything is clear. As we can see, the table structure has been completely preserved. And this is really one of the main features of Docling. When formatting from PDF, from Docx, from HTML, from almost any formats into, in principle, many formats too, but the most common is Markdown, it preserves almost 100% of the entire document structure. This is very cool. As you can see, the document structure here is not that easy. There are tables, various formulas, and the like, paragraphs, sections, it's not so easy to do all this, as you might have understood. The next table. As we can see, everything is clear, everything is cool. There are also dashes here, as if everything here, everything is preserved. As you can see, I am showing you with a clear example how it works. Everything is clear, as if all tables are preserved, and this is very cool. All sections, all paragraphs. Okay, I moved on. I chunked this Markdown file. I made, uh, vector embeddings from the chunks and made my agent RAG, that is, agent RAG. And now I have, accordingly, this, so to speak, file there, and I can ask questions to the model based on it. Look, let's go back here. For example, I am interested in, uh, training losses in this framework. And here we have three losses. The first is, uh, grasp detection loss, then, uh, semantic segmentation loss, and grasp refinement loss. I want to ask, accordingly, the Agent RAG, what losses does this framework have at all, and let's see what it answers us. For this, I have, of course, created my own vector database, chunked everything, Vector Embeddings, and written my agentic. Let's run it and see how well it works in conjunction with Docling. Super. Database connected. Let me remind you, now in this database there is, accordingly, this PDF simply in another format, in vector format. And let's ask a question. So. And here is my question: which losses does this deep neural network have? Which means, what losses does this Deep Neural Network have? Let's press Enter. We wait for it to process. Of course, this is RAG, it's not all that fast, but as we can see, it didn't take long. So, and we have an excellent answer to the question. This deep neural network has, uh, the following losses. First, grasp detection loss, as I said, learning the grasp detection task. Then semantic segmentation loss, cited. In general, they give a description, by the way, a clear description. And grasp loss, and also a clear description. These losses are combined in a composite loss function, as we saw the function, and by specific hyperparameter lambda. As we can see, it is so. Everything is clear. That is, we can already chat with this document using an LLM. It also indicates the source in case we have multiple sources. As you can see, Docling is a super powerful tool. With it, you can do really a lot of very cool things. I use it, in principle, on a permanent basis. I like it very much. In general, if you want to delve deeper into Docling in practice, all this will be in RAG University. The link will be in the description under this video. Be sure to check it out, there will be a lot of practice on Docling. We will also consider RAG and the like there. And now we are moving on to the theoretical part. I will tell you more about Docling. and we will consider how it works in theory, what neural networks are there, and the like. So, let's go. So, Docling. Once again. Docling is an open-source library from IBM Research. I think many of you know this company. And in general, their IBM Research created such an open-source library in order to transform files from various formats into, say, Markdown, while fully understanding the structure. That is, Docling is not just text, but it specifically understands the document structure using, again, neural networks. And the coolest thing is that these neural networks run completely locally on your end. That is, you can work with, so to speak, private data, because this data does not leave the boundaries of your hardware. That is, everything happens locally, nothing is leaked anywhere, everything can work without the internet, and this is super cool. This is a very, very cool feature. Let's consider how Docling works in more detail using a PDF file as an example. Again, as I said earlier, it can work with different formats, even audio formats, which is also super cool. I tested it, it works excellently with them. But let's consider a PDF file specifically. Let's say we have some PDF file, some research, there are some tables, pictures, yes, a lot of everything. So, first. The script simply receives the PDF file. Next, the second stage. This PDF file goes into a neural network, which they call the Layout Model. What does this Layout Model do, which, I emphasize again, runs locally, which is super important, that you can work with private data. And this Layout Model finds tables, it simply finds them. It creates, you know, bounding boxes, finds tables, headings, paragraphs, finds images, and the like. It creates such bounding boxes around, so to speak, certain things. For example, here we have a paragraph bounding box, then we have a table, yes, this Layout Model simply creates bounding boxes around, so to speak, things. After this Layout Model, we have a special model called Table Former. And it already extracts tables. That is, for example, this Layout Model finds any table, designates it with a bounding box. And then the Table Model already extracts this table. extracting data while preserving the structure, which is super important. All this, of course, is available with the help of neural networks. Next, optionally, they also have an OCR model. Let's write a little arrow here, uh, OCR, which, accordingly, also works optionally. It is, in general, mainly used for scanned PDFs, Easy OCR for recognizing text and images in PDF files. Well, this is also very optional, it is not always used, but by the way, they have a good OCR model, which also runs completely locally, which is super cool. Then all this is combined. All this is combined into one, so to speak, file, reading order, all tables, all structure, all paragraphs, metadata is extracted, and so on. In general, all this, so to speak, is formed into one format. A format called Docling Doc is assembled. Yes. Next, what can be done with this Docling Doc? All this can be configured in the script. They have a very cool library, and it's not super difficult to work with, by the way. That is, everything is super clear, in principle. There, figuratively speaking, in a couple of lines, you can actually convert PDF to Markdown. In general, this Docling Doc can then be converted, for example, into a Markdown file, a JSON file too, yes, it can be converted. Also, this Docling Doc can be used for fine-tuning, yes? That is, you have, for example, a lot of PDFs with some important information that will help train your LLM or another neural network. You use Docling to, so to speak, collect all this into one unified format and then use this data for fine-tuning, which is also very cool. Next, of course, you can use RAG, which is what I did. In general, guys, if you were looking for a way to convert various files to Markdown, JSON for fine-tuning, for RAG, I know that many have encountered this problem. In general, your solution is Docling. This is an excellent open-source library. Also, guys, who want to get practical knowledge, look at how these scripts work in practice, how all this works, how all this can be integrated, how all this, accordingly, can be put into a vector database. They have an excellent topic with hybrid chunking. I also talk about it in great detail. Also, those who want to delve into Agentic RAG, get real workflows that seriously work, which you can apply in your work, in your projects immediately. I talk about all this in RAG University, so be sure to check it out. This will be the first link in the description under this video. There we will consider Docling in more detail, specifically in practice. That is, there will be pure practice, which many really lack, it will solve many of your pain points, and you will be able to use Docling in practice in your projects, in your work, in your tasks. So, I thank you for watching. See you in RAG University. M.