Transcription
Friends, hello everyone. When we talk about using neural networks for our businesses, for our projects, we always assume that knowledge is limited in terms of understanding our own context. When we talk about limited knowledge, we quickly move to the topic of RAG, the topic of loading our own files, our own knowledge into neural networks using various methods. Sometimes we talk about fine-tuning models, but generally, we want to provide our own knowledge to the neural network so that it uses it as long-term memory. And when we talk about transferring such knowledge, we most often deal with files, PDFs, HTMLs, DOCX, or Word files. But the problem is that these files, in nine cases out of ten, are not structured. They do not have a clear, understandable structure that neural networks will understand well. Yes, of course, we can always use some external parsers. For example, we have often discussed Mir CR with you, or simply extracted texts using built-in NCMN tools or other agent builders. But we always had the question: can we process our files locally, which most often means securely, without our data leaking somewhere outside? And what, of course, is always important, is either cheap or completely free. Yes, and with such a level of quality that complex tables, PDFs with embedded charts, turn into beautiful markdown that is easily readable by neural networks. And moreover, that internal images and graphics in these PDFs retain their positions and are even described on the fly. Yes, this can be done. And I will even say more. There are tools that allow you to handle complex cases when images are difficult to embed in our files, and they can even understand cases when our tables are split across different pages; they can maintain the overall context and understand that it is a single table, and perceive it as a single object. This is where an open-source project like Docling comes into play. It is an open-source project that can parse a huge number of formats: PDFs, Word files, even HTML images, and also output parsed results in a huge number of formats. Well, what do you say? Interesting. Watch until the end, like, subscribe, and leave some comment. Thank you. If you read Docling's documentation, you can find information that it has a very interesting architecture. Firstly, it is a set of parsers. Each parser is used depending on the type of document you give it. Secondly, it is the definition of the structure of the parsed content. That is, it identifies objects within the document and tries to combine them into unified logical blocks. And moreover, these logical blocks are then used for vectorizing your document. So that your chunks for vectorization are formed strictly in accordance with the document's structure, and you don't have partial chunks that then end up in your vector database and degrade the quality of vector search. This project has been talked about for several months. It has a huge number of stars on GitHub, so you can find many ways to use Docling for your projects online. The documentation is excellent, and we can see that to parse our first document, it is enough to initialize a converter, give it the document, and specify some output format that we need. In this case, Markdown. Let's try to do this simple example right now. Well, and the sweetest part. We will need to write a little code, but don't worry too much, because the code will be simple, and we will slowly go through each part and discuss the nuances of this code. Let's find some implemented example online. Here, probably, is a repository known to all of you. And here are several examples of ready-made Python code where exactly what we just discussed happens. Here, the converter is initialized, the file is passed to this converter, and the result is exported in markdown format. Great, this suits us. Let's take this code right here, go to our cursor, and let's do this. We have a completely empty folder in the cursor, and we need to set up our project. Let's make the laziest option possible. Let's write it like this: "Set up a new Python project for me that uses Docling, and I'll give you an example of another working project. Make one similar for me." And here we paste the example we just found. Start execution. Look how well the cursor works. It immediately creates a to-do list for itself, and based on the code we gave it, it sees that documents should be taken from an input folder and results should be placed in an output folder. And look at the structure. There is a documents folder, there is an output folder, and there is even an already created script that is very similar to the one we gave it. Let's take a look. Yes, the path to some file is specified here. Then the results of the work are output to the console. Then there is the initialization of the converter. And, in principle, saving the results in Markdown. Well, let's try to execute this script. Friends, if you have never set up Python projects before, but are doing it with a cursor, don't worry. When you try to run it, you will likely encounter errors the first time. Something will be not installed, something will need to be initialized. The cursor will guide you through all the basic things, all the basic first steps you will need to take to run this project. Just follow the cursor's recommendations. And for such tasks, the cursor works very well, so you will definitely launch your project. By the way, all these links we are using now, I will put them in my free Telegram group. Join it, take them, because here are the source codes and links to documentation that will be useful to read. So, and now we open the terminal and type this command here. This is the launch of our script in advance. Look, I put a PDF file in the documents folder, because the script will take it from there. And I specified the path to this file here. Let's open this file, how it looks. There are only a few pages here, but we see that quite complex tables are present here, and even a chart. Well, let's try to run our script. We run it. We watch how it executes. All the current steps we are going through are displayed in the console. Great. We are waiting for something in the folder. So, look, our Markdown has appeared here. The first few lines. And here in the output, we have our complete markdown. Let's close the terminal. Some random words here, but then, look, the application of these corrections. Let's open our original file. It starts with the application of these corrections. It is an abbreviation. And then, look, it has preserved our lists at a good level. And look, even from the first attempt, beautiful tables in Markdown with headers. And, in principle, everything looks quite decent. And note, it is defined where the image was. And it is even visible that it is a chart called "Composition of income received in 2021". Here it is, "Composition of income received in 2021". But there is no image processing here, unfortunately. So, let's go back to the script. I'm not very satisfied that it takes files strictly by their names. Let's try to create a second script, but let's make it so that all files contained in the documents folder are taken automatically, and the results are placed in the Output folder. And I write: "Make it so that instead of a list, it just takes all available documents in the documents folder that have a suitable format." We launch the execution and let's see what it wrote. So, we see that a second script has just been created, called "multiple formats". And we see the correct construction here. It takes all files from the documents folder that match the formats with which, in principle, Docling works, and passes them for processing. Let's try to add another file and see how it works. Here I've added another file. So, next to the original file, there will be some strange file, but it's fine for our testing. And let's go back to our cursor, open the terminal, and now let's run the second script. That's what it's called. Let's run our script and look at the output folder. What will be there? So, look, we have two outputs here. Output file, the original file that we processed before. And look, the second output is from the second file, where we have the text itself, the gibberish text, but it has recognized everything well and preserved all the lists, all the bullet points, and put in the headings. I want to remind you that all this happens 100% locally, only on your computer. But I am, of course, very intrigued by this story. We see that it perfectly identifies the places where there were images. Here we see, yes, that at the very top there is an image, but I'm interested, can it describe these images in such a way that all of this is stored in a single markdown, in a single context? We return to the Docling documentation, go to the "Enrichment features" tab, and see that there is a separate item called "Picture Description". And here it says that if you want to describe images in documents, then all this can be done using visual models. Moreover, it says that models can be local, or they can be connected from outside, i.e., remote vision models. What interests me, in fact, is the ability to describe images using commercial models, because we know that local models can do this at a fairly low level, especially if we lack resources on our servers. Let's try to connect a commercial model from OpenAI to describe our images and insert the description directly back into the document to preserve the context of this document along with the original text. I take this code and return to the cursor and write very straightforwardly. I see in the documentation that it can process images from files directly. Add this capability to me. I would like to use a commercial model from OpenAI for this. Here are excerpts from their documentation. And I paste all the documentation I will need below, and launch the process. I see that it works with the correct variables in accordance with the documentation and creates a third script for me. Let's look at the third script itself. Note that the Picture Description API Options are imported here, and the correct pipeline options are set to use external models. And then our API from OpenAI is connected, where we specify the model we need that can analyze images, and set the correct prompt. Describe the charts or images in as much detail as possible, using all available data. I am giving you all the links and prompts so that you can reproduce this process on your computer. Well, and if you need the source code of these scripts, I will put it in my pro-group. By the way, take a look at it too, because we study artificial intelligence, tools, and in general, automation approaches for your business, for your career, very deeply there. It's very interesting there. Well, now let's try to run our third script and see how our file will be processed. The file remains the same, but I am very interested in the first page and the moment where we have the chart. We know how critically important it is to vectorize charts and their semantic meaning for RAG to work correctly. We go and run the script. We watch how the script is executed. And when it finishes, I will get back to you. So, the script has just finished executing. Let's go to our markdown and look for the first page specifically. Look, it has been determined where the image is located. And then it says: "The presented chart shows a pie chart showing the distribution of various categories of income or fees." And, for example, we see "Madrid system duties" with a volume of 82.4, which is 17.3%. Let's open it and see "Madrid system duties 82.4 17.3%". Actually, I want to say that this is, in principle, mind-blowing, if we consider the condition that most of these operations happen locally on our computer, and only the descriptive part of the graphics is done using commercial neural networks. And this can also be done using local neural networks, including. We will conclude the video here, but at the end, I want to tell you that this same Doclink library is integrated with the most popular RAG frameworks, for example, Langchain and LlamaIndex. And there is also chunking serialization. For example, there is a very promising hybrid chunking that breaks your document into chunks by logical blocks and vectorizes this data block by block and places it in a vector database. This helps maintain a very high level of vector search and retrieval of chunks. When you use a vector database, this topic is for a separate video. If this is interesting, be sure to write in the comments. We can develop this topic further, study how open-source libraries work, in particular Docling, because there are many more possibilities here than covered in this video. Again, look for all direct links in our free pro-group. If you need the source code for this automation, then get it in the pro-group. And that's all. Have a good day. M.