Transcription
In this video, I have a super quick tutorial showing you how to create a fully local chatbot. It uses Llama OCR, multimodal RAG, and a local LLM to make a powerful agent chatbot for your business or personal use.
I was scrolling through Twitter and came across an interesting project called Llama OCR. It's an open-source optical character recognition tool powered by the Llama 3.2 Vision model. This tool is designed to convert images of documents into markdown format, making it especially useful for developers and tech enthusiasts who often work with complex document layouts such as tables, receipts, or mixed-format files.
In a standard retrieval augmented generation (RAG) setup, input documents usually consist of plain text data. Llama OCR steps up by enabling seamless interaction with visual data. It allows the LLM to leverage in-context learning, retrieving chunks of relevant text from documents that match the context of your query. What should you do if the document contains pictures, tables, charts, etc., in addition to text data? Each format has its own structure and challenges, but the real difficulty comes from the sheer variety within these formats. For example, a PDF can be single-column or multi-column, can contain tables or charts, and can have headers, footers, images, or diagrams. This wide range of possibilities makes it impractical to create a one-size-fits-all solution. So let me give you a quick demo of a live chatbot to show you what I mean. I asked the chatbot a question: "What is approximately 85% of FY22 revenue in Canada?" The chatbot uses multimodal RAG to interact with PDFs and generate an output. This process combines text, visuals, tables, and charts for a comprehensive response. We also integrated Kaolin, a cutting-edge multimodal retrieval system, to seamlessly retrieve images instead of relying on traditional OCR or image captioning. Kaolin directly encodes image patches, simplifying text extraction from PDFs for embedding and retrieving images. We use Koin-Q2, which enhances the efficiency of this workflow.
Guys, if you're looking for advanced OCR combined with a Llama Vision chatbot, I've got you covered. I've documented my entire learning journey and shared the code on Patreon. Let me guide you through how this chatbot works. To add images or PDFs to the system, simply go to the "+ Add to Index" section, upload your files, and the system will take care of the rest. It automatically generates Koin-Q embeddings, checks for duplicates, and organizes everything in SQLite for seamless access. When you're ready to search, head over to the "Query Index" section. Just type in a natural language query, and the system will retrieve similar images while also giving you an in-depth Llama Vision analysis. Managing and exploring your visual data in this step-by-step guide, we will cover why OCR is still struggling to retrieve complex information, what Kaolin is, how Kaolin works, and how to implement all these techniques together. Definitely stay tuned throughout the end of this video.
If you guys haven't followed me, I highly recommend that you do so, so you can stay up to date with the latest AI news. Lastly, make sure you guys subscribe, turn the notification bell on, like this video, and check out previous videos because there is a lot of content that you will definitely benefit from. So that thought, let's get right back into the video.
Standard large language models (LLMs) ignore this additional information. RAG systems must rely on OCR tools to extract information from tables, images, etc. Although OCR technology has greatly improved recently, it is commonly used to extract text from scanned images. However, it still produces errors, especially in cases of poor scan quality and also struggles when dealing with complex layouts such as multi-column PDFs or mixed documents containing text and images. This will result in irrelevant or incorrectly informed text blocks being indexed, which will negatively affect the quality of the large language model synthesized answer when these text blocks are retrieved and added to the context of the large language model. We will use the multimodal large language model to infer information from complex PDF documents. Kaolin is a model with a novel model architecture and training strategy based on a visual language model that efficiently indexes documents based on their visual features. It is an extension of Llama 3B that generates ColBERT-style multi-vector representations of text and images. It is introduced in the paper called "Efficient Document Retrieval with Visual Language Models." Koin-Q2 offers a novel approach to dealing with complex document formats. Koin-Q2 directly converts screenshots of PDF pages, including images, charts, and tables, into vector representations for retrieval and sorting without the need for OCR, layout analysis, or any other complex pre-processing steps and without the need for text segmentation. All that is required is a screenshot image of the page. The Kaolin is inspired by ColBERT, where text is represented as multiple vectors instead of a single vector representation. Koin-Q is derived from Pema, a powerful visual language model. Koin-Q is based on two observations: multi-vector representation and late interaction scoring can improve retrieval performance, and visual language models excel in understanding visual content. Understanding documents with rich layouts and multimodal components has always been an important and practical task. Recent large vision language models have achieved remarkable progress in various tasks, especially in single-page document understanding. Llama now officially supports the Llama 3.2 Vision model. You can recognize the picture by dragging it in like this. You can see that the model has an 11B parameter version and a 90B parameter version. When choosing the 90B parameter version, the file size is about 55 GB. Of course, there are also some quantized versions. Llama 3.2 Vision 11B requires at least 8 GB VRAM, while the 90B model requires at least 64 GB VRAM. The biggest update is the support for the Llama 3.2 Vision visual model. The version of the Llama has also been v0.3.14 and has been directly upgraded from v0.4.0. After upgrading, run the following command to experience. Let's start coding.
I plan to use multimodal RAG to interact with last year's Accenture investor slide deck. The deck spans 17 pages and includes a mix of text, visuals, tables, charts, and annotations. Each page has a unique structure and template, making it challenging to process using traditional RAG methods. Multimodal RAG is perfect for handling such complex documents as it combines various data types seamlessly, ensuring accurate and meaningful interactions. We will use Baldi, a library from Answer AI, that makes it easier to work with an upgraded version of Kaolin called Koin-Q2 to embed and retrieve images of our PDF documents. And we use pdf2image to convert PDF files into PIL objects and with Popper you can read, modify, and change PDF files. We also need to install popular utils, an essential package for manipulating PDF files and converting them to other formats. We load Kaolin from Baldi using RAG multimodal. Then we retrieve from the 17-page Accenture investor presentation and use the `mv` command to rename it to `Accenture_presentation.pdf`. We use the Koin-Q2 model to index the content of the `Accenture_presentation.pdf` file. It assigns the index the name `atenta_index`, stores both vector representations and base64 images for retrieval, and allows overwriting any existing index with the same name. Let's query an index presentation using the question: "What is approximately 85% of FY22 revenue in Canada?" The top five most similar results are retrieved using the model's `do_search` method, which ranks pages based on similarity scores. The results are printed with document IDs, page numbers, and similarity scores for review. Finally, a message confirms that the search process was successful. Since we stored the collection along with the index, we also have the base64 encoded images of all PDF pages. This allows the system to retrieve the base64 image of the top result, enabling the matching page to be displayed visually. The retrieved page or its base64 encoded version is then passed as `returned_page` for further use. We use the Llama 3.2 Vision model, which simplifies the process by allowing us to pass in an image and either extract text or ask questions about the image. The Llama 3.2 Vision model is a 9 billion parameter model that performs well on most OCR tasks. I run the Llama 3.2 Vision model locally on my MacBook using AI. To use the Llama 3.3 Vision model, you need AI 0.4.0, which is currently available as a pre-release. Kaolin is a major advancement in the field of multimodal document retrieval, combining the strengths of VM with innovative architectural choices. It efficiently processes and retrieves information from complex documents, positioning itself as a valuable tool in evolving AI-driven data analysis and retrieval systems. Llama OCR is an excellent assistant for developers and content creators. With the help of advanced AI models, it easily meets the OCR processing needs of complex documents. Direct output in markdown format adds even more convenience and efficiency.