Transcription
Hello everyone. And today on our channel, I will tell you about our team's case, Nedra, at the Digital Breakthrough hackathon, which took place in November 2024, where we took a prize-winning place, becoming one of the winners of this hackathon. As always, our case was related to AI and large language models. We were implementing a Q Bot. So, now I will share my screen and tell you about the case we were solving.
So, we were implementing a bot with the function of neural search of information in a knowledge base for a media agency. The composition of our team, which worked tirelessly for 48 hours on the case, consists of all employees of the Digital Research and Development Laboratory of r.ru. The problem of the case was that media agencies have quite a lot of employees and a pool of personnel, but the most important thing is that they have very many projects for different companies. And, in general, it's a similar story with us IT specialists: the constant task of searching for information for new approaches to implementing solutions. Of course, it is important to reuse best practices and fight against "reinventing the wheel," which employees of media agencies often do, instead of using existing beautiful solutions that have already been made by their colleagues. Sometimes they invent everything from scratch, and in this regard, efficiency is lost, and most importantly, time, which is money for a media company. Therefore, for us, the main thing was to create a user-friendly interface that meets the main business scenarios, and we created it. We presented the agency with a pilot solution, AIIC, as we called it, which greeted employees in its web interface. It was very friendly and easily integrated into the corporate environment. It also connected to the file storage and automatically launched the process of indexing and summarizing previews of documents. This helped it work with up-to-date knowledge without accessing the internet. And under the hood, a smart model was used, built on the basis of technology. For those who are encountering this technology for the first time, RAG stands for Retrieval-Augmented Generation, which means large language models. The main essence of this approach is that from a pre-indexed knowledge base of documents, we extract relevant segments of this information. Subsequently, we transfer it to a language model along with the original question and a given system prompt that instructs the model on how to answer based on the found information. In this case, on the slide, for each incoming request, we pre-generate several variations of alternative queries, meaning we used this approach. Another important task was to be able to search for information in both Russian and English, and even if the question was asked in Russian, but the information was found in English, it was necessary to provide the answer in Russian as well. Therefore, at this stage, we also asked the model to generate and translate these queries from one language to another. So, under the hood of the model itself, the search component contained original documents in various formats: PDF and PowerPoint presentations. The training process for the company's search was carried out using a combined approach. That is, we indexed and searched by keywords using BM25, and we also had semantic search. In this case, we used the M3 model. After retrieving the six most relevant ones, we also used the cross-encoder approach and transferred the found document segments to the LLM model. By the way, about segments: since we mainly had presentations in PowerPoint and PDF formats, we indexed each page separately. In this case, everything was quite clear: one chunk, one segment, was equal to one presentation page. The LLM model generated answers with links to the source. We also maintained a dialogue history and took into account the dialogue history in the improved version of our application. I will demonstrate it to you based on the history and the last answer and question. The system also generated guiding questions that helped users refine their dialogue. The improved version of the application, which I will demonstrate to you, also included a pro mode using the function of multimodal search and information analysis. It's a very useful function in principle for chatbots, which I highly recommend using, at least studying. Its key difference from the approach where we use just a language model, not a visual one, a large language model, is that it can analyze graphs and not just graphs, but essentially any images. It can analyze them, essentially like a person, with their visual organs, their eyes. So, in principle, to the question "What is the loyalty dynamics in Yandex Market?", it can analyze this graph, see that Yandex Market is the blue line, read its axis, and provide an answer on how the dynamics changed. Well, subsequently, we will look at it further. And this solution, it used models like Cohere, Vortex, and also multilingual models. It works faster than the PDF plus AR approach. It better understands the segmentation and structure of presentations, and can analyze texts, graphs, and tables. A crucial component, of course, was in configuring our pipeline and in principle choosing which pipeline elements to create: metrics. We used a basic metric at the hackathon, our accuracy evaluation on a pre-prepared Ground Truth dataset of about 450 questions. We checked, with certain configurations of our pipeline elements, the number of generated documents with various functions of merging and ranking, and determined the one that gave the highest accuracy, and then used it as the final version of the pipeline. We also used the ROUGE quality metric, which allowed us to evaluate the metric of how complete the answer was for the given question, whether it contained additional irrelevant information for the question. We also evaluated how factual the [music] context was. Well, and the final result was the answer to the given question. In fact, it contains a larger number of metrics, but this is also a very interesting topic separately: evaluating the quality of the model. Below are some reference links to the original literature. It is flexible, scalable, and suitable for further development. The models themselves worked on the basis of VM. By the way, VM can be launched both in a custom machine and in Google Colab for ease of use. Well, let's look at the application itself now. It's very similar to the familiar interfaces of ChatGPT and Plexi, which are becoming a kind of standard in the development of such applications. So, the application itself provides the ability to search for materials by different types and categories. Upon loading documents into the material base, the bot summarizes and categorizes them using a large language model. It has two modes: normal and pro mode, which uses a visual language model. Let's ask it some questions. For example, we are researching podcasts in Russia. We ask a question, it searches for relevant information in presentations about slides, generates a synthesized answer with a source of information, provides the ability to view slides, also provides individual segments of found information, and provides the ability to go to the original file. Let's see what kind of files were in the corporate knowledge base. We see that these are presentations with graphs, with text, with images, with tables. Let's ask a question about brands' spending on podcast promotion for 2020, 2021, 2022. Such information is contained in this presentation. We ask a question in the same dialogue, without specifying that it is about podcasts, just saying what is the total amount spent by brands in 2022. The system takes into account the dialogue history, so it understands that we are talking about podcasts, and provides an answer specifically about podcasts. It provides the information read correctly from the slide, broken down by year. Let's ask it to calculate the total amount. Let's see how it handles this task. It essentially needs to add three numbers. It successfully adds them. Well, it's interesting to ask it to calculate, for example, the arithmetic mean. Under the hood, we are using the Kven 2.5 model in this case, and it also calculates the arithmetic mean correctly. So, the bot offers the ability to rate the model based on the dialogue context, questions, and generates clarifying questions that can be asked. Well, let's also ask another question, for example, "How much does it cost to launch a podcast?" and the LLM model performs a search for information and generates an answer that a podcast can be launched with a budget of zero rubles, but on average for branded organizations, it reaches up to 100,000 rubles. It's interesting to understand how it calculates this tabular information. Let's ask it to present this expenditure structure in text format. The LLM model presents it to us in a form that is not very convenient to understand. Let's ask it, for example, to sort it in descending order. It presented it in descending order, from infinity, 50,000 rubles, 30, 20, 15. Everything was sorted quite successfully. It's interesting how much the total cost for the host and sound engineer will be. The host is 20,000 rubles, and the sound engineer is 3,000 rubles. So, it says that it's 23,000 rubles. Great. So, I will also demonstrate the function of viewing all documents that are in the search index, correctly speaking, not in memory, but in the search index of our model, because its memory is basic parametric, and we took the model itself in a frozen state, using the technology. We essentially provide the ability to answer always up-to-date information contained in documents without using expensive, long model training processes. And here we see that we will use the pro model for answering questions about these two graphs: the bubble chart on audience structure and the linear chart, the loyalty graph of the marketplace. Well, let's open a new chat. Initially, without using the pro mode, let's ask the question: "What was the loyalty of the marketplace Ozon in July 2023?" and the model does not know the answer to this question. But if we connect the visual model, then as we see, the model already understands the answer. It analyzed this graph, saw where Ozon is, saw where January 2023 is, and gave us an answer, and this truly impresses us. Let's ask it another question: "Which genre is more popular among women?" and the model managed to analyze this diagram as well, which, obviously, is not written anywhere in text that this is the most popular among women. This information was read precisely from the graph. So, it says "Star." So, the interface developed by us, we also always collect statistics on usage and system performance, feedback, and provide dashboards to the client for convenient analysis of this statistical information. The entire dialogue history is stored, you can go back, look, enter the dialogue, continue working in it. So, here is a very convenient interface for work, for search, which takes into account both the dialogue context and, accordingly, the up-to-date information stored in the knowledge base. Therefore, dialogue context is no longer a prospect, it is already a reality. We can also handle complex questions. Further prospects for development for the media agency were, in general, generating presentations, generating web pages, reports based on dialogues, connecting new models, search techniques, training, auto-classification, query routing. The effect of implementing such solutions is, of course, saving time for employees. They are assistants for employees, they increase the accuracy of information search and analysis, and most importantly, they free up employees from routine tasks to solve important strategic problems for the benefit of the company. Well, this is the case we were implementing. I hope it was interesting for you. Write in the comments, let's discuss. We are ready to tell and show the solution itself if there are requests. Subscribe to our channel on Telegram. If you have requests for creating such solutions, contact us, write to us. Visit our laboratory's website. We are ready to discuss with you the ways of applying artificial intelligence for your business, for your cases, and develop pilot products that will at least allow you to assess the applicability and effect of such solutions. Well, we will be finishing this monologue on the channel of our dialogues this time. Thank you all for your attention. See you next time.