Transcription
Greetings. Let's break down the usage of one of my programs, a converter. I'll show you now. Here is our program. A link to it will be in the video description on the GitHub page. There, in principle, basic information on how to use the program is also described. But I will show it more visually, so that, well, perhaps it will be easier for someone to understand through video, rather than through a text description. Let's go in order. The program is in English, it can be changed. The language can be changed in the settings. Here is the settings button. Here you can switch to Russian. Everything changes in real time, or you can keep it in English. I personally prefer English, but you will be able to change it yourself in the program after it is turned off. Let's go in order. Configuration. This is our main configuration window. Here are our presets and the source folder. How to use the program? Currently, our Source folder is empty, meaning no folder is selected. To select a folder, you can do it in two ways: either you can simply drag and drop it from the file explorer, and then we will have the folder with our selected folder. The preset is automatically selected, so you don't need to do it manually, but I want to clarify that there are several presets in the program. All of this listed here is supported and converted without any problems. But, for example, with the C# preset, there is a problem that sometimes it is not detected automatically, and with some others as well. With the Godot preset, there is no such issue. That is, if you download the program solely for using Godot, there will be no problems. The program is initially designed for use with Godot. Special compression modes, made for "done report" are already made specifically for it. What is the purpose of the program? Its use case. Here is our Godot project, we drag it in. The program is designed to provide context to AI models, LLMs. In particular, I prefer to use Gini. You can use any other. I use Gini because Gini works well with PDF reports, well, with PDF files. Here is our PDF file, we then drag it into our AI studio. Let's move on to usage. Here our folder is selected. We can also select a folder through the "Select" button. The familiar window opens. And here we select the folder. Everything as always, everything is familiar. Preset. What does the preset do? Here we have the extension, which is the file extension. Here we have, for example, a file.txt. And what is after the dot is the file extension. So we have, for example, scripts. Scripts. Here we have UI script. Well, any script in Godot has the extension .gd. We set the extensions that we want to be converted. All these extensions will go into the final conversion folder. There is also support for ignored folders, so that no junk gets into our conversion output. How does it work? In particular, with Godot, it's not very useful. I'll explain with the example of JavaScript programs. Those who know JavaScript programs have Node, so the node_modules folder. This folder stores downloaded libraries. And in these libraries, there can also be data files with these extensions, but they are not related to the project context. If we provide context to an LLM program, it doesn't need information from these node_modules folders. So you just add the folder here that you want to be ignored. Well, in principle, I think everything is clear. Ignored folders. Everything is elementary. Let's move on to the checkboxes. Generate structure file. I'll explain what this is. Here is our file structure. Here it is generated. File structure is a summary of the entire project, which provides paths to all files that have been converted. This is very useful for LLMs, so that they can, considering for themselves not only the files that have been converted, but also the files that, in principle, exist in the project, so that they know the path to them. That is, so that Gini knows where this file is located. I think that's also clear. Compact mode. This also relates to the structure file. What does this checkbox do? If it is unchecked, then all project files that are in this folder will appear in our structure file. That is, all files that we have here will appear, their paths. So Gini will know not only the location of files with these extensions, but also the location of all files, including PNG, any models, and so on. I personally prefer to leave this checkbox enabled, so that these files do not end up in the structure file. Why? I believe that cluttering the model's context with paths to files it can never work with is impractical and simply unnecessary. Therefore, I recommend leaving it enabled as well. Let's move on to token compression. Token compression is responsible for, here we have several modes. None, meaning nothing will be compressed at all. Everything will be provided to the model in its original form. Smart save. This mode simply removes extra indentation, removes extra new lines that we create with the enter key. And it doesn't do much else. It reduces, well, token consumption by about, let's say, 10 percent. Maximum compression. This compression is focused on Godot. This is a unique feature, personally made by me. I'll explain how it works. Here is a scene, the main focus. Maximum compression is applied specifically to scenes. Now let's find one. Look. This is a scene converted with maximum compression. That is, the content of the scene is not provided verbatim, but a summary of useful information from that scene. Smart models like Gini will be able to easily, without losing any data or damaging the scene, convert the information provided here into useful context. Now, for a visual comparison, I will show you the difference between a maximally compressed scene and a scene without compression. Here we compress, and here is what we got. Let's open it. So, look. Here is the scene. See? This is how it looks uncompressed. There is a lot of information here, like this, this, this. Those who use Godot, I think, understand what I'm talking about. The model doesn't need all of this. So we can get rid of it and just use the compressed version of the scene. Like this. This reduces context consumption by up to 50%. I swear, I checked, there were many tests, especially in large projects, the use of scenes grows exponentially. And this kind of compression saves a huge number of tokens. This is beneficial in terms of cost savings, because the fewer tokens you send, the less you have to pay for input. And it's also beneficial because, for example, with Gemini models, the more context they are given, the dumber they become. We've covered compression. Generate merged file. This checkbox is responsible for generating this file. This file is a complete dump of all code collected from all files with these extensions. That is, all files in this folder that have been converted go into one such file. Why is this needed? Why not send many files to the model separately? I'll explain. Models and, in general, the interfaces with which you interact with models, do not perceive a large number of files well. So it will be better if you send everything as one file. This has also been tested. I conducted many tests. And, well, just use this. This is a text file. This is what is responsible for the text file. Let's move on to the next function. Generate PDF report. This function is only needed if you are working with multimodal models that understand images well, that understand PDF documents well, without converting them to text. That is, for example, with Deepseek, this function will work the same as with this. Well, for Deepseek, there will be no difference between a PDF document and this, because Deepseek simply extracts textual information. But, for example, Gemini models, which are multimodal, are able to work with images, converting image tokens more effectively than text tokens. I will show you the difference visually later. Regarding the modes, I will also show you visually now. So, let's do a standard generation in the standard mode without compression. I must say, I do not recommend using the standard mode. The program is not focused on it. The standard mode will be almost no different in efficiency from a text file. That is, the token consumption will be approximately the same. This is how an uncompressed PDF file looks. That is, it is literally a beautifully formatted version of all your text base. Here, the structure file and all other project files are included. See how many pages it has. I have a large project, and these pages were included. Well, look, I'll explain more correctly. Probably, the more pages a PDF document has, the more tokens it will occupy. Let's save this PDF document to show it visually later. And let's also run it now. No, compression. Let's also save it. And now let's switch to maximum mode. Compression to extreme. And you will immediately see a big difference in the document's formatting, and a little later in the document's size. Here are only 11 pages. See how it looks? It's made with very small font, but all the information is preserved here. You might say, "But this is some kind of nonsense. This is unreadable." But models don't care, I've checked. The context is extracted correctly. It's safe to use. The context is not lost. All information will successfully reach the model. Well, at least with Gemini models, I haven't tested with others. Here, 200-plus pages turned into 11. Let's compare in the AI studio itself in terms of tokens. Here is my AI studio. Let's drag the compressed PDF file with the extreme compression mode and see how many tokens it occupies. Look, only 6,000 tokens. This is a huge project. Now you will see how many tokens the text file occupies, and just compare the difference between them. 174,000 tokens. So you understand how much token consumption has been reduced. This is an incredible amount of savings. And let's now transfer the uncompressed version of the PDF document here, which has 200-plus pages. See how much it will weigh. 146,000 tokens. So the difference between the extreme compression mode, which has 11 pages, and the difference between the standard compression mode is colossal. That's why I recommend using the extreme mode. Information is not lost, I've checked. So just use the extreme mode and you won't have problems. I also wanted to say that specifically with this function of PDF reports in extreme mode, you will be able to load much larger projects into the LLM model's memory than through a text file. Well, it's already clear why. Well, in principle, I think we've covered it. I also wanted to show the difference between the compression of the compressed and uncompressed versions of the text file. This is also important, because this text file ends up in the PDF file. That is, a PDF file made from a text file without maximum compression will occupy more tokens than a file generated with this maximum compression. So, here we have generated without compression. Let's see how many tokens it occupies. There will also be some insane amount here. Look, 303,000 tokens occupy the uncompressed file. And let's also transfer the file with maximum compression here, 174,000. That is, the compression is 50%. That's why it's important to choose the maximum compression modes here and here. Other functions. Here we have "select files". When you select a folder, an automatic scan of that folder is performed. All files with these extensions are identified. So, in my project folder, in this one, there are 252 relevant files that match these extensions. "Select files" is a separate window that shows all these files. Here, if there are any files that you don't want to select, you can simply uncheck them, and they will not end up in the PDF or in this text file. Why is this needed? Previously, before I opened this method of using PDF documents, in order for the model not to be so stupid, I had to remove some files from the context and send only those files to the model that I planned to work with at that moment. So, you remove, for example, some script, and it doesn't appear here or there. The model will not know about it. Understood. This can be done completely. You can remove all scripts like this. GD, that is, by extensions. See, view by extensions, shaders, and so on. Everything is very interactive, everything is very convenient. I made this program for myself, which is why I vouch for the quality. There is also an alternative view, "folder structure". How does it work? Look. Here is Magic Sandbox, my project. Here we have the Resources folder. Let's open it and open it here. Here in the resources, it shows, well, that is, it's literally like a file explorer, you understand? That is, a more familiar option if you want to find a script specifically in the scripts folder. So this is a by-folder mode, that's why it's called folder structure. I think that's also clear. But I personally usually found it more convenient to use GD Extension. Oh, well, the view by extensions seemed more convenient to me, but sometimes the view by folders is also needed. I think everything is clear now. Next, "select". This mode, oh, well, it's not a mode, this button is responsible for, let's say, it's a smart mode for selecting files. It was also needed before. Now it's essentially obsolete and no longer needed. To use it, you can set a key, for example, "query". And a window will open. We write a request there, for example, "select all relevant files for automatic level generation, i.e., procedural generation," and the model will think and give you, well, the program has a special prompt. Based on this prompt, it will output a JSON structure from which the program will extract and automatically select the files that the model considered relevant. If you want, you can also use it. It supports not only Google Gemini, but also Nvidia. Those who know can also use it. Well, that's all. To start the conversion, I think everyone understands. Just press this button, and in your project folder, you will find our "converted to text" folder. Let's delete it so you can see its appearance visually. Let's start the conversion. Conversion is complete. Everything happens quickly. This is not some heavy program, and it doesn't load the system. I keep it running constantly while working. So everything is done robustly, there are no memory leaks. The program's RAM consumption is also low. So it's suitable for any system, for anything. Well, and here's our folder. At the top, we will always have the file structure, our text dump, and the PDF document. Why are they always at the top? The Windows file explorer prioritizes files that start with a dot or, I think, an underscore in the sorting order by name. Therefore, so that you don't have to search through this folder for the files you need, these two, well, in particular, PDF will be used most often. They have an underscore at the beginning so that they are always at the top. Well, that's all, in principle, the breakdown is complete. Thank you for watching. Enjoy using it.