Transcription
Hi everyone, this is Fahad Bza, and I welcome you to the channel. I'm very excited to share with you this newly released model from Microsoft: Omni Parser version 2.
We covered the first version of this model around three months ago, and it was quite sublime. That is why I'm quite curious to see what sort of improvements Microsoft has made to this new version.
Before we install it locally and play around with it, let's have a quick look at what exactly this model is.
We all know that graphical user interface, or GUI, automation requires agents that can understand and interact with user screens, whether it is your desktop screen or your mobile screen. However, using general-purpose large language model architectures as GUI agents poses challenges, including reliably identifying interactable icons and understanding the semantics of various elements within a screenshot.
For instance, on this channel's YouTube screen, it's really hard for LLMs to distinguish between these text headings or between these images or the titles. That is where Omni Parser tries to help out.
It addresses this gap by tokenizing UI screenshots into structured elements that are interpretable by LLMs. Omni Parser V2 takes this capability to the next level, achieving higher accuracy in detecting smaller interactable elements and facilitating faster inference.
This has been trained on an expanded dataset. Omni Parser version 2 reduces latency by 60% compared to its predecessor. Notably, the combination of Omni Parser and GPT-4 achieves state-of-the-art average accuracy of 39.6 on the recently released benchmarks, which they have shared on their model card. I will drop the link to it in the video's description.
So now we understand what this model is, what it does, and how it differs from its previous version. Let's try to get it installed.
For that, let me first thank M Compute, who are sponsoring this VM, which is 12224, and this GPU card, Nvidia RTX A6000 with 48 GB of VRAM.
Let me create a virtual environment with K. Make sure you are using Python 3.12. By the way, if you're also looking to rent a GPU at very affordable prices, I will drop the link to the website of M Compute in the video's description.
I'm also going to give you a coupon code for a 50% discount on a range of GPUs, and you can find both of these in the video's description.
Let's wait for this K environment to get created. Shouldn't take too long, and that is all done.
Let me clone the repo of Omni Parser, and you can find it from their model card too. Once that's done, all we need to do is install all the requirements.
For requirement installation, we just need to run this pip install. It is going to install Torch, Torch Vision, and a lot of other stuff which is already present in their repo.
While it installs this, let me also introduce you to the sponsors of the video, who are Entb. Entb lets you effortlessly deploy a personalized knowledge base across platforms like Discord, Slack, and others. It is ideal for open-source tech communities and startups that provide user support. I will drop the link to their website in the video's description.
Okay, so let's go back to our terminal. It is going to take a couple of minutes, so let's wait. You can see that it is installing a lot of packages there, and everything is installed now.
In the next step, we need to log into Hugging Face because we need to download the models from there. For that, let me show you what you need to do.
First up, just run this command: huggingface-cli login. It is going to ask us for our Hugging Face token. You can grab this from your Hugging Face profile after logging in. It's a free token. Just go to your Hugging Face profile, go to settings, and it's the token. Just grab your read token.
So I'm just going to paste my token here, and once I paste it, I'm going to press enter, press capital Y, and it should log me in. So there you go, I have been logged in.
Next step, we need to set up our stuff here. By stuff, I mean we need to download the model. For that, the first step we need to do is to make sure that there is nothing in this directory from the root of the repo.
This is the directory icon_caption_Florence. Don't worry about the name Florence; this was a previous model of Microsoft. Then you can run the following command to download the model.
This is just simply running the for loop, and then by using Hugging Face's CLI download, it is downloading all the model weights from this folder. So let me run this, and once you run it, it is going to download all the models here.
The model is not that huge, and I have covered heaps of GUI models. If you go to my channel, there are a lot of them, not only these but various others, and they're not big models.
Okay, so let's go back to see what is happening. Let's wait for it to get downloaded, and everything is downloaded.
Next up, simply move all of the weights to the Florence icon_caption_Florence directory, and then let me clear the screen.
So everything is set up now. Our installation is done, and the model is downloaded. Let's run the Gradio demo. The first time you run it, it is going to download some of the stuff here from the repo, so just wait for it.
I think these are the things for the Paddle OCR, just to do the optical character recognition from the screen. If you're curious, by the way, it uses the YOLO model for the detection.
And there you go, our Omni Parser version two is running on localhost at port 7861. Let me go here, and then we can access it at this port. It is loading, and there you go, our Omni Parser version two is running here.
Now you can just select any image from here. For example, I'm just going to select this page. This is simply a screenshot in JPG format, and this is present in their GitHub repo.
So I'm just going to select it from here. Let's go to that Omni Parser directory, from here go to image, and this is a demo image which I have just uploaded.
Then these are a few of the thresholds which you can set. I'm going to use Paddle OCR, and let's click on submit. It is working, and while it works on it, let's quickly check the VRAM consumption.
It's not even using that much VRAM—under 1 GB of VRAM, nothing. You can even use it on CPU quite easily. So let's wait for it to come back.
Let's see what sort of structured element it gives us from here. While I was talking, it already did that. There you go, so it has done the OCR. It has selected all of the elements from the page, as you can see.
Every element is there, and then if I scroll down, this is what the beauty is. If you see, these are all the icons and then bounding boxes, you know, all the coordinates of the screen.
You can easily pass it; you can use it in your application, and the use cases are limitless here. This is astounding—really, really good stuff.
I would say accurate. I know that some of you don't like me saying good stuff, but when I say good stuff, it means that the answer is correct, by the way.
Anyway, so let me press X here. Let's try out one more. I have now selected this Excel image. Let's click on submit. The speed is quite good too, really. I mean, quite impressed by the speed of—there you go.
It has selected everything because there is a blank sheet. It hasn't done anything; even it has selected this plus sign. I hope you can see it. Maybe I'll just make it a tad bit bigger.
Okay, so you see it has selected this plus sign here. Let me go down. Really, I think very accurate—very, very accurate. It's very hard for me to even identify any element on the screen which it missed to capture.
Okay, let's try out one more. Let's try out this mobile one. I'm just going to scroll down, click on submit, and let's see what it does.
You see, within a few seconds—like under 20 seconds—even this one, it was even like 7 seconds. Again, the mobile screen has been identified.
So given the VRAM and the whole footprint of this, you can easily run it on your mobile phone, even the older ones, easily. You can create an app, and then you can pass your screen very easily. Very nice.
Let's try out one more, and then we will see how it works. So maybe I'll just go with this iOS screen. Should be the same. Let me go down, click on submit.
By the way, if you are again still wondering what could be the real use case of this, look, while it has done wonderfully well, the use case could be: we can do automated GUI testing.
So it can be used to automate testing of GUIs, reducing the need for manual testing. This can be used to develop accessibility tools that help individuals with disabilities so that they could interact with these tools more easily.
Then there are a lot of other things which we can do. For example, we can do automated data entry with it. We can even do RPA—robotic process automation.
UI Path is one software which is very expensive, by the way, and you can replace it with this. So I think this is a huge threat for UI Path, and there are Automate Anywhere. There are a lot of them.
I think Microsoft's Power Automate is one, so there are a lot of them which are under threat, I believe.
So let's do this again. It's a double screen, and the other could be virtual assistant, game playing, game testing, automated customer support, GUI reverse engineering could be one use case, GUI security testing, pen testing, and then a lot of other R&D use cases could be done.
But look, I think we have tested it out with various sorts of screens there in multiple ways. Sublime model—sublime, really.
As I said, if you are interested in learning about its history, just check this one out. If I quickly search here, there are other ones too where I have done. You can search on the channel; you might have to just come up with a good one.
So you see this one was a good GUI agent model. I did show you I was quite good, but I think this one is the best so far this week, I would say.
So let me know what you think. If you like the content, please consider subscribing to the channel. If you are already subscribed, please do me a favor and share the video and channel among your network because this is the only marketing I do.
Send it through your email, share it on LinkedIn, Reddit, Twitter, X, or whatever, and that will be really, really a huge favor to me. Thanks a lot!