Transcription
Today we're building a copilot agent using computer use within Microsoft Copilot Studio. This is one of the newest tools available and using natural language. You tell this tool what to do and it drives a virtual mouse and keyboard and gets the task done for you. You have to see it to believe it. We'll be building this agent step by step so that you can follow along. My name is Shervin Shaffy and I'm a co-pilot principal engineer at Microsoft. Welcome to collaboration simplified.
Okay, so let's jump straight into building this agent. We're inside Copilot Studio and we're looking at an agent that I actually built in the past. It was called the HR Miracle Worker Pro. And if you want to know how it was built, it was basically in one of my videos in the past in this Copilot Agents one right over here. So if you want to see how that was built, it was demonstrated in this video. But the reason I'm why I'm showing it to you is because I'm going to rebuild it this time using computer use. So the reason I'm rebuilding it is because this particular agent HR miracle worker pro it basically connects to service now and it does it using connectors or API. So you can see under the tool section here there's this tool called list records. So if I open it up you can see that basically it connects to service. Now there's even more to it than that. This is actually the easy part. But on the top here, if I click this plus five, I can go to a section called topics. And when I go into topics, you there's a there's a section here called booking vacation that I created. And when I open that up, there's a flow in here that says, look, if somebody's asking you to book a vacation, this is the trigger. Ask them these questions. And ultimately, it goes ahead and it connects to this custom connector. And again, the reason I'm showing you all this is to kind of show you all the work that has to go into play to create a connection to service. Now, so this particular connector, if we go to the designer section of it inside of Flow, not only does it list the record over here, there's some work involved there, but for each transaction, it also updates the record. And when we look at updating the record here, it looks at hours that were booked. Someone's trying to book vacation. We want to see how many hours they had booked and hours they have left. And there's formulas in there. We want to bypass all this, right? We don't want to have to do all this work. We just want to tell this computer use agent, click here, click there, and just do it without having to do a little bit of low code, if you will.
Now, what do we want in terms of steps this computer used to do? So this is what we wanted to do. I have a service now developer login. And so what I want this computer use to do is first I want it to go to this website developer.service.com. Then I wanted to click on sign in. Then I wanted to sign in with my login of I love computer use at outlook.com and then put in my password of copilot studio rocks and then click sign in and then once it signs in I want it to go to this section here under app engine studio. I want it to select that. And then once it selects that, I want it to scroll down to this my app section and click on HR. And once it does that, I want it to close that out. And then go here and click on this HR DB, this HR database. And then once that's done, I want it to go over here on the top and I wanted to select preview. And then it gets this table of my employees. And basically I have four employees. I have Jane, Pete, Sammy, and Bob. And then I can see how many hours they've booked and how many hours they have left. And then I wanted to update these hours by what is being requested of it. That other agent, Miracle Worker Pro, did all that using connectors or APIs, but I want computer use to do it by just clicking and using the keyboard and mouse and getting to it and updating these fields cuz ultimately if I wanted to update Jane's profile, I wanted to click on Jane, I wanted to go to these numbers and update them and then click update. All right, so that's the use case. This is what we're going to build together.
To build it, we start off in Copilot Studio and on the top left hand side, I'm going to select create and I'm going to select new agent and I'm just going to select create on the top right hand side to get the template of the agent created. So, we'll just give it a sec here. Now I can go ahead and select edit over here to edit the agent name. And we're going to call it computer use for service now. And I'm going to click on change the icon over here. And then again here change icon. I have Clippy. So let's use this one. Let's go ahead and click edit again and give it a description for the agent. an agent that can book vacation days for employees. Okay, then we're going to click save on the top right hand side here. So, we got the name, we gave it a picture, we gave it the description, and it's using the GPT40 model. And by the way, if you were curious, if you select settings for the agent, you might be able to in your version change the models to different ones like 41, GPT5 auto, GPT5 reasoning, and so on. So, I'm going to stick with the default at the moment, but just kind of a heads up. And there's some other things you can change here like content moderation. Right now, it's set to high. Maybe you want to set it to low just to kind of experiment. So there's all these kinds of things you can check and also here sometimes people are like well it's using the knowledge of the large language models and then you can actually turn it off so that general knowledge isn't used but we're going to leave it on for now. So there's some cool settings kind of as a pro tip if you want to click on the top right hand side you get to all those sections.
Okay. Now what are we going to do in terms of instructions? Well, I needed to kind of flow where it asks a bunch of questions like, "Who are you booking this vacation for?" And then I wanted to get to the computer use. So, first what we're going to do is we're going to go to this plus five over here and we're going to create a topic. And here I selected this arrow next to it and I'm going to add from a description with Copilot. So here I'm going to call it um booking vacation and I'm going to create a topic. I'm going to tell it what I want it to do. So get the first and last name of the person looking to book vacation as well as how many hours they want to book. All right, let's go ahead and create that and see what it does. Close this out. close out this side here. And so basically it says this tool handles queries like book a vacation, plan my holiday. And then the first question, this is all autogenerated based on what I just wrote. What is your first and last name? And it says identify the person as their first and last name over here. And what I want to do is I just actually selected on the person's name. Just select it. And then I want to do multiplechoice option. So I just want to give it that information. So I'm going to select multiple choice and it says option for the user. If I actually go to my database for service now, these are the names of the people that I had. Jane Ingley, Pete PD. Let's put those people in. So go back here and for the first option we're going to put Jane Ingley and then click enter plus new option Pete PD. Don't tell me how I came up with these names. Uh click new option Sammy Balong. I'm going to put Bob Smith. All right. So hopefully those are all the the right names. So we're asking for the first and last name. We're giving options and then based this this all got autocreated. So based on what it equals like all of these feed into the next questions. How many hours do you want to book? It's a number and the response is saved as hours. The other response was saved as name. And so I think this is good for now, right? So I'm just going to go ahead and on the top right hand side I'm just going to click save.
So this is the topic. What we're going to do next is eventually we're going to pull all this together in the instructions, but at the moment that I've clicked save, I'm going to click on overview on the top here. And I'm going to scroll down to the tool section. So, we can use we can add our computer use. So, I go I'm going to go ahead and select add a tool. And this is where you would generally search for all these connectors, right? So, if I searched for Service Now, those different Service Now connectors that we used in the other agent, they all pop up here. But we're not going to do that. What we're going to do instead is I'm going to go ahead and select plus new tool over here. And then this is where all those tools show up, including computer use. Now, in the past, we actually did a video on the model context protocol. That's actually a cool one where we connect it to docysine. And then today we're going to do computer use which again you can see it's in preview at the moment. And by the way, real quick, we have some really good documentation on computer use. So if you go to the Copilot Studio admin guide, this is the documentation for computer use right here. And it was posted maybe about a month ago. And so definitely check that out. There's a good video in here. It talks about how to add it to your agent, which we're doing at the moment. All kinds of good stuff in this documentation.
Now, back to the agent. We'll just select computer use. And here it says, look, what are the instructions that you want to put in for the computer use? For example, go to this website, add an invoice, all this kind of stuff. or do you want to start with a template, you know, for invoice processing or or for data entry? And if you select invoice processing or data entry or data extraction, it actually puts a sample set of instructions in here and these websites are actually live. So, this is a good way for you to experiment if you don't have a place to experiment with. So definitely check this out and these are also in the video and in that link as well in the documentation. Now for now I'm just going to go ahead and erase all this. I'm not going to put any instructions. The other thing to note is that it says look how are you connecting to this computer that's running these instructions. And I'm using a hosted browser which if you look for what it is, if you click the I and go to learn more, it opens up that site that I showed you. And if you scroll all the way down to a section in the bottom here, it says uh configure where your computer use runs. So, I'm going to be using this hosted browser powered by Windows 365, which basically allows me to start quickly without any machine setup. And this hosted browser runs in a Microsoft manage environment, but it is not Microsoft Entra join to your tenant or managed by your intoune policies. So this means that if you're trying to connect to your shareepoint site or to your one drive or a website that's internal, it's not going to be able to do that. But if you want to connect to a public site like I am to this service developer site, it can connect to it. Right? There is a way you can actually connect to your introconnected domain manage machine which is right here and it tells you how to go about doing that. We're not going to do that, but the documentation shows you how to do that if that's what your use case is for. So for us, we're just going to leave it as use hosted browser and then go ahead and select add and configure on the bottom here.
Now we're in this section where we have details and and inputs. And for the name of it, I'm going to say this is computer use to service now. Okay. description. What does it do? Connects to the service now developer site that has employee vacation hours. All right. Now, here's where all the instructions come. Everything that I showed you, we expect this to do in terms of go here, click this, do that. This is where we're going to put the instructions. Now, I've come up with the instructions because there's a bunch of them. So, I'm just going to copy and paste them. Okay. So, I just pasted everything in and let's take a look at these together. Right. So, first we're going to go to the particular website which is at developer.service.com. So, we're telling go there and if you're not logged in, click sign in and then log in with my email and my password. This is not a secure way of doing it, but I'm doing it just because this is the demo. What you're supposed to do officially is you go to this stored credentials section here and you add your credentials and this adds them to your Azure key vault in a very secure way and then you can use them. I didn't want to go about setting all this up cuz it's a lot of extra work. So I'm just putting it straight in in the instructions which is again fine for the demo but not recommended if you're doing a production run on this. So, I put in my information, my email, and my password. And then step four, I'm basically saying, select the app engine studio. Go to this HR thing under your app. Select HRDV. Everything that I showed you earlier, select preview, and then search for the name that's being asked of you and click on that name in the first name column. Increase the hours booked by the field hours and decrease the hours left by the hours. So, we're basically updating the the number of hours in there and then ultimately we are clicking an update. We don't want it to ask for permission because it has a tendency to ask for permission and then it kind of like stops and cancels out. And then once the update is complete, tell me how many hours have been booked so far and how many are left. All right, so these are kind of all my instructions. If you guys are interested, I can put these in the comments or in the description somewhere so you have a copy of them. But that's basically what I'm telling it to do.
Now, if I scroll down, there's some other sections of this that are important. We looked at the stored credentials. You can turn on allowing specific websites and desktop apps so that computer use is only limited to go to certain places. If you toggle this on, then you can say, look, from a website, you can go to this website, but you can't. So you can kind of like tell it which ones it's allowed as well as applications. We're leaving all that blank. And then from an inputs perspective, so if you remember in our topics, we actually have that person's name as a variable called name and ours is ours. And so here I'm going to go and create some inputs. And I'm going to say name name of the person. Click done. and then add another input. And I'm going to put hours, number of vacation hours to book. So these are my inputs. So you can see that they just got put in here, hours and name. And I think at this point in time, we're in good shape. So on the top right hand side, I'm going to go ahead and click save.
Okay. Now, once this gets saved, you can actually go ahead and click test over here and test it out. But we're not going to do that. I'm just going to run it inside the agent and show you guys exactly what happens. Now, what we've done so far is we created this topic and we added the tool of computer use with all these instructions. Right now, what we're going to do is we're going to select overview and we're going to go back and we're going to give this agent instructions. We're going to come to this instruction section and we're going to select edit and we're going to say look first use this I'm going to put slash and I want to use this topic booking vacation topic to find out for whom and how many hours are being asked to book then use slash computer use to service now the tool that we just added the computer used to make the changes to the service now database okay very simple save it and I think with that that's our agent that we just built using topics and the computer use and you can see knowledge is blank the tool is just that one tool there is no triggers however ever. Autonomous is actually recommended for computer use probably because computer use takes a long time and you'll see in a second. And then in terms of agents like multi-agent orchestration that's blank as well. So there's a bunch of sections blank in this but that's okay.
So what we're going to do now is we're going to select test and then here I'm going to type in I'd like to book vacation and see what happens. So first this computer use comes in it comes in progress then my booking vacation topic gets initiated and as you can see it says what is the first and last name and so I can pick these names that we put in let's put Sammy belong and it says how many hours do you want to book and I say I want to book 10 hours okay and now this is when the computer use is going to start. And it says that computer use will begin as soon as a computer is available. It's a hosted computer. So, let's uh give it a second and see what happens here. Now, you can see that right away there are screenshots that are being generated of what this computer use is doing. And it's telling you step by step what it's doing. It's going to follow all the instructions that we gave it. So, first it's opening a browser to access service. Now it's going to file explorer and then opening the browser. So it's going to kind of give you a playbyplay and looks like it's actually in the browser. Now it's going to go to that website. And this does take a little bit of time. That's why autonomous is recommended for these types of computer use tools, but we'll see how it does as time goes by. All right, looks like so far it's actually made it to that service. developer site which is good and now it's clicking in on the sign in on the portal. Okay, so it looks like it actually got past this sign in process over here and so now it's actually following the rest of the steps. It's trying to get itself to this App Engine Studio section. So, so far so good. It's continuing on and it's giving us a playbyplay as it goes on. All right, looks like it's making good progress. It actually made it all the way to this database section where Jane and Pete and every everyone's there. And so now it's trying to get to uh Sammy belongs information here and it's updating the hours. It says updating hours left to 80. All right. So, this is it. This is the final section and it says Sammy's hours already booked has been updated to 120 and hours left to 80. And then the computer task is finished. Okay. So, we're looking for 120 and 80 for Sammy. So, if we go to our database and this is a moment of truth, right? So, it says right now it's 110 and 90. So, this should go up to 120 and this should go down to 80. Let's let me refresh the screen. See what happens here. Boom. That's awesome. It worked. Maybe I shouldn't be surprised. Well, that's awesome. I'm happy about that. And if we kind of close this out and go back.
This is one of the issues that I see with computer use. Right now, people are not that patient. You know, maybe a couple of seconds. And you can see that this booking vacation, the topic took 5 seconds, but this took 283 seconds essentially, right? It's a long time. And that's why it's recommended that autonomous is being used for computer use. think about like autonomous things that you can do versus uh a real time thing that could take a long time like that. So that's essentially it. But one thing I wanted to mention as well is I actually had to go to the settings and I went down to this section here which I showed before this content moderation level and I brought it down to low instead of high. Uh, and then I I ran it. And I don't know if I showed that earlier on, but experiment with that uh from high to low. Uh, if you can't get your computer used to work and it might be because I actually had the username and password in my instructions. If you enjoyed this video, make sure to give it a like and consider subscribing. Drop any questions you have in the comments and I'll catch you on the next.