Transcription
In today's video, I'm going to show you how to create your own Javis AI voice agent using Python and 11Labs as a voice component that is basically able to do anything that you can write a function for in Python, which is able to do stuff like this.
Hi, I'm Jarvis, your personal assistant. How may I help you, sir?
Hi, can you design me a new Iron Man suit, please, that is able to defeat Thanos?
I'll design a new Iron Man suit capable of defeating Thanos, sir. Though I suspect even your ego might not fit inside this one. Your new Thanos buster suit is ready for review, sir.
And actually, there are already a couple of these here on YouTube. Most of them use workflow automations like nitnomake.com. But of course, coding the business logic yourself gives you more control as to how the agent operates. So, let's just jump right into the tutorial.
So, let's start with the voice component. 11 Labs. For this, you can simply Google 11 Labs and then click on the first link. And then on the upper right corner, you can click on go to app. And here's where you would need to set up an account. Uh I obviously already did that. So I'm going to proceed by clicking on conversational AI. Then I'm going to click on agents. And here's where you can manage your agents and where you can create a new one. You can either click this button here or you can click on create an agent. And over here you can select from a few templates. For this video, I'm going to select blank template. And I'm going to give it a name. Let's call this Jarvis. And then I click on create agent. And there you go. We already have our agent set up. And we can also test this right away by clicking on test AI agent. And then we can click on call AI agent.
Hi, are you listening?
Yes, I am listening. How can I help you today?
So, as you can see, this already worked. And the cool thing about the 11 Labs is that it gives you a lot of different options to configure your AI agent when it comes down to the speech, the language, and the voice and so on and so forth. And you can even add different languages or different language capabilities to your AI agent. The way you would do that is you can add simply uh select some uh languages that you want to add. For example, I can select German or let's just add another language like Vietnamese. And in order to make your AI agent be able to switch between these languages seamlessly, you have to go down and enable a so-called tool. And for this you can click here on add tool. And here right away you can select language detection. And then over here you can click on add tool. And there you go. Now you have a different or additional tool which is which allows your agent to switch between languages. And here you can click on save to save your changes. And then you can test this out again. And as you can see now we have different languages which we can select from. And then we can test this by clicking on call AI agent again.
Hi are you listening?
Yes I am listening. How can I help you today?
So, as you can see, it worked pretty well. He seamlessly just switched to the language that I was speaking, even though my pronunciation, for example, in Vietnamese is not the best, I would say. So, that's pretty neat.
Another thing which we would uh which we would need to configure is a first message. The first message is the first sentence. the AI agent would say when you start a conversation. So you can just write something like, "Hi, I'm Jarvis, your personal AI or your personal assistant. How can I help you?"
Another thing that is quite helpful is a system prompt. This is basically a description as to how the AI agent should operate. So you can write something like persona, you are a digital uh AI butler and can write task. This is what the AI agent should do. Uh help the user by using tools. And then we can also specify some rules which the AI agent should follow. Um call the specifics. And these large language models which are behind the AI agent or basically the heart of the AI agent, they tend to answer in very long and lengthy sentences if you ask them something and that can get quite annoying if they just keep on and on and on and on with their answer. So in order to avoid that we can simply write um keep or answer in only short sentences. And of course we're trying to rebuild Jarvis here. So Jarvis in the movies is quite sarcastic all the time. So we can uh make our agent be like that as well by writing uh be sarcastic. And since it's an butler, I want him to call me sir. Uh, call the user sir.
All right. So, as you could tell, the voice of the AI agent was quite far from how the voice of Jarvis sounds in the movie. So, we can change the voice over here. And I played around with this a little bit and I found that the voice called Charlie is the closest thing to Jarvis from the movie. So, we're going to change it as well and click on save.
Now we can test this again.
Hi, I'm Jarvis, your personal acetant. How can I help you?
Oh, wait a minute. I wrote I misspelled this probably. Yeah. Okay, let's try this again.
Hi, I'm Jarvis, your personal assistant. How can I help you?
Hi, how are you?
I am doing splendidly, sir. How can I be of service to you today?
So there you go. Our voice component is complete and that sounded already pretty good. So there's one more thing which we have to do before we can start with the Python implementation. That is we have to copy this ID of the agent. We can simply copy this here. And then the second thing is that we have to create an API key. We can click here and then click on API keys. And then we can click on create API key. And here we have to give it a name. We can call this Jarvis and then click on create. And it is really important that you save this or copy this and save it where you can access this again later because once you close this window, you cannot open this again. So you cannot look at your API key again or copy it. So save it somewhere where you can access it again.
Here we are in Visual Studio Code. This is my preferred IDE for any Python implementations. You can also use this and I would advise you to do so. For this you can simply go to the browser and then Google Visual Studio Code, download it and install it. That's actually really easy. Um, and once you did so you can create a folder wherever you like and then you can open it. And what I would also advise you to do is to install something called Copilot. You can find it here and you can type in copilot and then you can click on it and install it. I already did that. And what this essentially is is an AI assistant that helps you to develop faster. It can autocomplete your code. It can even help you find bugs or generate new code. So it's really helpful and you able to develop way faster with this. So definitely do that.
So now we can start by creating our first file over here. We can type main.py. The next thing we would need is av file. And then we would need a requirements file. This is a txt file. And we would also need a tools file.py. And then we can press control shiftp to open a new terminal. We can type in create new terminal. I prefer to use powershell. So I'm going to select that. And now we have a new terminal down here. And what I would always advise you to do is to create the virtual environment by typing python- m venth v. Press enter. And now we can you can see here we have a new virtual environment. The next thing we would need to do is we need to activate it. You can do this by typing dot back slashlength back slashcripts and then activate. Press enter. Now it's activated. And then we need to specify our libraries which we're going to use. Of course, we're going to need 11 laps. We can go to our requirements file and type that in 11 laps. And then we would also need something like pi audio for audio processing. and then open AI and then envo and also something called langchain community. Then we can save that and normally you could uh simply install all of that in one go by typing u vpip install-r requirements. We can press the tap button to autocomplete here. So I'm going to tap that and then you could press enter to install everything. My virus software for some reason doesn't allow me to do that. So if you run into a similar problem, you can simply install everything manually. So you can write pip install 11 laps. There you go. That's my anti virus software. But now it's installing everything. So this is installed. Uh, I'm going to fast forward after I installed this.
Now we can set up our env file. For this we go to the file here and here we need to paste our 11labs API key which we created before. So we can type here 11labs API key. As you can see, Copilot already autocompleted that for me, which is really neat. And then we need to paste our API key here like that. And then we also need our agent ID like this. And then we can press Ctrl S to save this. And then we can go back to our main.py. Pi. What we can do now is we can go to the browser and then we can Google 11 Labs Python and we click on the first link. And here we can simply copy the tutorial here and paste it in. Here we can copy everything actually. We don't need this last part here. And the last thing we need is we need to import from env this function here. And then we need to invoke this like that. And what it essentially does it would load the variables that we declared in the env file into our local file here. So we can have access to our API keys and then we can simply save this by Ctrl S again. And then we can go to the console and type pi main. And then we can press the autocomplete by pressing uh the tab button and then press enter.
Hi, I'm Javas, your personal assistant. How can I help you?
Hi, how are you today?
Well, sir, I'm functioning at optimal capacity today, though I must say it's rather dull without any challenging tasks to tackle.
Can you end this conversation, please?
Very well, sir. I shall power down for now. Do call upon me when you require my services again.
Okay, as you can see, now we can talk to our AI agent through Python. Now we want our Jarvis to be able to perform actions for us. And for this we need to create functions and tools for it to be able to use. And for this I thought that uh I wanted to make something similar to something called Menace AI. I don't know if you heard about this but this is an AI agent that is able to perform certain tasks for you. uh it can for example create like a itinerary for a whole Japan trip for you or even analyze some stocks for you or even create entire websites for you. So it's really diverse and in case of the 7-day Japan itinerary it is able to uh create you like a very detailed plan for every day. So I want to do something similar with our Jarvis AI agent. And for this we can go back into our implementation and then type from 11labs.conversational AI import client tools. Um, next thing we would need is from lang chain community tools import duck duck go search run. This is a search engine like Google and it's free. So that's why I'm using it. This is um so we can search the web for information for our 7-day itinerary trip. And for this I need to define a function called search web. And I can pass it some parameters. And here I can write query par parameters.get query. What this means is that the AI agent would invoke this function and then pass parameters to this function and from those parameters I would access the variable called query. This is our search query which we can then insert into the um into the search engine like this query equals query and then we would need to return our result like that and this is our search function.
Next thing is that we would need to be able to save this result to a file. So we can create new function called save to txt because I want a txt file. We also give it parameters and this time we want a file name like that. And we also want the data that we would save into the file. Then we need to format that data. Format it data equals f curly bracket. And we add the data in here. This will format my data so it looks better. And then we would also need to save this file. I want a instead of W. And then I wanted to encode it with UT F8 like that. The way the reason why I'm able to create this code so fast is because of Copilot. So every time you have this like line here that's a suggestion for code and you can simply press the tap button to autocomplete that like this and our function are now done.
Next thing is that we need to register these functions to a tool. So we need to write uh client tools I don't know about that client [Music] tools client tools and then we need to register them like that. Then we have to give it a res registration name and then here also like a registration name and then we have to specify which function. So this is the name, this is the function and I actually like to uh write the name like this so I know the difference and then we can press Ctrl S to save this again. We can go back into our uh agent and it's very important to to copy this these uh registration names because once we go in here again know into our agent and we add a new tool. We click custom tool. We go up to clients we need to register or we need to specify these two tools now. So we paste the registration name in here. We give it a meaningful description so the AI agent knows when to use these functions. So for example, use this tool to create a or to search the web for info. And then we add the parameters which we defined in the functions. The first one was like the query, the search query and this is the search query. Then we click add tool to add this tool and then we add a new one. The second one was to save it to a txt file. So we paste this name as well. It was uh I think save txt or save to txt I believe. And then we also have to give it a description which is use this tool to save or to create a txt file. Then we have to add the two parameters. One was the file name, the name of the created file. And then the next thing is we have to add the second parameter which was the data. And this is the content of the txt file. And then we have to save this by clicking add tool. And then we save this. And before we can try this out, I was I would advise you to change the large language model from Gemini 2.0 flash to something uh some clot model because generally speaking, these work much better and are much more deterministic in their functions. So I selected claw 3.0 uh 3.5. And then we can go back into our environment and then we can go to our main.py Pi. And what we need to do here is we need to import those client tools from uh tools. Import client tools. And then we need to give this to our agent client tools at client tools. We save this. Ctrl S again. And then we can try this out. And hopefully this going to work by typing pi main autocomplete. Enter.
Hi, I'm Jarvis, your personal assistant. How can I help you?
Hi, can you create me a 7-day itinerary for Japan? It should include cities like Tokyo, Osaka, and Kyoto. And then, can you save that as a file, and after that, please end the conversation. Thank you.
Allow me to craft an exquisite itinerary for your Japanese adventure, sir. Though I must say, cramming three cities into seven days shows quite the ambitious spirit. I've saved your meticulously planned itinerary as requested, sir. Shall we conclude our interaction with a proper farewell?
Thanks. Goodbye.
So, as you can see, this is really um close to what Menace AI already was able to create. So, we have all of the specified cities in here as I as I told him to, and this already works quite fine.
Now I want our Javis AI to be able to create a website for us. And for this I need to again define a new function. For example I can call it create html file. Again I have to pass it some parameters. And here I need to specify the file name again just like before. The data as well. Oh no. And this time I also want a title. And then I can write something like formatted HTML equals F. This is like standard HTML code. So I have a frame for the potential website. And as you can see, using Copilot here is just completely overpowered because it just generates me most of the code like that. I want div here, not P. And of course, I also I also want this to be saved as a file. So can write file name just like that just like before and that's actually it. We save this and of course we also have to register this tool like that. I copy this name here. I go back into our agent and I add this client tool again. create or use this tool to create a HTML file or website and then I add these parameters again. One was the file name again just the name of the website the name of the HTML file. Then I add the data the data of the HTML file the created HTML file. And then I add another one. This is the title. the title of the a of the website. Then I add this tool. I click on save. I go back into the agent. Need to save this as well. I click on main. I don't have to go in there actually. I can simply try this out. Hopefully this works.
Hi, I'm Jarvis, your personal assistant. How can I help you?
Hi, can you create me a website of dogs, please? And also include some nice animations and also some nice background pictures, please. And after that, end the call. Thank you.
Very well, sir. I shall create a delightful website about our four-legged companions with some rather fetching animations. Your wish is my command.
So, there we go. We have the website. And as you can see, he added also some animations here. Uh, I have no idea why he added this, but as you can see, here's a pod that just goes from like from left to right. But yeah, this is a working website. That's pretty cool.
Now, I want our Jarvis AI to be able to create me a blueprint of a potential new Iron Man suit, similarly how it was in the movies. And for this, I'm going to use the Open AI image generation capability. But first, I would need again a API key. And for this, I need to also go to the browser and then Google Platform Open AI and then click on the first link. And normally, I would also need to create an account. I assume that you already have one as well. If not, then just create one. And then we can search for API keys. Get to this site here. And then here again, we can just create a new API key. Give it a name. And then again, copy this API key and make sure that somewhere safe so we can access it later again because you cannot look at this again. And then we can go back into our Python environment. And then we go to our env file. And again we have to also specify an open AI key here. And then paste our API key here. Just like that. And then save this.
And then we can create a new function. You probably already get the gist of this. We can call this uh generate image. Okay. This is too much. Thanks for that. We can specify the parameters here again. Okay, I need the prompt though, so I'm going to keep that. Uh, we also need the file name again like that. And this time we also need the size. Um, I'm going to give this 1024 though. What this is is a standard value. So um let's say you the LLM for some reason does not pass a value here then the code will select this as a standard value so you still have something to work with. Uh, what we also need or what I also want is to save it into a directory. So I'm going to add this too. Going to call this generated images. Um, the reason why is as you can see every time we generate something here I can just zoom in here. Every time we generate something here, uh, it gets just stored into the same folder and over time that gets quite messy. So I want to have a dedicated directory here. But before we proceed, we actually need to import more libraries. We can go up here and then we can say from our import OS and we would also need open AI. Uh, we would also need requests and then from pil we need images and then IO image import byes.io and now we can proceed. uh we can write uh os dot make deers that will create a new directory based on the name that's going to be passed. And then I also want to have a file path like that. This is going to create me the file path from the file name and the directory. So it's automatically saved into the directory. And then I need to load my API key so I have access to it like that. Then I can write client equals open AI images.create. This is going to create me my oh wait it's generate actually. This is going to allow me to generate an image and we can autocomplete this. What we also need to specify is the model like doll E going to use three. And what I also need to specify is the quality. I can say standard quality. I think that's actually it. Yeah. And then I can write image URL. I actually need the response I think. Oh no, wait a minute. Sorry. I need to write client equals open AI dot open AI. This is the response actually. And then what this does is that it will uh generate me the image. I get the response from this API call and then I can extract the URL of the generated image from that um response. What's not working here? Oh, forgot this. Um and then I can use that URL to download the image. uh data ul like that. Uh, let's also print the response to see how it looks. Uh, then I can get the image response response like that. I can download the image with image open byes io and then I can put the content of the image response in there and then I can save the image. All right, that was a lot.
Now we have to go back into our agent and then we have to add another tool. Oh, sorry. First we have to of course register our new function. Almost forgot that. Can save this. We can copy this name again. We can put the name here. Give it a description. Use this tool to generate an image. Again we have to add all of these parameters. File name and then we can save this again and hopefully this works. Uh normally that doesn't work uh if I try it first time. So fingers crossed we can clear this terminal again and then we can try this out.
Hi, I'm Jarvis, your personal assistant. How can I help you?
Can you generate me a blueprint of a potential new Iron Man suit and call it Mark 80, please? And then end the call after that.
With utmost pleasure, sir. I shall create a schematic of what could be your next masterpiece. though. I do hope this one lasts longer than the previous models.
Oh, there we go. It created the suit for us. Looks like a looks like a space suit actually or a combination of a space suit and a Hulkbuster suit. But there you go.
So, there you have it. Your own little Jarvis AI voice assistant. Now, you can add any sort of functionality you want. Of course, you only have to make other tools for that. And of course, as an AI assistant, you wanted to be able to make appointments or take notes or even send messages to people, which of course I could do, but this video was already long enough and I did not want it to be too long. And I can also add these functions if you want to see a part two of this. And as a side note, this was actually my first ever YouTube video. So if you have any sort of constructive critique you want to leave or what I can do better stuff, just let me know in the comments below. So yeah, I hope this was helpful to you and peace