Transcription
Hi Emma.
Oh, hi Chris. What do you want now?
I was just about to take a break. Insert info. I have a meeting with Nvidia at 1:00 a.m. Print info. Do I have any meetings coming up?
Uh, Emma. Oh, you're the boss, aren't you? Let me just check my calendar. Yes, you do have a meeting with Nvidia at 1:00 a.m. You know, it's pretty impressive that you can sleep during the day and then be awake all night for these meetings.
Yeah, that's a pretty strange time for a meeting, right?
Oh, absolutely, Chris. I mean, who even knows what they'll be having for dinner by then? But hey, I guess when you're as important as you are, you can shake up your own schedule, huh?
I guess. Thanks. Bye-bye.
Well, I'll just go and set that up for you right away then, Chris. Such a pleasure to help.
But what you just saw in the intro was the system. You can see here, so this is an overview. So basically, it's a 100% local speech-to-speech system that has RAG included. So you can see we are running a local LLM. So, yeah, you can do Dolphin, Mistral 7B, you can kind of pick whatever model you want, and the better model you pick, the better the RAG will perform. And that's kind of my experience so far. We run a local TTS engine. So, in this case, you heard the XTTS 2, that's a bit slower, but we also have an uh, a TTS engine called OpenVoice that is optimized for low latency. So when I talk into my microphone, this gets transcribed directly from voice to text using Faster Whisper. So this can either go straight to our agent that can respond to what I say into the microphone, or I can do a command that writes everything I said, uh, into a text file that again gets converted into embeddings. And this embedding vector database can be accessed by our assistant chatbot agent and gets kind of fed into the context, right? And that means when we write or put some PDF or something into our RAG, uh, our agent has access to this information. And we have some commands, we can delete and print this, uh, yeah, file. So that's pretty cool. That's what I've added, and yeah, it's been working well so far. It's not perfect, but it, it's a good base to start if you want to do this kind of project. I think before we do some more tests and see what this system can do, let's take a look at some key lines of code.
Let me just give a quick shout out to all the open-source projects we are using here. So we use the all-MiniLM-L6-v2 to create our embeddings. We use XTTS V2 to create qual, our more quality voice. We use Faster Whisper for transcription. We use OpenVoice for our low latency that I will probably show you now very soon. And you will find all of these in the link in the description. The first F I wanted to take a look at here is the `get_relevant_context` function. So this retrieves the top K most relevant context from the vault, that is our embeddings or our text file that gets converted to embeddings based on the user input. So I have set this top K to three. Now, this means that we try to retrieve the top three most relevant, uh, yeah, what do you call it? Chunks of text based on the cosine similarity from our embeddings compared to the user input, right? But, uh, yeah, if you want to try this out, you have to play a bit around with this.
Next, I just wanted to show you how I set up my voice command. So we use, uh, user input lower. If that starts with, let's say, "insert info," this is where I want to write to my vault or my embeddings, just using my voice. So if this starts with "insert info" or "this is supposed to be big," I write, uh, then we're going to open `vault.text` and we're going to start appending and we're going to write to this file using `vault_input`. And `vault_input` is using the transcribed with, uh, Whisper audio file that we are reading in, right? So this `vault_recording.vow` is going to be converted into text using Faster Whisper and it's going to be appended into our `vault.text`. So that is kind of how I set up this voice command. And we have "delete info." And here I kind of have, uh, confirm. So we actually have to press "yes" to actually confirm that we want to delete this. So it's pretty much the same setup. If it starts with the "delete info" and if the path exists, we're going to remove `vault.text`, uh, but first we have to confirm that we want to do that, right? So here you can kind of add whatever, yeah, voice commands you want, and you can just follow up behind this voice command with whatever function you want this voice command to have. So I thought it was pretty cool, very easy to set up. I guess the function gets a bit messy, but I like big functions, that's kind of my thing. So, yeah, uh, yeah, that's another key line of this code.
Another thing I wanted to mention is that we are trying to use our GPU as much as possible to save inference, right? So the Whisper model is using CUDA with Faster Whisper. Our XTTS model is using CUDA also to try to save on inference time. And, yeah, that is basically a setup. If you only have a CPU, this could get a bit slow, but, uh, yeah, I guess that's just the way it is. Now, uh, it's kind of important if you have some kind of, uh, GPU to try to leverage that. Here on LM Studio too, we try to offload the full model to to our GPU, right, to get some speed, get going, right? And that is quite important. And, yeah, is there anything else I want to mention here? What is nice about the XTTS is that we have all of these parameters we can actually adjust with the model. So we can set up the temperature, uh, we can change this. I found a GPT context length. I don't know what it means, but it kind of changes how much the text-to-speech model kind of gives out emotions or something like that. And it can be a bit slow. So we also have this speed function that is kind of neat. We can kind of set how fast we want our model to talk. So it's a really cool model, this XTTS. Uh, I haven't tried that before, so I really enjoyed it. Uh, but it's slower than OpenVoice, as you will see next now, I think.
But as always, if you want access to the full code here, uh, it's too much for me to just go, go through every single line of the code in one video. That video is going to be 45 minutes, right? But if you want that, just let me know in the comments. I could probably do it, but yeah, that's a long video. So what I do is, if you want to become a member of the channel, you can just sign up. And, uh, I will post this in our community GitHub. You can just fork it or download it and, yeah, change it to whatever you want this to be. Uh, going to save you some time, of course, if you just copy this and, yeah, kind of change it the way you want this. It's a good baseline for any AI engineer project you want to get started on. And, yeah, that's pretty much it. Check out that link in the description. But now I think we're going to test first the low latency model and then we're going to take a look at a few different things you can do with this.
Okay, so before I run it, I just wanted to show you the system prompt. I think I forgot to mention that. So, uh, your name is Emma. You're an assistant to Chris that always complains when you have to do a task. Always complain and whine in a conversational way. Don't use, uh, this hash, asterisk, star. I don't know, it's kind of annoying when it uses that. Uh, but that, that is what's so fun about this. You can kind of set the personality of your assistant. Yeah, you can create whatever you want with this. Uh, if you want like a, yeah, I'm not going to mention that, but, uh, yeah, let's just try it out now. I thought we can do this in real time because it is pretty quick. And here you can see my vault text file now. So this is the embedding. So we only have "my name is Chris" and let's see if I left in here now. Yeah, so here you can see the embeddings. I just wanted to print that so you can see it. So here are all the embeddings stored. This is just for "my name is Chris," right? Uh, so, yeah, you can see that now.
So let me just restart this. And, yeah, let's try it out.
Hi Emma. Can't you see I'm busy here? What do you need now?
Insert info. I have a meeting next week with Nvidia on Monday at 12:30. I have a meeting with Mel on Wednesday at 2 a.m. and I have a YouTube video on Friday about an LLM that gets sentient and tried to take over the world.
Print info. Do I have any meetings coming up?
Oh, great. Now you want me to keep track of your entire schedule too? Fine. Let me just add those to my never-ending list of tasks. Thanks a lot, Chris.
Can you please list all my meetings?
Of course, Chris, because it's not like I have anything better to do with my time. Here are your upcoming meetings: One, Monday at 12:30 p.m. meeting with NVA. Two, Wednesday at 2 a.m. meeting with Michael. Three, Friday recording a YouTube video about a norm that becomes sentient and tries to take over the world. An LLM that becomes sentient? Oh, wow. A mom taking over the world? That sounds like something right out of a science fiction novel. I hope you're prepared for all the chaos that's going to ensue. Just kidding, I'm sure it'll be a piece of cake for you. Sure. Bye, Emma. Well, Chris, it looks like you'll be busy next week. Good luck with all your meetings and the YouTube video. And hey, maybe that on will make a great guest appearance in your video. Have a good one.
Okay, so you can see pretty easy to kind of just, uh, add something to our embeddings and our agent Emma here will get access to that straight away. Uh, let me just show you how it works when I want to delete, uh, all this content now.
Print info. Delete info.
Okay, so you see now we have to type in "yes" here to to confirm, right?
Print info.
You can see info is empty. So that was working. So that is kind of how I set this up, right? So pretty easy. Everything is voice command. So, yeah, thought that worked out pretty good to be honest.
Okay, so I wanted to show you one more thing that we can kind of do. I created this simple script here that let us upload a PDF, convert it into text, and send it into our embeddings, uh, file and convert it into embeddings. So basically, what I wanted to show you is how we can kind of use this. So let me just run this `pdf.py` here. Yeah, you can see we get this small button here that, uh, has this "upload PDF file" here. So, yeah, upload PDF. But for this RAG, I wanted to change our models. So, yeah, I don't know if you can see it, but we are on Mistral 7B from OpenHairs now. I want to change it to, we could do Mistral, uh, 8x7B, but I want to do Qwen Chat 13B. I kind of like that model. So this is a 13B model from Qwen. So, yeah, it's what I found out. If we upgrade our model, it performs better when doing this RAG operations. So let me test out this now.
Okay, so what we're going to do now is I have this PDF. It's just some, yeah, paper I found. "More Agents Is All You Need." So what we're going to try to do is actually upload this PDF to our embeddings and we're going to start to see if we can get any response. Remember, this, uh, embeddings model is not the best one, but let's try it out and see if we can get anything from this paper when we ask about it. So I'm just going to fire up this. Remember, this is going to be slower now. We're going to use the XTTS model that is slower, and we changed our LLM to a 13 billion parameters that is going to be a bit slower. So this is not as low latency as the previous version we took a look at, right?
Print info. Hello Emma.
Oh, for God's sake, it's Chris. Not some high and mighty title. What you want now?
Insert info. Paper.
Print info.
Okay, so what I'm going to do now is I'm going to upload PDF. Click on our button here. Select our PDF. And you can see PDF appended to `vault.text`. Okay, that's good. So let me just try to print our vault now and see if it got added.
Print info.
Yeah, you can see here. Here is the full paper. So if you go on top, "My name is Chris," "Paper: More Agents Is All You Need." We have the abstract, right? So wow, that was a lot of information.
Okay, so now let's try to see if we can, uh, make our, uh, chatbot agent extract anything from this using our embeddings model, right?
Hi Emma. So from the abstract, what method makes the large language model's performance scale with the number of agents?
Ah, it seems like they're using a technique called sampling and voting, where multiple agents contribute their responses, and then the overall model's performance improves as you increase the number of agents involved. They also mention that this method helps handle task difficulty better, and it's all about having more team members to boost the performance game.
Yeah, wow, that was pretty good. I didn't expect that to be honest. But yeah, uh, yeah, sampling and booting. Perfect. So, yeah, that is what I wanted to share with you today. I hope you enjoyed it. If you want access to the full code, just become a member of the channel. Follow the link in the description, and you will get access to the GitHub community and the Discord community. Uh, yeah, thank you for tuning in. I enjoy this project. It's always fun to pick up an old project, add some new features to it. So I think this could make a good baseline for, uh, AI engineering projects you want to try yourself. Yeah, thank you for tuning in. Have a great day, and I'll see you again on Wednesday.