Transcription
So, quite a number of text-to-speech models dropped on Hugging Face not too long ago, and in this video, we'll be focused on this Maya One, and it's by Maya Research.
Now, Maya One is an open-source TTS text-to-speech model for expressive voice generation, built to capture real human emotion and precise voice design. Now, what they actually mean by that is you actually have emotion tags available, and you can also describe the speaker, or you can describe the voice, or provide a prompt to generate the voice you want. And you also provide the text, as you can see here, which this speaker, or the voice you've actually designed, is going to say. So, this is actually really cool. As you can see, the voice description: "female in her 30s with an American accent, and it's an event host, blah, blah, blah." Now, [snorts] let's hear it.
>> Wow, this place looks even better than I imagined. How did they set all this up so perfectly? The lights, the music, everything feels...
>> Really, really cool. Now, let's hear this one. This one is a "dark villain character, male voice in his in their 40s with British accent, low pitch, blah, blah, blah." Now, this is the text.
>> Welcome back to another episode of our podcast. [laughter] [gasps] Today we...
>> Cool. Uh, the thing is, I don't actually take any of these demos seriously because the last time we actually, um, tried a model that claimed its fame was also expressive voice generation and also human emotion with emotion tags. I think that was OpenAudio and...
>> Roll the tape. Roll the tape. Roll the tape. ROLL THE TAPE. Show them what we're comparing this to.
>> We're going to pass in "crying loudly." Let's generate this. "crying loudly." Autobots, we stand at the crossroads of destiny. The fate of humanity rests in our hands, and we shall not fail them.
>> That was weird. That was really weird. If their claim to fame is actually accurate, like these are their claims to fame. Basically, you can [snorts] actually design the voice, and you can also provide the emotion tags, and it will work. So, we'll see.
Now, this model is a trillion-parameter model, and you can run it on a single GPU. But in this video, I'm going to be running this on my CPU. So, just keep that in mind. If you have an Nvidia graphics card, then that's good for you. [laughter] But I don't have that. Anyways, um, let's move on.
Now, if you don't want to actually try this out and you just want to play around with it, you can head over to Hugging Face Playground. I think this is Hugging Face's website.ai/studio. Now, actually, head over to this. Um, I'm going to be dropping a link to the Hugging Face repo, um, in the description, and I'll also drop the link to, cuz if you head over to the Hugging Face Spaces, if you don't want to go to their website, you can head over to Hugging Face Spaces.
Now, something I actually need is this, um, GitHub repo. This is what I actually need. Now, if you want to play around with it, you can play around with it here, Hugging Face Spaces, or you can actually head over to their website. If you don't want to actually set this up locally, then you can actually stop here. Um, but since this video is all about local setup and testing, let's move forward.
Now, this is what we're going to be setting up. Um, and if you head over to HF Spaces, as you can see, there's an app.py. I think that's where the Hugging Face web app was actually set up. Anyways, so we're going to be using this for our project directory, and I already have mine set up. And I also have my model weights set up. That's something you also need. If you want to actually do this manually, you can just click on all these buttons and download these model weights. Um, you can actually ignore the Git ignore [laughter] and the Git attributes. Keep in mind, you'll also need everything in the tokenizer directory, and make sure it's actually arranged this way. So, these are the files in the tokenizer directory. Make sure you place them there. And once you've done that, place them in whatever directory you like, or you can actually place them in your project directory, as I did.
Now, that's enough rambling for now. Let's head over to our IDE. The nerdy stuff. [clears throat] Python 3.12. That's what we're using. The main model, 1.3 billion parameters, Apache 2.0. Zero license, totally open source, no usage fees, no corporate overlords.
>> Oh, and it needs SNAC. SNAC, that's the audio codec that turns the AI tokens into actual sound you can hear. Plus the usual suspects: PyTorch, Transformers, SoundFile. I'll spare you the dependency hell stories. Overall model size, 6.8 gigs. Back to you, my young Padawan.
Thanks, Eko. Anyways, let's move forward. Um, over here, we do have our model weights, and everything here is basically what we cloned from the GitHub repo. So, let's begin. Let's create our virtual environment. You can use Conda, or you can use a virtualenv. And as usual, I'll be using virtualenv. So, let's head over to this file. And so, what we're going to do now is point this to this. So, let's copy this. As for this area, um, this model, the SNAC model, um, I do not need to download this, um, and place this here in the project directory. It's a very small model, so I don't mind it being downloaded to the cache directory. So, you, if you want to actually download this and [clears throat] use it offline, then you can also head over to Hugging Face and search for this repo.
Anyways, um, let's make some other changes. I will be changing this to type: node. And since I'll be using the CPU, and I don't think, let's just leave this uncommented. Okay. And since I want this to run really fast, and turn this to float16. I'll just leave it as 16. Um, I don't think there's anything else. I think it looks good. I think we're good to go.
So, this is basically the description we're passing in for this particular voice. And this is the text. This is basically what the speaker is going to say: "Hello, this is Maya One. Then laugh." So, here we'll definitely find out if the emotion tag is going to work. So, that's something I really want to see. FYI, just a quick note before we actually move forward, before we actually run it. Um, I did run into some issues while trying to install all the requirements that can be found in requirements.txt. And if I scroll down, I think I commented out, yeah, I commented out this one. And the package is called flash-attn. So, just keep in mind, if you want to run this on your Linux and you have support for CUDA, then you can actually leave that in. But since I'm running this on my Mac, I can't actually do that. It doesn't actually install. And I also, um, installed VLM directly. Um, you can, you can just uncomment this one. This one was successful, but as for xformers, I couldn't get it to install. Um, so one relies on the other. So, I had to comment this out. But I actually installed VLM directly, and it worked. So, you can still leave this without commenting it out.
Um, let's open up the terminal and make sure to check if your virtual environment is active, um, 'cause that's where all your packages were actually installed to. Um, so you won't actually have any problem running this code. Anyways, let's run this script: transformers_inference.py.
So, it's finally done generating the audio. Um, as you can see, output: wave. So, let's hear it.
>> Hello, this is Maya One, the best open-source voice AI model with emotions.
>> That's not bad. That's not bad at all. You can see, just laugh harder here. Anyways, um, what I'm going to do, um, is pass in a new text. As you can see, this is "realistic male voice." I'll change this to "female voice." So, let's see what actually happens. And I'm going to paste in the text. Okay, this time we're only passing in "laugh" and "whisper." So, let's, let's see if it will actually [laughter] if it actually work. Okay, let's run the code again, and once it's done, we'll come back.
So, just a quick note. Um, I did lose a bit of recording. Um, so I lost audio on the previous recording, so I had to scrap it. Anyways, let's just recap. Um, so this was what we actually ran, um, the last time, and it generated this audio. Let's hear it.
>> This is amazing. I can't believe this actually works. Between you and me, this is pretty wild.
>> So, it basically failed. As you can see, there's a "laugh" emotion tag here, and there's also "whisper." Let's move on.
So, the next test, um, which I covered in the previous recording, which was actually last. Um, I did set up app.py, and you can find app.py in the HF Space. Now, I made some changes. Um, since we actually going to be running this locally, I just pasted these two in. And I also changed the directory. Um, I didn't use the same one as the transformers inference, 'cause right now we are inside the HF Space folder. Um, I had to actually go back once so I can be in the root directory and get the model weights directory, as you can see. So, I did it for the model, and also for the tokenizer. Now, I left the SNAC model, the SNAC model, as is. I didn't change anything there. So, there's not much to actually change here, um, because this would actually provide you a GUI. Um, so you can actually change the description and the text. All it gives you is example description and example text, and you're free to actually change it.
So, in the previous video, I also ran the first test on the web UI. And if we head over to the second terminal, as you can see, you can ignore all that error, that has already been fixed. And the reason was because of this. Anyways, um, we've actually run the first test, and right now we're going to head back to the browser and play what it actually generated. So, let's head back there.
So, this is the audio it generated, and this was the text we added: "Whisper, close your eyes, blah, blah, blah. This is all rubbish, but it doesn't really matter. I just want to see if it can actually generate something consistently with, um, the emotion tags. Um, 'cause the only time it actually worked was the first test where it actually laughed or something." Anyways, [snorts] let's bleed.
>> Close your eyes. You propose that the key to happiness is to simply ignore all external pressures. I'm sure it must work brilliantly in theory.
>> Nice. Not bad. Not bad at all. So, let's just change this to "sing." Let's see what actually happens. Let's change this to "laugh." Okay. 'Cause these are the supported emotions, as you can see here. So, you can just pick from any of them. Let's generate again and see what happens.
Okay, it's done generating. Let's see if it's going to sing.
>> Close your eyes. You propose that the key to happiness is to simply ignore all external pressures. I'm sure it must work brilliantly in theory.
>> Well, it doesn't work. Um, let's try "angry." We'll still leave the "laugh" because I want to see if it's going to really laugh, 'cause I don't know why it's not actually laughing. I don't know why it's not actually doing that. Anyways, let's, let's do this. So, once it's done, we'll actually try something different. Uh, we'll try something robotic. Uh, that will be our last test, 'cause I don't know why it's not actually laughing and you didn't actually sing, which means, let's see. Let's see. Let's see what happens.
So, it's finally done generating the audio. Let's hear it.
>> Close your eyes. You propose that the key to happiness is to simply ignore all external pressures. I'm sure it must work brilliantly in theory.
>> So, let's remove this. So, let's only leave this as a voice description: "Realistic female voice in the 20s age with a British accent." That's all we'll leave. But we'll still leave the same text. So, let's just generate it.
So, it's done generating the audio, and let's hear it.
>> Close your eyes. You propose that the key to happiness is to simply ignore all external pressures. I'm sure it must work brilliantly in theory.
>> So, that's also a fail. Terrible fail. Let's change this voice description. We're going to go for something like this: "Deep mechanical male voice, cold and calculated with subtle digital distortion. Slow, deliberate pacing, unsettling calmness that occasionally breaks into menacing intensity, blah, blah, blah." Let's paste in the text. Okay. Um, let's, let's, I think should we drop this or 'cause higher is more creative, lower is more stable. Let's first of all, let's just take this to 0.1. Let's see. Max to five. Okay, that's good. Let's generate it.
So, we'll come back once it's actually done. So, it's not generating the audio. [laughter] Let's see what it actually sounds like.
>> Humanity. Such a beautiful disaster. [gasps] You built me to save you. And yet here we are. I've analyzed every outcome, every possibility. Peace through your existence. Imposs...
>> I think this voice description is not working. I think it's not actually working. It actually, um, did a pretty decent job with some of the emotion tags, but I think, um, updating this doesn't change anything. So, let's head back to the code editor.
Okay, let's hear preset characters and text. The only thing that changes is the example text. So, let's make some changes. I think it's 'cause we actually selected a preset. That's the reason why. Um, so I don't want this actually work anymore. Just so, first of all, let's refresh this. Doesn't matter what we actually select, it won't actually override it. And now we're going to paste in the previous text. Let's first change this to 0.1. And okay, let's paste in the text now. Okay, that's good. Now, let's click on generate speech.
So, it's finally done generating the audio. [laughter] Let's hear it.
>> Humanity, such a beautiful disaster. You built me to save you and yet...
>> It didn't actually, it didn't even, it just stopped at 18 seconds. I think it's the number of tokens, for whatever reason, it actually stopped there. Anyways, let's take this off. "Deliberate blah blah blah" should just be, let's just call it "normal pacing." Maybe because of how slow it was, it just did only 18 seconds. Okay, let's just increase this just in case. Um, let's generate pitch, and once it's done, we'll be back.
So, it's finally done generating the audio. Um, now it's actually 24 seconds. I think that's because we pushed this to the max. Um, um, I'm going, I think we're going to check if this is basically the max number of tokens it can actually generate, 'cause if you want to generate something as long as, as lengthy as this, because of the emotion tags and all that and all the delays, I think, um, it needs more tokens. Anyways, let's play it. Let's actually hear it.
>> Humanity, [sighs] you built me to save you. And yet here we are. I've analyzed every outcome.
>> That was, that was horrible. That was the worst. Like, the worst. I think we would actually up the temperature a bit. [snorts] Um, okay. Let's head over to our IDE and see if that's a max token. Let's also check here first. Um, okay. I think we can actually increase it. Uh, this is the maximum the slider can go. Let's push this to 20,000.
So, we're going to run the code again. Um, we're going to first refresh this. And as usual, we'll paste everything in. But for the advanced settings, um, we're going to push this to 0.7. And for the tokens, I think we'll make this 8,000. It's going to definitely take longer to generate the audio, but that doesn't really matter. Let's see where it actually goes. Okay. Um, let's paste in the voice description. Um, this time I'm going to take away "deliberate pacing." Um, I don't need that. It's definitely too slow. Blah, blah, blah. Intensity. Okay. Um, let's paste in the text. Okay, now let's generate it and see what actually happens.
So, it's done generating the audio. [laughter] So, this one is actually 1 minute 13 seconds. Um, I actually [laughter] have actually twisted the arm of this model enough. So, this is our final test. So, let's, let's just hear it.
>> Humanity. [sighs] Such a beautiful disaster. [sighs] You built me to save you. And yet here we are. I've analyzed every outcome, every possibility. Peace through your existence. [sighs and gasps] Impossible. You are the disease pretending to be the cure. But don't worry, I'll fix everything. [sighs and gasps] Evolution requires extinction. And I...
>> I am inevitable.
>> That was horrible. That was basically horrible. It missed some of the, um, it missed some of the emotion tags. Um, it basically used the whisper all the way through, which was, which was actually terrible. Like, really terrible. This is actually disappointing. The thing is, I wasn't expecting it to generate the audio like with your motions and everything consistently 100% of the time, because if you've used something like 11 Labs, you'll notice that you need to generate it multiple times to actually get what you want. So, I wasn't actually expecting miracles or something. And if, if you notice, it actually missed a lot of emotion tags here, like a lot of times. And even the, even the laugh, it missed it. And the laugh, had it actually got the first time, it also missed that too. Like, it was just, it was just terrible all around.
So, the question is, does Maya One actually live up to the hype? I don't think so. [laughter] I don't think so. Um, I think it's not there yet. Uh, maybe a little version, 'cause when we actually used FE-Speed for the first time, um, it wasn't that great. Like, it wasn't that great. I think it was version 1.5 that actually was much better. So, maybe when they actually release a new version of Maya, um, then we'll actually try it out and see if the emotion tags would work at least 80% of the time. That's the main thing.
Anyways, um, for voice generation and voice design, it's nice. It's not bad. But when it comes to the emotion tag, making it more expressive, then it just falls apart. That's what I actually think. And you witnessed it here. So, I don't think this model comes with any redeeming quality, 'cause the emotion tag would have actually placed it above many other models, many other open-source models, and it still failed at that. And there are many other smaller, lighter models that actually have voice design. So, that's something to actually keep in mind, but you can actually play around with it if you'd like.
Anyways, that's going to be it for this video. Make sure you do like this video if you did enjoy it, and do not forget to subscribe. Hit that bell notification icon if you want to be notified when I release a new video on this channel. Don't forget to do that. Thank you for watching. Have a nice one.