Diarization, Voice and Turn Detection

This is an introduction to turn detection and diarization. Turn detection is where you want to know when a speaker is finished so that you can respond to them, possibly with an AI voice. Diarization is relevant to transcription, where you're going from speech to text and you want to identify which speaker is speaking when.

I'm going to show you how to prototype the common libraries for these two types of application. I'll go through Smart Turn from PipCat for turn detection, and then I'll go through the Pyannote and the Nvidia NeMo libraries for doing diarization. First of all, let me explain to you these two applications. I'll go through a few slides before then going through some scripts in the Advanced Transcription repo.

Turn detection, as I said, is primarily for real-time voice applications. It's so you can tell when to respond with an AI. Diarization, meanwhile, this is attributing the speech to various speakers: speaker one, speaker two, etc. Now I'll start by describing turn detection and then move to diarization.

The challenge with turn detection is often in handling long pauses, in handling certain tones or words that indicate the speaker's not finished. For example, I might say "uh" or "and," and you might think if you look at a pause afterwards that I'm finished speaking, but I'm not. I'm thinking, and I wouldn't want to be interrupted by an AI that interjects too soon. And likewise, there might be certain tones or certain words that indicate I'm finished. If I'm very emphatic with an answer and say, "Yes, that's right," that might be an indicator for the AI to respond immediately without waiting for much of a pause. And this is the big question with turn detection: how do you tell if a speaker is finished with high confidence?

The first step in turn detection is typically to identify whether a given sound is speech or non-speech, and this is done via voice activation detection, for example, using a common model like uh, Cerebro here. And quite simply, this looks at small frames of about 30 milliseconds and gives a probability on whether that 30 milliseconds is speech or non-speech.

Now, once you know something is speech, you can then decide whether the speaker is finished or not. The naive way you could do that is just to look at pauses. For example, if the speaker has got a multi-second pause, you might just say they're finished speaking and move to respond with an AI. Of course, that's not entirely robust, as we said, the speaker might be thinking. In which case, you need a more elaborate way of detecting the speaker being finished, maybe via a neural net. And that's what the Smart Turn project from PipCat is doing. It aims to detect completion based on the previous few words that the speaker has said.

The voice activation detection model is typically very small and it's very fast because it looks at this narrow portion of the speech. But that also means it doesn't have full context on all of that speech, so it's hard for it to tell, "Oh, there was a phrase that's likely incomplete, and therefore it's unfinished." This is difficult with voice activation detection because it just looks at such narrow windows. So the goal of Smart Turn is to use a larger model that looks at a bigger window and uses a more complicated neural net to try and tell whether the speaker is finished. Right now, the model is quite large, it's about 2.3 gigabytes. This is unfortunately a bit too large for speech applications, meaning that it's quite slow to respond, but I expect the model will be pruned or simple architectures will be found over time to improve performance.

The Smart Turn project, which I'll link below in the description, is from PipCat and it allows you to put in an input of audio and get an output of a probability of the speech being complete or incomplete. It's built using a Wave2Vec 2.0 BERT model. Now, Wave2Vec 2.0 converts a soundwave into a vector, and by further adding BERT, it transforms this vector into a space that is also used for word embeddings. BERT is a model that converts from words into two vectors that represent meaning. So by doing Wave2Vec 2.0 and then Vec to BERT, you're basically taking sound and putting it into the same subspace that would normally be used for the output of word embedding models.

This model is not trained from scratch by PipCat. It's actually taken from a Facebook project that was trained on a lot of audio data. What Smart Turn does is add in two more layers, uh, which I think are quite small. I've put in less than one megabyte, maybe it's a little bigger, but they're very small layers that convert this model to being a classifier and output a probability. I should say that the reason, the motivation for putting in this BERT part here is because, you know, well, when you finish or you're not finished a sentence, that's probably related to the words you use. For example, if you say "uh" or "and," these are words that indicate a speaker is not finished. And because we have words, we need some kind of transformer that is familiar with words and not just sounds. Whereas a voice activation detection model is looking at these tiny windows of sound, whether it's speech or noise, it doesn't have that context to say, "Oh, these words indicate we may not be finished speaking." So that's the motivation for going for a bigger model that includes this BERT portion here.

It's instructive if we take a look at the data that this model was trained on. Now, I mean the post-training data, because the main Wave2Vec 2.0 BERT model is already pre-trained. The post-training uses both synthetic and human data of complete and incomplete speech. You can see here about 4,000 training rows of the humans split of this data. So here we play a short snippet: "Could you explain how memories are stored?" And you can hear there's a little "and" at the end, and that's why this is marked here in the dataset as false for being complete. Whereas if we look at a complete section of voice, let's try this one: "The printer is like, um, like, uh, it's out of sync." I guess here the phrase ends with "I guess," but that probably means the speaker is finished. So this is marked as being true in terms of completeness. So you can imagine this full pipeline being trained now and generating a model or providing a model that allows you to classify audio inputs as either complete or incomplete.

I'll run through some examples where we'll pressure test the performance of this model, but first I'm going to describe diarization because there's just a lot of overlap between the model components and also the applications of diarization and turn detection.

The challenges in diarization are particularly acute when you have speech that's overlapping, so two speakers talking at once, when you have very noisy speech, and then when you have speech with even more than two speakers, or even worse, multiple speakers where each speaker is only saying something very short. The problem with short utterances is that often we may say them with a different kind of tone, like if I say "here, here," then the tone used for normal speaking, and that makes it quite hard to match short sequences with given speakers, apart from the fact that the shorter the speaker, the shorter the utterance, the less data you have anyway in order to make a match. So we'll test out these diarization models with some quite challenging overlapping and short speaker utterances.

The basic pipeline, and let me expand my screen, for doing diarization is first to do voice detection to tell whether you have speech or non-speech. Then you take little windows where voice is detected and combine them into voice segments. And this is the segmentation step, which I'm describing here. Doing a rules-based approach, you could also use a neural net, which we'll see in a second, but the basic approach is to just detect speech or non-speech for tiny windows and combine them using rules to identify speech segments. Then you take each segment and you convert it into a vector representing its style. So now you have a style vector for each of your segments, and you can cluster those to identify the number of speakers and then attribute each segment to a given speaker.

If you want to extend on that, rather than, before I explain how to extend, just note that the voice activation detection model is quite small. The embeddings models can be a bit larger, maybe 25 to 100 megabytes, and overall this pipeline is still fairly fast.

A slight improvement here, and this is what Pyannote does in its pipeline, is it uses, rather than a voice activation detection model, will use a dedicated segmentation model. And this is to split the raw speech into segments of speech. It does things that are a little bit different in a few ways. First of all, while voice activation detection just looks at very small snippets, 30 milliseconds, the segmentation model called PySegment in the Pyannote library, this looks at many seconds at a time, so it's got the full context of more speech. And this is helpful if you want to detect segments, whereas of course, just looking at very small windows then requires rules to come up with ways to have segments.

Now, not only is it looking at a wider span of speech, it's also a more complicated model. So whereas a model like Cerebro for voice activation, it has frequency transform, it will extract features in the frequency space, and it will use a small LSTM, which is a recursive model with memory elements, and then a decoder to come back out to the probability estimate for speech or non-speech. By contrast, the segmentation model will have more layers. It uses a bidirectional LSTM, so it will process the speech left to right, accumulating memory, and then right to left, and use both of those pieces of information. Furthermore, because it's more complex, it's able to handle actually up to three speakers and tell you even if there's overlap. So you can think of this as taking in more context with a larger model in order to give better quality speech segments.

So other than that, the approach with Pyannote is the same. It will use a segmentation model, then, and it will use clustering to identify which speaker is speaking in each voice segment. To show you in more detail here, you start with a voice wave. Here you would then convert it into windows, which would be speech or non-speech, and that will allow you to get segments, either done during using that approach with the VAD or using the segmentation model. And once you have segments, you get an embedding, which represents the style of each of those segments, and then you do clustering. So here I've got two embeddings drawn in the same direction, so I'm clustering those as part of the same turn, and I'm clustering this other embedding in a different direction as the second speaker. So that's Pyannote.

There's also a library available from Nvidia called Nvidia NeMo. I'll show you that in the scripts demo, and I'll describe it here. It's different in a few ways. First of all, it uses different default models, although you can swap those models. By default, it uses MarbleNet for VAD, which is a bit smaller and faster, and uses TitanNet for the embeddings, which is larger and probably therefore slower. It also uses rule-based segmentation, so it does not use a segmentation model in order to detect speech segments. But primarily where it differs is in two ways: first is multiscale embeddings, which I'll talk about in a second, and second of all, it doesn't just use clustering to refine the turns, it also uses a neural net to refine the turns and detect overlap.

So let me describe how this pairwise neural diarizer works, and then I'll describe how this multiscale idea can improve performance of embeddings. Now, just note again the model sizes. We have still a small voice activation detection model. The default embedding model is larger, and this refinement module here, which is in neural nets, it's also quite small at 5 megabytes and should be fairly fast.

So how, let me first describe how the pairwise neural diarizer works. It takes in the embeddings that we create for the speech segments. Here I've got some embeddings, let's say for speaker one in green, and then in yellow, and for speaker two. And this multiscale diarizer will output a value for each window for each speaker. So here on the top in green, I've got the first speaker, and it's saying, well, in this window, it's basically guaranteed the speaker is present, the speaker is present 80%, 30%, and then the speaker is not present here. And for speaker two, in the early windows, speaker two is not present, 60% chance of being present, and then definitely present in the later frames here. And you can see by providing outputs for each of the speakers, this allows us to define windows where there is overlap. For example, here there is likely some overlap because we've high likelihood presence of speaker one and of speaker two here.

Now, this is just when you pass in pairs. So you pass in the embeddings of two speakers, but you can generalize this and pass in the embeddings of three speakers or four speakers, and the model will handle them pairwise and then average across those results. So yes, if you have more than two speakers, this pairwise neural diarizer will still work because it will handle them pairwise and aggregate results, which allows you in principle to identify highly overlapping speech. In practice, this is still extremely challenging, and it's just difficult to get good results when you have overlap, and we'll see that in practice.

One tip that you can use, and that Nvidia NeMo uses, is multiscale embeddings. And the idea here is that speech often has different scales. You may have sentences that span multiple seconds, but you may have very short phrases. And the problem with just creating an embedding for each segment, which is what we do in the Pyannote model, is that it's just not granular enough because there might be information from smaller words and there might be information from larger turns. So the one approach to improving on that is to use multiscale embeddings, which just means for each segment of speech, you split it into, say, half-second windows, but you also split it into one and a half second windows, or two and a half second windows, or scales of windows at maybe four different scales. And you calculate the embeddings at each of those scales, and you actually run the diarizer at each of those scales. So you will run it at this half-second scale, like this here, you'll take in the half-second scale embeddings, you calculate the probabilities, and you do the same for each scale. And then once you've got the presence of the speaker for each scale of embeddings, you can combine them back in. For example, if you want to know the chance of a speaker being present in this small window here, you will consider the result of the diarizer for this small window, but then you will add in a weighting for the result from this larger window of which the smaller window is part.

The default is to weight each of these scales equally, and this in principle should allow you to capture information from the different scales of the voice. Now, whether it gives you an improvement in practice, we'll see that when we compare the scripts.

So this is the overview of the main models: the turn detection from Smart Turn by PipCat, then the Pyannote pipeline, and last of all, Nvidia NeMo. Next, we're going to run scripts for each of those with some challenging examples to see how our models, see how the different approaches perform. And what you should think of doing is creating some sound snippets for your application that are of representative difficulty and running those scripts through the same kind of playbook that I show you here to understand which models are doing well and where the gaps are. And from there, you can think about where you need to improve your data or your models, and even doing some fine-tuning on the models that I'll do in a later video.

For this part of the video, I'm going to run through some scripts and examples. I'll go through Pyannote and I'll go through Nvidia NeMo. I'll also go through Smart Turn from PipCat, and I'm going to do this in the Advanced Transcription repo. This is available if you go to trus.com/advanced-transcription. It's a repo that now contains scripts for doing everything from transcription to voice cloning, speech-to-speech models, and now diarization and turn detection. It includes fine-tuning for all of these, and in the future, I hope to cover some fine-tuning so you can improve performance of diarization and turn detection.

Now, I should mention, yes, this is a paid repo. It's the business model that supports the Trus Research channel, but if you're not in a position to buy this, I'll describe in enough detail you should be able to follow along by yourself by building similar scripts, and I'll provide links to the key open-source repos that I make use of.

I've cloned Advanced Transcription, and you can see here the full contents, which now includes a section for voice detection. If you go to the voice detection folder, now you should see a README here that lays out all of the contents, and this has both the turn detection and the diarization. So we're going to work through, first of all, turn detection, first voice detection, then turn detection, then we'll go through Pyannote and that full pipeline, and then we'll go through the Nvidia NeMo model, and that it's actually in a dedicated folder here that I've called NeMo, along with a specific README for going through NeMo.

The first thing we do is we'll understand how voice activation detection works using this Cerebro VAD model. So we're going to download the model and we're going to then pass through a voice snippet that I'm going to record right now. So if I scroll down here to the Quick Test, I want to do some of the installation first. I'll move into the voice detection folder like this here, and I'm going to use `uv` to set up a virtual environment and install some of these dependencies here that we're going to need.

Now, the first thing I'll do is record a simple sound. So let's just run this script here, which is importing `sounddevice` and `soundfile`, and it's getting me to record just for a period of 10 seconds. You can see the duration is set here, and then it's going to write that to a file called `conversation.wav`. So on this here: "Hello, please do a recording of 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12." Okay, so I have now this recording here: "Hello, please do a recording of one, two, three, four, five." So that's me recording myself, and what it should detect is that there's actually me speaking briefly at the start here, then a pause, and then there's basically me speaking for the rest.

So let's see if we run a simple script on voice detection. Let's see if it's able to detect that. And I can show you what the script is doing. It's simply loading this Cerebro VAD model from the `snakers4` repo on GitHub. It's moving the model to the device, which is going to be CUDA if you had a GPU, but I'm using Mac, so I'm going to set it to MPS. If you're not on Mac, you should just swap this to CPU. CPU, and it will run fine, it's a pretty small model. Then it's going to get the speech timestamps, load the audio from the file I just recorded. Here it's just setting up, get speech timestamps via a utils function, and then it's going to get the speech timestamps by running `get_speech_timestamps`, and then it'll print out each of those timestamps here along with the, well, it's just going to print the timestamps.

So let's run this. So you can see we're downloading the model, which makes sense. And now it has calculated the segments. So roughly speaking, yeah, it looks like it's found a very small snippet of speech at the start. It thinks there's a pause then till about 2 seconds when there's more speech, and then a very brief pause before there's some more speech detected again. We can maybe double check our conversation here: "Hello, please do a recording of one, two, three." Yeah, so it's detecting a pause after I say the word "uh," very brief, and then it's noting that there's speech then basically till the end of that conversation. So this is voice detection, and yeah, it's not just doing voice detection on the frames. We're also applying some rules in order to combine little sequences of voice into full segments. So this, you can think of as the naive way to do voice segment detection.

The next thing we're going to do is take a look at PipCat. So what PipCat does is it takes in segments of voice that have already been detected as coherent segments, and then it will give us a probability of that voice being complete or incomplete. So there are a few updates we need to make sure we've done here around PortAudio, and then, yeah, we'll install PortAudio using Homebrew. So, sorry, these are the instructions if you're using Ubuntu, and these are instructions if you're using Mac. Then we will clone the repo, the Smart Turn repo, and move into that repo and install the requirements.

So let's go ahead and do that. Smart Turn already exists because I've run this before, so that makes sense. We just make sure we're in the Smart Turn repo, which we are, and then I'm going to install the requirements and run a small script that's called `record_and_predict`.

The `record_and_predict` script, which we can take a look at if we just go here to Smart Turn and we look at `record_and_predict`. So here's the function. So it's going to listen for speech, record a small snippet of speech, and then it's going to process the speech segment. And let's see what that function does. So the `process_speech_segment` is going to extract the audio segments, and the `record_and_predict` function here is going to allow us to record some speech, then it's going to process that speech segment, and within the `process_speech_segment` function, it's going to call on `predict`, which is here, `predict_endpoint`, which will tell us whether the prediction is either going to be complete or incomplete.

So let's take a look at running this script here: `uvicorn run record_and_predict`. And I'll expand this up here just so we can see. So I'm hitting some issues with the installation, and the solution I'm going to use is to install Python 3.9. So I've created a virtual environment with Python 3.9, and let's see if the installation runs smoothly now. And with that installed, I'm going to run `record_and_predict`, and it's going to take about 30 seconds here because we need to download the models and then load them into the Mac.

Okay, so we can see that the voice activation detection model is loading. That's going to allow the windows to detect speech or non-speech and then create the voice segments. And it's actually recorded me there now. So let's see. It thinks it's incomplete, which I guess is a good guess because I was still speaking. I'm not sure. Well, the thing about that is, so you can see I've recorded a few examples there, and it actually did fairly well. So I gave an example of speech where I was definitely not finished because I said "uh, hmm." The next example, I was definitely not finished. This last example here, I was also not finished, but it thinks my speech was incomplete. Now, if you read through the Smart Turn documentation, so far it's been trained on a number of keywords that typically indicate incompleteness, and it's doing quite well on those, but it's not a general model in the sense that it can just go by tone and for arbitrary choices of words where the user is likely not complete because of the tone that they're speaking with. It's going to find it harder on some of those. So most likely, if you want this to work for your use case, you're going to need to fine-tune the model. Furthermore, this model, as I said earlier, it is kind of large, so it isn't the fastest at doing the turn detection, so you'd have to factor in whether or not that's going to be an issue.

So that's an indication of how turn detection works and how to install it. I'm going to move now through the diarization example, starting with Pyannote and then moving to the Nvidia approach. Now, because the pipeline for diarization does a few things, I'm going to run through, I'm going to run a script for different parts so you can see how the parts of the pipeline work before you can see how it all runs together. So here we go, first with the segmentation model. This model here is an alternative to using just voice activation detection. So let's see how this model does if we run it. First of all, we're going to load the segmentation model and then we're going to save the model info to file. Actually, I've already run that already. So if you want to check out here within the segmentation, let's see `segmentation_model_info`, you can see what this model looks like. It's got `synnet` at the start, which is taking in the wave, it's got an LSTM that's bidirectional, and then it's got some linear layers and then a classifier head on the top here. So this is the segmentation model, which you can just print out if you want to inspect it, and you can read more here in the README about the more complex architecture compared to the voice activation detection model.

Now, we do a quick test here, and I'll make sure that I've installed all the requirements. Let me just `cd` back backwards from the Smart Turn folder, ensure that I've run all of these requirements, and again, I can record some voice and save it as `conversation.wav`. Now, what I'm going to do is I'm going to use a prior conversation that I've stored, and it's one where I have two different voices. I took them from, I think it was VoxCeleb, one of the open-source datasets on Hugging Face, and the dataset I have is this one here. I'm going to start with a noisy non-overlapping voice. I'll just play you this sample here: "Let's break the problem down. The old cinemas behind the mall were demolished for park." So this is two speakers. The first has got a very deep voice and says, "Let's break the problem down." The second one then says something about the mall. So we have a pause, a speaker, a pause, a speaker. So this is kind of noisy, but the speakers are distinct, so they're not overlapping, and it's not too challenging of a little test.

So I'm going to just copy this and rename it as `conversation`. I can delete my `conversation` here and let's rename this here as `conversation` so that it can be used. And now we'll be able to run our segmentation model on it. So let's, let's first actually, from comparison, just see what the VAD on its own with the rule set does. So we can scroll all the way back up to the VAD model, which is this one. It's going to run a lot faster now because we've already downloaded it. So let's quickly run just the VAD with the rule set. We're running it on the `conversation`. Let's see if it picks up the two speakers, a segment for each. And yes, it does. So it gets the first segment from 2.5 to 4, and then from 6 to 10.

Now let's run the segmentation model, which is a more elaborate way of detecting segments, and we can do that by running the script here, which is, sorry, that's for recording the test audio. We want to run the actual segmentation. So here we go. And yeah, keep in mind the pipeline we're using here, it's being loaded from the segmentation model, which you can find on Hugging Face. It's under `pyannote/segmentation-3.0`. So we run that script now, and let's see if we get we get similar timestamps as the previous. So the timestamps we're expecting here are going to be roughly from 2.5 seconds to 4, and then 6 to 10. So let's see what we get. The model's currently downloading. Yeah, so the speech segments we get are very similar. Instead of 2.5 to 4.4, and then 6.2, I think it's incredibly similar. The only difference is 6.2 versus 6.3. So the voice activation detection is starting a tiny bit later with detecting the second portion of voice, but basically they're performing very similar on this test here.

Now, this segmentation model can also be used to detect overlap. So for that, let's feed in some voice that is a little bit more challenging. So here what I've done is I've recorded, I've actually taken the recording I just played, which is the one with the two separate speakers, and I've recorded myself on top of that, just saying something. So let's listen to my overlapping voice segment here: "Hello, my name is Ronan, and I come from County Kildare in Ireland. It's nice to talk to you. The old cinemas behind the mall..." So I'm speaking, I'm saying, "Hello, my name is Ronan McGavin, I'm from County Kildare. It's nice to speak to you," and that is overlapping fully with the first speaker. It's not overlapping with the second speaker.

So let's now take this piece of speech and let's delete the `conversation` we have and replace it now. Let's rename it here just as `conversation`. Now, of course, you could just pass in a different file name when you're running the command, but I'm just going to run it like this for now. So let's run the Pyannote model using the overlap detection, which is possible using the segmentation model. So here we're going to run, and you can see it's detecting an overlap from 4.5 to 5.1 seconds. Now, if we listen to the speech, that's not exactly right. If we listen here: "It's nice to talk to you. The old..." So it's getting some of the overlap, but it should be getting overlap from 2.5 seconds, "and I come from Ireland. It's nice to talk to you." So it's finding part of the overlap between the first speaker and myself, but it's not finding the full portion of that overlap, which would really be from 2.5 seconds up to a little bit higher than that, it should be from 2.5 to 4.4 because that's when the first speaker is speaking, and I overlap entirely with that speaker. So you can see this is not exactly right here.

Now we can move then to more sophisticated overlap detection when we go to the Nvidia NeMo model, but that's just to show you that if you do a pure, a pure segmentation model, which is the 5 megabyte model, it's not going to accurately be able to identify these speaker overlaps.

Now we'll move to doing Pyannote diarization. So here what I want to get back is an indication of which speaker is speaking when. And for this, we're going to run the script. I've provided an `inspect_pyannote` script that can be run here, and this simply allows you to print what the Pyannote model looks like. We can do that by going to the `pyannote_model_info` folder, we can check `pipeline.txt`, and you can see that the pipeline has got a few things. It's got a segmentation model, which is used to get the segments. It's got an embedding model here that's used to provide a vector for each of the embeddings, and then it's got a clustering approach that's used to cluster those embeddings. So this is how the Pyannote segmentation model works. It's a bit better in principle than just voice activation detection because it's got a segmentation model, which is a bit more elaborate and looks at larger input snippets, but then it uses embeddings and clustering of those embeddings to find which speaker is speaking when.

So let's try and run. You can read more about the architecture here if you like, but we're going to try and run this. So I'm going to make sure we've installed all of the requirements for Pyannote, and I need to make sure that I'm logged into Hugging Face or that I've got a token, because some of these need to have the user conditions accepted. Now, I think I am logged in in this terminal, so I don't need to go get the token. So I'm ready to record a recording if I want and then run that recording.

Now, rather than record something new, I'm just going to run this script directly here, and I'm going to run it on `conversation.wav`, and actually I'm going to run it this time on the noisy non-overlapping, just to see how it does. If we have no overlap, it should be able to identify the speakers. So let's run the full pipeline here, and it's going to convert into speech segments, get the embeddings, and then attribute the embeddings or those windows to different speakers.

So here you can see it's outputting speaker 0, 1 for 2.5 to 4 seconds, and then speaker zero, a different speaker for 6.2 to 10. Notice that it's not telling us one speaker is this person or not because we're not inputting any reference sets of embeddings. We're just finding out which speakers are different, essentially, and how many speakers there are. If you wanted to go further and match those, say speaker one with a given speaker, you would need to compare the embeddings of a given speaker from some out-of-data, out-of-sample data with the specific samples that are in the data. So here it's correctly identifying each of the speakers. This is pretty good, and it's performing well because we've got, even though it's noisy, we've got speech that's not overlapping.

Now, if you want to take an example of some overlapping speech, we can run that as well, and you can see here it's identifying a first speaker going from 1.6 to 6.5 seconds, that's myself, and then it's identifying a second speaker going from 4.5 to 5.1, which is probably part of speaker one, and then it's identifying speaker two, it's identifying that same speaker then speaking from 7.2 seconds to 10. So this is just all jumbled up. In reality, I am speaking for most of the recording, and then there's speaker one who's speaking towards the start, and speaker two towards the end. So you can see that pipeline here is not effectively able to get the overlapping speakers in this case.

Now, there's one more example that I can show which is even more difficult, but it's not able to get it. I'll just play the recording. It's the same two speakers speaking separated, but with me then speaking over them with very short words. I'm actually counting, and this is very difficult because there's a lot of pauses between my different words, and this makes it hard for the model to attribute whether this beat should be me or whether it should be one of the other speakers. So I'll just play this noisy sporadic overlapping speech. It's kind of the most difficult, or one of the most difficult you can think of. I guess it'd be more difficult if you had even more than three speakers, but let's listen to this here: "One, two, three, four. The old cinemas behind the mall were demolished." I've even put in a cough at the start, which the model does well in getting rid of, but because I'm sporadically talking with the counting above the other two speakers, it just makes it very difficult for to be able to distinguish between those speakers.

So I've shown you here the capabilities of the Pyannote pipeline, and basically it does well when you have separation between speakers and when you have reasonably long chunks of speech. When you move to multiple speakers, even though in principle the segmentation model is able to detect different speakers, and even though in principle using the embeddings you might be able to detect different speakers overlapping, this is very difficult in practice. So we'll move now and take a look at the NeMo model and see how that performs in some of these snippets using its multiscale embeddings, which should help with granularity, and using the multiscale diarizer, which should allow for better recording of the overlaps.

Now we're going to run NeMo and we'll do diarization using the Nvidia NeMo model. I've put the `.ipynb` files within the NeMo folder here, and there's a README that goes along with it that explains how to get started, and I'll explain that in a moment. But basically, we're going to run a notebook that you can get online. It's available here. I'll put the link in the description. So we're going to run this `.ipynb` notebook, and to do that, we first need to have some installs. If you're on Ubuntu, you'll want to run `apt-get` and make the `libsocks` and `ffmpeg` installs. On Mac OS, you can use Homebrew, and then set up a virtual environment using Python 3.10, install `ipykernel`, set the kernel to have to use the virtual environment and give it a name. Then optionally, and then you'll be able to open up your notebook. You can run that by doing by using Jupyter Lab or Jupyter, and make sure to select the kernel. So I'm going to select the Python 3.10 kernel, and now we're up and running.

Now, there are a few things to note about how this script is set up. It's set up in order to allow you to separate the performance of different modules, and that's best explained by looking at this chart of how NeMo works, which is a variation of the one, the simpler one I showed earlier. Basically, NeMo will take in speech, it will use a voice activity detection model, MarbleNet, as I said earlier, in order to detect speech or non-speech. It will then segment that using rules, not using a neural net. It will then extract embeddings for each of the speech segments. It will actually use a multiscale approach, as I described earlier, and then do close clustering in order to identify how many speakers and which segments are attributed to which speaker. And it will take those based on the clustering, it will then pass in pairwise sets of embeddings for pairs of speakers to the neural diarizer, which will then output the final speaker labels.

Additionally, you can decide to use an ASR model, that's a speech-to-text model, like say Whisper, in order to provide timestamp information that can improve a little bit your speaker labels, but generally this is not going to solve problems with overlapping speakers or say very short turns. That's still going to be challenging if you turn on your Whisper model here to help out. So I'm not going to turn on the assistance of the ASR. I'm going to just run the main pipeline here.

Now, I said this script is set up in order to isolate the performance of different parts of the system, and the specific parts it tries to isolate are the voice detection model from the performance of the embedding and the neural diarizer. And the way it does that is by allowing us to run these scripts in two modes. In the first mode, rather than using voice activation to provide to provide segment labels, we provide ground truths. So we literally say, listen to the audio and assign what the ground truth start of each speaker segment is and what the end is, and we use those ground truths in order to then determine which speaker is speaking when. So you can think of diarization as kind of a two-part problem: one is identifying what are the speech segments, and two is identifying who's speaking during those segments. And this way, if we provide ground truths for the segments, we're just testing the performance of attribution, attribution of speakers to segments. Whereas if we run the full pipeline, we're testing two things at once: the combined performance of the segmentation part, which is the voice activation detection and segmentation, and second of all, the performance of the speaker identification.

So, yeah, when you run this script, as I said, there are those two ways of running it: with Oracle VAD, Oracle means ground truth voice activation detection, which literally means you input what you know are the correct timestamps for segments, and then you can run with system VAD, which they mean here using the NeMo VAD models, namely Cerebro VAD, that was for Pyannote, or here the MarbleNet model. So we're going to run through this script fairly fast. We're going to run through it using the simple separate speakers that are recorded in 10 seconds, and then we'll do it with me talking over them in the overlap and see how the model performs there.

So I'm going to move down fairly fast, just doing Shift+Enter to run the different cells. I will stop at a few different points so that we can see the different configurations that we're printing out. Now, the script allows you to download a sample piece of recording, sample recording. It's from an Amazon server. Rather than downloading, I've just pasted in the sample that we recorded earlier, which is this one here: "Let's break the pr down." So this is the sample earlier, and I've put it in the data folder with the name `diarization_test`. I know that's a bit misleading. I'm kind of hacking this by just putting my file in place of what the download file would be, but it just allows you a simple way to kind of playground the results you would get.

Next, we're going to set up the ability to listen to the audio. So this is simply going to allow us to inspect the data that we're going to run. If I play this, it will just play the exact snippet, and here you can see a nice waveform of what we recorded. So you can see a lot of noise here, then at 2 and a half seconds, you have the first deep voice, the man here, and then the second man here, slightly, well, normal or less deep, more average, say pitch voice, and this here goes right till the end of the clip here. So you can see clear separation between the speakers, which is going to help get good performance.

So next, what we're going to need is the ground truth, which is stored in an RTTM. This is a Rich Text Form of transcript, sorry, Rich Text Form of timestamps, and it's located in the same folder here. So when I've got the WAV file, I've got the RTTM, and here I have speaker, and I've put in basically the timestamps, which from 2.5 seconds to 1.5, and then the timestamps for the second speaker, 6.25 to 3.75. And notice here how there's an "A" and then there's a "B." So this denotes different speakers. Now, the NAs here typically they would be filled in with the transcript itself, like the text, and also some metadata around whether it's a man, woman, young man, young woman. So that's a standardized way of encoding this information around timestamps and turns and transcripts for a given sound. But since we're not looking at doing the transcription, we're just interested in the timestamps, we've got NAs here in multiple cases.

So let's just save this file and we can move on. So here we've got, we're reading in basically this RTTM file and printing it out, and this allows us a way to inspect what that looks like. And when I say what that looks like, I mean the timestamps of the speakers. So here on the X-axis you have time, sorry, it's probably a bit difficult to see, and on then along the time we're showing which speaker is speaking when. So here you can see at about 2 and a half seconds, we have speaker one speaking to about 4 and a quarter, and then at about 6 and a quarter, we have the second speaker speaking right to the end here. So this is a visual representation of the speaker timestamps.

Continuing on here, we're going to run, we're going to run diarization using the reference timestamps, the ground truth or the Oracle. And for that, we're going to need to have a model configuration file, and the configuration file, which basically sets some of the thresholds for diarization, is going to be this.

dire INF for telephonic um file. So this is a readymade config that's available, uh, from GitHub. You can, this is for a telephone conversation between two participants, so it will be optimized for that. Actually, sorry, it's for more than two participants. I think it should work too. You can also use a more general, uh, template that's for more general conversations, but we'll just go with the default, which is this, um, telephonic conversation. And when you run that script, it should be downloaded. So we should be able to see here, uh, what that file looks like, as this one here. And you can see indeed, there are just a lot of parameters here, not just around sample rate, but also around some of these hyper parameters that deal with kind of thresholds for overlap and detection of the separate speakers here. We've, I think, just printed out, uh, what that configuration looks like, and we should be ready to run diarization now.

There's one note I want to make here, which is a modification to the Python notebook, and that's swapping here the config number of workers, uh, to zero from one. And this will prevent, if you're getting any multiprocessing or pickling issues, uh, you should be able to solve it in that way. So here we're going to do some final configuration before we run the model. We'll make sure that we're taking as input, uh, a JSON file. We can take a look at the JSON. This is a configuration file that gives the audio file path we want to pass in, and it also gives, uh, some of the high-level hyper parameters like number of speakers, um, and also, I think, the RTM file path, which is the ground truth labels we need in order to run then the diarization. So we're going to update this to make sure it takes in the right parameters. We're going to use Titan at large, about 100 megabyte, uh, embeddings model. Um, we're going to read in the JSON file and make these updates to it. We're also going to define, um, the scales for the embeddings. So here you can see we're going to calculate windows all the way from 1.5 seconds to 0.5 seconds, and we're going to define the shifts. So the windows aren't just, um, they aren't just straight across, as I showed in the PowerPoint. You actually use a sliding window type approach, and you define, uh, the slide of that window here with the shift. Lastly, here we've got weights. So this defines the weighting of each of the scales. We're going to use equal weight, which is the default. And here, because we're using reference, uh, ground truth data for the timestamps, we'll set this Oracle VAD to true. This, as I said earlier, should be set to zero if you're facing issues with, uh, with pickling.

Okay, now that we've the configuration set, we can now start to run diarization. And we're actually going to run diarization in two ways. Sorry to add to the layers here, but we're going to run it first just using clustering, and then we're going to use at the MSDD. Let me maybe show with the diagram again up at the top, uh, so we're going to run once. And in both cases here, we're using the reference, the ground truth for the segmentation. We're going to run once where we just do clustering. We won't use the multiscale dizer. And then we'll run again, but we will use the neural net. So let's go back down to where we've set the configuration, and we're now going to move to run, uh, this speak diarization with the Oracle VAD. That's correct. We've set the configuration, and we're going to run now with clustering dizer. This is just clustering without using the multiscale dizer. We're not using a neural net here. So let's run this cell here, and let's diarize. And notice that the, not well, yeah, notice that the diarization will be done at five different scales. So we will do clustering at five different scales, uh, that I've defined in the configuration. And you can see the results, uh, will come out here. And all we're seeing now, if we print out, uh, the results, this is the results of our dizer, and this is the ground truth. All we're seeing is that the model has correctly identified that there are two speakers. So there are two different colors here. We can see the first speaker in green, the second in purple, and that matches the fact that there are two speakers in the ground truth data, which one is red and one is blue. And by the way, it's not impressive that the exact timestamps are the same here, because we're using ground truths for the start and end of segments. What we are finding out as new information by running this model is whether there are one or two or more speakers, and it identifies there are two speakers via the two different colors.

Now, this is done with just clustering, but we can also run this using the neural dizer, which you might think we get F better fine-grain boundaries. So let's run that now. And this will, uh, make sure we use the same path here for the configurations. We do need to set some thresholds and some new thresholds that relate to the neural dizer. If you set this, um, high to a value of one, um, you can basically block out any overlapping speech if you want to ignore overlapping speech. Or if you lower the threshold, you can allow for overlapping speech to be detected. So we are allowing for overlapping speech here with, um, T equals 7. And actually, we run it again, then blocking that overlapping speech by setting it equal to one. So let's now run the neural dizer, which also will be run at each of the five scales that we've defined. And if you want to look at the architecture of the neural dizer, you can print it out here. You can see that the model has got a pre-processor that goes to a Mel spectrogram, then there's the MSDD module, which includes an LSTM and as con some convolutional layers, and this model here. And running this, then, if we move down to here and run Di, should again run at each of the five scales. So, yeah, you can see the embeddings being computed at each of the five scales, and then we should have, uh, the output results. So if we scroll all the way down here, we can print out, uh, the results, and we can print out the ground truth. And easier is just if we actually print, uh, we show the graphs. So again, we'll take a look at the graphs, and the neural dizer also is identifying their two different speakers. And again, again, it's not impressive that the timestamps are correct because we're using the Oracle ground truth. So nothing particularly, uh, of note here so far, but we'll put the, the model to more of a test now by getting it to actually compute the segmentation using the voice activation detection model.

So we'll move through running the exact same scripts. The only difference here is that we're going to set, uh, to true the parameter that, um, defines whether we use an Oracle VAD or not. So we'll cycle through this here, and sorry, we're going to set it to false because we're not using an Oracle VAD. We're going to use MarbleNet this time as the pre-trained VAD, and we're going to use TitanNet large for the embeddings. We used that also when we use the Oracle VAD. And with that, um, we should be able to set our configuration this time with the Oracle VAD equals to false, which means we're not using ground truth. We're not cheating, if you like. And now we can rerun this using, um, the clustering dizer, which is solely using embeddings clustering, and then we can run it again using, um, the neural net. So we're just running now the five scales for the clustering dizer, and we can hopefully see those results now shortly. So here we can actually get a detailed print out. And again, I know my screen is quite small. Let me try and increase this. We get quite a bit more detailed information here. So what you see is that voice signal are recording. You can see the first voice, the second voice. You can see in green. Now, it's hard to see. Green is the speech probability from the VAD. So if this is above zero, it's more likely than not speech. If it's below, then it's more likely than not noise or or silence. So the green has being used to identify the segments, and then by kind of averaging or combining each of the frames of the green, the prediction on voice versus noise, we're then able to make a prediction on whether there's actually going to be a speech segment. And then in red, you can see the label, which is the ground truth. And in blue, you can see the prediction. So the red and the blue seem to overlap fairly well, but there's two mistakes actually. And these mistakes are not present if we go back to the Pyote pipeline. The first mistake is that the label, which is the correct value, is correctly capturing all of this sound here, which is the speaker, but actually the model is cutting the speaker off early. So it's truncating end of the first speaker and what he's saying. And in the second case here of the speaker, the model is truncating because you see the blue goes down here, the model is truncating part of the speech just in the middle of the second speaker's turn. So there's, yeah, there's a portion of about half a second that's just being cropped out because the model thinks that that portion is, uh, probably noise. So what it seems, uh, from the performance here is that the, the, the voice activation detection combined with the segmentation, um, is not not particularly accurate, and that's resulting in us losing some of the speech when it comes to doing, uh, the segmentation, and then ultimately the speaker identification.

So here are the predicted output values. You can see that it detects the first speaker, and the first speaker is going from, uh, 2.4 seconds for a duration of 1.5. And sorry, if I misexplained that earlier on the RTM file, it's not the start and end time, it's the start time followed by the duration. So the first speaker starts at about 2.4 for 1.5 seconds. Then, um, the next speaker, speaker zero, starts at 6 for 2.8, and then restarts again at 9.4 for 0.5 seconds. And yeah, this is the gap. There's basically a gap here that the model thinks there is in the speaker, but that's incorrect. There isn't really a gap. And furthermore, here on the first speaker, this first speaker is really speaking for more than 1.5 seconds, but the model thinks it's just 1.5 seconds. Uh, so that also is an error in this case.

Now we can take a look, uh, then at the clustering dizer result. So this is the visual form. It thinks there's a first speaker. It's truning it early. And when you look at the visual form, you can see it thinks there's a gap here in speech, but there actually isn't a gap. Uh, so this is incorrect. So this is a different visualization, the same, uh, different visualization of what I've said here above. Now, if we look at the ground truth speaker, this is what it actually should be recording. It should just be cleanly recording the first speaker, but starting, but finishing at 4 and a quarter seconds, not finishing here. And the second speaker should just be straight through. So this is the ground truth correct.

Now we can run, uh, the neural dizer and see if this, uh, has any difference. Unfortunately, because the segmentation is poor, it's, uh, going to be restricted in a similar way. So when we print out, and it's still going to run here, it's got to run on the five layers of embeddings. When we print out the results, um, which should be present momentarily, you can see it's also detecting that the first speaker is ending early, and, um, there is a gap in the second speaker. But yeah, this is to do with the segmentation model. The fact that we use a neural dizer is only going to improve detection of which speaker is speaking, and basically both the clustering and the neural approach have got this correct. There are two speakers. The problem here really is in our segmentation approach. So if you wanted best performance, maybe you should take the segmentation approach from Panote and then combine it potentially with the speaker detection from Nemo here. Again, we can see in graphical form, um, the same result, which is not surprising because it's is the fault of the segmentation model. And then the same result here in this case, rer, this is the ground truth, which it should be, uh, contrasting with the result from the segmentation plus neural dizer. So basically, the Nemo model here is weaker than Panote, uh, for the case of of the non-overlapping speakers.

Let's now put it to the test with overlapping speakers. So I'm going to take this, uh, conversation, the noisy and overlapping, and I'm going to just move it into the folder where we have the data. So let's move it into the data folder for Nemo. And actually, I didn't mean to move it, I want to copy it. So I'll copy it here and we'll delete this file, or rather, just rename it first and replace, uh, the noisy overlapping here. So now we're in a position to rerun all of our scripts, but this time we're going to rerun them using, um, this noisy overlapping speech. So I'm going to run everything again, but using this sample. I'm not going to redownload because I don't want to accidentally overwrite. So I'm not going to download, uh, this baseline, this baseline sound from Amazon, but I am going to run the rest of the scripts. You can see how this looks now. Whereas previously there were just two kind of clean portions of speech, you can now see me speaking overlaid with the first speaker here. And we now need to adjust the RTTM file because we need to add in me as the third speaker. And when we add me in, I'm going to add me in as going from about 1.5 seconds to about, let's say, 6.3. And here we have the RTTM file, and I have a third speaker now. So I need to put in a new RLE. And because the third speaker, which is me, starts off first within the recording, which is me right here around 1 point, probably 1.5 seconds, and then let's say 1.3, talking all the way until about 6.3, because I am the first to speak, I need to put that in as the first row here. So 1.3, and I'll put that in for a duration of 5 seconds. So the first number is the starting point, and the second number is the duration. And I'm now speaker A. So that means this has to be speaker B, and this is speaker C. By the way, A4 is just, um, is just the file name. So I've now got my RTTM ground truth data here. And now I'll go back to my Python Jupyter notebook, and I can run these cells. I'll just check that the TM file is indeed what I typed, and it is. And now I can visualize that RTTM file, and this looks good. You can see here my voice followed by Speaker 1, and followed by Speaker 2. There's a tiny bit of overlap here, and then there's full overlap for this, uh, for speaker with me.

So we'll move now to using the ground truths to do the embedding-based embeddings-based diarization, the clustering-based, and then we'll do the neural net-based. So first, we'll create a JSON file to set up our configuration. So we'll just run this here. It defines the input file for the manifest.json. And now we'll do that diarization using the Oracle data. And again, we're going to use the dire in for telephonic, so the settings for telephonic multispeaker conversation, that's set. And we're going to do some configuration now. So we have defined our output directory, the manifest JSON input file, the embeddings model here. We'll have, uh, five different scales that will be equal weighted, and we're going to use the Oracle, uh, dizer. Now, there's one extra setting here, which is we're going to provide as Oracle the number of speakers. So this is again helping, not again, but well, again, yeah, it's helping the model to know how many speakers there are, rather than it trying to determine from the embeddings. You can also run this is false, which is the default setting here, but in that case, it will need to use the embeddings to figure out how many speakers there are, and that's actually quite difficult. In fact, we can just run it with false so you can see how that looks. And if we run with false here, we'll get it to diarize, and let's see how many, uh, speakers it comes up with. So here you can see, if we don't feed in the number of speakers, if we don't say how many clusters there should be of the embeddings, it thinks there are now seven different, different speakers. Uh, so clearly the performance is, is pretty difficult. Now, it's definitely a drawback because if you have to put in how many speakers there are, that means you need to know in advance, and in some applications you will know, and some you won't, so it won't be an option. But if it is an option, I definitely recommend specifying how many speakers there are and passing in that Oracle value.

Now let's try that again, but we'll run it this time without, um, a Roder, with passing in the Oracle amount of speakers. So we're going to pass in that there are, uh, three speakers. And by the way, I did define earlier up here, I don't think I showed it, but I did swap the number of speakers, or I passed by quickly. I, I defined it as three. So this is the ground truth number of speakers. And with that, we can again run here and see if we can improve performance by passing in the Oracle amount of speakers. Okay, so using embeddings with the Oracle amount of speakers, uh, we have a detection of three speakers, uh, which is not surprising, cuz we passed that in. Now notice that when we do the clustering approach, we're not able to detect overlaps. We're only able to detect overlaps in the multiscale diarization model, the pairwise diarization approach. So we're going to run that next, and we'll see how it goes. So I'll keep going here. Notice that by default, um, this script will run two different thresholds for detecting overlap. If you set the threshold to one, it will basically set the threshold to full for overlap, so none of the overlaps will be detected. If you want to detect overlaps, you need to lower this from a value of one. So we're going to run it at 7, which is the other suggested value, which should allow for some overlap detection. So we'll set that, and now we'll run the dizer, and we will inspect and see if it's able to detect some overlaps now.

So here we've got the results using the neural dizer, using the ground truth timestamps, and using the ground truth number of speakers. And you can see it does detect some timestamps, although I wouldn't say this is particularly neat. It's detecting that there are multiple, uh, speech fragments here, some of which are overlapping, both at the start and here at the end of this, uh, second speaker here. But broadly speaking, the layout of the speakers is not correct. There is a long portion for this third speaker, but the first speaker here is quite short. This is me speaking, and then there's only one speaker in the middle when there should still be two speakers here in overlap.

Now I'll go through the details of how the accuracy is reported. That's shown when you run the diarization itself. There's a long output block that you can view, uh, in text editor form. That's going to be the easiest way to look at it. And the results are actually calculated in three different ways. The first way, which is, which I'll show at the bottom here, is the strictest comparison. And here it is directly comparing the results to what the ground truth is. And really, that's just comparing the diagrams here, so this diagram with this one. And if it's not matching exactly, it's going to penalize it. So when you do that comparison for which you would have to get the overlaps correct, the current accuracy is, uh, 30, 34% error rate. And what this, uh, block of results is saying, there's 10 seconds of audio here. I don't know why it's 10.5 actually, but you have 34% of that time is actually an error in terms of how you calculate, uh, the diarization. And the breakdown of that error is primarily, it's primarily due to two things. One is confusion, where it's thinking it should be one speaker, but it's actually the other, and that accounts for 20 or 18% of the error. And the other problem here is, um, a MISD detection, so there should be voice, but it thinks that it's noise, and that's 14% of the error. And sorry, there's also a small error contributing here for 2 seconds, or 1.9% of the time due to false alarm, which means the model thinks there's somebody speaking, but it's actually noise. So it's the opposite of the MIS detection. So you'll see this breakdown here. And the settings for this breakdown, which is the strictest, are a collar of 0 seconds, which basically says whatever time science stamps I've specified in RTTM, they are to be considered the ground truth timestamps, and there's no allowance for these being slightly inaccurate. If you want a less strict comparison, you can allow for a small collar, so you can allow for 0.25, uh, seconds of inaccuracy in the RTM timestamps. And because that's less strict, you'll see that the error rate is slightly lower, cuz it's 29 instead of, uh, 34, cuz we're just ignoring basically 0.25 around the ends of the RTTM ground truths. Now, if you want to make it even less strict, or if you want to evaluate models that are not capable of doing overlap, then you can ignore the overlapping portion and evaluate on that basis. In which case, generally, the error rate will be again lower. In principle, it could be higher or lower because when you cut out pieces of overlap, you may end up having a higher lower percentage error, but ignoring the overlap is going to be a less strict way of evaluating the performance, and it's relevant in the case of maybe testing just pure embedding clustering approaches where you're not able to detect overlap. So that's a quick explanation, and ideally, we'd like it to be zero. So we'd like the diarization error to be zero here instead of 34%.

So let's move on now from the case where we've done, um, ground truth-based predictions, and let's now look at the case where we use the system voice activation detection. So here we'll just print out the configuration again, and we will make sure to select the VAD model and also the embeddings model. And now for the configuration again, we're going to allow for passing in the number of speakers to make the results, uh, a bit stronger, but we will not use an Oracle for the timestamps around the segments. Let's first run, uh, the clustering dizer. So we'll run the same scripts, and we can inspect the performance and compare the graphs. So here, when we run this, first of all, we can see if we look at the green line, this is the voice activation detection prediction of when there's, uh, speech versus non-speech. It's cutting out a little bit of speech here around the 5-second mark. It's correctly getting silence here, but then it's also cutting out chunks of speech here from the final speaker. So that's going to be source of error. Now, if we compare what the output is of diarization, you can see the model predicts, uh, three speakers. So we have, and this is clustering, remember, so it can't find overlap. It's kind of broadly doing an okay job. We can look at the performance, and the performance should be measured up here in terms of, uh, actually, it's only been measured ignoring the overlap and with a collar, which is one of the less strict metrics, but makes sense because the clustering approach, we can't anyway detect overlaps, and it's got a 36% error rate, which is primarily attributed to missing some spots where there's voice, and we saw that visually, I pointed it out, and then some confusion among the speakers. But of course, this won't be good enough for overlaps because clustering is not capable of the detecting those overlaps.

So we'll move to use the neural dizer. We'll again use the threshold of 7. You could play around with lowering that to make it, um, more sensitive to overlaps, and we'll again run the results here and see how they compare. So now we have the results, and we do see some overlap, um, here where there is indeed overlap between between two speakers. But it seems like there is confusion between the the same speaker. In fact, this portion here of the deep man's voice is kind of roughly overlapping with this portion here of speech, but then the model is missing out that I'm speaking all the way through here, and the model is also dropping out pieces, which is due to the segmentation algorithm. It's dropping out pieces of the final speaker here. We can check those results. Let's look in the text editor. We'll check the strictest results with no, uh, collar, which allows for error in RTTM and no overlap. You know, probably it's a good idea to allow for some error in the collar definition, so you could maybe look at these intermediate results. The error rate is still high, though, 35%. I mean, as we can see visually, it's not a great diarization, and this is due to a lot of MISD detection, particularly with the third speaker, and then there's confusion as well around which speaker, uh, should be speaking at a given time.

So I know this isn't showing you how to solve the problem of multiple speakers, but hopefully it helps you to set your expectations around the baseline performance before you've done any tuning on these models. I have also put this test, which is quite difficult, this little snippet of voice. I've put it into, uh, OpenAI's real-time voice. I've actually just played it live so that the model responds back to me. I've also put it in as some voice to, uh, Google Gemini. And basically, both of those also ignore, uh, the background speaker that has the short, uh, deep voice. And my guess as to why that is, is simply because the models, the real-time voice models are probably not trained on a lot of overlapping speech. They're probably mostly trained on single voice speech. So there isn't really a way for it to pull out the fact that there are multiple speakers when you transcribe something like this using Whisper, and you're going to see it if you look at the time St, or if you look at the, the captions here on this video now, you'll see that it will typically just transcribe one line of voice. It won't transcribe, uh, any underlying voice. The kind of deeper, quieter voice typically won't be captured because it's just going to pick out, it thinks there can only be one voice if that's the data that it's been trained on.

So ultimately, my recommendation here, if you're starting off new on real-time voice, or if you're starting off in diarization, is to run all of the pipelines like this in a playground type manner so that you can get out results and a benchmark of a baseline rouer of where performance is. And from there, you can start to think about which parts of my pipeline are weak and why, and what can I do to improve those parts parts of the pipeline, either by using stronger models, like testing different embedding models, for example, or eventually by doing some data preparation and fine-tuning, which I hope to cover in a later video. That is it for this video. I'll put some of the key links below in the description, and you can get the scripts at tr.com Advanced transcription. Let me know as usual any questions below in the comments. Cheers.