Transcription
Hi everybody, and welcome to a new exciting video in the audio signal processing for machine learning series. Last time, we extracted the discrete Fourier transform with Python and Librosa from a bunch of audio files. This time, we're back to theory; specifically, we'll be addressing a key topic in AI audio: the short-time Fourier transform.
So why is the short-time Fourier transform so important? Well, that's because it enables us to extract spectrograms, and spectrograms are really the—probably the—most important feature that you can feed to deep learning audio models. But before we get into the ins and outs of the short-time Fourier transform, I want to remind you once again about the Sound of AI Slack community. On this community, you'll find people with interests in AI, music, audio, audio digital signal processing, and so if you're there, you can ask for feedback, you can share your projects, and network with a bunch of very cool people. So if you're interested in joining, I'll leave you the sign-up link in the uh description section below.
Okay, now on to the real cool stuff. So before we get to the STFT, I want to remind you about the discrete Fourier transform, and here we have its mathematical formulation. Now, I'm not gonna get too much into the details here, that's just because I have a whole video on the discrete Fourier transform, so if you're interested, you can go and check that out. But what we need here is the high-level intuition. So we start with our signal in the time domain—so a waveform like this—then we apply the discrete Fourier transform, and what we get back is basically a picture of the presence of the different frequency components in the original signal, and usually we get like a magnitude spectrum like this. But this is a still image. What do I mean by that? Well, it's a still image in the sense that it only provides us like one picture that averages the presence of the frequency components across the whole duration of the signal. And here we actually have a problem because we know what frequency components are present in a signal, but we don't know when they are more or less present because all of them are averaged across the whole duration of the entire signal. And this is a little bit of a problem because we know that with audio data, it's all about the evolution of frequency components like over time, and so audio data is very, very dynamic, and we want to know how these different frequency components evolve over time. And this is the whole point of the short-time Fourier transform—so moving from a still image to a video that provides us information about the um different frequency components across time.
So how can we do that? Well, that's the whole point of the short-time Fourier transform, and the high-level idea here is that we don't perform the Fourier transform across the whole duration of the signal, but rather we consider small segments or chunks of the signal, which technically we call frames, and then we apply a discrete Fourier transform for each frame. Now, I know this could sound a little bit like abstract, so let's visualize this. So we start with a uh signal, audio signal like this, then we consider only like the first chunk, the first frame, and at this point, on only on this—the samples belonging to this frame—we apply the discrete Fourier transform, and we get back like our nice magnitude spectrum. Then we slide on to the next frame, and once again we apply to that frame the discrete Fourier transform. Third frame, same thing, until we pass through all the duration of the signal. One way we can use to derive these segments is through windowing; in other words, we apply a window function to a signal. What does that mean? Well, it means that we take the original signal and then we multiply that by a window function sample by sample, and we obtain a windowed signal. Now this feels a little bit abstract, doesn't it? So let me give you an example. So we start from an audio signal, and then here we're going to be applying a rectangle windowing function, and this is the result. So the rectangular window function is this—a red curve here—and yeah, it has like a rectangle shape, right? And this function is zero everywhere apart from a segment where it is equal to one. So if we multiply the signal with the rectangle window, we obtain this windowed uh signal down here.
Now uh I want to introduce a couple of parameters that are very important for what we are discussing today and now. So one is the window size; the other one is the frame size. They're both measured in number of samples, but they refer to two slightly different things. So let's take a look at the window size first. So the window size is basically the amount of samples we apply windowing to. The frame size, on the other hand, is the kind of like the—the number of samples that we consider in each chunk of the signal when we segment the—the signal, and then we pass it to the—the short-time period transform for just like calculating the Fourier transform for each frame, for each um segment. Okay, usually the window size and the frame size coincide; they are—they have the same value, the same number of samples, but sometimes it happens that the frame size is larger than the window size. Now this is like quite unusual, I would say, and most of the time, in most of your applications, the window size and the frame size will coincide, and this is like so—so true that, for example, in Librosa, when we um extract the short-time Fourier transform, we are not uh forced to pass the window size, and the default value for the window size is the frame size, right? Okay, but what happens if the window size is smaller than the frame size? Well, still, we—the chunk so we—we apply the Fourier transform to is the whole frame, but then the windowing happens only on uh the window size number of uh samples, right? And we apply the window function on those samples, and then the remaining samples, which are the difference between the frame size and the window size, are going to be zero-padded. Okay, but for the sake of this video, we'll assume that we have the window size which is equal to the frame size. So if that happens, what this is at this point is already the windowed—so we—we have like one frame here, and we've also applied the window function here, and at this point is when we apply the uh Fourier, discrete Fourier transform so that we can get uh the frequency components out of this frame. Then we move on, we slide to the right, and we get like a second frame like this, sliding also like the window function, and here once again we apply the discrete Fourier transform. We move to the third frame, same thing, until we get to the end of the signal. But this is a simplified um version; what usually happens in a short-time Fourier transform because the frames are overlapping like this, so the second frame is overlapping with the first one, as you can see from here.
Now I need to introduce another parameter here that's called hop size or capital H, and it's given by this visually, and this basically provides us—it tells us how many samples we slide to the right when we take a new uh frame. Okay, if you want to know why um the hop size is like—it's so important and we need overlapping frames, I really suggest you to go check out my video on audio feature extraction pipelines; it's up here, and there you'll learn about a lot of topics about, for example, like spectral leakage and a bunch of other things that are related to like the points that I'm making here, and I'm not gonna get into the details here because I've already done that.
Okay, moving on, it's time to move from this kind of like visual intuition of the short-hand Fourier transform to its mathematical formulation. So what I want to do here, and don't be scared, is to compare the digital Fourier transform or its mathematical form, which is this formula in the top, with the formulation for—mathematical formulation for the short-time Fourier transform uh which is this one down here. Okay, so let's go uh item by item and see how they map to each other. Okay, the first one is just like the definition, right? The output that we get. So in the case of the discrete Fourier transform, and we get like this x hat as a function of k, where k is a proxy for a frequency, and um so in other words, like uh the discrete Fourier transform depends on the frequency, and this means that uh each of these like formulas is gonna give us a complex Fourier coefficient for the kf frequency, and the complex Fourier coefficient provides us information about two parameters: the phase and the magnitude. Now, if you want to really know what phase and magnitude mean in terms of the Fourier transform, I suggest you to go check out this video on uh the Fourier transform, and there like you'll get like a better picture, but I hope like you've already watched that. Okay, so this is for the discrete Fourier transform. What about the short-time Fourier transform? As we can see here, capital S depends not only on k, on frequency, but also on m. So what's this m? Well, m is a proxy for time, so that the short-time Fourier transform depends both on frequency and time. So now let's understand a little bit better like what this n is and uh like nominally uh it's just like the—the frame—the frame number we are currently in, but let's visualize this. Okay, so here we have like—we are back again with our um original signal and like the different frames here, and so for the first frame we have m which is equal to one; here we have for the second frame m equal to two, and m equals three; you get the idea, right? So m is just like the frame number. So in other words, uh the result that we get from the short-time Fourier transform is the Fourier coefficient for the kf frequency at the nth temporal uh bin or nth frame. Okay, but still like the—the Fourier coefficient that we get, it's still like a complex number that has information about phase and um uh magnitude.
Okay, moving on. So the next step is to compare these two sums. If you guys remember from the DFT um so what was happening here is we were summing all across like all the uh samples, so basically we were summing across like all the time, all the duration of the signal, and we do like something similar also in the STFT, kind of like intuitively we are doing the same thing; we are summing like across time, or given like we are in a discrete uh domain, across like all the different samples. But what's different between these two sums is capital N. In the case of the discrete Fourier transform, we are summing across all of the samples in the signal, so n is equal to all the samples in the signal. In the case of the short-time Fourier transform, capital N over here is equal to the frame size, and that's because we're not considering all of the signal but just one frame, and in one frame we have a number of samples that's equal to the frame size by definition, right? So now you start to get the idea of how like this STFT works, but to see this like even more um specifically, we need to move to the next element of this uh formulas, which is the signal itself. So let's analyze the one in the top formula. So for the discrete Fourier transform, x of n is just like the signal considered—I mean the whole signal, so all of these samples. Uh in the short-time Fourier transform, by contrast, we are only considering the signal that's uh present like in the current frame. So in other words, we're considering all the samples that are present in the current m frame. And so uh why is that the case? Well, that's the case because first of all, this m multiplied by capital H is the starting sample of the current frame, because m is the current frame and h—capital H—is the hop size. So if you multiply two, you realize that this is like the starting sample of the current frame, and then we add n, but this n moves like from zero to uh the frame size uh minus one, which basically means like that we are kind of like covering all the samples in that frame. Okay, now if we want to visualize this, we can just go back to the signal, and here like we have like this rectangle, and here we have like all of the—at the signal like for one uh frame uh and here like on the left of this rectangle, this vertical uh line here is the starting sample of the frame, which is equal to m multiplied by capital H. Okay, so now um we—the next step though, in the case of the short-time Fourier transform, is that we should multiply this signal by the uh windowing function, and we do that with this uh like representation here, so we have like the signal where the signal for one frame, and we multiply that by the windowing function, and again, so we already saw this, right? And so we are multiplying the original uh signal like for a specific frame by the windowing function, and we obtain the windowed um signal okay for that one frame.
Okay, moving on, we have the last step, and the last step is the same for both the discrete Fourier transform and the short-time Fourier transform; in other words, we are multiplying by a pure tone that has frequency given by k divided by capital N, and so by doing so, what we are doing is we are taking the uh the signal uh and then we are decomposing it and projecting it onto uh the pure tone with frequency um k divided by uh capital N. Okay, so here you have the comparison between the math behind the discrete Fourier transform and the math for the short-time Fourier transform. But now you may be wondering, okay, now more or less like I get like the math here, but what are like the outputs? So what do we get out of a DFT and an STFT? Let's take a look at that. Okay, so for the DFT, we extract uh a spectral vector, and which uh for a number of like frequency bins; in other words, like we—we get a Fourier coefficient for each of the frequency components we've decomposed our original signal into, and this is a one-dimensional array; it's just like a vector, right? And there's no mention of time in here because everything like is averaged across the whole duration of a—of a signal. But with the STFT, we have like something that's quite different. In this case, we don't have a one-dimensional uh value, one-dimensional array, but rather a two-dimensional array, or in other words, a spectral matrix that has a number of frequency bins and a number of frames, and in other words, we—we get a complex Fourier coefficient for each frequency bin that we are considering for each frame. Okay, and so in other words, we have both uh reference—a reference to frequency as we had with the discrete Fourier transform, but now we've gained also information about time through the different frames, which are proxies for time. Okay, but um you may be wondering, but can we calculate the actual number of frequency bins and number of frames that we get out of an STFT? Well, yes, of course we can, and we'll do that.
Okay, so how do we get the number of frequency bins? Well, this is quite easy to get, and uh you have the formula here. So you get the frame size, you divide the frame size by two, and then you add up one, and this gives us the number of frequency bins. Now let's try to understand why this is the case. If you guys remember from my earlier video on the discrete Fourier transform, you should know that the number of frequency bins that we get out of a discrete Fourier transform is equal to the number of samples that we have in the—in the—on the whole signal. Now, in the case of an STFT, we don't uh average—we don't consider the whole samples at once, but rather like a frame size number of samples, so we would expect that the number of frequency bins uh for each Fourier transform that we get is equal to uh the frame size, but we don't get that; we get the frame size divided by two plus one. Was that the case? Well, if you remember once again from the discrete Fourier transform video, we saw that the discrete Fourier transform is symmetrical—has a mirror symmetry around the center frequency, which is the Nyquist frequency—and what happens there is that's basically like the—the first half has some information, and then that gets like mirrored in the second half. And so in a short-time Fourier transform, we—we considered that, and so we don't need to take information about all of those bins because it's just like redundancy; we're only taking from the information about like the first half—so frame size divided by two plus one. So that's the reason why. Now, if you haven't followed along uh completely, I highly suggest you to go check out my video on the discrete Fourier transform to understand what it meant more specifically there.
Okay, now let's move on to the number of frames, and so here we have another very nice little formula, and the number of frames is given by the total number of samples that we have in a signal minus the frame size divided by the hop size plus one. Now I'm not gonna get into the details of explaining this visually, and I highly suggest you as an exercise to play around with this and understand why this formula gives us the number of frames. Okay, but I know this can feel a little bit abstract, so let's go move on with an example. So here we have like a bunch of like STFT parameters, and we want to find the actual output shape. So we have a signal with 10,000 samples; we have a frame size which is equal to 1,000 samples, and we have a hop size of 500 samples. So for the number of frequency bins, so we take the frame size, we divided it by two, we add one, and we get 501 frequency bins. Okay, but these are frequency bins, and we know that they divide a certain frequency range um equally, and so we have like a frequency range that's divided in 501 bins in this case. Uh now what—what is that range? Well, that range—and the frequency range—is between 0 Hertz and the sampling rate divided by 2 Hertz, and that is the Nyquist frequency once again. So if you want to know why that's the case, once again, just go back to my video on discrete Fourier transform. Okay, so moving on, the number of frames, so here we have like a little formula, and so we have to take the number of samples in the signal, so 10,000 minus the—the frame size, and this is going to be divided by the hop size, and all of this we have to—all of this we have to add one, and the result is 19. So we have 19 frames this signal is going to be divided into. So the overall uh output shape of the STFT in this particular case is going to be 501 and 19. So it's a two-dimensional array; the first dimension uh provides us information about frequency, the second uh provides us information about the temporal bins or the number of frames.
Okay, so now uh I think like we should take a look at the short-time Fourier transform and try to understand the different parameters. So the important thing that you should understand here is that really the short-time Fourier transform depends on a bunch of parameters that we pass. So depending on the parameters that we pass, we're going to get an output that's going to be different. So one of these parameters, and we've already encountered it, is the frame size, or in other words, how big are the chunks we divide our original signal into, and this is measured in frames, and the usual values that we have here are like—like this—like 512, 1024, 892. As you can see, these are power of two numbers, and as we already discussed in a previous video, it's important that the frame size is a power of two number, and that's because um with that specific number we can use the fast Fourier transform to calculate the discrete Fourier transform, which is a very quick and computationally efficient um way of extracting the discrete Fourier transform. Now there's an interesting aspect uh in when we choose the frame size, and it's called the time-frequency trade-off. So if we get like a larger—a large frame size, what usually—what happens is that the frequency resolution is going to uh increase, and the time resolution uh is going to be degraded. Um so was that the
Case well, so we know that if we uh, uh, enlarge the frame size, so we take more samples, we're going to be having like more frequency bins. And so if you have more frequency bins, it means that your frequency resolution overall improves. But if you take more, more samples in one frame size, it means that you are taking like a larger, you're considering a larger chunk of time, because samples like our proxies for, for time, okay?
And in other words, if you're taking like, like a larger chunk of time, it means that the time resolution goes down, right? And uh, the opposite is also true. In other words, if you take a smaller frame size, then the frequency resolution is going to go down. And that's because you're going to have like a, a smaller number of frequency bins as output, but you're going to have a higher, a better time resolution, just because you're considering less samples, which equates like to, uh, like a less amount of time. And so you're gonna be calculating like the, the, the Fourier transform like on smaller, uh, chunks of time, so your time resolution is going to be better. Now this is like a time-frequency trade-off, as you can see here. So when, when you try to improve the frequency resolution, then the time resolution is going to go down, and vice versa.
Now how do we solve that? Well, we don't really solve that; we just have like some heuristics. Most of the time you want to find a value of the frame size that's okay, and it's a good trade-off between frequency and time resolution. But this really depends on the type of application that your, um, or problem that you are interested in. So certain problems, for certain problems, it's more important that you have a higher frequency resolution, and in that case you should take like a bigger frame size. Uh, for other applications, like for example onset detection, you're not really super interested in the frequency, frequency resolution; perhaps you're more interested in just like having like a very precise or like high, highly resolved, or like time, so that you a very good like time resolution, so that you can really know what happens like at each point in time, okay? So I think like this is like very, uh, important to keep in mind when you decide like which frame size like to, to take, because I mean the two things, frequency and time resolution, are related together and inversely related in a sense, right?
Okay, so now, um, let's move on to the next short-time Fourier transform parameter, and that's the hop size. We already saw that multiple times in this video and in earlier videos, and we know that's the number of samples that we slide to the right when we want to take a new frame. So usual values here, once again, 256, 512, I mean all power of twos most of the time, and we can also define this as a fraction of the frame size, so a half of the frame size or a fourth or an eighth of the frame size. So you have like both definitions, absolute and relative, okay?
Now moving on, uh, a third very important parameter is the windowing function. Obviously, the short-time Fourier transform is not only like a function of the signal itself, but it's a function also of the windowing function that we choose, because different windowing functions are going to, uh, kind of like modulate the original signal in different manners, and then this is going to have like an effect on the short-time Fourier transform results, okay? So we introduced the rectangle window function, but that's not really used at all in, um, in digital signal processing, and that's because like it creates, uh, discontinuities like on the edges. Like all of the windows, rather, to avoid those, you want to use like a bell-shaped curve, one of which the most important probably is the Han window. So 90% of the time, probably you're going to be using the Han window when you perform a short-time Fourier transform, perhaps without even knowing that. And so this, uh, function is given like by this formula here, which is obviously like a periodic, uh, formula, a periodic like function over here, and here like you have visualization of this. So, um, so let's see this like in action. So here we have like a signal, here we have like our bell-shaped Han window. So when we apply the Han window to the signal, you see that the signal gets modulated, and towards the end, the, um, the values of the, of the samples tend to get squashed, right, towards zero, so that we avoid discontinuities on the edges. Once again, if you want to know why that is so important, you should go check out my video on audio feature extraction pipelines, where I talk about spectral leakage, okay?
So now let's move on to the final topic of this, uh, video, and this is like what you probably came here for, and that's the spectrogram. So through the spectrogram, we can visualize sound. So but how do we get to the spectrograms? Because up until now, we know that we have the short-time Fourier transform, and that's a matrix that has like complex numbers or Fourier coefficients for each item in the matrix. So what we do is we take the squared magnitude of the short-time Fourier transform, and what we get is a matrix which has the same shape as the original short-time Fourier transform, but the difference is that now we have, uh, all, all of the items are not complex numbers anymore, but they are real numbers, and now we can visualize them using a heat map, and the visualization is called a spectrogram. And this is, I mean this is like so important for all applications in AI audio, because like so many times we're going to be using spectrograms as features that we feed into the algorithms.
So now let's take a look at the spectrogram here. So on the x-axis we have time, then these are like discrete times, and you can see it here that you have like this tiny like discontinuities, and these are like all the frames, all the temporal bins. And on the y-axis we have, uh, frequency, with all of the different frequency bins. And so what we are seeing here is how the different frequency bins, how the different frequency components evolve all the time across the different frames that we have in the original signal. And so, and now this is actually the dream that we wanted to come true. So now not only we have information about the frequency components, which was something that we already had with the spectrum, the magnitude spectrum, but we also have information about the components evolving over time, which is the information that we usually get from the time domain. And this is why a spectrogram is called a time-frequency representation, and this is why spectrograms are so important in AI audio.
Now I'm not going to get into the details of the implementation and all of these things like, uh, for spectrograms, because that's the, um, topic of the next video. So in the next video we're going to be using Python and Librosa specifically for extracting spectrograms; we're going to be looking into different flavors of spectrograms and understand which ones to use; and then we're going to be examining like different audio samples and comparing them, perhaps like different musical genres, and how like their spectrograms differ, okay? So I hope that you found, uh, this video instructive and useful. If that's the case, please consider leaving a like, and if you haven't subscribed yet to the channel and you want to see more videos like this, please consider subscribing. So if you have any questions, please leave them in the comment section below. I think that's all for today; I'll see you next time. Cheers