Transcription
This is the first lecture of a two-lecture series on source separation methods that are used in neuroscience. This first lecture corresponds to temporal source separation, so we're going to be talking about source separation methods that involve time series data. And in the next lecture, I will focus more on spatial source separation methods.
So let's first start with a kind of generic and high-level discussion of source separation and what source separation means in neuroscience. I'm going to make the claim here that basically all of neuroscience is source separation, or I should say all of neuroscience research is focused on trying to isolate or separate different sources.
So what does this mean? Well, first let's talk about what this term "source" means. So what do you think a source means in in the brain, in neuroscience? Well, there are many possible ways to interpret this term "source." You can think of it in terms of like EEG or MEG forward modeling. You can think of a source in the brain as being a dipole or one location in the brain from which electrical activity is emanating and projecting to the scalp to be measured. You can also think of a source as being a neuron. So let's say a neuron is the source of the action potential that you measure from some electrode. But we can also abstract this a little bit. So we can talk about a source as being a particular computation that's happening in the brain, or maybe the source is a particular cognitive function like attention or language production.
Okay, so then the idea is that there are many, many sources that are happening in the brain, different kinds of sources in different brain regions and different time windows and so on. And what we want to do in neuroscience is to separate those different sources so that they can be studied well, separately from each other. So we have sources that get mixed together. They are mixed together in terms of the the measurements, they are mixed together in terms of the fMRI voxels, they're mixed together in terms of the cognitive processing. You can think about attention and memory are happening at the same time, so different sources. And we want to separate them to understand them individually.
So with that as a kind of loose definition of source, how do we go about separating different sources in neuroscience? So there are many ways to try to experimentally tease apart or isolate different sources. Here's a non-exhaustive list. Here we can do anatomical source separation. So this would mean, for example, studying the something that's happening in the prefrontal cortex versus something that's happening in the cerebellum. So there we are anatomically separating the different sources. We can also design psychological tasks that would be cognitive source separation. So let's say you design a task where you want to separate semantic memory from declarative memory. So there's different kinds of memory. So we can design an experiment that will help us isolate different sources in at a cognitive level.
There's temporal source separation, and this has a few different interpretations, which we are going to discuss later in this lecture. But a simple way of doing temporal source separation would be again to design an experiment where you can separate out different components of um of processing in time. So, for example, let's say you show a picture on the screen and you want your research participants to make a response according to the content of that picture. Normally, they do that as fast as they possibly can. But you can ask your participants to first make their evaluation of the picture and then five seconds later, they make a response to that picture. So now you are temporarily separating the processing of the visual stimulus from the processing of the motor response. So that's kind of it. That's one way of interpreting temporal source separation.
Then there's also spatial source separation, which is actually very similar to anatomical source separation, except the spatial distributions can be overlapping. This I'm going to talk more about next week. And then statistical source separation is, you know, basically another way of uh or a method of applying or obtaining these different temporal or spatial source separations. So essentially, we're looking at the statistical characteristics of the data and using some assumptions and some statistical methods to separate the different sources.
All right, so this is a bunch of words. It's actually uh mostly the same words here. So let me uh describe what I'm talking about in a picture, in a diagram that I hope will serve as as kind of a the broad outline for this lecture and the next lecture in this series.
So imagine, so here we have some microphones, but imagine these are just sensors. These are things that you measure from the uh from the real world, from the universe. So these could be microphones, they could be EEG electrodes or MEG sensors. But they don't even need to be, you know, sort of electromagnetic things. They could be voxels. Each one of these could be a voxel and an fMRI, and you're measuring the hemodynamic response. And at each voxel, or these could be some kind of, you know, cognitive or experimental outputs. Like, for example, imagine you you're looking at questionnaire data. So each one of these microphones here would actually correspond to a question on a questionnaire that that people are answering. Or maybe this is reaction time in a particular uh task. So you ask your participate research participants to respond to a stimulus as quickly as possible. So this would be the reaction time. So we can abstract this a little bit. They don't have to be microphones. These are just any ways that we have of measuring things in the outside world.
Now, these are called manifest variables because they are the variables that we actually have direct access to. These are things in the universe that we can measure. And what that means, so, you know, there's there's a whole branch of science and philosophy of science about measurement and what it actually means to measure something. But we can simplify that discussion here and just say that a manifest variable is a is a number that we can attach. It's a way of attaching a number to some some phenomenon in the universe, maybe it's a voltage fluctuation or something like that. So these are things that that give us numbers that reflect something about the real world.
Now, the thing is, in neuroscience, in most of science, we don't actually care about the sensors. We don't care about the manifest variables. What we care about is the things that are producing data that the manifest variables are measuring. So let's take an example of EEG. If each one of these corresponds to an electrode, you know, you don't really care about the electrode data, the voltage data. What you care about is the brain process that is producing electric fields that are then measured by the electrodes.
And here's where mixing comes into play because you can have one source. So this could be a source, one source that projects to multiple sensors. So in the case of EEG, this would be one dipole in the brain, and that dipole's electric fields are projecting to multiple electrodes on the scalp. So they project to many different electrodes. But this could also be a cognitive process. So let's say this is attention, and that's the thing that we're interested in. You know, we want to understand how attention works. But we but we cannot measure attention directly.
So here we have the the the action potentials from one neuron, and this is the activity from a different neuron, and a different neuron. So there is some latent construct in the brain called attention, and and we cannot measure this thing directly. But we can measure the effects of attention on these neurons. So here's where things start to get tricky because this is one source. And now, you know, in real brains, we have multiple sources contributing to the signals at the same time. And again, if this is the case of EEG, you could have you could think of multiple different regions in the brain that are generating electric fields, and all of those electric fields are propagating to the different electrodes that we measure on the head. Or, you know, if you abstract this to be some kind of computation, some cognitive process, that maybe this is attention, and this is memory, and so the effect of attention and the effect of memory are are altering the the activity of individual neurons in different ways, but they they're sort of contributing to the same population of neurons.
Now, we might have another source, and maybe this is just noise. Maybe this is sensor noise, maybe this is like muscle noise that's contributing to the electrodes. So all of these different sources, there are multiple sources that we are interested in, maybe there are some sources of noise, and all of these things are separate in some, you know, platonic plane of existence. But in in reality, the the way that they affect the the manifest variables all get mixed together. So this is a problem. Here we have the sources, and here we have mixing. So that's why we need source separation methods.
Okay, so these are the sensors, or the manifest variables. We can call these the true sources, or the latent constructs. And again, these are not things that we can directly measure. These are things that we want to understand, but we can only study them by uh via the manifest variables, which are things that we actually can measure. So again, these are sort of clouded in this like, you know, they're in a sort of other plane of existence or something where we cannot directly access them.
So how do we study these different sources given that they are mixed at the sensor level? Well, you know, obviously we need to do some some source separation. So you could naively try to do a one-to-one mapping and say, well, this sensor corresponds to this source, this sensor corresponds to this source, and this sensor corresponds to this source. Now, that approach may or may not be valid. And sometimes, in some cases, it's not necessarily a terrible approach. If this source is mostly contributing to this one, and these two other sources have very little contribution to this sensor, then it, you know, it might be reasonable to just try to do this one-to-one mapping. But in many cases, that's not desirable and it's not really optimal.
So therefore, what we try to do is come up with a way to combine these different sensors such that their combination is maximizing the contribution from what we think is one particular source. So then this would be the contribution for this source, this would be the contribution for this source, and maybe we assume that this thing is noise. So we just don't even try to reconstruct it. All we try to do is minimize the contribution of this source to these estimated sources.
So then the idea of source separation is that we can analyze these. These are the things that we can analyze, and we hope that this is based on or that there is a close connection between this reconstructed source and this true underlying latent construct. Of course, we don't know. Well, in simulated data, so we can simulate data where we know the ground truth, and then we can confirm whether our data analysis methods are are appropriate, are valid. But in real data, of course, we don't have access to the ground truth. So, you know, we make some assumptions about what these lines look like. We make some assumptions about what the characteristics of the data here might look like, how they might be interacting with each other, how they might project to the different sensors. And based on those assumptions, we reconstruct these these are sources in quotes, or we can call these components. And then we hope that there is some, you know, reasonably close resemblance of this reconstructed component to these original sources. And then these are the things that we use to try to get over or separate these different sources that are being mixed here at the level of the manifest variables.
So I hope that makes sense. I'll be referring back to this diagram later on in the course as well, uh, in this lecture. So now the question is, how do we define these lines? And that's the tricky part. And that's basically the the magic of source separation analyses is trying to come up with how we define these lines, how we we combine the information across these different sensors, these different manifest variables, in order to arrive at these these components here. But what you can see here is that we are taking a weighted combination of these different sensors. So the data from this sensor times some weight, plus the data from this sensor times some other weight, plus the data from this sensor times some other weight, and we combine all of those. And from three different measurements here, we arrive at this one measurement over here, this one component.
Okay, so now to make this a little bit more concrete, I'm going to talk about how these arrows are, how these weights look in time filtering and in space filtering. So let's just focus on this part of the slide. Now, we have a broadband signal over here, so it's a signal that has energy at lots of different frequencies. And here is the filtered version of this signal. So it's a narrowband filtered version. You can see it looks a lot smoother than the original signal, but all of this information is technically contained inside this signal. It's just mixed in with a lot of other signals.
Now, how does this relate to what I showed in the previous slide? Well, in the previous slide, so in this slide, I showed these three sensors here, and you can now think of these as being different time points. So this is time point one, so the the data value at time point one, the data value at time point two, and the data value at time point three. So here we have one microphone, you know, the second microphone, and the third microphone. So for this whole time series, maybe this is, you know, 200 data points or whatever, then this would correspond to 200 different manifest variables that I showed in the previous slide.
And so how does filtering work? Well, so there's several ways to do filtering, and I'll talk about those in a little bit. But in this case, we are doing filtering in the time domain. And the way that works is by constructing this little wiggle here, which is called a kernel. But the the name isn't important. So we take this little wiggle here, and then you multiply this time point in the data by this time point in the kernel, and then this time point in the data by this time point in the kernel, and so on. This time point in the data by this time point in the kernel, and all the way back here. So now you're multiplying the signal by the corresponding data values in the kernel, and then you sum all of those weighted values together, and that gives you a single time point in the filtered signal. So a single time point in the filtered signal is the weighted combination of the original data, the raw measured data, times the uh, this this kernel here, which we construct according to, you know, some some spectral features and so on.
So I hope you can see that this is just a different manifestation of this here. So here we have the weights, I'm drawing them as arrows, and then I said that the weighted combination of all the data points at the different sensors gives us one thing that we can measure here, one component. And that's essentially what we're doing here. We are taking the original data that we measured from the universe, and we are passing it through a weight kernel or a time series of weights, and then we multiply each of those, we sum them all together, and that gives us one time point here. And of course, then, you know, this would slide throughout the data to give us our filter time series.
Okay, so this is temporal filtering, and how this relates to source separation is something I will talk about in the next slide. And then here we have spatial filtering, and I'm going to talk in more detail about this in the next lecture, but just to introduce you to the concept now. So here we have, this would be, you know, like an image of EEGs. We have these EEG electrodes all over the head, and then the data from each uh electrode is a time series. That this would be, you know, one cut, the data from one electrode. So here we have lots of electrodes, 250 electrodes. And now what we're doing is combining all of the electrodes together at each time point. So here we are combining one electrode time series across many different time points, and here we are combining the information across many different electrodes at a single time point. And again, the weighted combination of all of the channel time series gives us one component.
Okay, so again, don't worry too much about this. I'm going to talk about this in the next lecture. Here we're focusing more on temporal filtering and source separation by via filtering.
Okay, so let me start by talking about the assumptions for doing spectral separation, which is basically just a different, it's a variant of temporal source separation. So what we're looking at here is the power spectrum of a signal. So here we have frequency on the x-axis, with lower numbers corresponding to slow fluctuations in the signal and higher numbers corresponding to faster fluctuations in the signal. How you get something like this is through the Fourier transform, and I'll talk about that in a moment. And on the x-axis we have, or sorry, the y-axis we have power, which is basically the amount of energy in the signal.
Okay, so when you look at this at this power spectrum, you can see that it looks like there's two features. So I'm going to call this two features. One feature is this kind of decrease in power with increasing frequency. That's sometimes referred to as one over F because it has kind of a shape of one divided by F, where F is frequency. So that's one thing. And then superimposed on top of that is uh is this feature here, this peak here at, you know, somewhere between 10 and 17 Hertz or whatever.
Now, let's say that this is actually a mixture of two different sources. So we have two sources, and they get mixed together in the frequency domain. One is this d this one over F, and I'm going to say that that's the noise. And then we have this peak here, and that is the signal. So now the question is, can we separate the signal from the noise in this spectrum? Can we do source separation in the frequency domain to isolate, to separate the noise, which is the one over F, from the the peak, which is our signal?
Now, unfortunately, it's not possible for me to like, you know, have a discussion with you about this right now. But if you like, you can pause the video when I'm asking these questions. You can pause the video and take a moment to think about this. So the answer to this question is partly. It's not fully yes, and it's certainly not no. And the reason why it's partly true that we can separate the different sources here, the signal from the noise, is that the noise and the signal are overlapping at this part of the spectrum. So that might look something like this. So we have the pure noise here, and the pure signal here. Now, just by looking at the by filtering the data from 10 to, you know, 17 Hertz or whatever, we do partly isolate the signal, and we do get rid of a lot of noise. But you can also see that the noise and the signal overlap in the frequency domain here, in this area. So there is going to be some some mix between noise and signal that just by doing spectral separation or temporal separation, we are not going to be able to separate.
And that is that leads me to the key assumption underlying spectral separation methods or temporal separation methods. The key assumption underlying spectral separation methods is that the different sources, it could be signal and noise, or it could be multiple different signals, they must be separable in the frequency domain. If they have representation energy at different frequencies, then it's possible to apply spectral separation methods. And if they don't, if they overlap in the frequency domain like what you see here, then either spectral separation is impossible, or in a case like this, spectral separation will still be, you know, pretty good, but but not perfect. So that's just something we need to accept.
So again, the idea here that I'm depicting in this slide is that when different sources are are are separable in the frequency domain, meaning that they have energy at different ranges of frequencies, then we can separate them. And if their energies overlap in the frequency domain, then either we cannot use spectral separation, or we can, but we just have to accept that spectral separation is not going to be perfect. We are still going to have a little bit of mixing even with the with our best attempts.
Now, there would be in theory some ways to further separate the noise from the signal. For example, if we had multi-channel data, we could leverage the multi-channel data to try and further separate the signal from the noise. But that's for another discussion.
Okay, so here we see a power spectrum, and here on this slide, we see what's called a time-frequency plot. So here we have something that's like a power spectrum that I showed on the previous slide, but now frequency is on the y-axis, whereas in the previous slide, frequency was on the x-axis. Now on the x-axis, we have time. And so now what we see is basically a way to decompose our signal or represent the signal changing over time and also changing over frequency.
So now, in the previous slide, when there was more energy in a certain frequency band, then you saw the height of the graph was higher, the height of the line was higher. And here, that corresponds to color. So when you see red, that means there's more energy at this frequency at this particular time point. And when there's blue, that means there's relatively less energy at that frequency at that time point. So these time-frequency plots are really great because they allow us to do both spectral source separation and temporal source separation at the same time. So we can separate different sources in time, and we can separate different things happening at different frequency ranges.
So what I'm going to tell you about now for the rest of this lecture is basically how you get a Fourier transform to do a spectral source separation, and how you can create a time-frequency plot to do both temporal and spectral source separation at the same time. Now, a full understanding of the Fourier transform to do spectral source separation and a full understanding of how to create a time-frequency plot requires more depth, more time than what we're going to cover in this lecture. But but I will give you some basic semantic understanding. So you will have the general idea, even if you don't get uh through all of the details.
Okay, so let me start by telling you a little bit about the Fourier transform. The Fourier transform is the main method for getting from the time domain into the frequency domain. In other words, to get from a time series signal into a power spectrum. So here is our time domain signal. You can see it's like, yeah, sort of wobbling up and down. There's some big fluctuations, there's small fluctuations, maybe some of this is a little bit of noise or something. But so this is our time series signal, maybe this is, you know, the output of uh of an EEG electrode, for example.
Okay, so the way the Fourier transform works is we take a sine wave and we line up the sine wave with our signal. So that looks something like this. And now we ask the question, are these two time series similar to each other? Do they look the same as each other? Now, obviously, you you look at this and qualitatively, you say, yeah, they look really similar. So how do we go about quantifying this? Well, we can start by thinking about these not as pictures, but as a list of numbers. So I'm drawing these as lines, but the way that they're represented on the computer is just as a vector of numbers. So this blue line of the EEG data or whatever, this is just a list of numbers, and this cyan dash line is also just a list of numbers.
Now, if I gave you two lists of numbers and I asked you to tell me how similar are these two lists to each other, you might think of doing something like a correlation coefficient. So that would be the case for, you know, if I gave you a list of heights and a list of weights and I said, is height related to weight? And then you would correlate the two, you compute a correlation coefficient. So the Fourier transform does not involve correlation coefficients per se, but what we do is actually compute something called the dot product. And the dot product is the element-wise multiplication and then sum. So we multiply this data point by this data point, this data point by this data point, and so on for all these data points, and then we sum all of those data points together. That gives us a single number, and that's called the dot product.
Now, the dot product, it turns out, is also the computational backbone of the correlation coefficient. So the correlation coefficient is essentially just a dot product that is normalized in two different ways. It's variance normalized and it's mean centered. But the point is that the mechanism of the Fourier transform is really, really similar to the mechanism of the correlation coefficient.
Okay, now I can imagine what you're thinking right now, which is that, yeah, okay, you know, these two time series look like they're really strongly correlated. But if you would shift this cyan time series a little bit, shift this sine wave a little bit, then, you know, they wouldn't correlate with each other. So we can even uh, we can even look at that here. So imagine we take the this this cyan time series and just shift it a little bit, and then you would find that actually there is no correlation, or maybe, you know, a weak correlation between the signal and the sine wave, even though they're at the same frequency. In other words, there is a phase relationship between the uh, the the signal and the sine wave, and the resulting dot product or something like a correlation coefficient. And that is true. And to overcome that limitation, we need to use something called a complex-valued sine wave. And a complex-valued sine wave has a real part and an imaginary part that corresponds to a cosine and a sine. And it turns out that because cosines and sines are are orthogonally related to each other, they're the same wave, they're just shifted from each other, we can actually use a complex sine wave to avoid this problem of phase shift.
So what we can do is use a complex-valued sine wave instead of a real-valued sine wave, and that not only tells us about the correlation between the signal and the sine wave at that frequency, but it also will tell us about the exact phase relationship. So what phase of a sine wave would maximally correlate with the signal? Okay, so that's getting some of the details about the Fourier transform that we don't need to discuss uh here. But suffice it to say that this phase shift problem is solved in the Fourier transform.
Okay, so anyway, so we compute the dot product between the sine wave and the signal, and that's going to give us a dot product result. And then we plot that here on the graph with at an x-axis location corresponding to the frequency of the sine wave. So what we we can do then is repeat this procedure, except we use a sine wave with a different frequency. So we take a sine wave that's a bit faster, and we go through the same procedure. So we kind of correlate the two, you know, the sine wave and the signal, except it's not exactly a correlation, it's the dot product. But in this case, you can see that the correlation between the the yellow sine wave and and the blue signal is much smaller. So therefore, we get a much smaller signal here. And this is basically how the Fourier transform works.
So we compare the signal to a bunch of sine waves at different frequencies, and then we compute the dot product or the similarity, something that's like a correlation coefficient between the signal and the sine wave at each frequency. And when you repeat that for lots of different sine wave frequencies, that builds up a power spectrum that looks like this. So essentially, what this graph is telling us is that the signal, now I'm not showing you the original signal here, but the signal looks an awful lot like a sine wave at 14 Hertz, and it really doesn't look anything like a sine wave at 40 Hertz. And if you if you correlate the signal with a sine wave at 2 Hertz, then yeah, it looks kind of like a signal at 2 Hertz. So that's a quick verbal introduction into the Fourier transform.
Here I want to show you some pseudo code to illustrate how the discrete time Fourier transform can be implemented in a loop in practice. The Fourier transform is implemented through an algorithm called the FFT or the Fast Fourier Transform. But uh, but but you can also implement the Fourier transform in a loop in a for loop like this. This is not as efficient computationally, it's actually considerably slower than the FFT, but this is a valid way to compute the Fourier transform. And so I just want to show this to you so you can see so you can, you know, just get a better picture of how the Fourier transform works.
Okay, so you run a for loop over frequencies for different sine waves, and then you create a complex sine wave. So again, a complex sine wave is is a regular sine wave, it just has two parts, a cosine part and a sine part. And then you compute the similarity between the complex sine wave and the signal, which is actually the dot product. So it's the element-wise multiplication and then sum. And then you just keep running over lots of different frequencies. Now, how many frequencies you do and exactly what those frequencies are, that gets into more of the details of the Fourier transform, which I'm not going to discuss here.
Okay, and then once you're all done, you can extract the amplitude spectrum and you can extract the phase spectrum, which this tells you about how the uh, the the the dot products, these Fourier coefficients are correlated with a cosine versus the sine part of a complex sine wave. So in general, most people are just interested in the amplitude spectrum or the power spectrum, which is the amplitude spectrum squared. So that's the basic idea of the Fourier transform.
Now I want to discuss two limitations of applying the Fourier transform to do spectral analysis and also spectral source separation, particularly for complex systems like the brain. One limitation is that the results of the Fourier transform, so the power spectrum that you get out of the Fourier transform, are easily interpretable only for stationary signals. And for non-stationary signals, the results of the Fourier transform are valid. Fourier transform is always valid, but the results of the Fourier transform become increasingly difficult to interpret for non-stationary signals.
Now I'm going to talk more about that, including defining what stationary means in a moment in the next slide. First, just let me state the second limitation, which is actually basically just a different way of of saying this first limitation. Fourier transform kind of hides the temporal dynamics. So it's it's difficult to understand how a signal is changing over time, how its spectral characteristics are changing over time, just from the uh, the results of the Fourier transform. Now I put hides and apology quotes here because technically all of the temporal dynamics are in the Fourier transform, they're not really lost per se, but they're kind of hidden in the phase spectrum in a way that's just very difficult to interpret.
Okay, so let's let's talk about stationary. So here I have a definition of stationarity. A signal, a time series signal, is stationary when its statistical characteristics do not significantly change over time. And a signal is non-stationary, basically when stationarity is violated. So non-stationary just means that there is a lack of stationarity.
Okay, so I would like you to think for a moment about whether this definition of stationarity is really precise, whether you know, you think it's a totally unambiguous and no one can interpret it in any other way than what I've written here. Well, obviously the answer is no, otherwise I wouldn't have asked the question. But I would like you to take a moment to think about what are some of the ambiguities in this definition? What what did I write here that is ambiguous or not totally clear?
So there's several points of ambiguity here in this definition. One is, so what exactly I mean by statistical characteristics? So these descriptive statistical characteristics include features like the mean and the variance and the the spectral content and things about distribution like skewness and kurtosis. So there's all sorts of statistical characteristics, and here I'm not saying which statistical characteristics I'm referring to. And then here, whenever you're talking about significantly changing, that is some kind of inferential statistic. So you need to define a threshold and a p-value and an appropriate statistical test. And when things are changing over time, we also have to think about the size of the window that we're looking at and so on. So it's actually quite a few ambiguities in this definition, and that's a little bit intentional. The thing is, you know, we don't really have unambiguous definitions for stationarity.
Okay, but I do want to illustrate this concept a little bit more because it's a really important uh idea in neuroscience time series signal analysis. So so let me try to build some visual intuition about stationary. So here we have a time series signal, and now what I would like to know is whether the mean, so the average value in this time window, is the same as the average value in this time window. Now, to be clear, I do not mean that the average here and here have to be exactly, exactly numerically identical to, you know, 35 significant digits after the decimal point. But, you know, within some reasonable tolerance, is the average value here the same as the average value here? I think we can agree that it most likely is. Maybe you could do a t-test on these values and these values and see if the means are are significantly different. I would guess the t-test is going to be non-significant.
Likewise, we can ask the question about this window. So now we have a larger window, and is the average of the data values in this time window going to be roughly the same, again, within some reasonable tolerance, as the average value within this time window? And I think we can all agree that probably the answer is going to be yes. In other words, when you look at this signal and you're thinking about the mean, the average value, you would say that the estimate of these statistics, so in this case, I gave the example of the average, is independent of the size and the location of the time window. So therefore, we can say that this signal is mean stationary. So the average value is basically the same wherever you look in this signal, and with various time windows.
Now, of course, you can take this to an annoying anal retentive extreme and think about the average of just this one data point, or a window that contains only this data point, versus the average of a window that contains only this data point. So, you know, again, we have to be, let's be reasonable here. But uh, in general, we can say that, yeah, the average is not really changing over time. So the signal is mean stationary. You know, the the variance also looks pretty similar over time. So I would say this signal is probably also variance stationary.
So now let's have a look at a counter example, and we can play the same game here. So the question is, is the average value in this time window the same as the average value in this time window? Now, here the answer is obviously going to be no. Now, it's not to say that the average value in every single time window is different. The average in this time window and the average of this time window, if you just shift this box over, those are probably, you know, the means, they are are going to be roughly the same. So but we can look at different time windows, different sizes of the time windows, and we can say that the estimate of the descriptive statistic, in this case, the mean, is dependent on the size and location of the time window. So this signal, we can call mean non-stationary. Again, this also depends on the location and the size of the windows.
So the term stationarity should be interpreted more in terms of a an idea, a concept, and less in terms of a precise mathematical definition. But so in contrast to mean non-stationarity, this signal, we'll call mean non-stationary. I think this signal is roughly variance stationary, which is to say, if you would look at the variance or the standard deviation of this signal in different time windows, I think in nearly all time windows the variance is the same. Maybe except for here, if you had a time window that included this sharp drop here, the variance is going to be relatively large. But the mean, you know, comparing this time window to this time window, the means are different, but the variance is the same. So maybe we can say this signal is variance stationary but mean non-stationary. So that means that a signal can be stationary in some characteristics and non-stationary in other characteristics.
So so here you see an example. This signal here is frequency stationary because the frequency is not changing over time. But it's amplitude non-stationary because the amplitude is increasing over time. And this signal, in contrast, is amplitude stationary because the amplitude is the same, but the frequency characteristics are changing over time. So this is frequency non-stationary.
Now, why do I bring up all of this stuff about about stationarity and signal stationarity? This is a really important consideration for neuroscience data analysis because data, you know, brain time series data are incredibly non-stationary. In fact, it is no understatement to claim that uh, that pretty much all of neuroscience is focused on trying to understand the non-stationary features of the brain. It's the non-stationary features that allow you to think and and act and remember things. Anytime something is happening and something is changing in your brain, then those are non-stationary, not non-stationarities in your brain. So these are the things that we're interested in, the the temporal non-stationary, spatial non-stationarities, spectral non-stationarities. So we want to be able to understand them.
So the fact that the results of the Fourier transform are difficult to interpret for non-stationary signals is a major problem for the Fourier transform, or well, yeah, it's not a problem for the Fourier transform per se. The Fourier transform is always perfectly valid. It's a problem for using the Fourier transform to try to do spectral analysis and spectral source separation in neuroscience time series data.
Okay, so now let me discuss the second limitation that I identified before, which is basically just a different way of phrasing the first limitation about stationarity. So here we have a time domain signal, and you can see there's a lot of stuff happening in here. This is actually a recording from V1, from primary visual cortex, and here there was a stimulus that appeared on the screen, so you see a visual response. And here there was a different visual stimulus that appeared on the screen, so you see another response. And this really sharp peak, this turns out to be gamma oscillations here. And then this was when the stimulus came off the screen, and you see this offset response. So there's a lot of stuff happening at different points in time. There's a lot of temporal dynamics in this signal.
This is the power spectrum of this signal, and I don't have the frequencies labeled here, but that doesn't really matter. This is low frequency. This bump here is somewhere, I think the peak was around 40 Hertz or so, but it doesn't really matter. So this peak here corresponds to these ripples here, these fast oscillations here. Now, this power spectrum is perfectly valid, but it is hard to interpret because from looking at this power spectrum, you don't actually get a sense of where the this dynamic is happening. It could be just from this power spectrum, it could be that there is consistent 40 Hertz oscillations over time, or it could be that there's a huge burst of 40 Hertz activity and then no 40 Hertz activity, and then another burst, and then more quality, and then another burst, and so on. But that's actually not what's going on in the time domain.
And the frequency domain representation, the power spectrum hides all of these temporal dynamics. Again, they are they're technically there, they're in a different dimension of the results of the Fourier transform, which is in the phase spectrum. But even if I were to plot the phase spectrum, it would be uninterpretable. You wouldn't be able to understand that this 40 Hertz feature happens over here and not over here. So you already know the solution to this problem, which is instead of just looking at a a power spectrum, we can look at a time-frequency plot. And now that allows us to do both spectral separation and temporal separation at the same time.
So now the question is, you know, you know how to create a a power spectrum. So I told you about the Fourier transform. And now the question is, how do we create a time-frequency plot? And that is what I'm going to tell you about for the next like 10 minutes or so. There are in fact many, well, several ways to create a time-frequency power plot from a time domain signal. Some several of those methods are all closely related to each other, both conceptually and analytically, and some methods are a little bit different from each other. And so what I'm going to do now is just explain one method to get from the time domain into the time-frequency domain to allow you to do temporal and spectral source separation. And that method is, I'm calling it here, Gaussian filter Hilbert.
So we have two really strong, influential German mathematicians contributing to the method. So, you know, it's got to be good because, uh, yeah, German, Germany has produced a lot of really brilliant mathematicians. So we have Gauss, great mathematician, one of the arguably one of the most intelligent humans that ever existed in the history of humanity. And Hilbert, it was also a very brilliant mathematician. Anyway, so now I'm going to walk you through a step-by-step procedure for creating a time-frequency plot via the Gaussian filter Hilbert method.
So we start with our time series data. So this would be a broadband signal. This would be the thing that you are measuring from the world over time from a sensor. So maybe this is from an EEG electrode or an LFP electrode or something like that. This could also be the BOLD time series data from an individual fMRI voxel. So this is our time domain signal. And the next thing we do is take its power spectrum through the Fast Fourier Transform, as I mentioned before. The Fast Fourier Transform is uh, basically just a more efficient implementation of the Fourier transform, which I already told you about, where we just loop over a lot of frequencies, we create sine waves at each frequency, and we correlate the signal with the sine wave at those frequencies.
Okay, the next step is to take this power spectrum of the signal and multiply it by a Gaussian. So here we are saying that we just want to preserve these frequencies here in the signal. And the so the Gaussian basically goes to zero here. So we're going to essentially zero out or at least really strongly attenuate frequencies higher than this curve, and also frequencies lower than this curve. Again, this is based on the assumption that this frequency range here contains some signal, and the other frequency ranges, so lower and higher frequencies, contain maybe other signals or maybe noise, but it's certainly something separate, something different from this signal here that we are interested in.
Okay, so here you see the power spectrum and the Gaussian. When you multiply them together, you get a spectrum that looks something like this. So it's basically the features in the time series signal that are overlapping with the features of the Gaussian. And then you take what's called the inverse Fourier transform, and it's just literally the backwards version of the Fourier transform. So the Fourier transform gets you from the time domain to the frequency domain, the inverse Fourier transform gets you from the frequency domain back into the time domain. So we take the inverse Fourier transform of this multiplied spectrum, and that gives us a time domain signal, which you see in yellow. And well, it looks like a bandpass filtered version of the original signal, and that's exactly what the Gaussian filtering is doing. It's giving us a bandpass filtered version of the signal.
Now, for a time-frequency plot, we don't actually care about this narrowband filtered.
Signal all of these quick ups and downs instead. What we want is the amount of energy in the signal that's varying over time. So that's kind of like, you know, if you would imagine connecting or like drawing a line that goes from across all of these peaks like this. So, uh, from peak to peak to peak to peak to peak. And then when the the the line is higher, the peaks are higher, then the signal has more energy at that frequency. And that can be obtained by something called the Hilbert transform. So the magnitude of the Hilbert transform is going to return this envelope. It's sometimes called the the amplitude envelope of this signal over time. And essentially that's it's, you know, you can think of it as being a smooth line. So a line that goes from one peak to the next peak and then that line gets smoothed out. And that's basically what the result of the Hilbert transform is. So this is called the amplitude time series or the amplitude envelope. And you can square the amplitude time series to get the power time series. And this amplitude time series is actually one row in the, uh, time frequency plot that I showed earlier. So imagine taking this here, it's going down and up a little bit, and then down. And here it goes up more. And imagine mapping this onto color. So this would be blue when it's low, relatively low, and it would be red when it's relatively high. So here we go blue again. And then this would be, you know, maybe orange. It's not quite as high as this peak up here. And then instead of being a line going up and down, it's a row of colors. And that gets mapped onto this plot. So this would be, you know, the amplitude is going down at this frequency. The energy is, there's less energy here. And here there's more energy. So the the line is going up. And that gets mapped onto a reddish color here. So this procedure gives you one row in the time frequency plot. And then to create the next row in the time frequency plot, you have to go back and iterate through this entire procedure. Now, the time domain signal is always the same. What's changing at each step is this Gaussian. So this would be the Gaussian for for one step, for one frequency. And then for a different row in a time frequency plot, you just slide this Gaussian up. So you would use a Gaussian that's like up here, would be overlapping a little bit with the previous Gaussian. But you can see that it's going to highlight different features of the signal, different spectral features of the signal. And that, in a nutshell, is how we create this time frequency plot. Again, there's a lot more details, as you can imagine. There's all sorts of stuff I haven't really talked about. But I hope you get a general sense of how to create a time frequency plot, which is really great for neural time series data. It allows you to do both temporal source separation and also spectral source separation at the same time.
Okay, so let me give you a really quick recap of what we covered in this lecture. So essentially, I started with this concept that there are these these true sources, these latent constructs. These are the things in the real world that we as scientists are interested in understanding. And now, maybe this is a part of the brain, maybe the source is a cognitive computation like language production, uh, or or motor planning, or maybe these are some kind of computations that are taking place in the brain. Here we have these sensors. These are the things in the universe that we can actually measure, the manifest variables. These might be the action potentials of an individual neuron, they might be responses on a questionnaire, a personality questionnaire, they might be voltage fluctuations from an EEG or MEG sensor. And what we try to do is use these sensors to study these latent constructs. Now, these are invisible, and these are the things that we can measure. So what we do is we make some assumptions. We make some guesses about what these sources might look like, how they might behave, and how they might project to the different sensors. And based on those assumptions, we use some statistical methods to create a way of combining the data from all the different sources to give us the different components. And in this particular case, in this lecture, these different components, these estimated sources are this power spectrum. Now, the thing is, we don't know if, you know, in the brain, the signal really looks like a sine wave. You know, if it really looks like this wiggling up and down like a sine wave. So we make that assumption that the data in the brain look sort of kind of sinusoidal-ish. You know, it's a separate discussion, but the brain doesn't have to produce pure sine waves for this method to be valid and appropriate. But we make some assumptions that the signal in the brain is both narrow band and frequency and also time limited. And based on those assumptions, we can generate the weights for these little arrows, which is the Gaussian filter Hilbert method that I discussed a few slides ago. And then we can actually study the components, and hopefully these are telling us something about these underlying sources. So I hope you found that lecture interesting and thought-provoking. So that was all about temporal filtering and spectral source separation. In the next lecture, we are going to discuss spatial filtering.