Transcription
All right. So, in this video, we're going to be looking at the relationship between regime detection methods, portfolio risk, and geometry, as well as kind of correlations as key quantities for determining these things. We will build up to some very neat visualizations like this, but we're going to start out from kind of the ground up.
So, the first thing we need to talk about when we're considering doing regime detection is really non-stationary distributions. So, I'm going to be hopping back and forth between the tablet and the Jupyter notebook in this video. Basically, the definition of a non-stationary distribution is just one whose moments, whether it be these kind of lower order moments like mean and variance or higher order moments like skew and curtosis change over time.
And usually this comes in two forms. So, you either have like continuously changing distributions. So, at some time I have this kind of normal like distribution and then some short time later it's changed a bit. These sort of things are tough to model. You need to use things like the wave equation. You may need to use ideas from quantum mechanics or any sort of like uh methods to like propagate um wave packets. Uh and really this is not something you want to be doing. It's uh you can of course do it. There are many places in finance where it is done. But if you can avoid this, it's a nice thing to do.
And the key promise of regime detection is actually exactly this. We're saying that we can take a look at this non-stationary distribution and model it using discrete jumps between uh well like individual stationary distributions. And this is from a very bottoms up approach exactly how people argue for hidden marov models. So you don't need to know anything whatsoever about hidden marov models to understand this video. I'm going to explain the motivation for them uh to you right now.
So before we even look at that diagram, suppose we just have some system and it's kind of denoted by this index I. And what it does is it generates for us uh an entry into a time series. Let's say those are returns for some stock price. And so at each time step, it does this. And it also tells us um how likely am I to transition states basically this index I going to some other index and that's exactly what we are looking at here.
So, we're going to look at a two-state system, which again at each time it spits out a return and then tells us if it basically going to change state. And the the first state that we have is a normal distribution. So, our returns are drawn from this normal distribution. And then we check if we transition over to this Koshy distribution. And uh likewise at if we're living in the Koshy distribution regime then it will generate for us a return and it will check if we transition back to the normal distribution. These transitions in this model are happening with a 5% probability. And this is kind of a perfect example of what I mean by these discrete jumps between individual stationary distributions which are of course approximations in and of themselves. But we'll hop back to the Jupiter notebook and I will kind of introduce whoops introduce for you both of these cases.
So this would be an example of a continuously changing distribution where we have the mean oscillating around zero and the long run variance also oscillating around some value. And what this does is it clusters the returns. So the returns cluster below zero then they cluster above zero because we're oscillating around zero for the mean. Uh and likewise for the volatility you have kind of highvall and lowvall clustering and again this is the perfect example of the moments of the distribution changing with time. It's very clear visually and it's easy to understand. That's the continuous case. Now what we're really concerned with this is sort of just included for completeness sake. Um what we're really concerned with for this video is uh that discrete flipping between data generating distributions.
So in the red shaded region you have the returns being drawn from that Koshy distribution and in the rest of the regions they're pulled from the normal distribution. And if you look at something like the change uh in the price series over time right uh of course the price can't go negative but let's say this was fixed at some initial value then in this region you know the the balance or the uh what do you call it the price is relatively flat. Uh however, when we're in this Koshi regime, things are a lot more volatile, right? And this is the the quintessential use case of like a hidden marov model and why regimes are seen as so useful.
So why are they so appealing? Well, they're kind of appealing for the reason that any mathematical framework is appealing to a practitioner, which is that they make things easier to compute and they go well with our intuition. So, we'll hop back to uh the tablet for a moment here. And what we're going to look at is this claim in more detail. So, how does it make things mathematically easier? And how does it make things uh well, how does it align with our intuition?
So, again, if you have this real explicit time dependence in your data generating distribution, you often need very complex methods to propagate that. However, if we're able to kind of factor out that time dependence into something which just tells us at a given point in time which regime am I living in then all you need to do is learn two in this case you know of course it depends on the number of regimes you have but in this case you just need to learn two conditional distributions which as an approximation you keep fixed over time. So that greatly simplifies the search space. It greatly simplifies uh the optimization procedure uh with the added complexity of the fact that we have this very very clear dependence on these regimes which are sort of illdefined so far.
Now how it makes things uh more intuitive for us? Well, you just look at a return distribution. You say this is kind of low volatility. You say this is high volatility and you make some regime detection algorithm which classifies those for you effectively. Similarly, it could be a bull bare market. Um, it could be three regimes. You could have bull bar and sideways. Um, inflation, stagflation, deflation. And it doesn't have to be related directly to price. It could be conditioned on rate cycles, macroeconomic variables or what we'll look at later in the video which is this sort of risk on riskoff profile uh as dictated by asset correlations and sort of geome we're going to look at a geometric interpretation of that.
Okay. So back to the notebook. This is where hidden markoff models come in and where people become very very interested in using hidden marov models. It's because the key goal of a hidden maroff model for regime detection is to say at a given point in time uh people will sometimes call this state decoding. What is the sequence of these latent or hidden states which we're calling our regimes right so give me at a point in time am I in a bull or a bare market that's mainly the goal of a hidden maroff model for regime detection and of course we've kind of gone over how this is intuitive for us here are the key mathematical components of a hidden maroff model and then we'll analyze one of these trained on S&P uh price data basically You have the states, those latent states, which for us are just the regimes. You have the transition probabilities. Back to our example between S1 and S2. Those were just 5% at any given point in time. You have the emission probabilities. These are basically telling you how probable the observed data is given the regime that you're in. And you can use this, of course, then you would imagine to sort of guess which regime you're in. Um, there are all sorts of complex algorithms for how you do this, all the mathematics that go into this. And I'm happy to make a full video on that. It's kind of a topic I've gone into very deeply. So I would be happy to do that if people are very interested. Just let me know. And you also have some sort of initial conditions which we don't need to worry about for uh the purposes of this video. But again, the key task is this state decoding. Basically tell me what latent regime am I in at each point in time in my time series.
Now the pain point is that when you deal with these HMMs, you don't get to tell the model exactly what you want those states to mean. So I've trained this here and we can say that we have a a two regime model. That's a hyperparameter that you set. Um and maybe we hope that it picks up bull and bare markets. Well, it's kind of doing that. These green ones are sort of in market downturns, but you know what does it mean to be in a market downturn for like 4 days? Is that a real thing? Why is it changing so frequently? Why is it uh flickering back and forth between regimes? And do we really have this? I mean, it's green here, but then it's kind of going up again. You know, what does this mean? How do we interpret these regimes? And this is a key practical concern of hidden markoff models, especially applied to financial data.
So, I have a few uh challenges associated with these models. One is parameter instability. So, uh again, those conditional distributions are not truly stationary, but that's sort of a problem for any regime detection model. Um, something that's very particular to HMMs is the regime flickering. They're very sensitive to outliers and so it's frequently the case that if it sees something that it deems to be highly unprobable for a regime, it will just collapse to the other regime um for a day and then flip back the next day. And that is very problematic, especially if you want to use these things in, let's say, an automated trading system um where your something is conditioned on what regime you're in. maybe entries and exits are happening far too frequently when it's flickering. Um, and really just the utility is is kind of mitigated by these three questions. One is how do I interpret the states? We've already touched on that. How do I get good out of sample performance? It's very easy to fit these historically dialed the parameters in such that you're happy with the regimes. You've mitigated the flickering. You've kind of increased the stickiness of a regime transition. But you know then when you go to use it out of sample, anyone who's actually done this will know it's very difficult. These are very difficult to dial in for something other than historical explanability and keep them stable over time. You can do it. There are plenty of ways to do it and I've done it. I think many people watching the video have probably done it. But it's you have to admit it's often very tricky. And then of course how do you mitigate outliers again which cause these uh infrequent flickerings often or rather these these kind of like uh impersistent flickerings I should say.
There are all sorts of other concerns and am I saying that hidden maroff models are useless? No, not at all. I'm just saying there are plenty of practical concerns with how you implement them um and monitor them as they're kind of in their model life cycle. Right? And I'm going to propose to you an alternative which is that in many scenarios you're using a hidden maroff model because you know what kind of regime you want to identify. And when that's the case I often think the the approach is overused. It's often easier to just build the regime detection model for what you're trying to target. Um and so that's going to be the alternative. That's what we are going to do. Again, you have all these different types of of regimes you could be detecting, but we're going to be looking at this risk on and risk off uh type of regime. And again, the key idea for this next part is if we know we want to identify this, why don't we just build that? You know, why does it have to be this complex model? Um well, it's not even about the complexity, but why does it have to be this thing where we don't really get to easily template what we want the regimes to mean, right? And this is where we get to the the second part of the video where you'll see some of these nice visualizations that I showed in the beginning. Risk correlation and geometry.
So I'm going to play for you this video. And what we're looking at here is the S&P 500 as well as a sector correlation matrix. Uh if you're not familiar with a correlation matrix, it just plots the pair-wise correlations of these assets over some kind of rolling historical window. And what you'll notice is that uh blue here means high correlation, yellow means no correlation, red means negative correlation. Often cases in sort of stable up times for the market, the correlations will be closer to zero, right? And we can look at that for example uh towards the end of 2019. You know, we have uh some sectors are you know relatively low correlation. Um maybe this is a better snapshot to look at. There are of course positive correlations as you will frequently see across sectors. However, you could build a relatively well diversified portfolio out of this, right? Even a long only portfolio, you could have some decent amount of um sort of like hedging against broad market moves.
Now, what happens when we get to uh early 2020, March 2020, this is when lockdowns happen and the market crashes. Basically, this correlation matrix goes to one. And this is uh the age-old adage people say of in a crash correlations go to one. This is exactly what they mean. Plot the correlations of the different sectors building out your market index and there are going to be very high when the market crashes. And so if you have this long only portfolio, this long only unhedged portfolio and you think you're diversified four months ago, well, you're in for a very rude awakening when everything crashes in tandem. And this is the story of pretty much every market crash that is not like explicitly politically driven. Um, of course this is the case of 2008, right? Um, sort of beyond the scope sort of like a tangent for this video, but this is the idea is that in some way we can see that correlation has a relationship with risk. And this is the thing that we want to sort of build this intuition that we have into our mathematical model that we're going to use for regime detection.
And here I just want to caution against something that I see all the time. People are very obsessed, especially people who are new to the space and I was like this as well. So it's not a critique, but people often get obsessed with purely algorithmic systems or purely systematized systems, everything automated, no human touch whatsoever. Um, and you know that's something that can work. And if you have a very complex model, if you're in a frontier AI lab building models with a trillion parameters, often you have a big enough model with high enough capacity to learn those highly complex and nonlinear relationships between uh you know, elements of the training data. However, usually for people doing uh projects, right, or you know, even just most work in the industry, that's not the case. So getting your hands dirty with the assumptions going into your modeling framework, the data itself and building up your kind of context in this domain is very very very important and I think it's something people kind of rush to skip or they see it as a negative. Uh the people who do this and do this well they build better models they build more robust and resilient models. So this is something I always sort of caution against.
Now we're going to look at this correlation risk relationship and we're going to do it through the lens of geometric algebra. So this idea is not mine actually you can see a blog post from August Saganto. He'll be linked in the description. He calls it the geometry of a market crash. And we're going to be looking at effectively ideas from his post um with some added visualization, some added interpretations um and you know how these things can be useful to us to build a more resilient regime detection model for high-risisk low-risk uh environments from a portfolio diversification uh point of view.
So we have three things that we do need to introduce. First is the wedge product. I'll get to in a second. Second is the gram matrix which is just a matrix built out of these inner products. So that the J element of this matrix is simply the dotproduct between RA I and RJ. Now what are RA and RJ? These are simply uh individual returns for the sector I. So this could be like financials, right? Or this could be industrials, this could be consumer staples, something like that. And we're going to be looking at effectively this thing uh over rolling windows. Now you also need to understand a bit about determinants to understand that we get a very useful volutric quantity from the determinant while the square root of the determinant of this gram matrix. And one uh note computationally is that I'm normalizing these returns such that the inner product between um one sector and itself is always one. That will give us a very nice bounding between zero and one of this volume.
Okay. So, what is the wedge product? Well, the wedge product I'm going to point you to a lecture series on Clifford algebra, geometric algebra uh by Mathoma on YouTube. Very interesting topic. You don't need to understand much from it for the purposes of this video, but if you're interested, I think it's it's one of uh my favorite areas of mathematics. And this is one of the rare occasions where it becomes useful for uh quantitative finance. So kind of excited to make a video on this. What the wedge product does I can give you an example in two dimensions is let's say we have two dimensional vectors v_sub_1 v2 and this is a bit handwavy. However, let me actually draw these so they persist for more than a second here. Let's say we have v_sub_1 v2. Well, what a wedge product does, and it's just denoted by this wedge here, looks something like a lambda, um, is it defines for you an oriented volume. So, if you've worked with a cross productduct, it's very similar to that in this specific example, though the wedge product is a much more general quantity. Uh, so it gives you this oriented volume, which in this case is a plane, right? So it gives you this plane that is in a sense spanned by these two vectors and it also gives you the magnitude or the volume which in the two-dimensional case is simply an area. This generalizes to higher dimensions and people always call these higher dimensional uh magnitudes volumes. They're kind of volumes in some larger dimensional space. Okay, but that's really the the idea for um the wedge product is we get this oriented volume as well as the magnitudes. And these magnitudes are what we are going to care about because we have this very interesting relationship between the magnitude of that wedge product of our sector returns and this gram matrix which is very easy for us to compute. And the relationship is this. The magnitude is equal to the square root of the determinant of the gram matrix. And just to be very very explicit so no one is confused, the gram matrix is literally just the inner products of those sector returns. Okay, so imagine you have like a rolling window of 60 periods, 60 days. Well, then this is just the dotproduct between like two sectors and that's what builds out this matrix. Okay. And so it gives us this great geometric interpretation of in a sense the volume enclosed by these different uh return time series. And we can use that to relate to correlation and portfolio risk because if we have zero correlation across our assets, then this volume quantity is going to be one. And if we have perfect correlation across our assets, well then think about how you you know what happens if that's the case here in terms of the determinant. The volume will go to zero. And as an added thing, I've actually built out a framework such that we can visualize this volume in three dimensions. What we're going to do is we're going to look at three sectors, compute the gram matrix, factoriize the gram matrix. Uh this is just simply a cholky decomposition. And then the vectors the column space of this cholski factorization uh the L matrix here which is a lower triangular matrix. This gives us basically a three-dimensional volume that we can use to um view these market dynamics really in three dimensions as market crashes happen and things. So that will be sort of like the seinal visualization for for this video. So we can hop back to the Jupiter uh notebook for now. Uh just give me a second here.
Okay. Now what I've done here is first we're going to look at this as an alternative to doing the regime detection. So I've computed the uh wedge volumes. Basically what I'm calling this square root of the determinant of the gram matrix. I'm calling it interchangeably the wedge volume and the gram volume. And what I've done is plotted it here in orange. So you can see that in these sort of market crash environments, we're very frequently getting a very very stable signal that something is happening geometrically in the market as dictated by these sector returns which is persistent, which is consistent, and which is really not doing too much flickering. Sure, it happens sometimes depending on how we threshold this gram volume to denote our two regimes, but it's really quite consistent, especially I mean just for for contrast, uh going back up to the HMM, it is nowhere near as volatile as this, right? It's much much much more persistent. And so if you were to trade on something like this, you'll find far fewer issues with transaction costs and just like the model constantly flickering back and forth between uh regimes. So that's the idea here. And you'll notice that sometimes it does not pick up crashes. And what that means is that in this quote unquote market crash, we could zoom in. The gram volume is not really doing anything too uh too exciting, right? Sure, it dips a bit, but it's nowhere near as pronounced as in the March crash, March 2020 crash from uh the lockdowns. And why is that? Well, it's because this crash is not the same geometrically, right? These sectors are still rotating. there's still basically a relatively high volume enclosed by those u vectors you know sort of uh generated from the uh return time series of these different sectors. It's a lot to think about if it's the first time you've been exposed to this concept of geometric algebra or to linear algebra if you're not too sound on your linear algebra. But I think if you are, these things are sort of intuitive uh to you. So just pause the video, think about these as you need, uh drop comments down below and I will be sure to uh respond and kind of help if anyone needs help understanding these things.
So what I want to look at now is well is this any different than just looking at the pair wise asset correlation. So what we're going to do is we're going to compare this wedge volume to the average of the correlation matrix. So we're going to just run through and this is again for those sectors feeding into into the S&P 500. And what we're immediately seeing is wait, okay, uhoh, this is like the same thing. Why do I need all this complex geometric algebra? Can't I just take the average of the correlation matrix? I mean, they seem to be basically inversely correlated um perfectly, right? Well, that is true in some scenarios and of course it's handpicked to be true in this scenario, but I'm going to show you one where it is not true and I would encourage you to pause the video and think about the fact that this is a volume, right? And what do you know from linear algebra about uh volume collapse? How can we generate that? And how may that appear in something like this determinant but go relatively unnoticed by a metric such as the average of these pair-wise correlations? I'm going to spoil it right now. So, if you want to think about it, uh, make sure to pause. Uh, I'm going to run this video for you. This is just synthetic, uh, price data for four assets. And I'll just let this run. And what I have done is at this break point, I have almost perfectly correlated assets A and assets D. So, in orange and purple, you can see these movements beyond this point are highly correlated. Okay. And what does that actually mean for the pair-wise correlation? the average rather of those pair wise correlations versus the wedge volume. Well, I'm going to show you exactly that. So, I've computed for these the pair-wise correlations in this correlation matrix as well as those wedge volumes. And we will run this and think for yourself what's going to happen. What do you think at this point where A and D become perfectly correlated? What's going to happen to the average correlation? And what is going to happen to this wedge volume? Um and I will play this for you right now so you can see well at this point where the correlations go to one and they sort of phase in because we have a rolling window uh being uh fed into these computations um by the point that this rolling window has has basically phased out and the one to one corre one correlation is constituting the whole of the history. Well, the average correlation has gone up a bit, but it really hasn't gone up that much. However, this wedge volume has taken a significant significant and very statistically prominent dip, right? And this is the utility of this and why it's fundamentally different than why it's a fundamentally geometric property in this in in a totally different way from uh just taking average of pair-wise correlations. So the reason this is useful is because let's say you have a few asset portfolio and you're monitoring the average correlation as a risk parameter, but two of your positions become almost perfectly correlated. Well, you are significantly under diversified relative to what you would expect from this average correlation, right? However, if you use this truly geometric quantity, that will pick that out for you very very significantly uh and almost immediately.
Okay, so now I promised you kind of a cool visualization of the true market volume enclosed by these return time series. And what we're going to do is we're going to look at this as applied to three sectors, materials, finance, and industrials. I'm going to plot for you the S&P index as well as this wedge volume uh over time. I'll first play this and then we can think about it. Um actually before I play this, I'll explain to you what this volume is. So this is the decomposition of the Graham matrix into a Cholski decomposition. And then again, we're just plotting the uh column vectors of the Cholski matrix basically. Um and so yeah, that's that's really the idea here. If you want a refresher on this, I'm happy to do kind of more basic linear algebra concepts. But this volume here really tells you on a rolling basis, what is the geometric volume enclosed by these three return time series? And we can look at what happens in market crashes. So here where we will have a dip in just a second, you see the volume greatly suppresses and the correlations go up. And this becomes even more prominent here in the March 2020 crash. Right? You see the volume is greatly greatly suppressed. So I'll go back to uh right around here. This thing has almost compressed to in this case a 2D object where by definition the determinant would be zero and the volume would be zero. And again if you look at just the uh sort of average of the correlation matrix in this example it would be similar but we've shown you that this is really picking out a fundamentally different quantity which you may or may not prefer. Um, and so that's kind of the end of the video.
And so what's the takeaway? Is the takeaway that the wedge volumes better than hidden markoff models for regime detections? That these geometric approaches with some fancy math are always better than just pure statistics and modeling? Um intuitive methods are better than complex methods? No, of course that's not the takeaway, but the takeaway is just to do what works best. it is to expose yourself to as many concepts as possible. Be hands-on with the data and always try to let the data do the talking for you. The more things that you are familiar with, the more interested you are in kind of neat new concepts, the more tools you will have in your mathematical tool belt and the easier it is to sort of look at a data set where you have some uh outcome that you want, right? And just try different things and uh build up an intuition for kind of research direction. what is the right thing to do here? That's a tool that it's a skill rather that you build over time just by trying a lot of things and being in the weeds in your data. And so, uh, yeah, I'm just going to wrap it up here. I think I had a lot of fun with this video. I hope you got, uh, some some interesting ideas. I hope you liked the visualizations. They were fun to make. Um, but, uh, that's going to do it for this one. So, I'll just close the video here. Say thank you very much for watching and I hope you have a good rest of your day.