Transcription
Hello everyone, and thanks very much for joining this webinar today. Um, and thanks Ria for your introduction. Um, so my name is Tom Stark. Uh, there's probably not much more to say about uh my uh CV and the things I've done.
So today we will talk a little bit about uh deep reinforcement learning for trading, a topic which I've been interested in for several years now, and it's quite a tricky, difficult topic if you are new to this uh to start with. So today we can only really look uh in a more superficial way into this, but I really hope uh you will get a lot out of it and it will motivate you to look a little bit deeper into all the ins and outs of this.
So as I said, uh this is my uh CV; we can easily skip this. So the question is, what is reinforcement learning really? Um, it's not easy necessarily to answer because um we already have uh deep learning and machine learning, which a lot of you will be familiar with. So what's the difference to reinforcement learning? Well, basically RL is a technique to make a machine solve a task without supervision. So what that really means is we can give the machine a task, and the Machine just will learn by itself what to do and successfully perform these tasks and hopefully produce favorable outcomes.
Um, so when we usually perform a task, we go through a number of steps during this task. So let's say in this typical reinforcement learning example, a machine would like to learn how to play a computer game. The way it works is usually the computer game starts, and then we find ourselves in some gaming scenario, and while we are in this scenario, we go from one step in the game to the next, and while we're doing this, the state of the system changes, meaning what we see on the screen changes.
So when we play a computer game, there's one thing uh that is basically us, and another thing is that um we have a screen that we look at, which is effectively the state of our system, and based on that we respond; we make a we make decisions to do something uh inside this computer game. So in in other words, we act according to what we see, and then as we step through the steps, several things happen in this computer game, and finally the game somehow ends, and we end up with a success or failure based on the chain of the decisions that we made. So we may have done a lot of really good steps and actually scored some points, or maybe we have actually not done very good steps, and we failed in our game. So we could, for example, as a reward, win a number of points; on the other hand, we get a negative reward where we lose a number of points. So depending on depending on what um what we've done, it goes one way or the other. Okay, let's go to the next step.
So what this really means is we go through what we call a Markov decision process; we're basically stepping through a Markov decision process, and U you probably, if you are familiar with this a little bit, uh you know what a Markov chain is. So I talked about different states; so here uh the states are uh shown in in those letters a, b, c, d, e, f. We will in a little bit uh find out how all this applies to trading; just just just uh uh let's for a moment focus on this diagram, just just to see what this looks like. So we basically have a state a here, and then we take an action um and and the action of that's the letter a uh which is different from the a um so we take an action, and so for example, we could take an action uh along this line, and we end up with State e, or we could take another action; we end up with State C, or another action; we end up with State B. So we're basically uh having a transition from one state to the next, and then at each transition uh we may get a reward, and finally in order to determine which of the states or which of the uh actions we choose to get from one state to another, we have to have some sort of policy that determines which action we choose and why. And so, for example, that policy could be that the action that immediately gives us the highest reward uh would be the action that we choose. So this choice of of uh uh finding a particular action within the set of actions is called a policy, and as we step through these, let's say we just step from here from a all the way to F uh we finally uh get a reward, and so on each step along we get an incremental reward, and we end up with a final reward here uh F.
Now this is all good and well, um and the our our objective is basically to maximize uh this final reward F, and we significant uh signify this here with this little equation: um we want to maximize our expected reward given a certain policy and a certain State at a certain time. Okay, now um when we when we do this, um we're basically have an issue in that, for example, in trading when we enter a game, we're not necessarily getting a reward straight away. So we're stepping through a couple of steps, and finally our reward will be if we close our position in the simplest case. So let's say uh we buy stock X, and then you know the price goes up, the price goes down, finally it ends up going up a little bit, and we close our position, and we make a small profit.
Um, so in trading basically what what we can see is, other than a game, our states are our Market parameters. So let's say uh we are a Trader, and we look at a chart, and so let's say on the chart we have a number of indicators and and Technical uh things. So here here you can see this, for example, we have like a trend channel uh which is this channel here between those two lines, and maybe we have uh exponential moving average, or we have support and resistance lines, and all of these things help us to identify what state we are in with our trade. So so in some sense it is equivalent to look at a screen on a computer game. So all these different indicators, for example, they tell us something about the state of our system, and based of on the state of our system, we make a decision, and decision the decision that we make is basically one of three actions; it's either buy, sell, or hold; that's pretty obvious. So we really have those three decisions in trading; of course, we we can become more complex than that if we deal with, for example, portfolios, but let's uh start with the simplest case uh which is basically buy, sell, or hold, and again that that's already uh fairly complicated if we go into the minute details of it.
So let's assume um we are down at this point here and we go long. So we stay in the trade; we stay in the trade; our prices go up, and then they drop a little bit, and then they go up again, and they drop, and then they go up again, and then finally uh we exit our trade, let's say up here, and then we get a reward which is the p&l of our trade, and our goal is to basically get the best reward possible. So in this case uh which is really the ideal case, we enter here at the lowest point, and we exit at the highest point, so we get a maximum reward from this trade. Now, of course, that's the ideal world.
So what's the problem here? The problem is when we are in a trade and we decide, for example, uh to stay in the trade even if the trade goes against us, we basically have uh we we have the scenario where we actually might lose money, but we still have a conviction that this trade will really go in our favor in the long run, and so we don't want to exit it now. In traditional machine learning, what we normally do is we have a state at a certain time, and we give that state a particular label, and that label would be, for example, the immediate profit or loss of our trade, but the problem with this is if the trade in the immediate um future goes against us, then this label is actually not really correct.
So here is an example of that. Let's just assume uh we have this very strongly mean reverting price curve. So this could, for example, be the portfolio value of a pairs trade, and let's say uh this red line signifies two standard deviations of above our mean value, which is here at zero. Okay, now let's assume we're entering a trade once we Bridge the barrier of two standard deviations. So for example, that would be here. Now, as it always is in reality, just because we enter a trade doesn't mean the trade exactly goes in our favor immediately, and in this case you can see that our price curve of our portfolio moves away from the mean, and whilst we're reaching the two standard deviation uh line and we enter a trade, we will still move further away from the mean, as you can see here. And so what that means is we're actually ending up with a negative p&l or at least a negative unrealized p&l. So we haven't closed our trade yet, but we know, and this is what our experience perhaps our back testing tells us, that eventually this position that we entered here will revert back to the average line. So even though the trade goes against us, we're staying in the trade, and we're waiting and waiting and waiting until the trade reverses and finally perhaps again breaks through this uh line here, and once we entered here, for example, in a short position, then we can exit this short position at this point. Now what that means is somehow the machine has to understand that even though the trade goes against us and we have negative or negative p&l labels here, perhaps we still have enough conviction or the machine has enough conviction to then um stay in the trade and eventually end up uh with a positive profit. We could also call this delayed gratification. So when we enter this trade, we're not immediately getting gratification out of this trade; we're actually seeing, Oh my God, you know, we're actually in a drawdown, and then the gratification only comes later as our trade reverts back to the mean.
So the question is, how do we label uh such a decision-making process? And in traditional machine learning, that's really not that's really not easy to do because what we would normally do is we would say, well, we come here, and then we label every step along the way with a specific p&l, and so if our pnl just in this case goes negative, we have all these negative labels, and especially if it moves far away Enough From Here, we will just uh have to close the trade because it goes negative; there is nothing that can tell us, oh, in the future it might go the other way again.
So now we enter reinforcement learning; we apply a technique called retroactive labeling. So imagine we build a little table here just like this; have a look, look at that. So in this table um we can see, for example, uh here in this um in in this uh row, we have different price levels that our trade can have, and then here we have two actions: hold and sell. So what happens is at the beginning of the trade, we get into a long trade, so we buy. So the two actions that we can actually carry out are hold or sell. So at first, as we enter, let's say at $1,200, nothing happens initially. So if we or if we hold or sell, we still at a zero level. Then at 1250, if we sell at 1250, we make $50; uh if we um do hold, we make 0. Then at 12 then theice price drops a little bit; if we keep holding, 0; if we sell here, we end up with the $20 profit, and so on. And then finally, let's say at 1300 uh we really uh decide to sell; if if we don't hold anymore but we sell, we make a $100. Okay, so that's all good and well, um but basically here when we look at the hold column, we see all zeros, but in some ways these columns, these zeros are not they're not really real in some ways. If we hold, we actually have some kind of conviction that we would still be able to make some to make some money. So what can we do? We can actually use this equation here called the Bellman equation, and instead of just assigning a zero to all of these uh entries here, we use this equation to retroactively label each one of those actions with a certain value uh which is nonzero. So let me just quickly explain this equation. Basically, um RSA is in some ways the table that we've seen before, so we see s and a; this is s is the state, which is in this case just our just our um price levels, and a the action, hold or sell. Just a quick one—I've got some—okay, wonderful; I just had to check something. Okay, so now if we apply the Bellman equation here, we see GMA—this is a factor which we will explain in a bit—and then um we have q s A, and Q is basically this table here. So Q has um a state which is the state of the of the future uh trade and an action. So remember this is really a way of labeling a process for machine learning, and so this is not in that sense uh a way to to move forward, but basically to move backward to build a system that we can feed in our machine Learning System and actually find good labels to solve this problem of delayed gratification.
So when we look at this equation uh in this case, we set gamma at 0.9, and then we have this uh table here qsa; so the maximum of those decisions or the maximum of those values, we end up actually labeling each of these whole decisions here with a u a p&l value. So this is not a real p&l value, but and a sort of Quasi p&l value; it gives us an idea of how much our decision to hold is worth with as opposed to decision to sell at this point. And so if we do this in this case, we can see that if we make a decision to hold, that decision to hold is actually worth more than the decision to sell. So if we see this, we can say, well, let's make this decision to instead of of selling uh to actually uh hold our stock, and then finally when when it comes to the next level here, we make a decision to sell, and we go to this point uh where we actually close the trade. Okay, so this is the Bellman equation. Now let's um let's just uh dissect uh this a little bit more. So r, as you can see here, is the immediate reward of an action a. So the immediate reward is effectively the table that we've seen first, which had all the zeros in it. Q is the cumulative reward of an action a. So so that's effectively the other table. So we have a cumulative reward for each action, which is not a real reward as such, but it's an imaginary reward that we would get get if we look at a specific action uh at some point. Then S, as I said before, is the state that we end up with when we perform an action; so it's actually our our future State, and then uh so this is S Prime, sorry, and then s is the previous state. So you can see that this equation Works retroactively; we go from the future State uh and calculate the previous state, and then we have this GMA, which is called a discount factor, and Gamma is effectively giving weight to this future state with respect to the previous state. So if gamma is zero, then we can clearly see that all we have is that uh table that we started with, and the larger we make GMA, the more we actually uh give weight to Future outcomes. So for example, if gamma is zero, then we say, well, we can either um um hold, which gives us a zero reward, or we can uh sell, which gives us a ro for example of of whatever that reward was that we saw in the first table. But then if we apply a high gamma, for example uh the one that we seen in the previous table of 0.9, then based on based on our labeling um um we will see that that our reward of holding this position will be higher than the reward of closing that position. So this is really what's helping us to make decisions that are based not on a what we call Greedy decision-making process where we only look at the decision that gives us the highest reward right in that moment when we make the decision, but we're basically saying, oh, okay, even if our decision looks immediately not very favorable, we will still keep holding that position for a little while, for example, if if the price drops, and we will have the conviction that we end up with a higher reward later.
So that's reinforcement learning in a nutshell; it basically helps us to make decisions based on future outcomes rather than based on outcomes of immediate actions. So the interesting thing is in traditional reinforcement learning, we use tables as the one I've shown you, and we build those tables uh based on what we've experienced and based on the learning process. However, in trading, for example, we have a large amount of states; for example, if if we look at different indicators in the market and different, you know, different uh other other influences uh that that we might be interested in, and so we go through a lot of different states, and the complexity of our system becomes incredibly High, which means we won't be able really to build these tables anymore and store them somewhere. So what we're actually uh wanting to do then is find another way to build these tables without having this huge complexity of creating enormously humongous tables, and this is where deep reinforcement learning or where deep learning comes in because basically what we can do is, rather than building these these tables for real, we approximate these tables with deep learning or in other words, neural networks. So for example, um I just put a little diagram of a neural network here. Let's say we have four input States um which is let's say a moving average uh difference here; maybe it's the relative strength index; maybe uh X3 is some sort of Candlestick chart pattern; and maybe our fourth input is the time of the day at which this trade occurs or at which the time of the day at which we see this particular State. And then everything goes through a neural network, which is effectively an approximation of our complex table, and then we end up with three different outputs, and our three different outputs are buy, sell, and hold. So this is the interesting part; we use this neural network as a proxy for these tables that we've seen previously. Now we've seen now that we can have these tables; we use the Bellman equation, and then we approximate these tables with neural networks.
Now how do we implement this? Uh, this is very interesting, and the first thing is that we use something which is called gamification of trading. So effectively turn each trade into a separate uh computer game. So we see each trade effectively as a computer game. Now when we implement this, one of the difficult called bits is to find the right reward function. So I've already talked about this; I have that I have said that um the reward function could be just the uh profit and loss, but that's not necessarily always a good choice of a reward function, and I explain a little bit later why that is in more detail.
Now I've seen some people ask me a few questions here. Um, usually when I run this as a webinar for Quant in, I often answer the questions during my talk. However, uh today because we have limited time, I will actually uh ask these uh sorry, answer these questions after our talk. So if you ask if you have some questions, please bear with me, and what I would uh really prefer is if you hold off a little bit with your question and then send me the question at the end of my talk, and then I can answer them; that there will be plenty of time uh to answer some of your questions. Okay, so when we implement this, we really need to find the right reward function. Then um we also need to find the right inputs to Define our state. So of course we could use a wide wide variety of inputs; a very typical example in TR are technical indicators, but we could also use all sorts of alternative data, sentiment data, um results from satellite.
Images and on and on and on, um, and of course, some of them will be valuable, and others will not be valuable. Now, if we only have inputs that don't really have much of a predictive value, then we also won't be able to make good trading decisions, of course. Um, and we follow the principle: rubbish in, rubbish out. So when you use those, of course, make sure that you actually create inputs that have some sort of positive predictive, uh, behavior. Okay. And then, um, what we need to do is once we have built a system, we need to test the system for known behavior. So, for example, before we apply the system, uh, to use in market data, what we want to do is we want to first test the system with other things such as sine waves or or random, uh, price data or just just random, uh, functions in order to test whether our system actually behaves the way it's supposed to behave given, uh, the known inputs.
Then we also need to choose the type of neural network that we would use. We need to choose the size of our neural network, and finally, we also need to choose the cost function of our neural network, which is really, in some sense, dependent on which reward function that we choose. Okay. So let's, uh, talk a little bit about gamification, and I've already, uh, explained a little bit about this. So in the normal computer game, we have a state, which is what we see on the screen. We have a, uh, what we call a policy or our actions, which is our movement of the cursor. So our movement of the cursor is basically up or down in this case or stay where it is, and then the policy, of course, means, uh, why we would move the cursor in one direction. And then finally, uh, we get a reward, and we can see here these numbers usually when we score a point, uh, we get a reward, uh, based on the score. And so for trading, I already explained that, um, the equivalent would be something like, uh, a state: historical and current prices, technical data, and alternative data sources; buy, sell, and hold; and then finally our reward would be the P&L of our trade when we exit.
So how do we train, uh, the system? So first of all, uh, when we start training a system like this, what happens is the system will just randomly make decisions. It won't know anything about what's happening at the beginning, and so it makes a decision, for example, to enter a trade, and then it holds that trade for a while, and then at some point it just makes a decision to exit a trade, and there's, uh, a reward at the end of this, and then our reinforcement learner will use the Bellman equation to basically, uh, label our observations with, uh, retroactive, uh, labeling, uh, according to the equation we've seen, uh, before. But then over time, uh, the more we apply this sort of retroactive labeling and the more we update our tables, which are approximated by our neural networks, slowly but steadily the machine will learn to make better and better trading decisions. So in the simplest case, uh, we do this for just one instrument, and this is really what we're discussing today. So we're just, say, want to trade one instrument, let's say, um, the S&P 500, uh, index, the SPY, uh, in the US. So we feed the price data; we apply our system; we step through time; and then bit by bit the machine, uh, will learn how to trade.
Another way to do this also is rather than stepping, uh, through time sequentially where the machine learns, we can also just take our training price series and then just randomly enter at some point in this price series, play a game, and then exit, and then enter and, uh, randomly enter the price series at a different point, run another game, and then exit, and so on. So rather than running through the price series sequentially, we can just enter and exit at random points for training. So which one of those two you choose is really up to you; both are really possible, and and both work, work, um, and there is no really good or bad; it it really depends, uh, what your preferences are and how you would do it.
So I talked about the difficulty of designing a reward function. So there is really, uh, many, many ways, uh, to build a reward function, and obviously the easiest would be just to use a pure percentage P&L. So basically just the difference between the exit price and the entry price divided by the entry price, so just how much, what is your percentage profit? But quite often this is not necessarily the ideal way to build a reward function. Why is this? Because if you use P&L, then the machine could just decide to get into a buy and hold state and just basically decide not to exit the trade anymore at all, but just buy and hold. So, for example, if we had, uh, the, um, the US, uh, SPY index, uh, ETF, which has basically been going up since 2009 for a long, long time, your reinforcement learner might decide, well, the best strategy really is to buy and hold, which is not necessarily, uh, a bad decision to make, but sometimes this is not really what we want; perhaps we want to trade a lot more. So another reward function we could choose, for example, is just simply, um, rather than pure P&L on exit, we choose how much profit did we make per tick. So let's say rather than, um, maximizing the total P&L at the end, we maximize the profit that we make at every time step.
Another way to determine a, a reward function is to use the Sharpe ratio, also called the risk-adjusted return. So we might make a profit, but the risk during this trade was extremely high, so we're not actually really happy with that trade because we exposed ourselves to a lot of risk. So rather than using profit, we basically use a risk-adjusted return, which means we use the standard deviation of all our, um, profits at each tick divided by the, uh, standard deviation of these returns at each tick, and that gives us the Sharpe ratio, and that can often be a much better way to, um, determine a good, um, a good trade. Then another way to design a reward function is to punish our learning engine for long hold times. So as I said, we might just get into a buy and hold regime, but we don't want this, so we somehow, um, use our P&L, uh, for example, but then for every, for every, uh, time step that we stay longer in a trade than we, we would like to be in this trade, we punish our P&L by some value or another. Um, another way is to punish our function for drawdowns. So if we have a big drawdown during our trade, we could introduce some sort of punishment for that, which is really, in some sense, similar to the Sharpe ratio. Another way to re, to do the reward function is just, uh, having a binary win and loss. So, for example, if we win, we choose to give a reward of plus one; if we lose, we give a reward of minus one. And then there's some more complex, um, reward functions as well. So, for example, this one assigned categorical. So instead of using just pure P&L, we split our P&L into different categories, in basically P&L bands. So, for example, um, the first $100 P&L is band one, the second $100 is band two, and so equally, uh, if we lose $100 or between zero and $100, we are band minus one; if we lose minus $100 to minus $200, we are in band minus two. And then finally, um, we, we could do the same with the categorical bands, but instead of that, we use, uh, exponential, um, so rather than, um, using some sort of sign, niche, we could say, well, if if we, um, if we have really, really small numbers here in our win and loss, if we have really big negative values, then, um, our, we, we will be basically be in the zero band, but if we have really, really high P&Ls, we will be in a much, much higher positive band. So none of these, uh, solutions is better than the other, but it really depends on your own work and how you optimize, uh, your system to find the right reward function that works for you and the style of trading that you would like to use. All right. So, um, the next step is, uh, what features, uh, should we use for our trading? And so there's many, many things, uh, that we could look at. So one of the most obvious ones is, for example, open, high, low, close, and volume values. Uh, we could also, uh, say, for example, candlestick patterns. So let's say, uh, we would like to base our, uh, trading decision based on the open, high, low, close values or the candlestick patterns of the last n candlesticks in our, uh, trade. So, for example, here we see, let's say this candlestick pattern, and then we say, well, if you see this, we have a big drop in price, and if we see this candlestick pattern, uh, we might assign that that we have a really big, uh, drawdown in price at this point.
Then, uh, the next, uh, is technical indicators, of course. We all know, uh, technical indicators. So, for example, the difference between two moving averages, as we can see here, or other technical indicators like the relative strength index, uh, for example, and all the others, uh, that you have probably heard of. Some interesting features also are time of the day, day of the week, or time of the year, um, so, for example, if all things being equal, we take a trade at the beginning of the day, that might have a different outcome than if we take exactly, if we take the same trade or with exactly the same state otherwise at the end of the day. So, so time of the day and so on, they could be really important, um, input features. Then, um, another one are different time granularities. So let's say we have as our state technical indicators, uh, we could have these technical indicators on minute bars; we could have them on hourly bars, but we could also have them, say, on daily bars, and so all these different granularities, they produce different results. And even let's say we have the relative strength index, uh, this may give us a different value at the one-minute granularity than on the one-day granularity, obviously. So we will get different types of information with the same technical indicators, for example, at different granularities. And then, of course, we could do the same inputting, uh, the prices and technical indicators for other instruments. So, for example, if we, uh, use the, uh, the SPY ETF, we might be interested to also input the interest rates, the price of gold, the price of oil, and so on as other inputs to our, um, to our state. And then, of course, as I mentioned before, alternative data such as sentiment, um, maybe you know things like satellite images and so on. And so from all of these, we can construct a state, and don't forget that state is quite a complex input feature, and it's in some way similar to what we would input when we see a computer game. So this, the sum total of this, is equivalent to when we look at a computer screen in a computer game, and based on, on the sum total of this, our machine makes the trading decision. All right. Then in the next step, um, how do we test our, our reinforcement learner? So, um, when I build reinforcement learning machines, I want to know first of all if they work at all. So I have a series of tests that I apply before I even use price data, and the first one that I normally use is a clean sine wave. So I put this clean sine wave in rather than a price data, and then I do all the calculations, uh, um, and and basically build my state based on the sine wave, and normally what I would expect is very quickly our re, my reinforcement learning machine will learn how this sine wave works and very quickly start, uh, becoming very profitable, and because a sine wave is obviously very clearly predictable and easy to understand for a reinforcement learning machine.
Then in the next step, uh, for example, I could apply a trend, uh, trend curves, uh, and again I should see very quickly positive results from my reinforcement learning machine because we are obviously in a trend, and the reinforcement learner will pick this up very fast. Then, uh, on the other hand, I also will apply, uh, something called a random walk or like basically a randomized, uh, price series where where I don't have any structure in it other than just, just randomness, and with this we would normally not see, uh, the machine making consistent profits. And so if the, uh, machine makes consistent profit with a randomized time series, then usually that means, uh, that something in our coding isn't correct, and and we probably have to go back and figure out what the problem is. Then here we could, could also input different types of autocorrelations, um, then what I also like to do is I add noise to my clean test curves. So, for example, instead of using a clean sine wave or a clean trend curve, I use a noisy sine wave or noisy trend curve. And then finally, we could also input recurring patterns. So, for example, uh, we could, uh, input recurring, uh, chart patterns in our test system that the machine may or may not pick up and then trade on it. Okay. Then, um, the next question is what type of neural network, uh, should I use? So, um, in most of my work, I use just standard, uh, neural networks, but there's also the possibility to use convolutional neural networks or, and convolutional neural networks are often used in image processing or LSTM, which means long short, long or short-term memory, uh, neural networks. LSTMs are often really good in dealing with time series; however, the problem with LSTM is that the time series, uh, that it likes dealing with, uh, should be stationary, and this is often not the case for financial data. So in my, uh, own work experience, I find, uh, LSTMs don't really show much benefit; they're not worse than, uh, standard neural networks, but they don't usually provide a benefit, uh, for reinforcement learning, and the same in fact is actually true for convolutional neural networks. Now, of course, uh, what I'm building, uh, may not be, uh, ideal, and I'm not in that sense a machine learning guru; I'm first of all a, a trader, and I'm operating in the financial markets to make profit, and I use all these algorithms as tools, and I usually, uh, apply, uh, the things that work and don't take too much computational effort and just function in the way that that I want them to function. So in my experience, simple neural networks are just sufficient for what we're trying to do.
All right. So now let me show you some results, and, um, I talked about, uh, the testing phase, and so this curve, for example, is a, um, when we apply a sine wave with no, and and a trend with no noise. So you can see here at first, this, this is actually, this is actually at first where we apply a sine wave, and then approximately around here we apply a trend, and this is really interesting. So at first the machine starts learning, and it doesn't actually make much profit, so it's not doing very well because it's still in a stage where it tries to figure out what's actually happening, and then at some point it actually, it starts picking up the, um, the sine wave, and then once it's learned the sine wave, it consistently starts making profit, and then way they have a transition over to the trend, uh, which is, uh, around this, this, uh, halfway mark, it actually very easily transitions to, uh, trading a trend as well and keeps making consistent profit. Now, of course, um, these curves are very idealized because they don't have any noise, and so unfortunately, uh, the reality is not, not as good as this, and we can see this here if we apply a significant amount of noise to the sine wave, which is shown in this box, then, um, our reinforcement, uh, learning machine will already have a much harder time to, to learn and to trade, and we can see this here. So, so quite this amount of noise, uh, we can see that the machine already starts struggling to learn, uh, what's happening in this scenario. The problem, of course, with, um, neural networks is that they very easily overfit, which means that neural networks actually stop learning the noise rather than the signal. So, so that's that can be a hard thing. Um, so here similar, we actually have, we have a trend; we have a really, uh, strongly upward, uh, trending price curve, but we really add a lot of noise in into the daily, uh, trend line. So whilst the trend is clearly up, uh, over periods of time, every day has actually a significant amount of noise, and again you can see that our reinforcement learner really struggles, uh, to learn this. If the trend, if the noise gets, or if the signal-to-noise ratio gets worse and worse and worse, it will just learn the noise rather than the signal, um, and that is one of the big problems in reinforcement learning, the fact that we can't really necessarily, or the machine can't really easily distinguish between the signal and the noise unless we do something about that.
Okay. So now this is interesting, um, so what I've done here is rather than, um, putting in the noisy price curve, I actually made the machine learn a five-period simple moving average of the price curve. So what I did is really I took some of the noise out of our price curve; it's actually not a very big, um, moving average; five periods, it's it's quite small, so it smooths just a tiny little bit, but we can see that even with that little smoothing, which when you, when you actually look at the price curve overall, you won't even be able to see that the difference between the, uh, the smooth price and the actual price, um, but what it, what, what we can see here is that the reinforcement learning machine does a much, much better job in actually, uh, making profitable trading decisions. And perhaps when you build those machines, that little bit can give you a good hint on when you build your own machine what you should be looking at in order to make your, to make your reinforcement learning work well for you. So, so this is, this is one of the really, really interesting, uh, results that we can, can see.
So now, now that we've seen some, um, results here, um, I want to talk about a few challenges that we can see in reinforcement learning, and one of them is that financial time series are clearly very noisy, um, and and they will give us a hard time. The other problem is that financial systems are generally very dynamic. So the rules, uh, often keep changing. So there could be regulatory rules, rules, uh, market rules, for example, to transition from pit trading or open outcry trading to electronic trading and so on. Then we have changes in market regimes, for example, from trending to mean reverting, um, and one of the problems that we have with when we use neural networks is that we get this problem of local minima, which are really hard to escape. So once our, um, once our machine has learned a specific, um, a specific market regime or specific market behavior, then when the market behavior changes, it will sometimes be hard for our reinforcement learning machine to get out of this and actually learn a new behavior. Another problem with reinforcement learning is complex; so there's quite a few hyperparameters that we need to tune, which can be quite complex. And then the finally, uh, we also, so our reinforcement learning is quite simple, inefficient. So we need a lot of training data, but on the other hand, training data, uh, meaning market data, are usually quite sparse; we don't have an endless amount of market data available, so that's why, uh, training can sometimes be quite challenging as well. Now finally, here we see a full simulation, um, and this is, this is effectively our reinforcement learner learning with a really straightforward, um, P&L, um, reward function, and what you can see here, this, this is the US, this the US market, the SPY; what you can see is that it just gets into a very typical buy and hold pattern, which is actually not too bad because what it tells us is, hey, you know, um, buy and hold pattern is quite a profitable trading strategy on the US market, but it may not be, uh, really what we want, uh, so if we see this, we may choose to, uh, build a different reward function or use different input features, for example.
So what are the lessons, uh, that, uh, we've learned so far from this? Um, reinforcement learning overfits very easily; it mostly learns very basic, uh, market patterns. So what we often see is with reinforcement
Learning it doesn't necessarily converge on learning very complex behaviors. But often times, neural networks are a little bit dumb, and they learn very basic market patterns. Um, and so they only really act on those, uh, really basic ideas, which is not often what we want. We can overcome this with building more complex neural networks, but then again, if we do that, we can easily overfit the noise rather than, um, the signals. So, um, we end up with a complex—I call it a zoo—of hyperparameters, and this can be really challenging, um, and takes a lot of time to get right in when we build our system.
Reward function design is very challenging, um, and again it depends on what type of trading style you want. Uh, you have to build your own different reward functions. Um, so one of the most important takeaways from this is reinforcement learning is not a silver bullet. It's not something that you just apply, and then you let it run, and it will just automatically, uh, produce trading profits. We're not quite there yet, um, and we will probably never be there simply because someone else could come along, build this most intelligent machine, and just performs better than your machine, and you will just be losing. So reinforcement learning is a technique that we can use to trade on the markets, but it's definitely definitely not a silver bullet that will just produce us, uh, highly profitable, uh, trading returns with us not doing anything or with us not putting much effort in.
And finally, if you want to use this, I really really encourage you to get good market experience. I call it also domain-specific knowledge. Just being, um, good at machine learning, in my opinion, is clearly not enough, uh, to, uh, produce profitable outcomes. If you really want to do this, make sure you really understand the market and, uh, you really understand, uh, what you're up against when you do trading. And then, uh, in the end, if you want to learn how to code these systems, um, please check out my Quantiny lecture that I'm running. In this lecture, I will basically present you and walk you through a whole piece of code on or with with, uh, that basically builds a reinforcement learning machine. I will show you how it all works, what I explained to you in terms of actual real coding.
If you want to contact me, probably the easiest is on LinkedIn. If you have any questions that you would like to ask me on LinkedIn, please make sure you formulate your questions well and not just send me a garbled one-liner. That makes it much more likely for me to answer your questions because I understand it. And if you want, I've also got a couple of other instructional videos on YouTube. Um, all you need to do is just go to my channel, Dr. Tom Stark. Thank you, Dr. Thomas, uh, for such an enlightening session. There's so many questions that that I won't be able to answer them all, and I apologize in advance if if I cannot answer all these questions. Um, so, um, so let me go through a few of them. Um, so the first question here from, um, poop, uh, poop is, will the Bellman equation introduce look-ahead bias as s, uh, prime is the future state? Well, actually, uh, not quite because what we do is we only, um, we only do this analysis, uh, with the Bellman equation once the trade has actually finished, and then we feed this analysis into the neural network. So we're not actually looking ahead; we're just, um, we're just, uh, apply, uh, a retroactive labeling, uh, in, uh, going, going backward. But, but we're not looking ahead as such. Once we applied this retroactive labeling, we then use this going forward, the same labeling going forward into the future, but, uh, this does not actually introduce a look-ahead bias. I, I hope that makes sense. It's a bit complicated to explain in a short time, but, um, so so just so you know that that you know you wouldn't introduce a look-ahead bias.
Then someone says, do you actually use technical analysis indicators input for a deep neural net? Um, well, yes, you, you can sometimes, uh, use technical indicators as inputs, uh, but, uh, you have to do your due diligence of whether those technical indicators are actually useful or not. Uh, usually you do this, uh, with some other type of analysis, but, um, you have to be, you know, you have to be quite diligent. What is the significance of satellite images? So I mean satellite images? Well, um, some people have used satellite images, for example, the number of, uh, cars parked in front of a famous big department store in the US, uh, I think it was JC Penney, and they found that over time the number of cars in the car parks dropped, and this was quite strongly correlated to the stock price, and they could basically predict the demise of JC Penney based on those cars. Other people use, for example, they, um, satellite imaging of, of, uh, floating lids of oil tanks, and they can determine by the sunlight and the in the lids how high those oil or how much oil is in those oil tanks, and, uh, that that helps them to predict the oil price. So that's typical uses of satellite images. What's the time frame suitable for reinforcement trading? Can one minute or or even smaller, uh, time frame be used? Yes, absolutely. Um, the limit really is the calculation time of your neural network. So it depends how how much time, uh, your neural network takes to run. So if you try to do high-frequency trading with it in in the microseconds, that probably won't work because the neural network is just not fast enough. Um, so I need to read through some of those questions. How sensitive are reinforcement trading algos with respect to market anomalies? Well, uh, it really depends how well you train them. Um, they can be quite sensitive to market anomalies, but as I said before, uh, if you just do it in a sort of straightforward naive way, then quite often what those algos learn is quite, uh, quite dumb, should I say, or simple. So it's not necessarily, uh, uh, always great, but of course they need to be some sensitive to market anomalies; otherwise, we wouldn't make any money. Okay, next question. You focus on your networks. Can random decision forests, tree methods—so you focused on your networks. Can random decision trees or be trained using reinforcement learning framework? Have you tried this? Um, in principle, yes, but actually it doesn't really, uh, make so much sense. So you could theoretically do this, but it would be, uh, complex and not actually work, uh, very well. So so neural networks are really very well suited, uh, to this problem, and I would I would recommend, uh, to use those. I don't think, um, you would decision trees or or support vector machines or any of those would really work very well, uh, for this application. Um, so what about the cost? So what question here? What about the cost or loss, uh, functions for neural network training? Which ones do you use in case P&L based, uh, uh? So so you know there's, um, so for example, uh, for P&L, it would make sense to use the mean squared error as a loss function, whereas if you use some categorical, uh, categorical, um, categorical output, then, um, you probably want to look at, uh, categorical cross-entropy or or something similar. So so really you have to, uh, tune your, um, your loss function, uh, based on on the reward, uh, function that you use, and that that could be quite different, and and it requires a lot of testing to to do this. How much did you earn using reinforcement learning? I, uh, yeah, that's that's obviously proprietary information, but thanks for the question.
So there's a few more questions. Someone asks, what about high-frequency trading? Did you ex—what about high-frequency trading? Did you expand actions, uh, space using the market or, uh, uh, to using, for example, for market orders, limit orders with some distance from the market and so on? So so basically using reinforcement learning in the high-frequency space. Well, people have tried that, but again I should say that that there's a difficulty with this that, um, in generally, uh, neural networks are too slow to to respond fast enough in the high-frequency trading space. So, um, if you can if you are able to build really fast, very, very specific, optimized networks, you might get away with it, and it it might actually be very good. So I have seen people that have done some, um, basically theoretical work. So what they used to do is feed, um, the, uh, uh, price data into the neural network in in a back test, and then, um, run that through and and and make, uh, and have to reinforcement learn and make predictions. But the problem is that this is really really hard to do in real time. So if if you have the technology to do this in real time and you're fast enough, uh, then, uh, it's definitely worth trying. I I think, uh, you could probably achieve some really good results with this. So, um, someone asked if I'm taught on the Indian stock market. Actually, unfortunately not; I'm not very used to this. Um, so someone asked, I'm new to finance, already, uh, pursuing AI, so please suggest how I can combine my current knowledge and your offering to pursue a trading career. That is a very long and difficult, uh, question. I said if you're already in AI, then then I really suggest, uh, you get a lot of market knowledge first. So you could do this, for example, by actually, um, building a system which does some real trades in the real market. So you will learn a lot simply by sitting down and doing some trading. Um, there is a lot to learn in the markets; it's not as straightforward as it seems. And if if if you're really mostly in the machine learning space, I highly highly recommend, uh, getting a lot of market knowledge first before you really start, uh, going this way. There's obviously many ways you can have a career in finance, but this is a very, very long discussion which unfortunately, um, we won't be able to discuss today.
Okay, so this, uh, few more questions. How can you model a neural network to choose the best options trades? More inputs, strike, so basically, uh, someone asked, uh, can you use, uh, neural networks to get, you know, get good, uh, for option trading, and could you use strike prices, expiries, deltas, and all of this? Yes, of course, uh, that would be a really good way to do this. You could also, for example, uh, use, um, options data as an input for, uh, trading the underlying instrument. That would be, uh, another type of input rather than just technical indicators and all the other stuff I mentioned. You could use option chains and all the options, um, data and and see if you can, uh, find a way to to use that, uh, as inputs, uh, for trading the underlying, or you could also just, uh, trade options data based on on the options values. So, um, two more questions, uh, I answer here. There's a few more. Can you use a dual network to decide the number of, oh, can you use the neural network to decide the number of lots to sell and buy? Uh, yes, uh, technically you could, but you got to be really careful with this. Um, so one of the things you can do is rather than choosing entries and exits or, you know, like we had the the three outputs, buy, sell, and hold, um, you could actually use the, uh, numerical output values, uh, for example, if you trade a number of instruments, you could use the output values, uh, to to size your trade volume and then update the sizes of your portfolio, uh, weights when you trade. So so assume you you have instead of trading one instrument, you trade five instruments, and then your neural network gives you a value output for each of the instruments, and then you can use those values to to size, uh, your portfolio accordingly. So this is a way of of doing this.
So last question, um, just let me just have a quick look. How do you incorporate spread? So one someone asked how do you incorporate spread, commission, slippage into, uh, the algorithm, um, and are are they very sensitive? So so what, um, the way to incorporate spread and commissions, uh, is is really twofold. One is in the reward function where you basically instead of, um, where basically when you when you get a P&L reward, you just subtract your spread and slippage, um, from your system, and then obviously when you, um, when you then, you know, when you then look at at your trades, you have to somehow also understand, uh, what what what those are. So so so you need to, uh, yeah, so you need to build a model for your slippage, for example, because you don't know when you when you back test exactly what that slippage might be. You obviously know your trading commissions. So once you have built this model, then you can incorporate, uh, these things as part of the reward function, and that would be a good way to do this. Okay, so, um, unfortunately, uh, I haven't got any more time to answer questions, and, uh, thanks very much, uh, for staying with me, and I'm really sorry if I couldn't answer the questions from all of you. I really hope you enjoyed this talk, and, uh, good luck with your trading, and hopefully I see you sometime again in the future. Bye-bye. Uh, thank you so much, Dr. St. It was a really great, uh, Q&A looking at multiple questions that we have got. Uh, thank you so much, audience, for being and poods for this entire, uh, session that we have. Bye-bye.