Transcription
Have you ever noticed that things always take longer than it should? Despite planning more carefully and being more realistic and overestimating how long everything will take, things still take longer than you anticipate.
I've been maintaining a daily schedule for now over 16 years. And it's really only in the last two maybe 3 years that I feel like I'm able to plan things accurately, which helps me feel like I'm not always falling behind on my schedule and playing catch-up.
So, in this video, I want to share with you the things that I've learned over the years, both personally and as a cognitive performance coach, about why at a neuroscientific level, we as humans just suck at planning and what I have been doing over the last few years to finally plan accurately. And it's actually very simple.
And the first part is actually understanding is the problem that we are taking too long on executing on the task or is it that we are underestimating how long it would take? Cuz the solution is very different. And for me the answer became very apparent when I was in my fifth year of university. At the time I was doing my honors in medical science. It was with the department of opthalmology which is eye medicine and surgery. It's the first time that I had done my own research and published something by myself. And I remember my supervisor telling me in December even though you don't technically start until February you should start on your literature review now because it's going to take you longer than you think. And in my mind, I'd kind of done similar things in the past before. I thought it's not going to take me that long, maybe 1 month at most. So, being young and dumb, I ignored the advice of my very experienced supervisor and I left it until January. And lo and behold, she was right because it ended up taking me until March to actually get it done. But the thing is, during that period, I was very focused and it still just took me way longer than I had estimated. And what that taught me is that the issue is that I'm underestimating how long something is going to take. It's not just that I'm slow at executing on it.
And this actually has been studied. Three researchers by the name of Buler Griffin and Ross researched 465 undergrads predicting how long that they thought it was going to take for them to finish their thesis. On average, they thought it would take 33.9 days or in the worst case scenario, 48.6 days. But in reality it took 55.5 days. And this has been seen across multiple other industries and domains. A study in 2014 by Bent Flubag who is one of the most cited scholars in large scale complex project management found that 90% of largecale infrastructure projects like building roads and bridges and dams exceeded their cost and scheduling estimates often by more than 50%. So this seems to be this universal human phenomenon.
And what's really interesting about this is that when you look deeper at the research, we are not just bad at planning randomly. We do not equally overestimate and underestimate how long something is going to take. And you look at the research over the years across multiple studies, underestimating how long something is going to take is four times more common than overestimating. And in fact, when I coach people for years, I've been telling people estimate how long it's going to take and then add 50% on top of that. And even then, when I tell people to overestimate, it is still more common for me to see that they are underestimating. And so, we have to ask ourselves why. Why is it so powerful and so universal? And therefore, what can we do about it?
Well, the common reason that I've heard most people talk about is to do with this thing called the planning fallacy. So I'll explain what the planning fallacy is because it is important but I'll also then explain why that's not really the full picture and for most people it's actually not enough. So the planning fallacy comes from this research that was done in the late '7s by two researchers called Connean and Tverki. And they defined the planning fallacy as the tendency to underestimate how long it will take to complete something even when you know that similar tasks have taken longer in the past. And the core reason we think this planning fallacy occurs is because of this differentiation between something that is called inside view reasoning versus outside view reasoning.
So with an inside view what we're doing is we're actually taking each component each step of the process and we're walking through it step by step and estimating how long we think this is going to take us. We are imagining a mental simulation of going through the process and coming to a time estimate. And this feels like a very detailed and comprehensive way of planning. On the other hand, an outside view reasoning looks at the task that we're trying to complete and it says, "Hey, this task is really just an example, one instance of a wider category of other similar tasks." And so instead of working through each step of the sequence, we say, well, on average, these types of tasks tend to take me a certain amount of time. And we use the average completion rate for this entire category of tasks to say, okay, well then this task will probably take a similar amount of time. And so naturally using an outside view reasoning approach feels more generic, oversimplified, maybe not suitable for this particular context in this particular project. And so naturally because the inside view feels much more rigorous, much more detailed, more stepbystep, we strongly tend to by default favor an inside view of planning. And that that very strong neurobiological tendency is the root of the planning fallacy. And the reason is that when we use the inside view, our mental simulation of how everything is going to unfold is built on an optimum path. We simulate a smooth idealized path which tends to be very optimistic. If you wanted to make that simulation realistic, you would have to deliberately imagine obstacles and distractions and things going wrong, which people don't tend to do. And even if you do, it's hard to estimate how much extra time this unforeseen hypothetical distraction is going to add on to your overall time. On the other hand, an outside view approach tends to be more realistic because it's actually using realworld data of how long similar things have taken. Those naturally take into account the impact of common distractions and obstacles and barriers.
And so most of the time when people talk about this topic, they sort of stop here and they say, "Okay, well the solution is to make sure that you're not using an inside view reasoning approach and instead you make yourself use an outside view reasoning approach." But I personally have not actually found that to be that effective. And in fact, in planning for this video, I actually found this uh research by Roy Christenfeld and McKenzie, published a long time ago, 2005 actually, who looked at the research around the planning fallacy, and they found that just telling people about the planning fallacy and educating them, like I've educated you on it right now, does not actually eliminate the planning fallacy. And strangely, in some studies, it actually made it worse. And so what I have found to be a much more effective solution for eliminating the planning fallacy and actually giving you control of your time again is to dive a little bit deeper into why the planning fallacy occurs. And when you dive a bit deeper, what you find is that the planning fallacy is not just about inside view or outside view reasoning. And I believe that there are at least four other factors which are important that you need to address if you want to get rid of this planning fallacy.
The first of these other important factors is that the inside view is automatic. When I was learning about psychiatry and clinical psychology uh in medical school and also as a doctor, one of the things that I was taught was this phrase, don't think of a pink elephant. There's a variation of this which is don't think about a yellow jeep. And the idea is that when you start to not think about a pink elephant or not think about a yellow jeep, it's very hard not to think about it. Pretty much as soon as I say the phrase to you, if you were actually listening, you immediately start thinking about it. And the brain has a lot of these automatic responses. They're cognitive shortcuts which help save energy. It means that you're not actively having to initiate every cognitive process throughout the day, which would be exhausting. But the issue with planning is that using an inside view reasoning, going through the steps in the sequences is an automatic way that we tend to plan. And often when you're first learning about a task and what you need to do to complete it, let's say it takes you a minute to wrap your head around what this task involves. For that minute, as you are understanding what this task involves, automatically you are already using that inside view reasoning. After all, you can't use outside view reasoning until you actually understand the nature of the task in the first place to see what category it belongs to. And so, as a result, by the time you realize, oh, I shouldn't use inside view reasoning and I should switch to outside, your brain is already anchored onto those estimates that you made with inside view reasoning. And with complex projects that maybe it takes you 10 minutes, 30 minutes to fully wrap your head around what you need to do, you've been anchoring for 10 to 30 minutes with inside view reasoning. And this framing and anchoring is a very strong bias. It's what distorts your own logic. So that even if you know eventually that using an outside view reasoning this typically takes one week not one day you look at it and say well yeah but in this particular instance because of this and this it should be faster and smoother and easier than all those previous times. But again this is only one of the four important factors.
So the second factor is that all memory is biased. So if I ask you uh to recall some event that happened in your childhood, that memory doesn't live somewhere in your brain and is being pulled and extracted out for you to remember. Every single time you recall a memory, whether that's something that you've learned formally or it's just some event that happened in your life, that memory is being reconstructed by your brain. And the exact details of this reconstruction process is very complicated. But what you need to know is that it is not a perfect process. There is always the chance of introducing errors. And so over time, the accuracy objectively of what we're able to remember changes. And when it comes to planning, a really interesting thing that actually happens is that negative emotions fade in our memory more quickly than positive emotions. This is the reason why so many people have nice nostalgic memories. even though you distinctly remember it not being a nice experience at the time. In fact, when I first learned about this probably over 10 years ago now, I actually started deliberately trying to remind myself that a certain experience is not positive just to see if that years later I would still think of it positively. And years later I would recall that experience and it feels positive and nostalgic. And I distinctly remember telling myself this is not a positive experience. And when it comes to planning and estimating how long something's going to take, Rory and Christristenfeld, those researchers I talked about earlier, they a few years later also researched how people recall the experience of completing a task. And they found that when you complete a task smoothly, people tend to remember that more vividly than when the task completion is messy and full of distractions and interruptions. So, when you're deliberately trying to use outside view reasoning and you're trying to recall previous examples of similar tasks being completed and how long they took, your own recollection of how long those tasks took might not be accurate. They may be compressed. Now, FYI, emotions are not the only way that our memory is biased. We know that negative emotions fading quickly is a big part of this around why messy task completions are remembered less vividly. This is called fading affect bias. But even if there isn't a strong emotional component to it, reconstruction of memory is always a very biased process because when your brain is reconstructing a memory, it has to fit a certain structure or a pattern. It's almost like building a story. A story has a beginning, a middle, and an end. So, if you're thinking about a previous experience where you completed a task, events like being interrupted and then getting distracted and waiting a week for someone to reply to an email, these don't clearly fit into any of those places in that structure. So, in the reconstruction process, those parts of the event may not be recalled. They're just left out. And it's only when someone asks you and prompts you, hey, didn't it take ages for this person to respond to your email? Do you think, "Oh, yeah. Now, now that you ask me about it, I do remember that happening." And so, even if you're not, you know, on an emotional roller coaster every time you're completing a task, all memory is biased in some way, which is a risk to the accuracy of outside view reasoning.
And at this point, you might be thinking, well, I'm kind of screwed because it seems like my brain and my biology is really working against me. And this is only two of the four important factors. Uh but don't worry because at the end I will give you my actual solution of what I do on a daily basis which allows me to kind of mitigate all of these to a point where it doesn't really bother me at all anymore.
The third important factor is context switching penalty. So you know how I said before that when we use that inside view reasoning we're often not simulating interruptions and things that can go wrong. We're we're simulating this idealized path. One of the most common interruptions is context switching penalty. When your brain is working on a task, especially a complicated task, it has to build this context around how to think about the problem or the task that you're completing. And with complex tasks, it can take you a long time to build up this context. Some people call it a ramp up time. But your brain is not good at holding lots of things in the forefront of your mind at once. And so if you're working on a task and then you get distracted, an email comes in, someone calls you, a message, your dog walks in and you give it a pat and play with it for five minutes, these interruptions incur a context switching penalty because you have broken out of that flow and the context. And I'm not saying don't pet your dog. You maybe it's worth it. because you've broken out of that flow. When you come back to that task again, your brain has already dumped that context and you have to generate it again. It's kind of like clearing the cash on your computer. And research on attention residue has found that the process of rebuilding this context can take between 5 to 25 minutes. And again, it's especially severe when it's a complex task that requires a lot of focus and concentration. And so what that means is that if you're working on something for let's say 1 hour and you're interrupted just once because it takes you 5 to 25 minutes to rebuild that context again when you get back to your task your output across that hour may go down by 20 to 40%. Now, across an entire day, if you're getting interrupted, switching contexts five to 10 times a day, which is honestly pretty conservative, that can easily make a 5hour task take 7 8 9 10 hours. And when we're planning, we're not usually planning based on a distracted, ineffective version of ourselves.
Now, by the way, I've talked about some of these points previously, like the fact that memory is biased and how it can be biased. This idea of context switching penalty. These are points I've raised before and if you're finding this insightful and you wish you had known about these earlier so you weren't wasting time before, one of the best ways that you can make sure that you're on top of all of these insights is by joining my free weekly newsletter. Each newsletter takes about 3 to 5 minutes to read and it's kind of like a condensed version of some of the points that I'm mentioning. I'll take one particular principle or insight that I think is really useful for you to have known, things that I find that my clients really resonate with uh and then I share that and I unpack it in that newsletter and I give you a practical takeaway to work on for that week. Again, it's completely free. You can unsubscribe at any time that you like. So, if you're interested, I'll leave a link down below for you to join.
But moving on to the fourth important factor is something called motivated optimism or motivated reasoning. What is that spelling? Opimism. So motivated optimism is another kind of cognitive bias uh which says that we tend to expect better outcomes when it's concerning events that are meaningful and important to us. So if you're planning something important and you want that plan to go well and you want it to go smoothly, then that will bias us into forecasting and predicting that it will go smoothly. And like pretty much all cognitive biases because these things are hardwired into the way that our brain thinks. It is very difficult to avoid even when you are aware of it. It's just the way that your brain works. It's like telling someone not to be afraid when there's a tiger running towards you. Yeah, you probably could train yourself not to react with fear, but it's going to take a lot of dedicated consistent training.
But now, I did mention that for the last few years, I'm not really that affected by the planning fallacy anymore, at least not to a level that really bothers me. And so, how have I managed to overcome these things? This is what I do.
First of all, I ask myself how long something is going to take based on start and end times. So, I'm trying to use outside view reasoning, but at the same time, I understand how unreliable and biased it is because of all these factors that I've mentioned. And so, what I found to be helpful is not just asking myself how long did a similar task take in the past, but being specific about when did I start and when did I finish? And usually when I ask myself that question, I can't actually remember. I don't actually remember exactly when I started and when I finished. And what that tells me is that my memory of how long a similar thing took in the past. So for example, if I think it took around 3 weeks, if I can't label the start and end time, that 3 weeks is probably not reliable. And so by forcing my time estimates to be explicit, it catches errors in my recollection. If I've been working with other people on similar tasks, I might ask them about those start and end times. Uh we use software to track a lot of our projects as well within the team. So sometimes I can just look on the software to see when the start and end time literally was. If I'm able to get a time estimate from objective data of a similar previous task, I will try to use that as closely as possible, even if it feels like it's too long. There are a lot of risks that come with underestimating how long something is going to take and not many risks with overestimating. And the risks with overestimating are usually much more solvable than the risks that come with underestimating.
Now let's say that that time estimate is not objectively available. In that case, number two, I am always going to overestimate by at least 50%. And this scales with uh difficult tasks. So if I think something is going to take me 30 minutes, I'm going to estimate 45 minutes. Block that out into my calendar. If I think it's going to take an hour, I'll block out an hour and a half. Now I set the scales with complex tasks. For me, what I found is that when I estimate something is going to take about two hours, that's a danger zone because it means I'm aware that this is complicated enough that I'm going to need to be working on this for a decent amount of time. And so, intuitively, it can feel like, all right, I'll give myself a buffer. I think it's going to take 2 hours. I'll give myself 2 and 1/2 because 30 minutes is a pretty decent buffer. But 30 minutes is not 50% more. 50% more means that if I think it's going to take two hours, I schedule in three hours. And when I first started doing this, it felt excessive. And most of the people when I recommend this, they feel like it's just too much time to dedicate. But what I've actually found is that the longer the task is, because there are more opportunities for things to go wrong, you actually need an even bigger buffer. Sometimes for a 2hour task, I'll schedule 4 hours, maybe even 5 hours.
But that brings me to the third strategy that I use, which is that if possible, I try to split sessions. I try not to schedule anything in my calendar that's going to be longer than a 2hour block. And it's the same reasons that I already explained 2-hour blocks. In 2 hours, there are a lot of opportunities for things to go wrong. And so, if I genuinely think something is going to take 2 hours and therefore I need a schedule in 3 hours, I try to split that session. I might do two one and a half hour sessions or maybe three 1-hour sessions. I'm being realistic here about the fact that my concentration and flow might fatigue. There's more opportunities for context switching penalty to interrupt me. And so by creating a smaller, more focused session, I reduce that risk by a bit. As an added bonus, I found that it also allows me to see what is required to complete the task in more detail, and it makes it more likely that I'm going to pick up on little tasks and steps that I actually previously missed. So that's the third strategy which is to split the sessions.
The fourth thing that I do, I do this anytime I think that something is going to take me about a week or longer. So these are slightly bigger tasks, maybe entire projects that you're working on with multiple sessions across them. And in these situations, I try to do a premortem, even a quick one. So if you're not familiar with what a premortem is, premortem is basically saying, imagine you failed. Why would that have happened? So imagine a task takes you twice as long as you anticipated. Why might that be? And you think through all the reasons that are even somewhat likely to contribute to that? And if you write down a list of all the things that might inflate your time, you can make a decision about whether there's something that you can do to prevent that risk or mitigate that risk. And sometimes there are, which means that you're eliminating distractions in advance. Or sometimes you look at it and you think there's either nothing that I can really do to prevent that barrier or interruption or it's just not worthwhile. Like it would take so much work, so much time for me to just remove this barrier which may not even occur. And in those situations, I overestimate even more. I basically say there is a real barrier interruption, a risk here that I cannot control for. So, I'm going to additionally factor that in into my time estimate. I thought it was going to take a week, but this particular issue I don't think I can do anything about. So, I'm going to give myself a week and a half. I'm going to give myself two weeks just to account for that risk.
And when you apply all four of these strategies, you usually end up with creating schedules where everything that you're doing seems to be really conservatively overestimated. And that is actually my fifth strategy, which is well, I just call it a sense check. Go through the schedule that you've created and ask yourself, does this feel a little bit excessive? Do I feel like I've given myself a bit too much time, a little bit too much margin of error? The answer to that should be yes. When I look at my schedule, I try to make sure that the way that I've scheduled the things that especially things that are important to me that that really matter like I need to deliver a certain type of output by a certain time. I want to feel uncomfortable at how much time I'm giving myself because like I said the risks of overestimating are usually small and manageable and the risk of underestimating can be sometimes unreoverable. So if I'm looking at my schedule and I'm thinking, "Yeah, this should be reasonable. I should be able to get all this done as long as I stay locked in and focused." For me, that is a red flag. I might have one or two days every now and again that's like that and I really focus in and you know really lock in for that day. But if that is the norm of your schedule, it is very very rare for me to see someone who feels that way about their scheduling and actually feels like they are on top of things and they're consistent. Almost every single person that I've ever coached that talks about their schedule in that way, they're usually falling behind on their schedule. they've got a backlog of things that they they're not able to keep up on. They feel like they're not in control of their time anymore. And so when you are uncomfortably overestimating, yes, sometimes you will finish task a little bit earlier and that becomes like a bonus. You can do some more tasks. It's like a stretch goal, but it gives you so much more control over your ability to plan for things. You are no longer reacting to yesterday's version of you who didn't deliver what they promised they were going to deliver for you. Delays are usually fine as long as you can predict them. And by using these five strategies for the last couple of years, I've been able to eliminate the planning fallacy from about 90 to 95% of all my work.
Now, another great strategy or set of strategies that complement this really well is learning how to do really effective prioritization of your tasks. I think most people actually prioritize just based on urgency, which is a a very bad way of doing things. So, if you want to learn a little bit more about that, then check out this video here where I go into that topic in a bit more detail. Otherwise, I hope you enjoyed this video. Thank you so much for watching and I'll see you next.