Transcription
[Music]
Right. My name is Dan Davis. As said, I am a former economist and banker. And when I say former, I mean that I haven't done any economics or banking for 723 days. I'm in the process of drawing up a list of people to apologize to for the harm I did when I was an economist and banker, and I'm taking it one day at a time.
Uh, what what kind of got me out of the economics though was the banking because I spent six years of my life doing degrees to analyze a world of perfect information, rational expectations, and then I went onto a trading floor where there was no information and no rationality. And that's what got me interested in how big organizations make weird decision. Decisions because the kind of point that I'm going to try to make over the next, uh, kind of 20 minutes to half an hour, is that a lot of the time it is the organization which is making that decision, not the people in it.
We often get this situation where all the people are great, but the organization is very bad. And I'm going to try and explain why that is.
But I'll start off with, here's a really useful slogan. You know, often we find ourselves in a meeting where you want to say something that sounds kind of intelligent, maybe a bit profound, a bit cynical, worldlywise, but you're thinking, 'Happens isn't really appropriate for this meeting.' You're thinking, 'Now it is what it is,' is a bit like five years ago. I can absolutely recommend this posid: "The purpose of a system is what it does." You know, just at the end when everyone's trying to trying to ask what's going on, you can just sit there and go, "Yeah, well, you know, the purpose of a system is what it does." It's a great slogan and it does actually mean something. And that's what I'm going to try and explain now.
But I think first, you know, always start off with a couple of good war stories. And this is, yeah.
When you write a book and your publisher says, "We need an engaging, exciting, and amusing anecdote to motivate this whole thing." I would say bear in mind that you're going to be talking about that for the next two years at talks. And it gets a little bit sickening.
This story relates to the airline KM and 400 cute little squirrels. It doesn't end well. If anyone is really sentimental about squirrels, you might want to cover your ears for a while.
These poor creatures arrived in Shiphole Airport in 1999 as imports, basically, to a pet wholesaler in Athens from a breeder in Beijing. And they didn't have the right paperwork, and they were in the animal hotel, as it is called, at the cargo terminal at ship hall, and there was they created a problem for the people working there.
They couldn't be sent on to Athens because they did not have the relevant biocurity paperwork. They couldn't be sent back because the, uh, breeder hadn't filled in their details and wasn't paying for the flight back. Can't go on. Can't go back. Can't really stay here. What do you do with them? It's a tough question. It's hard to believe that there wasn't a better answer than what the KM handlers actually did, which is, unfortunately, all 400 of them, apart from two or three that had escaped, were thrown into an industrial shredder that they happen to have hanging around in a corner of the audience.
How do I know about this? I know about this because I went to business school, and the press release that KLM sent out apologizing for this disaster after questions were asked in the Dutch Parliament is still studied as a masterpiece of crisis PR to this day.
And it's a really kind of strange thing that happened because they concluded that no one had really made a mistake. The Dutch Ministry of Agriculture had implemented a European directive on, uh, the importation of rodents. KM had put in place policies saying that everything had to have the right paperwork or animals would be euthanized at the expense of the, uh, customer. And the people in the shed just carried out the orders in that pragmatic Dutch style. They went, "Better buy a shredder then."
And the problem was that no one was able to communicate that something bad had happened. If you know, there are two social classes in the world. There are people who are able to challenge corporate policies and people who work in sheds. And this was, you know, what had happened, and 400 squirrels paid the tragic price.
Okay. If anyone next to you has got their hands over their ears, just nudge them. That that concludes the squirrel part. There will be no, there will be few more mentions of animal peril or harm in this talk.
Another example of the kind of phenomena I'm talking about, and I will bring these together in a couple of slides' time, is the British education system, which is a great example of how good people trying to do the right thing can produce results that they simply didn't intend.
For example, you can have a combination of two unbiased systems making a heavily biased system. What do I mean here? Consider the exam system of A levels, which, uh, all the British people in this room will have gone through. We have an unbiased exam system. It is relatively unbiased in terms of the social demographics of people who pass those exams and get grades. It is subject to an appeals system which is also unbiased.
However, the combination of two unbiased systems means that you now have a system which is heavily biased in favor of people who can get their mistakes corrected. Incompetence is a form of bias because it's bias in favor of people who can get mistakes corrected. So, for borderline grades, which are the only ones that matter for university admissions in the UK ad education system, the system is heavily biased in favor of private schools. It is heavily biased in favor of the middle and upper classes, and it is biased to a ridiculous, comical extent in favor of the children of teachers because they know how to handle the appeals system.
That's the education system on its best day. And the coronavirus pandemic, uh, A-level season in 2021 was far from its best day. They couldn't hold the exams because of the pandemic, but they needed to provide people with grades in order to keep the university admission system functioning. And so the Department for Education handed it over to their education researchers to design an algorithm that would give people grades based on their teachers assessment.
Now, there's a funny thing about education researchers. They're often very good, very smart people, but they care about grade inflation more than anyone else in the world. Most of us forget about our exams a few years after we've taken them. Most of us regard the fact that the grades keep on getting higher and the exams might be getting easier as a bit annoying when you're comparing your results to your kids, but not something to think about all the time. Education researchers really care.
And so they designed a system which would minimize grade inflation and maximize comparability. What they created was a system that had two heavily offsetting biases: a very large bias in favor of small classes and a very strong bias against schools which had bad historic results. And so it blew up. The, uh, they produced an algorithm that was guaranteed to ensure that there were maximum number of headlines of the smartest kid in a tough comprehensive who had dreams of going to medical school and was deprived of them by the algorithm. Uh, as is happens always in the British system, the people responsible were either sacked or kned depending on their level of seniority in the organization.
Again, what had happened here is that a kind of information had been considered that wasn't related to the actual goal of the organization.
So, what do these two things have in common? This guy, Stafford Beer, I've spent like like the last five years or so kind of trying to resurrect some of, uh, his work from the 1970s. He was a management theorist. He was a yoga guru. He, uh, helped to reorganize the economy of Chile for the Aende government. A really weird, interesting guy.
But his key insight is that organizations are information processing systems. They take in inputs from the world and they produce decisions, which is kind of interesting in the context of some of the things we heard about this morning.
Um, if you're worried about vast nonhuman decision-making systems with completely alien priorities taking over the world and potentially destroying human life, the time to start worrying was probably several hundred years ago with the formation of the East India Company.
The modern corporation is a decision-making system because it makes decisions. It's non-human because it's not a human being. It is an artificial intelligence.
And what Stafford Beer said is that, like human beings, organizations have weird behavioral ticks. Organizations can hallucinate like an artificial intelligence system can. And they do so for the same reason, which is that the data they are working on isn't fully representative, and their method of processing that data just heads straight for median solutions which don't correspond to reality.
So Stafford Beer said that if a corporation is an artificial intelligence, then you should be able to apply the mathematics of information theory to it. You can't apply them in a straightforward manner because, unlike a computer, an organization doesn't have a finite and formatted set of information inputs. You have to constantly be engaged in the work of deciding what information you're going to pay attention to and what information is going to have a role in the organization.
And there's one particular kind of message that I think is really important. This is the red handle from a train. What happens when the driver pulls the red handle? Right? I kept on saying that. I even wrote it in the book. It's a really embarrassing mistake. When I did this presentation to these transport people, when you pull the red handle, the train goes. It's a dead man's handle. What happens when you let go of the handle? The train stops.
But I would argue that this thing here is mainly an organizational tool. It has the physical property that it stops the train moving. But if you're running a railway and you just physically stop one train and don't do anything else, bad things are going to happen on that railway. The red handle starts a process of reorganizing and rewriting that day's timetable. A single driver isn't usually able to rewrite the timetable. But by providing this signal, they say, in contact with the real world, something has happened which was not part of our system and not part of our plan. The organization needs to reorganize and respond to that.
One of the frustrating things about those poor squirrels is that I was actually kind of lying when I said earlier that you have people who can rewrite policies and people who work in sheds. If you work in a shed for KLM doing engine maintenance and you drop your spanner and it hits part of a jet engine, then you can stop things. You can even rewrite the entire flight schedule for KM that day if you have, uh, screwed up your maintenance badly enough. For safety critical applications, KM did have the kinds of communication channels between the consequences of the decisions and the people responsible for making the decisions that they didn't have in that benited shed of the squirrel apocalypse.
There's a name for these kind of connections. I would argue that the connection between the the consequences of a decision feeding back to the decision maker is what we call accountability. When it is absent, that is unaccountability. Where you have power without accountability, you have power located in a decision maker who is living in an information environment where they do not know the consequences of their actions. That is dangerous.
So why do we do it? The answer is that we always have to cut some of our information connections. The only people who pay attention to every bit of information they get are babies, and it makes them tired and cranky after a very short period of time. As we grow up, we start ignoring things because we start concentrating at things. And deciding what you're paying attention to is identically deciding what you're not paying attention to.
Organizations get more complex as they grow. They get more complex at a multiplicative rate just simply because when you add something to a group, you multiply the number of connections between things in that group. And our capacity only grows at best linearly. Every now and then we have a technological revolution like the computer or AI. Even the filing cabinet was an IT revolution in its day, and we get a step up. But always we are struggling to reconcile our own capacity to manage information with the world's capacity to generate it to literally make things manageable.
So we are always designing our information connections. Shouldn't we pay attention to how we do that?
In most cases, Safford Beer said, and there is a whole load of science going on in the background here that I am sparing you from: You have one of three big problems. You have important information channels have been cut because the messages they were giving were constantly bad news and which were constantly kind of crossing hierarchies and giving people messages they don't want to hear.
Uh, my daughter the other day made the incredibly tasteless joke after moving into her student accommodation that they had a carbon monoxide arm, but it kept making a beeping noise that was making her feel drowsy and giving her a headache, so she turned it off. I said, "I'll do that at the next presentation, but never say that to your mother." Um, organizations do similar things. If you have a channel that gives bad news to powerful people and those powerful people are overloaded, they will tend to cut that channel.
Organizations are also quite bad at understanding that they are creating their information environment in everything they do. Often people who are incredibly important in transmitting and processing information, even more so people who are important in translating information from one form to another, look like they're not doing much. They look like all they're doing is having meetings. They look like they don't produce revenue directly. They tend to get treated as deadwood and cut out. And that's how organizations basically debrain themselves.
Most importantly, though, organizations can set themselves up so that they are no longer able to understand or even formulate key information. And of course the main way in which this happens is with financial information. Financial reporting and financial data is one representation of the company. It throws away a huge amount of information. Notoriously all accountants will admit that they don't handle intangible assets at all well. But data about branding, about customer interactions, it's just extremely hard to translate that into financial numbers. A company which manages itself purely by financial reporting will lose touch with reality because it is no longer managing its business. It's managing an imaginary object, a sort of shared hallucination that's about the same size and shape as its business.
So I'll now unpack that slogan I started the meeting the presentation with. The systematic output of a system, what it actually does reliably and repeatedly, is defined by the kind of information it responds to and the relationship of that input to the decisions it makes.
If your company has a set of values, then either those determine the information it pays attention to, or they are not actually part of the purpose of the company.
The location of power within a company is entirely the power to decide what information the company is paying attention to.
Organizations are information processing systems. It's just that more and more of them these days are badly designed information processing systems in which most of the important parts have been cut out because they were causing too much trouble.
And, uh, I think my counselor's run down there. So I'll leave it to you.
Switched on. Yes, I am. Fantastic. What a joy. Um, absolutely tremendous. Also, I'm I'm very pleased to say, in his fantastic Substack piece, which I think is called D Squed Digest, isn't it? Is that right? It's it's called Back of Mind. Back of Mind. Sorry. D Squed Digest is your name on Twitter. You're absolutely right. Um, you also make the point that you got particularly angry with economics because it pays no attention to marketing.
Um, it is really, okay, this is actually embarrassing because I spent six years of my life and did, uh, two degrees in this subject. The best theory of marketing that economics has is that it is a way of showing to the public that this product must be good because we spent so much money on marketing it. You know, that that is literally, you know, people, that that's a Nobel Prize, is aloff, is it? That Gary Becka Nobel Prize? I mean, Akalof is slightly better, but it's, you know, won a Nobel Prize. And the thing, the reason is that economics works on price and quantity. They have decided to model the world, which is a business of deciding what you're not going to include in your mental object, on the basis that everything is either a price or a quantity of a standardized commodity. If you want to tell a story about how something's not a commodity in economic or financial terms, you just can't do it. It's like kind of trying to, it's like trying to express a sound that doesn't exist in a language and be using the Roman alphabet, you know.
And there's a fantastic example, by the way, of "the purpose of the system is what it does," which was last week when I met the editor of The Drum, which is an advertising publication, who said, "You can always tell when someone's had media training because they never get any press." And the reason for that is that once someone's had media training, journalists can immediately tell they've had media training. So, they know there have there is no likelihood they will ever say anything of interest. So, they immediately ignore them and never contact them ever again. Absolutely. That we're a perfect case of something which is notionally has the has one intention; the actual what it does is the complete opposite.
Yeah. I mean, that was as as staff of beer said in the, uh, designing the system for organizations is you cannot say that the purpose of something is an outcome that it systematically fails to achieve. Yeah.
And this, I mean, this is, it's fascinating because there are, I think, large, worthy parts of bureaucracy, by the way, which aren't necessarily that expensive in terms of the people they employ, but the net effect is just to slow things down to an extraordinary degree. So, you add this huge degree of complexity. You were getting particularly annoyed, I think, about some aspect of planning legislation online, which is that you're allowed to discuss a planning application with your turn.
Yeah. The, the planning inspectorate for "why doesn't the UK build infrastructure?" Um, part of the problem is that if you're building an infrastructure project in the UK, you can consult the planning inspectorate and say, "What are the big problems I'm going to have here? What are the kind of key issues that are likely to affect this?" As soon as you do that, the person you spoke to is regarded as having a conflict of interest and cannot be part of the examining authority for your project. The, the UK has set up a thing which is: knowing something about a project immediately disqualifies you from making a a judgment on it. And just in general, we have loads of situations, in particularly in government, actually less so in corporate organizations, where because of perceived or easily identifiable problems of conflict of interest, we force everything through the tiniest of information bottlenecks. And when you force everything through that bottleneck, your decisions are either going to be slow or bad or both. We're just not paying attention to the fact that fundamental theorem of information theory, you can't put more down a channel than it has bandwidth capacity.
So, interestingly, someone's asked here, "You mentioned you think the private sector's better." I'm not sure it is, actually. I mean, they do things differently. Um, and they, but they're kind of bad in different ways. The, the worst thing in the private sector is financialization. And the biggest driver of financialization is debt because once you have a company that is heavily indebted, it's a signal that swamps everything else. It can't pay attention to anything other than immediate cash flow in order to make the next debt repayment. And that's an organization that is bound to make mistakes because it is no longer living in the real world. You know, at the end of the day, customers live in the real world. They go, they go out there every day. A company that is driven by its customers can't drift too far from reality. A company that's driven by its accountants, who are, you know, great guys, many friends, married one, um, their job is producing a systematic set of illusions. That's what they do every single day. You know, as someone said, "Much more fiction is written in Microsoft Excel than is ever written in Microsoft Word."
Um, you know, and actually, on that point, do you think that software and the standardization of information is a significant contributor to the problem? Because it looks like an efficiency, but two two issues. The spreadsheet effectively gives numerical information vastly more weight than other forms of information, with the PowerPoint deck. The problem I have is that nobody in a shed uses PowerPoint. Okay, so there are two classes. They're the, I mentioned this on Evan Davis yesterday, they're the class of people who can present things, and then below a certain level of the organization where people just don't do PowerPoint effectively, you're frozen out of the decision-making loop.
Yeah, absolutely. And it comes back to Roger's point about the conflict between finance and everything else in reality. Once a number's in a spreadsheet, it really looks like a fact. You know, you pick a number out the air and put it in a spreadsheet, everyone else will treat it as a fact. Uh, because otherwise they would have to make up their own number, and no one wants that job.
Um, because Will, I I wrote a book a couple of years ago with Will Butler Adams, the CEO of Brmpton. Oh, bright and Will said they don't do revenue forecasting in Brmpton because his experience is that if you have a revenue forecasting process in which the projections are challenged and form the basis of decisions, what you are running is a training course for your employees in how to lie to you.
Perfect. That's probably our time limit. All I can say, all I can say is, if you think that's bad, try try working in an advertising industry where there is no correlation between the value you add and the money you make because you charge by the hour. Okay? That makes it worse still. Thank you very much.
[Music]