Transcription
Ultimately, no company, no country, no person has ever succeeded by just focusing on the same thing over time. You need to expand your horizons and that's what's helped make America successful. And I think the successful companies are the ones that do that as well.
Hey everyone, that was Eric Brinen talking about the risk that comes from resisting change. He is a professor of economics at Stanford University and director at the Institute for Human-Centered Artificial Intelligence. Brook Bryan and I met up with him on the Stanford campus. We talked about AI and GDP, what organizations can do differently to realize new productivity gains, and how we should rethink what the initials AI stand for. This is Mackenzie Talks Talent. I'm your host, Lucia Rahilly, with talent experts Brooke Wed and Brian Hancock.
Professor Eric Bolson, welcome to the podcast.
Thank you. Good to be here.
It's great to have you. And Broo and Brian, great to see you as always.
Great to be with you.
Great to be here.
So, let's start with a little context. At McKenzie, we talk a lot about the AI paradox or what we call the AI paradox. The gap between this massive investment that organizations are making into AI and the um appearance of meaningful at-scale enterprise value and returns. Your research shows that we may be turning a corner on that. Talk to us about the J curve and and where you think we're going.
What we're seeing is what we call the productivity J curve that these powerful technologies to get the full benefit of them you also need to make process changes you need to reskill the workforce sometimes you need to invent new products and services and that all takes time that's actually the hard work of it not you know buying the LLM while you're making all those intangible investments um there's a lot of work and it doesn't instantly translate into new output but then once the intangible investments are in place then you can start harvesting them and things start taking off. I'd say we're seeing some evidence that it's just beginning to show up in the national productivity statistics. I think it's actually too early to say for sure, but we're definitely seeing it at the micro level for specific companies. When it's rolled out, you know, effectively, you can get really big gains. The issue is that that's not something that's happening pervasively through the economy yet. I'm sure it will though.
What we're finding is that, you know, when we're working with organizations, I would say 70 to 80% of the work isn't on what should be automated or what is the tech enablement part of the workflow. It's what are the new roles, what are the behaviors of those people in those new roles, what are the skills required, even what are some of the underlying mindsets that are going to enable success.
Is that what you mean by doing the work?
Absolutely. Yeah. you have to sort of get in there and get into the nuts and bolts. But one way that you can get more visibility into it is break the occupations down into tasks. Every occupation is a bundle of tasks.
And typically AI can help with some of those tasks. You know, helping you write a memo or analyze um some documents. It can't help with other parts of the job. You know, if you have to lift a box or something, an LLM is not going to help you. And so, every occupation we looked at has some parts of it that are going to be more effective than others. And that means managers need to sort of reconfigure and reorganize and rebundle those tasks. That's actually takes a lot of creativity.
Yeah. I was I was actually intrigued by something I I heard you say in a different context, which was when you think about the agenda for AI and AI innovation, we should really be thinking about trying to hone AI as a capability and tasks that humans aren't good at doing. Right.
Yeah.
And so as you think about the work you've done on what is the automation potential, how do you factor that point in?
So many technologists are obsessed with making AI that imitates human. Like Alan Turing said, the Turing test, the ultimate test of AI is if you can make an AI that you can't distinguish from a human that perfectly imitates it. I think that's maybe kind of a cool concept, but it's a terrible business strategy. Um, instead what you should be doing is having the AI focus on what AI can do well and not what humans are already doing well. Um, and that's something that, um, makes AI more of a compliment rather than a substitute. And most of the managers I talk to, um, their first instinct is to have AI sort of replace the workforce. I'll tell you, I was talking to a CFO not that long ago, and she was saying, you know, we really need to measure the ROI of AI better. And I was like, "Absolutely, you got to measure it." And she said, "And therefore, we're going to go through each department and see how much headcount reduction they're getting from." And I was like, "Well, okay, that that's one measure. It's it's a very narrow measure, but it's so pervasive, this sort of cost mentality, because let's face it, it's the easiest thing to measure." And people think of AI replacing people. But the bigger upside is if you can get AI to allow those people to do new things that they've never done before.
Yes. So, so what are some practical examples of where AI can extend the humus?
I mean, one of the places where we're seeing it happen the fastest is in coding. And it's nice because you see there you can see some of it substituting and some of it augmenting often within the same occupation for different parts of it. And what we see is that some of the routine tasks AI agents can do those, but what's happening is the senior coders now are finding themselves augmented. they're able to do so much more. They are working with fleets of agents, not just one or two. Some a lot of the routine execution, the machines can do that better. So there's a reconfiguration, a rebalancing of what people do. And ultimately, it sometimes leads to, you know, creating more similar code. But the really exciting thing is when you're able to do new things that you never did before.
What are the implications for wages for folks at that higher end who are using AI to augment what they do versus the replacement context.
So we worked with our partner ADP, the world's largest payroll processor. Uh they gave us access to data so we could analyze it real carefully and what we found was that there were some pretty noticeable effects on the employment side. Some occupations we had falling employment, some we had growing. that depended a lot based on the age and how exposed it was to AI.
On the wage side, we did not see a big change in the wages.
I wonder if one of the things causing the lagging effect that you you might be seeing as one of your hypotheses is many of the companies I'm working with today are just trying to figure out the new way they do performance management of humans first and foremost. So, what what do I really want to assess, right? Are there new behaviors? Are there new skills? And you know, some uh people are have been talking about for some time now. How do I do performance management of agents too to kind of go in a completely different direction?
Yes.
The companies that are most successful are the ones that are best at measuring things. This is really the frontier that that we're very much in the front of right now is getting better metrics of what's happening. You know the old saying you can't manage what you don't measure and um we need a new set of metrics.
So I have to ask what what are the metrics that you would prioritize?
Well, so you know we want to boost productivity and the way economists define productivity is you know it's output per input and too many people as we were just talking focus only on the input part of it. So you need to be creative about coming up with better measures of the output side and that includes uh better quality better customer service new products um actually less employee turnover can can be part of it as well. I think most companies underestimate how much administrative data they already have. Ultimately, as you instrument the company more, whether it's a manufacturing company, sales, um you start being able to have all sorts of fine grain metrics and you can see how they move as a company rolls out the technology. A related part of it is is getting causal inference from that is you don't want to just have the correlations between these things. You have to really see whether they're causally related. And that means using different techniques like differences and differences and instrumental variables that economists have been using. There's been a credibility revolution. And one of my missions is to bring that to business and have that same kind of credibility in the business data.
How have you seen kind of the asks that you have of clients evolve over kind of the past three or four years?
What we're seeing is we're using the client's own data and doing a little bit more of a bottom-up assessment. Every company we've worked with has the same pattern which is this sort of this power law distribution where there's a few people that are performing at 10x 20x. If you can capture what those best people are doing in each of the different occupations and kind of codify it and replicate it and share it with the rest of the organization, it's a really easy win to level people up.
Yeah. The point about the you know the mechanism for learning and and how to do that at scale. My observation is that is a huge need right now because I think many organizations, you know, especially leading ones that said we're going to really unleash AI. We're going to let a thousand flowers bloom and now they're saying, okay, that was great. There was also a cultural byproduct of people kind of did some upskilling around AI, but now how do we move towards more of a valuebacked maybe top down agenda on it? And so they don't want to throw away all of that, but there has to be a more continuous way to scale.
Oh, absolutely. That's such an important point is is that while it's great to capture this knowledge, you also want to direct it. You got to have I think it's from both sides. You got to have the bottom up and the top down direction so you understand what's possible, but also make sure it it's directed at some useful outcome and not just playing around.
So Eric, you're talking about accelerating um the Jcurve trajectory, progress on that J curve and genification obviously would be a huge accelerant in that context. And you did some interesting research on the costs of agents and the surprise that um humans are really bad at estimating how token intensive and how costly certain tasks can be relative to others. Do you want to talk a little bit about that?
Yeah, there's a big disconnect there. You know, just really just a few months ago, people were token maxing and they weren't worried about like the token cost and now it's getting to be a you know, first order thing. companies are spending, you know, tens of millions of dollars on tokens. The problem is that you don't know how much uh token cost is going to go into a task and the agents themselves are very bad at estimating. And so this is a a big gap now because, you know, how can you start a project and you don't know what it's going to cost or you get halfway into and it doesn't finish. Um, and so there's there's an agenda there for doing more work on understanding
how much you're going to have to spend, you know, on humans, on tokens, etc. going first.
You and many others, Mackenzie included, have called this the most consequential transformation since the industrial revolution. Do you think we should be doing more to prepare for this kind of consequential change?
We absolutely know what I say.
Are we ready to take stock?
We are not ready. We, you know, um it's I'm impressed by these improvements in capabilities and how rapidly things are getting better, especially around here in Silicon Valley, people are spending literally hundreds of billions of dollars pushing those capabilities even faster. Uh at the same time our understanding of what's going to happen to you know employment, productivity, wealth, income, wages um you know uh centralization of power or decentralization uh even meaning we don't have a good grip on that at all and there's very little being invested in it and that I think is the biggest challenge for the next 10 10 years or so is figuring out and how to close that gap between these capabilities and our act our economic understanding. I worry a lot that we're going into a very disruptive period. I mentioned that I'm optimistic about productivity, but I'm worried about uh inequality and centralization of power and the disruption that's likely to happen even if we have a bigger pie. It could be very disruptive for in fact it probably will be very disruptive for a lot of people and uh and we're not ready for that.
And it's also one of the things I'm worried about because when I kind of aggregate across the client conversations I'm having when you hear people talk about augmenting talking about the front of office tasks these are people in sales or innovate and creating the products and when we're thinking about automating it's the back office and then if you start to layer on well who's in the front office who's in the back office it no longer looks like hey now all of a sudden this wave of automation is coming for people with mast's degrees or you know advanced capabilities. It looks like it's a different disruption pattern. What are you seeing in the research?
And it seems like it's cutting across lots of different groups. Now I was uh you know stunned when a student came to my office a month ago. Um she was a graduating senior from Stanford and she said you know she didn't have a job her friends didn't have jobs. She said quote is my generation doomed and I was like oh my god that's really hard. Um and and I see that kind of angst. Uh people are really worried. Um what I tried to give her some hope because I think that yes, there's lots of jobs being destroyed. I don't want to sugarcoat that. There's also all this opportunity being created. You have to have much more agency and aggressiveness to lean into that. It's not that someone's going to tell you what your job is. It's that you have to be the creator. At the last my final class for my Stanford students, I told them that you know when you hear the words the when you hear AI, you should not think artificial intelligence. You should think amplifying intention because that's what it does is if you have some intention. This is going to allow you to do a lot more than you ever could have before. I understand that's not what the conversation is about, but I think we need to change the conversation and remind people that this really could amplify whatever intention they have.
Yeah. I actually want to loop back to, you know, understanding how to prepare better for an AI future. What are your hypotheses on what could be a potential solution or set of solutions here? Is it solving a collective action problem? Is it an incentive problem?
Every company's going to have to reinvent invent itself. Part of it is a collective action problem like you defined. Um what we're seeing is that you know the pyramid in lots of companies McKenzie and and in universities elsewhere where people come in and they kind of work their way up is becoming more of a diamond and that base isn't there anymore and you know that saves them cost in the short run but that means where are those middle managers middle skill people going to come from and where are the senior people going to come from if they're not people in the entry and that's a real challenge part of it I think you have to have public investment in education and supporting training part of it. I think enlightened companies are leaning into it and AI itself can be a great teacher. Um you can get a lot of personalized education. You were talking about that earlier in terms of the mass customization. Um so there's an opportunity there.
You did find in your research that early career professionals are being disproportionately affected. Do you want to say a little bit about that?
Yeah, we did find that the groups that were shrinking employment were the early career uh entry level workers aged 22 to 26 in the most exposed occupations. You could rank all of the occupations about 700 of them uh by how exposed they were. And if you take the top, you know, 150 most exposed occupations, the top quintile, um what you found is that the young workers in those categories had about a 13% fall in employment when we first did the study. um we've been collecting every month since then is up to 16 or 17% now. The effect has just been kind of growing over time. And so that's why when that student told me about her troubles, you know, I didn't, you know, dismissed it. I I saw that in the in the data.
Do you think the trend will expand to more senior workers in the sense that it might only be a matter of time that more senior employees could also be susceptible to these kinds of
Well, that's a big concern. That is a possible future. Um, but another possible future is that we lean in and and show them how to use these technologies to augment what they're doing, discover new products and services, expand the market. But one of the things I really stress in talking to the senior executives and for that matter the policy makers and economists is, you know, this is a design problem. Let's figure out the right incentives, the right structures, the right intangible investments so we come out on the winning side of that. Um, we should not be passive.
Yeah. I think I think a lot of organizations though I completely agree are are a little stuck there and they want to know immediately how should the pyramid evolve and the it's a it's a choice just like any operating model or organizational design there is no one right answer so thinking through how is value created in the organization and different parts of the organization what would that mean or imply for what humans do versus what agents do either one can work and then only then once you have a view of that can you say, "Okay, here's the talent system to support it." Right? But that's a process.
Um, and I I think many organizations are moving straight to the efficiency part of this versus
that's kind of like going straight to the efficiency part. Honestly, I think it's a little lazy. You know, it's it's the easy way is to cut costs. It's not so hard to measure, but I don't think it's as sustainable as if you find new sources of value and you invest in your workforce. And you're going to get more resistance from your workforce if you know you tell them this is a tool for cutting heads. Um, and so I I really think there's a much bigger opportunity. I know McKenzie is leaning into this to try to find ways of creating new sources of value. It's what happened with the earlier technologies. You know, uh, all through history, wages and employment grew even as new technologies came along. And we want to keep that record going.
But when when you're talking about design, what I hear you say is we've got to be thoughtful in the design of multiple different levels. There's a societal level. How are we investing in the education and the training? How are we creating the right signaling mechanism so that the skills you gained in one place that are particularly relevant, another employer can pick them up? But there's also the
company level design. Yes. When you're reinventing, how are you thinking about to use an analogy, you know, you you've talked about before, you know, electricity is another, you know, general purpose technology. It's not about just about putting electric lights in the factory. It's about actually electrifying the entire factory which requires you to rethink the entire way the factory works powered by electricity. But that's a you know workflow level design but it requires both the societal level and you know the company and workflow level design for it to work.
Yeah, we've got work to do at all those different levels at the uh level of specific companies. I think you need to understand the tasks so that you can then do the restructuring. Those two kind of go together. Um, I think of the task as the atomic unit and once you understand where AI can affect each of those, then you're in a better position to do the reinvention that needs to be done.
Yeah.
And I I think there's a skill gap there. I think there's I think companies are under pressure in some ways to use AI to drive efficiencies. But I also think reimagining a workflow or an enterprise is not something that most leaders know how to do, right? And and so one of the things that I'm working with a lot of organizations on is creating a playbook for doing that and doing that consistently, right? What's that methodology?
I don't I don't think we we actually really
No, it's it's it's an inherently harder problem. It just takes more creativity. You you can look at what already exists and say how are we going to automate that? You know, that doesn't take a ton of creativity. But to imagine something new, that's harder. But my view is actually it's more risky to not do it. that if you just focus on what you're doing and trying to, you know, hang on to things, that's actually the riskier thing that ultimately no company, no country, no person has ever succeeded by just focusing on the same thing over time. You need to expand your horizons and that's what helped make America successful and I think the successful companies are the ones that do that as well. I think one of the areas that's exciting uh for AI is uh small businesses
because now they're able to access, you know, talent and capabilities, you know, that they wouldn't have been able to access before or to access them may have had to, you know, sell their physicians office to a private equity practice in order to have the back office taken care of. But now, hey, there's AI tools that can help them, you know, keep their business and grow their business while it's still their business. So,
it's exactly what we talked about. in my my master class. You know, you've all heard about the one person unicorn. And maybe that's kind of a little bit of an exaggeration, one person with a billion dollar business, but it's not that much of an exaggeration to have a small group of people who now can use AI to leverage so much.
On this topic of small businesses, I'm just thinking Brooke and Brian, uh, do you see companies or Eric, do you see companies actually successfully building AI native businesses and within their larger organizations to innovate and test and then integrating them? Is that happening yet?
One of my clients described as engine one, engine two. You know, engine one, the current business, the way it's going, engine two, hey, I'm going to invest in the disruption. So this is going to be my new AI native fill-in the product or offering here. And some others are saying, hey, actually I'm uncomfortable with that because what does it imply for people left in engine one as we're gearing up engine two? Does everybody know that that they are in engine one and engine one's on the wrong track,
you know? So do we need to actually invest behind the lead innovators that we have today across the business and then scale that up? And I think the you know the jury is a bit out among the clients of which way is are we able to do it because the engine two gets a lot more tangible output faster
but is that going to be the ultimate answer that you know gets you to scale I think that's the question that that you know our clients are wrestling with. Yeah, I I do know um one large um global banking client actually in their they you know kind of took this experimental approach and some of those experiments became famous within the organization for the problems that they were solving and so they actually have one that is kind of an internal go and see for how to do AI transformation well and how to use it as a force for augmentation and they're of course using that as an input into this learning scaling model that we've been working with them on. So I think it's exciting. I think that's a kind of a more nent approach or in this case that that model is is nent.
We've been talking a lot about augmentation about ways to push novel thinking etc. in order to continue to drive acceleration in uh productivity and also in um to keep humans engaged and employed and so forth. There's also a lot of discussion about um information collapse and the potential for information collapse if AI becomes too much ballast for humans in the workplace. How do you assess that risk?
Well, there is this concern that AI is generating more and more of the content on the web and more and more content in books and more and more content everywhere else. I mean, we're all humans. I can vouch for that. I'm pretty sure. Um, check.
One of the things that we need to do is make sure we have incentives for content creators, human creators, to continue to create the value so the LLMs can continue to become better and better. And right now, existing copyright law doesn't really necessarily reward the content creators. Um, um, and you want to strike a balance. If you give the content creators too much incentives actually, then the downstream people don't um, have incentives to use it as effectively. On the other hand, if you don't give them enough, the content isn't created. And so, I think going forward, we need to sort of rebalance the way we reward creation versus use of different kinds of knowledge in order to have kind of a thriving ecosystem.
We're we're seeing that very live happening um at a firm level in professional services organizations. You know, okay, you're the expert. Your expertise is what generates the client demand and the move. if you're not appropriately incentivized to do that to put your information into the firm systems and do it, it kind of collapses. So, we're seeing a lot of client interest in, okay, how do I set up a knowledge management system to make
Well, you've got smart clients because I I've also seen the opposite problem where uh or the opposite approach and it's it's really destructive. You know, in our call center paper, uh what we found was that the less skilled workers got the biggest boost from the LLMs and they were now performing almost as well as the most skilled workers. and some of the folks uh who talked to me at using these cult. Oh, that's great. Now we don't have to hire as many of those most skilled workers anymore because the less skilled workers are doing almost as well. Um and that's a very short-sighted approach because exactly as you're saying, where does that knowledge come from in the first place? It came from the most skilled workers. In some ways, they're even more valuable now because they're not only answering a question for their own client, they're answering a question that then gets replicated throughout the organization. So you want to make sure you invest in those highly skilled workers that are creating the really valuable content. That's how you upskill the rest of the workforce. But you need a a company that sort of is sufficiently uh forward-looking that they understand that and don't just reward them on what they're doing today, but also how their knowledge creates value in the future.
Earlier in the conversation, we talked about the CFO client of yours who reflexively turned to headcount reduction, cost cutting,
right? What kind of capabilities do you think leaders need to build? As Brooke said, it's an entirely new set of capabilities that leaders need to build to reinvision processes end to end and um reap the benefits of these intangibles. Are you seeing leaders do that successfully? And what kind of capabilities do you would you cite that they would need to?
It's really hard. There aren't that many that are doing it successfully. I mean, we've been working with NASDAQ, and I think it's one of the companies that's been doing a great job of this. Um, they've got some great leadership there that is um for a long time has been reinventing organizations even before LLM came along, but now they're continuing that track record. But most companies are are struggling with it. It's understandable. I think people um underestimate how difficult that is.
So, you mentioned agency, entrepreneurship. I go to you know thinking boldly because many of these workflows are cross-disciplinary and so in my experience getting a group of cross-disciplinary people in a room first of all everyone kind of is a little defensive well my part of the workflow is working great right so we don't need to reinvent that and there's a there's a lot of you know like learning in those spaces that require openness to actually get to the other to the truly reimagined end of that and I think leaders that can set the conditions for that in a way that allows for that thinking in a bold challenging way. Anyway, those are the qualities that stand out to me.
It's really hard, you know, and and CEOs have to step up and play a role because a lot of the rest of management is just not selected for having that kind of big transition. They've got something that's working and they want to preserve that. And so, it really needs a jolt to the system. And you also need the openness that you talked about that, hey, you can do this. It's going to be okay. um we don't expect it to go totally smoothly and let's be bold.
Yeah. Actually I was with a group of it was um you know business executives including CEOs and then also some military leaders and we were discussing you know some of these behaviors that are required for change and the one that or it's it's kind of a behavior but kind of a role looking for the mavericks in your organization and those can exist at multiple levels but a maverick is someone who is going to challenge who is going to be bold who isn't going to be defensive and you know a question that might be worth asking is you know to a CEO or or CH HRO is do you have a good sense of who your Mavericks are, right? And are you strategically allocating them to drive this transformation?
I like that. This is a time for mavericks.
Yeah. Do you want to answer the $64,000 question that is plaguing parents from shore to shore?
What are the skills that kids will need in the future? Kids who are in college now?
That's a really tough question. My framework is is less to go into specific skills and more into this broader framework. And it gets back to what we said earlier, having that intention, figuring out what is it that you really want to do. I think that's a skill that can be engendered even, you know, taught um and having that kind of openness to let's be the person who directs the project. Let's be the person who comes up with new ideas. If you can have kids who have that kind of an attitude, they're going to be much more successful in the future because I think almost everyone is going to be managing not just an agent, but a fleet of agents. They'll be like the CEO of their own little entity there and they'll have to have those leadership skills leading agents to to direct them and the ones who are good at pointing them in the right direction and then evaluating them are going to really thrive.
So from a policy standpoint should we have more investment in things that enable entrepreneurs?
I think so. Yeah. One of the things that um you know we look at over the past couple of decades is that you know technologies advanced but productivity hasn't grown all that much and part of the reason I think have to do with less dynamism in the economy. There, you know, even though we see a lot of it here in Silicon Valley, there's actually uh fewer startups overall in the United States. And if we can get more entrepreneurship and more dynamism, then I think the technologies are going to have a bigger uh beneficial effect.
Eric Reen, thank you so much for joining us today. This was great.
My pleasure.
And Brooke and Brian, great to see you as always.
Yeah, it's great to be here. Thank you.
Thank you.