Transcription
You know, I don't blame people when they are confused about AI because the studies that are coming out are also confused. This week, October 28th, Wharton came out with a study on generative AI return on investment and implementation at very large companies. If that sounds like a familiar subject, it should because MIT studied the same group of companies just a few months ago.
The kicker is this. MIT study had a 95% failure rate on AI projects and Wharton came back with a 75% success rate. Now, it does not take a lot of mathematical skill to figure out that these are not compatible numbers. You cannot be both correct. And so I want to spend time unpacking what is really going on at the enterprise, how we put these two numbers together and what is a reasonable path forward that cuts through frankly the headline nausea that I get from all of this just top lines that don't make sense and that keep changing all the time. Business needs consistency. Business needs clarity and business needs to be able to actually build in a way that makes sense. So, my goal is to ground you by the end of this so that you don't get spun and confused when people are saying, "Is it 75%, is it 95%." Here's what's really going on.
95% came out of the extremely tight screen that MIT put on Project Success. That is one of the ways they effectively engineered a headline that would go viral. And yes, I'm just going to say it. I think they engineered the headline because the screen is tighter than almost any other internal software measure I have ever seen. In this case, what MIT was saying is every project is by default a failure unless you can measure a dollar and cents impact on the bottom line, not the top line of the business within just a few months. It was like 6 or 12 months or whatever it was. If you can't do that, then it's useless. That is no other software that I have seen. If you're buying software is measured that way. You always measure it on internal metrics that you think will map to larger business value.
And that brings us to the Wharton study and the 75% success because Wharton took more of that approach. Wharton's approach was to talk to executives and let executives tell them how they're measuring ROI. And what executives said overwhelmingly is that they're using other metrics. They're not just using dollars and cents on the bottom line. They're looking at productivity. They're looking at time saved. They're looking at throughput. And when you look at all of those, execs feel like you get a very clear measure of success. And that's where that 75% number comes from. So really, if you want to know first what the heck is going on and why they're different, it's apples and oranges. You have a very hard profit measure from MIT and you have a looser, more conventional software ROI picture from Wharton.
Here's what both of them are not getting right and where I have sympathy. I think MIT is correct that we need to hold AI to a pretty high bar. This is a transformative technology. It's also an expensive technology. It is on the verge of being 10x or more more expensive per employee than any software was before. Yeah, we're going to have different ROI measures. So, I think that MIT is getting at something when they're challenging leadership to think differently about software purchase ROI. But I think Wharton is doing a great job actually analyzing the the reality on the ground and the way execs by definition and by convention and the way they usually act really measure stuff.
So, my ask to you, if I were to like take all of this away, my ask is that you not pay too much attention to these kinds of headlines, I get inbounds, right? I get emails, I get messages coming in. I get it. It is confusing when the news media loves to report contradictory information. But the reality at organizations that are succeeding with AI is a lot more steady. And that's the piece I want to leave you with from a grounding perspective. When we build with AI systems and they actually work, there are a few things that do align really well with both of these studies and I'll sort of explain that, but they don't like the studies don't get at them, right? They don't get at how to positively build and that's you know me, right? Like that's what I love to do.
The first piece that I want to lay out for you, think of these as sort of building blocks that you can build institutional fluency with. So, I talked about individual fluency a couple of weeks ago. I want to talk about institutional fluency today. I think that is one of the missing pieces that connects these two studies. And I think that understanding how it works will help you to not get swept and pushed around when the next whatever study comes out with whatever number.
The biggest piece of institutional fluency, if you want to set up a sort of whole companywide fluency on AI, your company has to get good at understanding and shaping context awareness for teams and individuals. And I think teams are really the atomic unit here. Individuals come and go, but teams are steady. Teams have a particular vertical they take care of. Teams have a particular domain ownership. and institutions that are fluent in AI understand that the value of the team is the context they inhabit and specifically the context they're able to articulate to AI systems. So when we talk about context engineering typically that's a job. I'm suggesting that we think of it less as a job and more as everybody's job. Context is something that we all bring to the table. Context is something that teams need to deliberately maintain. What do I mean by that? Right? Like if you understand at a very deep level, this is the way my domain actually works. This is the way I actually drive value for the business. These are the unique processes and workflows that I can use. These are the areas of uncertainty and the areas I need to explore in my domain to get better. And if you can articulate that intentionally to an LLM as a team, you are going to be in a position to deliver multiplied value to the business relative to individuals working on AI alone or relative to the work that we've done before 2022 sort of pregenerative AI. Context is king here. Context helps us to feed an AI with what's needed to be useful at a local level within the business. If you can't figure out how to help your team to articulate context to the AI, you're going to have trouble with everything else. And this is one of those things where like if you look at the Wharton study and the success, part of what's going on here is that leaders are saying that they are seeing accountable acceleration amongst teams. Like the way I read that is that leaders are starting to see teams pick up and use context in their disciplines to drive value and the executive just kind of gets to measure it, take credit for it potentially and count it as a success. So context is the first piece I want to call out. Institutionally fluent organizations in AI understand that context is local that context operates at the team level not the individual level and they are deliberately fostering team level context fluency.
The second piece that I want to share with you that institutionally fluent organizations have is problem solving skills. And this sounds really obvious because we've been talking about problem solving skills as an element in sort of employee training and upskilling for for decades, right? Way before generative AI. But socializing those problemolving skills is something that managers, directors and above are coming to me privately and saying this is really hard. This is not easy. And I think that part of why we see the discrepancy with the Wharton and the MIT measures is that the MIT measure, the 95% fail rate measure demands that a entire organization be so good at problem solving that it meaningfully upshift the bottom line. That is an extremely high bar. You can get a whole bunch of teams who are good at problem solving and if you have two or three bad apples, you will bottleneck somewhere in your process and have trouble delivering value to the bottom line. And so what we need is we need to treat problem-solving skills as a critical patch on team fluency that we cannot live without that we must have on every team and that we will hire for if needed to get done. In other words, AI problem solving is becoming all of our problems today now and it doesn't get better until we actually fix it.
So what does AI problem solving look like in practice? We can say it, but what makes it something that a team can reasonably learn? Because keep in mind, if teams know context, teams are going to know problems and teams are going to be able to sort of learn to solve problems. I want to suggest that problem solving is really, if you peel the onion back and you think about it deeply, a function of understanding how AI thinks about and processes information. Because if you think about problem solving conventionally before AI, we're really manipulating information in order to unlock ambiguous problem spaces. And so traditionally, it would be like, I'm going to write my product requirements document or I'm going to do this data analysis. And we're manipulating information in order to get closer to unlocking a complicated customer experience or a painoint in operations. And all of the stuff we talk about like critical thinking skills, good writing skills, those were all ways that we could scale up manually so that we could successfully manipulate information as an individual and as a team to solve these problems. And in that world, individual skills mattered a lot because individuals pushed information fluency forward. Right? If the individual could write well, they might write well enough that the whole team was elevated, right? And then ownership resided at the team level. And so a team manager would be responsible for would own solving the problem, driving around obstacles, all of that stuff you want good managers to do. That is starting to flip. And I have never shared this before. I think this is really interesting. I think that what we are starting to see in the age of AI problem solving is instead the individual needs to index really highly on ownership and the manager or the team needs to index highly on skills and that's sort of a reverse of the usual. So the problem solving skills, the ability to understand how LLM works, those actually can reside at the level of the team, but the ownership piece has to rest with the individual if we're going to make progress. And I'll explain why that flip has happened.
When you think about solving a problem in the age of AI, what you really are doing is you are understanding enough about AI to feed the AI the problem in a way that it could understand and work with. And I've talked about this part before where you're sort of chopping up the problem, decomposing it so that the robot AI can pick it up and manipulate the problem and help you get through the problem space faster, which is the whole goal. It is easier to solve problems if the robot intelligence is working on that problem with us. Here's what I haven't talked about before in practice with real teams building real AI systems. What I'm seeing is that ownership is irreplaceable at the level of the individual working with AI. If you don't have a very strong sense as an individual, as an individual contributor of ownership and quality and assessing the bar that AI is using to solve and insisting that the AI isn't doing good enough when it really isn't, you're not going to be able to add any value at all. Whereas in the past, you could have that bar set at the team level and the manager would be able to sort of manage the informationational standard and it would be okay because all of the humans were working together and information was moving slowly enough and we were exploring the problem slowly enough that the manager could act as a quality bar. In this day and age, that's not true. AI is giving everyone so much superpower that you have to devolve ownership down to the level of the individual contributor. And I think that at root is one of the reasons why organizations are struggling so much with the AI transformation. It demands more of our individual contributors than it ever has before. And we're not used to a world where the individual contributor is the atomic unit of the corporation as opposed to the manager. Corporations are founded on management theory. The idea is that the manager is accountable to for the domain for the department. They are the representative of the business. They work with the individual contributor. That's how how we've done it for hundreds of years. I am beginning to think that that is not how AI native organizations are actually going to be configured. The power you have with AI resides so heavily with the individual. I don't think you can do it any other way. I think you have to put ownership at the level of the individual contributor. And that has profound implications for how we train people. Because really what we need to train people to do is you need to start by taking ownership of your domain and your situation, of your problems, of the way you work with AI, of the bar you use it, everything flows from that. And ironically, what we previously had at the individual level, this sort of skill, hey, this is a really skilled writer, right? This amazing writer, uh, and we couldn't do it without him. And he lifts up the whole team. That kind of thing can now reside at the team level. Look at how teams are sharing prompts with one another. Sharing clawed skills with one another. How teams are sharing custom GPTs with one another. AI is enabling the commoditization of a lot of those skills. And when it comes to AI problem solving, you can encode a lot of the technical skills and understanding of AI in sharable format. And so let's say someone isn't super familiar with how transformer architectures work and how you want to chunk problems so that the AI can read the problem coherently. That's okay. You write a prompt for them. You share the prompt with the team. You have a brown bag where you talk about what it does, but they can just immediately run the prompt and the skill translates and they can gain skill over time as they socialize with the rest of the team. But what you can't do is give them the skill and they don't have the sense of ownership. That breaks. That does not work.
So we've talked about context. We've talked about problem solving and how it sort of inverts traditional team and managerial norms. There's one more piece that I want to talk about today that I think underlies this concept of institutional fluency that that isn't talked about very often. I think that previously the concept of taste, the concept of is this excellent, is this extraordinary, is this something that is an incredible offer for the customer, we could delegate that to a small, call it a priesthood within the company. The Steve Jobs of our company is over here. He has taste. He's an extraordinary builder. He's an amazing inventor. We'll run this by him and that will be fine. I think in the age of AI, taste is something that doesn't work that way anymore if you really want to move quickly. And so, one of the things that you want to do is actually give and socialize a sense of taste down to the team level so that teams are empowered to move autonomously without sacrificing extraordinary quality. And I think that that quality tradeoff is one of the pieces I really have been sitting with in the Wharton and MIT studies. I feel like MIT essentially had an extremely high quality bar and Wharton had a more relaxed traditional software quality bar. And if you want to thrive and build an AI native company that actually works, you have to figure out how you can socialize that insane almost founder level obsession with quality and taste to the point where the team has it built into their DNA because they have so much power with AI agents with uh AI tooling to launch their own products to drive their own corner of the business. This might look different at different companies. Maybe you say it's at the department level, not the team level. But the point stands, right? Taste is something that shows up at a much more democratized level than it did in the pre-AI age. And what's interesting is it's not just is this product good taste. It's taste in problems. Which problems are spicy that we should choose to solve? It's taste in problemsolving skills. Taste in learning methods. What I'm saying is you have to develop a sense of where the juice is in the profitability matrix of the organization. Maybe the most effective thing your team can do is to scale up for the next 3 months and other teams don't need that but yours does. Maybe the most effective thing you can do is double down on problem space discovery and other teams are building product or maybe it's a more traditional definition of taste and you're working on an excellent product. But the reason that matters is because the team has to have taste or the tooling they're using with problem solving with high um high ownership is wasted. Taste is effectively a fancy way of saying pick the right thing to work on and make sure that you are really really good at knowing what good looks like. That's taste. When we talk about someone with high taste in fashion, they pick the right thing to wear and they know how to wear it so it looks good. very very similar idea and I think that that's something that we could previously delegate to just a handful of people to a tiny sort of collection of folks when IBM was at its height IBM had taste makers they were a group of 10 or 15 people who were licensed to break all the norms of the organization and they were licensed to do that by the organization so that they could introduce creative thinking well their their taste has to be democratized that idea does not work anymore more. We need to build institutions that socialize a sense of taste. And I do want to suggest this is not universal. Right? The way LLMs work is universal. The ability to learn to solve problems with LLMs is also a universal skill. The sense of ownership is a universal skill. Taste is not. Taste is specific to your vertical. Taste is specific to your situation. Taste is more like context, which I mentioned at the beginning of this video. taste requires you to know your local domain very very well and have an excellent taste in problems.
So there you go. I think what we're really talking about between Wharton and MIT is institutional fluency. And I think the three keys are context and then the ability of teams to start to flip the traditional relationship between ownership and skills. Ownership residing now at the individual level, skills at the team level and then finally taste. I think that taste is something that we have to push down into our organizations and that's also new. What do you think you're missing or I'm missing on AI fluency in institutions? This is an evolving field. I'm learning and seeing this in real time. What are you saying?