Transcription
Harvard University just caught most people using AI at work, including elite consultants at one of the world's top firms actively making themselves worse at their jobs without even realizing it. And it brings up a question with serious consequences for all of us.
Is the AI that we use every day actually helping us work faster, smarter, better in the way that it's being promised? Or is it actually making us less valuable and less productive even if it feels like it helps? A study published by Harvard Business School shows that for most of us, it's the second.
So, the research followed 244 professionals from Boston Consulting Group, one of the largest consulting firms in the world, across nearly 5,000 individual AI interactions. And beyond concerns of trend slop, which is generic advice that ignores your specific situation, which we covered in a previous video, what this new study found is that the most common way we're all using these tools isn't just introducing minor mistakes from hallucinations. The way that we're using AI is actively eroding the quality of our decisions, the depth of our expertise, and ultimately the value that we bring to our work.
And it comes down to how we use these tools. The AI industry is selling these tools as an easy button. Press it and you get smarter, faster, and more productive. But when you look at the data, the people in companies pressing that button are the ones actually getting worse. So, let me show you what they found and what it means for your career. I'm Brendan Dell. This is the leverage class. Let's see through it.
All right. So, a few days ago, before Memorial Day here in the States, I finished a video covering MIT research and why AI can't cause the mass layoffs that CEOs are claiming. And after finishing that video, I was faced with a question. Would I get better traction publishing that video into a holiday weekend on the Saturday or waiting till after the holiday?
So, I asked VidIQ and VidIQ is an AI tool that's built with Claude and Gemini and it's on top of YouTube data and it accesses all my channel analytics as well as YouTube's data broadly and at large. So, I asked the AI, should I publish or should I hold? And it gave me a very clear declarative answer. It said hold. It said mid-career professionals don't watch YouTube on holiday weekends. That I should wait until Tuesday, maybe even Wednesday. Give people a day to catch up and so forth. So, I mean, that sounds kind of right, right? I mean, it it sounds reasonable. Anyhow, it kind of sounds like advice someone would give you, but given what I know about AI, I decided to check for myself.
So, I went and pulled the actual YouTube usage data. And it turns out that the opposite was true. Holiday weekends are some of the highest traffic windows on the platform, including for my audience. So, I published that video and it took off. However, the reason that this failed is not, you know, the AI itself. What I did was treat the tool like an easy button. I wanted an answer and it gave me one and it was completely wrong. And as one quick caveat to all this, I just want to say I actually like Vid IQ and if you're on YouTube, I find it very useful. So, nothing against the tool. And by the way, I get nothing for saying that. The failure in that interaction wasn't the tool. It was how I used the tool. And as it turns out, this mode of working with AI is the single most common and single most destructive method of using this technology.
So here's what Harvard found. So how exactly are most people using these tools in a way that actually makes them worse at work? Harvard answered this by studying 244 management consultants from Boston Consulting Group and these were associates and consultants. So this by the way is the junior tier and exactly who you know AI is supposed to be eliminating because it can just do the work for them. So they built a custom chatbt style platform that logged every interaction that the consultants had with AI. And what's particularly interesting about this study for me is that the question that these consultants were tasked with is exactly the kind of quote thinking that AI needs to be good at if it's going to create, you know, a white collar bloodbath like all the headlines say, if it's going to kill entry-level work and everything else.
So, here was the task. There's a fictional retail company with three brands. It's men's, women's, and kids. And the researchers handed each consultant the same set of interview notes with company insiders and the same financial data, same market context, all the same stuff. And then they posed a strategy question. Which brand should the CEO invest in to drive the most revenue growth? And then they let them work. So across the project, consultants logged almost 5,000 AI conversations. And then the researchers conducted 237hour long follow-up interviews to understand the process that each person used when doing the analysis. And what they found is there are essentially three patterns that all of us use and probably we each blend some of these patterns across our usage when working with AI.
So the first is cyborgs and this was about 60% of the users. These folks used AI for almost everything. They fed at the data and they asked for analysis and recommendations and they, you know, wrote the the recommendations and so forth. The important thing to see with cyborgs is this. They iterated with the AI. They didn't just take the answers at surface level, but they also relied on the AI. For example, a cyborg wouldn't actually stop to verify data. They would say something like, "Okay, doublech checkck this finding. Is it correct?" The AI was part of all the steps. And while they didn't blindly trust it, the AI was still the source of truth. Now, according to the study, these folks, the cyborgs, they got better at AI specifically. While they actually got worse at understanding real business problems, however, there is a confounding factor here, and it's one that only becomes obvious when you look at the broader AI research. Because while not improving at your job is obviously a bad thing hypothetically if AI is smart, then getting good at AI might actually counteract that, right? Maybe your actual value is still the same. But that is not the case. And we'll see why in just a second.
So the second group was centaur and this was about 14% of the users. These folks used AI selectively. So they would ask it for background things like uh what are key trends in men's wear or what's the Excel formula for compound annual growth rate but they then would go and do the actual analysis themselves. So this was like build the you know do the Excel spreadsheet or build the model or whatever and then they would write the argument and AI was the research assistant not the decision maker and unsurprisingly this group became more knowledgeable professionals but didn't deepen their AI skills according to the researchers.
Now the third group was the self-utomators and this was the remaining 27%. This is the category that you don't want to be in. They basically dumped the They dumped the entire problem into AI in one prompt and just took whatever came back. One participant pasted literally every interview transcript and every financial table in one single shot. They asked for the recommendation, the rationale and the memo. They took it and then they just moved on. So that group obviously gained nothing in either direction. Do not do this.
So the surface level read on this study is 60% of people using AI are actually stagnating in their profession while becoming better prompters. 14% are using the tools to be more effective professionals and progress faster and 27% are screwed. Now it is worth noting that those percentages add up to 101. I don't know maybe Harvard used AI for their math. Anyhow on its own this finding is not that surprising. And if that was the only implication, I wouldn't have made this video. But when you look beyond this one study across the continuum of research and at my experience working with and implementing these tools at scale and at all of our personal experiences actually using these tools, you see something which is much more important with broad implications for not only our own AI use, but how our companies use AI as well as our own careers.
So before we get to that, a lot of folks out there are trying to decide what to do next in their careers. I personally want three things out of work. I want work I enjoy doing. I want relative time freedom. And I want diversified income far in excess of what I need. The way that I originally built this for myself was with consulting. And after many years, my clients were signing my SOS and wondering how I'd been able to put that business together. So I built some modules for them which started getting shared which I turned into a course called the freelance formula. It's a program for mid-career professionals who want to build their own independent business. So, right now you can get that full program for $99. The link is below. With that, back to the content.
Now, back to the beginning of the video. So, the overwhelming majority of people in this Harvard study are using AI as an easy button. Knowingly or not, they are placing a bet that the AI itself is smart enough to be an oracle. They think they can ask it how to do something or to do something on its own. And then the AI knows what to do, how to do it, and can check itself to make sure that it's not wrong. And this only works if the quote artificial intelligence is in fact intelligent. But when we look at unbiased research, what becomes clear is AI cannot do that in its present state at all.
So to see this clearly, we need to look back quickly at a previous video I made covering another study that was published in the Harvard Business Review called AI has a trend slot problem. And for those who haven't seen it, here's the 60-second version. So researchers tested every major AI frontier model, Claude, Chad GBT, Gemini, etc. across 15,000 strategy scenarios. What they found is these tools don't quote reason but instead consistently recommend the trendy business strategy regardless of if it fits the business context or not. And the researchers called it trend slop.
Now the most important thing to highlight from this study is this. The researchers tried a variety of different prompting techniques to produce more tailored responses. And no matter what they did, the bias remained. Clearer prompts only move the bias 2%. Rich context was said plainly means giving the AI a bunch of unique details that should influence a recommendation moved it only 11% and telling the models to reason more carefully did almost nothing. The only thing that moved the answer significantly was flipping which option appeared on the page first. That moved the recommendation by 19%.
And there's a myriad of studies and data and anecdotes beyond trend slop pointing in this same direction. As one example, a research nonprofit called Meter, who runs many of the benchmarking studies, ran a randomized control trial with 16 experienced developers where half the time they used AI and half the time they didn't. And what they found was the developers using AI were 19% slower than the developers who weren't. But what's even more interesting is the developers predicted AI would make them 24% faster and after the study they reported feeling 20% faster but the measurement showed they were 19% slower. Now that study is small and it leaves much to be desired but what it illustrates is a phenomenon that's consistently validated across much larger populations. There is a huge gap between how much time people think that they're saving with AI and what they actually do. Studies from Duke, from the Federal Reserve Banks of Richmond and Atlanta, follow-up studies by Meter all consistently show that workers feel faster than they are.
But here is the thing that we need to register. The 87% of professionals who are using AI with some blend of cyborg and self-utomating behaviors are not only not moving their skills forward, they are developing a set of skills, namely AI fluency, that does not actually produce more accurate answers. And worse, for the whole narrative, they aren't even training their replacements. Right? The tool is giving bad advice regardless. Now, this is not necessarily an LLM problem. It is a big general purpose model problem that we'll discuss more in an upcoming video. But the most important takeaway for us is this. If we treat AI as an oracle rather than a sparring partner, these tools will erode our professional insights, provide bad recommendations, and erode our long-term growth as professionals.
But the implications of this finding go way beyond just, you know, become a centaur. The bigger question is this. If AI isn't quote intelligent in the way that it's being marketed, is the underlying technology itself, the large language model, actually useful at all? We can turn to one of AI's previously biggest skeptics, recently turned proponent for some insight. So Ken Griffin is a billionaire and he's the founder of Citadel, which is one of the world's most successful hedge funds. And for a very long time, he was a big AI skeptic. In January of this year at Davos, he was quoted as calling AI quote all garbage when you dig below the surface. And he framed the 500 billion in projected 2026 data center spending as quote hype pushed to justify the capex. His exact framing was, you're not going to generate this kind of spend unless you're going to make a promise that you're going to profoundly change the world.
Now I will go deeper into this Citadel story for an upcoming video but for now the important thing for us to understand is that he has recently completely changed his point of view. He said this at the Stanford Leadership Forum in the last few months there has been a step change function in the productivity of the AI toolkit. It is profoundly more powerful than it was just 9 months ago. And for us at Citadel, this has allowed us to unleash a much broader array of use cases for AI. He continued by saying, to be blunt, work that would usually be done with people with masters and PhDs in finance over the course of weeks or months is being done by AI agents over the course of hours or days. These are not mid-tier white collar jobs. These are extraordinarily high-skilled jobs being automated by agentic AI. And he went on to say, "I went home on Friday actually feeling fairly depressed by this because you could just see how this was going to have such a dramatic impact on society for the first time. AI is real."
So there it is. The skeptic conceds, right? AI does PhD level work. Case closed, right? But here is what the mainstream press is missing. This is not clawed or chat GBT licenses distributed to employees or some shallow data layer plugged into an algorithm. Citadel has spent decades building one of the largest proprietary financial data sources in the entire world. They then spent years building on top of that infrastructure. And all of this is happening within a very narrow domain with trackable data points and clear win loss states. Most companies can't tell you what the open rate on their last email was. That kind of intelligence isn't available by scraping the internet. It can't be solved by scaling to get to the real promise of this technology. As individuals or organizations, we cannot use these tools the way they are being sold to us.
So then when are these tools actually useful? In a phrase, AI is an aggregator, not an analyst. They pull together what's already known. They can't tell you if it's right for your situation in their present state. And here is this brought to life. Stanford and MIT researchers studied 5,179 customer support agents at a Fortune 500 company. AI access boosted productivity by 14% on average, but that 14% hides the real story. Novices got a 34% boost where experienced workers got almost nothing. The AI helped the people who didn't know what they were doing yet. It did almost nothing for people who already had expertise. And it only worked because the domain was narrow. Right? Customer support has very specific scripts and known right answers. The AI didn't have to figure out the answer. It just had to surface it.
This is the same pattern at Citadel. It's a very narrow domain. It's validated data. There are experts around the system. And the AI is an interface, not intelligence. And that's the honest picture of when AI actually works. LLMs can be amazing tools when it comes to things like translating between languages or summarizing text you already have, drafting routine documents or helping noviceses in very narrow scripted domains, acting as an interface to validated proprietary data. But what they cannot do as the research shows, but they don't help with strategic decisions where the data is ambiguous. Anything where you have a real consequence on the answer and anything where you don't have the expertise to catch it when it's wrong.
And I see this every single day with Vid IQ. It constantly feeds me title and thumbnail recommendations that score well on its algorithm. And those scores aren't random. They are based on patterns from, you know, videos across the platform that performed. But a high score is not a causal output. It's just a correlation. It tells me this pattern worked somewhere, but it can't tell me whether it's going to work for me. And in fact, it frequently recommends things that will not work for me in this moment of time. And I know because I've tested extensively if I hadn't spent years actually learning YouTube. I'd have no ways to know which of these confident recommendations to trust and what to throw out. In a sense if you want to think of it this way. I've spent years building my own context window and it's far larger and it's more specific for me than AI can be.
And that is the trap of treating these tools as easy buttons. They give you interesting inputs but they cannot give you causal outputs. And if you don't have the expertise to tell the difference, you're going to treat correlations like conclusions. Which brings us to the single most important takeaway of all. If we as professionals or organizations approach these tools as an easy button and we believe the hype that they can, you know, code autonomously and make our decisions and replace our workers and predict the future, then we will see short-term gains that create long-term and potentially catastrophic consequences because we become cyborgs who stop getting better at our actual jobs. We become the self-mators who hand over the thinking and just keep the output and we create technical debt that nobody will know how to dig their way out of because we are making decisions based on confident answers that can't be validated.
And the way out isn't to stop using these tools. It's to stop using them as oracles. So the 14% in the Harvard study who actually got better the centaurs, they were more skilled in their own domain. And that's the same pattern we see when this is done right. Which means that the most valuable thing any of us can do right now isn't getting better at prompting. It's building deeper expertise with AI in its correct role. It means understanding within your narrow domain what these tools can do, what they can't do, and then being able to tell the difference. It's about niching down, building your own frameworks of what good looks like, your own data sets, and then using AI to apply that expertise faster and more effectively than the generalists who are relying on whatever the tool gives them. And that's how we use these tools as leverage instead of becoming their leverage.
Now, we've seen through it. If you want to see how I'm thinking about this in my own career, watch the I'm 43 video next. If you want to learn more about the trend slob study mentioned earlier, watch the Harvard video next. And if you've seen those, I'll include a few more videos down in the description that I think you'll enjoy.