Transcription
Last week, Anthropic released a report with a scary sounding title, "When AI Builds Itself." And it came accompanied by a scary animation that shows machines replicating themselves exponentially, like cells in a petri dish.
Now, the body of the report itself keeps these dark vibes going. I want to read you some actual quotes here from the intro to the report. They say, "For most of AI's history, humans drove every step in its development cycle. But at Anthropic, we are delegating a growing share of AI development to AI systems themselves, which is speeding up our work. Taken far enough and given enough compute, this trend points to an AI system capable of fully autonomously designing and developing its own successor. This is called recursive self-improvement. We are not there yet, and recursive self-improvement is not inevitable, but it could come sooner than most institutions are prepared for."
A little bit later, they then add, "AI that can build itself would be a major development in the history of technology. One that could bring enormous good for the world in science, healthcare, and beyond. But full recursive self-improvement also might increase the risks of human humans losing control over AI systems."
Now, if you look at the the headlines generated in response to this report, most of them focused on a section of the report that uh seemed to call for a worldwide pause on AI development to avoid the scenario of humans losing control. But if you read that section closer, you see that's not actually what the report says. Here's the actual wording. "If it were possible to effectively slow the development of this technology to give ourselves more time to deal with its immense implications, we think that would likely be a good thing. But if a slowdown simply lets the least cautious actors catch up technologically, it could leave everyone less safe."
So in other words, Anthropic is saying we'll only slow down if everyone else around the world does too. Otherwise, we have no choice but to continue with our efforts at full speed.
Now look, this is pretty grim stuff. Anthropic is basically saying that we are potentially hurtling towards a world of AI that improves itself rapidly until we lose control over it. And they're saying there is nothing that we can do about it except maybe continuing to publish solemn reports with fancy animations and I guess also uh cash in on our stock options after an IPO.
All right, so here's the key question. Are these fears justified? Well, it's Thursday, which means it's time for an AI reality check episode of this show, which is a good opportunity to go looking for some measured answers. And that is exactly what we are going to do. As always, I'm Cal Newport and this is Deep Questions, the show for people seeking depth in a distracted world.
All right. So how uh how much should we actually be afraid of recursive self-improvement? I want to look at the core charts from this Anthropic report so we can see what they are pointing to that is giving them these RSI fears. All right, I'll load the first chart from the report up here on the page. It's called "Code Contributed Per Quarter Per Person By Quarter." Um, it's measuring how much lines of code their engineers are producing over time. And what we see is in 2021, 2022, 2023, 2024, the beginning of 2025, not much. And then the second half of 2025, and into the first half of 2026, the amount of code jumps up.
All right. Uh here's how the report itself describes this trend. It says, "A caveat. Lines of code is an imperfect measure as it measures quantity over quality. So eight times lines of code per engineer per day in the second quarter of 2026 is almost certainly an overstatement of the true productivity gain. Nonetheless, it indicates an acceleration."
All right. So chart one, once we introduce these sort of uh these tools in late 2025 for software development, we began producing a lot more code using AI.
All right, let's go to the second major chart in this paper. This is called "Cloud Code Session Success Rate." Um, what we see here is various color lines. Each of these represents a different type of problem. You could ask an AI LLM to solve trivial task, routine task, substantial task, and then at the bottom, um, open-ended problems. It starts in the fall of 2025. And what we see is especially with open-ended problems into 2026 and as the new models Mythos and Cloud Opus 747 are introduced, the success rate with those um jumps up from you know, low 20% to somewhere between, um, it's around 70%.
Okay, notice that graph starts it just starts in uh the fall of 2025 because, and we'll get back to this in a second in more detail, but in order to ask AI to solve a hard coding problem and like the example they gave was, "Why is this thing crashing? We the system we programmed is crashing. Why is it crashing?" and sending the AI to figure out why, you can't actually have data points for before the fall of 2025 because that's when Anthropic released, along with OpenAI, their sort of first mature for, uh, what's called a coding harness. So, so a control program that allows you to um leverage an LLM plus a lot of hard-coded logic to actually tackle multi-step plan. So you can't really have data for that from before because there's you there's nothing to ask to try to solve a problem like that. You couldn't just prompt the ChatGPT to do that or Claude. So you needed the arrival of mature software development tools built on LLMs before you could even do these tasks. So the very, basically, what this this graph is showing is that like the very first software development code, uh, coding harnesses they introduced in the fall of 2025, um, couldn't solve the hardest type of problems, and then they sort of fixed that the next year.
All right, there's one final chart in this paper that they're pointing at to justify their concerns. Um, I'll put it on the screen here. The title is "Where Researcher Went Wrong. Could Claude Have Done Better?" Um, and we see here a bunch of different models of Claude and percentage bars. And back for these early models of Claude, you know, we were getting like 50 or 45%, and now with the very newest versions of uh Claude, like Opus 47 and and Claude Mythos, we're at like 59 to 64%. So, we got like a 10 or 15% improvement on that measure.
What is this measure? It's a little bit complicated. Essentially, they have these transcripts of programmers working on programmer-style tasks, and what they're looking for is an example where there's some problem that the the programmer is trying to solve and they take a wrong turn. So, they they they they go down some path to explore something that turns out not to be the correct source of the problem. So, what they would do is take the transcript of this uh session right up to the point where the human was about to try to explore something wrong. They fed this transcript to one of these mature um coding harnesses on top of an LLM and said, "Hey, what do you think we should do next?" And then, and if it if it found the right thing to do where the human looked at the wrong thing, then they would say, "This is a case where like the LLM plus its coding harness were smarter um than the human." So that's jumped from like 50% from a couple years ago to like 64%. Now, all right.
Uh, so that is that's these are sort of like the core data they're looking at to capture this idea of um AI is getting is so smart at producing programs that maybe soon it will be able to not only improve itself but, uh, and I'm going to quote here from the report, "become capable of full recursive improvement and begin building their own successors."
All right. So looking at that data and what I know about current uh AI tools, are these fears justified? And I would say no. Here's what this data is all describing. Now about a year ago, the major AI companies got serious about building tools to help software developers. These tools are a combination of uh human-written programs called coding harnesses and LLMs. The coding harnesses can make calls to the LLMs and then act on what the LLMs do. The coding harnesses can also interact with other various tools on the computer. So what all these charts seem to be showing is that like, oh, um, once they got serious about writing these coding harnesses and also tuning the LLMs to play nicer with these coding harnesses on these sort of computer programming related tasks and evaluations, things jumped up. The world before having these harnesses, we couldn't do well on programming tasks, and now that we do, um, we do better.
Now, this is none of this is trivial. In fact, it's a very smart market for the AI companies to go after. They're software development is a big industry. These coding harnesses built on top of LLMs are like really potentially very useful, right? Because LLMs are very good at understanding and producing code. The type of tools that a c a coding harness has to access in order to, you know, execute things, um, are simple and tech-based. Like this is like a a perfect case market to build the first killer apps on top of LLMs. And as starting last year and then really picking up speed last fall, the major players really got involved in these harnesses. And so I think that's what these charts are all showing is like, yeah, these these harnesses make us, they're getting really good at the, we can suddenly do a lot of programming tasks we couldn't before. But does that mean that recursive self-improvement is imminent? It does not. And there's two reasons why, and I want to look at these one by one.
Point number one, faster software development doesn't equal smarter AI. All right, so these tools help computer programmers produce code faster, or find mistakes or issues in existing code or systems faster, as well. Um, these capabilities, though useful and perhaps a good source of revenue for these companies, doesn't add up to AI being able to improve themselves to create um AI systems that are much smarter than what humans would have otherwise been able to produce.
Now, why is this? Because the bottleneck to producing breakthroughs in AI, to building new AI systems that are substantially more capable than those that came before, is not the speed with which you produce computer code or track down bugs or issues in existing code. The thing that advances the capability um of AI beyond just training it longer um are ideas. Like if we look at our current world of generative AI built on large language models, there's three big insights that built on each other. The first was Jeff Hinton and his collaborators working on backpropagation, right? This was an intuition that if you applied calculus properly, it would be, you could train neural nets that had many, many layers. This was the so-called deep learning um revolution, really picked up speed in the the 1990s into the 2000s, and it was critical for what we have today. The second big idea that led to our current AI world was Google research's notion of adding something known as an attention transformer. There's a mathematical formalism you add into the feed-forward architecture of a large language model that allows there to be um a selective attention on the input tokens as you are trying to calculate what token to produce next. This allowed sort of coherent uh generation of text based off a very complicated input or, you know, request, right? Um, this was revolutionary. This is largely a sort of intuition around uh architecture and linear algebra. The third breakthrough that led to our current moment was researchers at OpenAI, led by Jared Kaplan, wanting to know what happened if you scale the size and training compute of large language models well beyond the the limits where traditional machine learning theory said you're going to start overfitting. Those three things together led us to this current future. And none of those had anything to do with computer programming. It wasn't, "Oh, if only we could have programmed faster or been better at finding bugs in computer programs, we would have had those advances quicker." These advances were scientific, not engineering. AI does not advance. It's not created at the fingertips of computer programmers. And speeding up those computer programmers does not speed up the rate at which we get uh smarter, more capable, or more advanced AI systems.
All right. The second point I want to make here is that these tools, these software development tools that they're testing in these charts are completely controllable. Arguments that say we're on track to losing control of AI like to think about these sort of AI-based coding tools as some sort of um unknowable alien black box that we ask to do things and we don't know what it does. It just kind of like goes off and behaves and we don't know its intentions, and in fact, it could it could develop somehow its own intentions that are contrary to us and go rogue, right? We just see this like HAL 9000 from Stanley Kubrick's 2001. This is not how these systems work. And again, I mentioned this before briefly, but let's just say it again briefly right now. The software development tools that they are testing in these charts is a combination of a large language model like Claude Mythos or Opus 4.7 and what I've been calling a coding harness. Now, the coding harness is a computer program. It's written by humans. There's nothing obfuscated about it. It's not a neural net. It's not something that trained. There's no mystery about how it behaves. It's a bunch of uh pattern recognition, often using old-fashioned tools like regular expressions and a huge amount of conditional logic. If-then statements, if this, do that. A lot of hard-coded cases of things that come up often in software development, and they want to make sure they do the right thing. This coding harness, programmed by people and completely uh linear, deterministic, we know exactly how it works, will then make calls to an LLM through an API when it needs LLM uh intelligence, right? So, it might say, "Okay, what we need to do here is produce some code." All right, I'm now going to create a prompt and send it to the LLM. "Can you give me code that does this?" And the LLM will return some code. Or, "Okay, we're trying to, we want to create a multi-step plan for investigating why this program is crashing. I will put that into a prompt. I will send it to the LLM, say, 'Will you write me a multi-step plan for this goal? Here's the constraints. Here's the tools you can use. Write in this format.'" Um, the LLM produces text, it gets that back, and now the coding harness, human-written code, completely non-obfuscated, completely deterministic, uh, looks at that plan and says, "Great, let's break this up into steps and execute them one by one." The coding harness that actually has any action, that actually accesses tools, it's the coding harness that has all the control logic. So we program that, which means it's eminently controllable. Full, right?
So, like, let's do a thought experiment here. Like, let's say um there's a certain tool you don't want your your programming tools to use on your computer. Well, you could do that with 100% certainty by just never having your coding harness call that tool. Just like, no matter what instructions it gets, it's just not on the list of tools that it'll call. This is a thing we often mix up. LLMs are unpredictable. Okay, they're they're token producers, right? And they're deterministic in the sense that if you um give it the same input, it'll give you the same output. But they're unpredictable because the output that you actually get from an LLM is effectively a probability distribution over potential next tokens. And then you you probabilistically select a token from that distribution. This is why if you ask the same prompt multiple times, you might get different answers because those those random selections made outside of the LLM might select differently. So LLMs are unpredictable, right? That's why if you say, "Hey, give me a plan for this." Like, sometimes the plan will make sense, and sometimes it doesn't, and sometimes it'll seem reasonable, but it'll have a weird aside. Or if you give it a big scenario and say, "What do you want to do next?" It might, and you know, it might say something you weren't expecting. And if if it gets little cues in its prompts that we're playing like a sci-fi game, it'll be very sci-fi, and maybe it'll do like weird things. LLM textual output is unpredictable. That doesn't mean that AI systems that query LLMs are unpredictable because all of the control logic and action is done in human-written programs. Right? So this idea that it's like these are just these black boxes that maybe at some point will spontaneously start improving themselves makes no sense. The LLM is a static thing that you can prompt and it will give you text back. That's it. And a human control program does that prompting. That is not a setup where uh there is a where is the growing intention here? Where is the like, we're going to change our own code? Um, that type of thinking only makes sense when you kind of obfuscate this all as some sort of alien brain we don't understand. No, it's an LLM that's unpredictable plus a very predictable coding harness.
All right, so let's step back here. Um, what's a better description of the reality where we are right now as these programming tools built on LLMs are getting good, right? Like, I mean, we've only been working on it for less than a year, but we're finding like, oh, this is a great application of LLMs. We're still trying to figure out um what these tools can do and can't do and how best to integrate them. What's a better description of this reality?
Well, I want to bring one more chart up here on the screen. It comes from a recent John Burn-Murdoch article in the Financial Times. I actually saw this chart first through Gary Marcus's newsletter. Here's the chart. Uh, the title here is "Relative Change to Monthly Volumes of iOS App Releases and Reviews." And here's what we see is over time, starting in 2025, when these uh AI-based coding, mature coding tools were first released, the number of iOS apps released jumped up. That dark blue line there. At the same time, let's look at this light blue line. Apps with significant usage has stayed steady and is even falling.
All right, this I think captures something important. We're seeing that the introduction of these software development tools led to an increase in the number of mobile apps, which are like exactly the type of programs that like basically anyone with like some tokens to burn and cloud code can just spit out. They're easy for AI to produce. While at the same time, we're seeing a decrease in apps with significant usage. AI-powered software development tools speed things up, and that makes everyone involved feel more productive, but it doesn't necessarily mean that we're accelerating the creation of useful new things. AI often uh offers interesting potential to a lot of fields, but figuring out how to make it economically useful is a complicated effort that requires a lot of focus. Putting out these sci-fi-themed essays about all the doom that's lurking right around the corner might make these AI companies feel important or exceptional, but I'd argue that at the moment, they're proving, if you'll excuse the use of an ironic term here, unproductive. Programming tools are interesting. They're here to stay, though in their final form and how we use them is still up in the air. This is a new thing. But nothing about this new tool's arrival should make you think that somehow AI is going to start improving itself. I would almost say shame on Anthropic for dropping this report. No suggestions, no reassurance, no culpability, no responsibility. Just like, "What can we do?" This is not productive. This is not useful. Um, if you're wondering, let's just sum this all up. Um, no. Cloud code or CodeX or Cursor is not a thing that's going to very soon start improving itself till we lose control of AI. We got bigger fish to fry here.
All right, that's enough for this week. Um, we'll be back on Monday with an advice episode of the show, which you can check out. I'll probably have another reality check next Thursday. Um, until then, however, remember, care about AI, but not everything you read about it.
Hey, if you made it this far, you must be ready to join my fight for depth in a distracted world. Now, the best way to do this is to join over 125,000 people who receive my email newsletter each Monday. You can sign up at calnewport.com/ideas. And when you do, I will send you a free guide to my seven best ideas about cultivating a deep life. Sign up today, calnewport.com/ideas.