Transcription
On March 8th, Andre Karpathy released a 630-line Python script that created an entirely new paradigm for AI development. He pointed an AI agent at his own training code, gave it a single metric to optimize, and went to sleep. Two days later, the agent had run 700 experiments, discovered 20 genuine improvements, and cut training time by 11% on a codebase that one of the best ML researchers alive had already optimized for months. It found a bug in his attention implementation that Karpathy had missed. Not because the agent was smarter, but because it tried more things faster without getting bored after the 15th failed attempt.
On April 2nd, a small YC startup called Third Layer took the same pattern and applied it to Agentic Harnesses, the prompts, the tools, the orchestration logic that determine how agents behave. A meta-agent rewrote the task agent's entire scaffolding overnight and claims to have hit first place on two major benchmarks. Every other entry on those leaderboards was hand-engineered by humans. The pattern is called the Karpathy loop, and it changes how you should think about AI in your organization.
So, I'm going to walk you through exactly how that works, what the constraints are, and how you can implement it in your org, and how this Karpathy loop escalated us from optimizing training code to optimizing agent behavior. We're also going to talk about what local hard takeoff means in practical business terms and why you need to care about it now. And we're going to talk finally about why most orgs will fail to take advantage of that auto-optimize loop and what a real path looks like for teams that want to get auto-optimizing agents going in 2026.
Now, if you watched my initial video on this in March, this is the sequel to that. That piece defined auto-research as one of four distinct architectures. This piece explores what happens when auto-research grows up and gets implemented in your org. First, I want to spend a minute on why auto-research works. Because the mechanism is the thing that most people are getting wrong about it. They see the idea that AI does research while you sleep, and they assume that there's magic in the agent's intelligence. But that's not it. The magic is actually in the constraints.
Karpathy's setup is deliberately minimal. There are just three files. One of them, train.py, is the only file the agent can touch. So the agent proposes an edit, runs a five-minute training experiment, checks a single metric, it validates it, and it either commits the change or it reverts it and says, "No, that didn't work." That is the entire loop. I just described it for you. It is one file. It is one metric. It is one fixed time budget. That is it. The human's job is just to write a plain English instruction file that tells the agent what to explore and what constraints that it must respect. And so the human needs to aim the research direction while the agent executes the search. And that's the bones of the Karpathy loop right there. It just has three components: an agent with access to one editable file, a single objectively testable metric, and a very fixed time limit per experiment. That's the whole architecture.
The minimalism isn't a limitation. It's the entire point. By constraining the search space to one file and one metric, Karpathy made the problem tractable for an agent in a way that a very sprawling, multi-file system wouldn't be because the agent can read the entire codebase or whatever you're trying to optimize in a single pass and it can understand the full context of any change. It can evaluate whether the change worked within just a few minutes. And it can do this hundreds and hundreds of times over and over again without fatigue, without distraction, without the sunk cost bias that makes human researchers really cling to ideas that aren't working anymore.
In the first run, the agent executed about 12 experiments an hour, or roughly 100 overnight. Of those, maybe 20 produced genuine improvements that stacked into an 11% speed-up in total. So the hit rate, that's not high, but the iteration rate is absolutely inhuman. A productive human researcher might manage 8 to 10 of those experiment cycles in a working day. And most of that time would be waiting for the GPU, not actually thinking. The agent doesn't have to wait. It doesn't have to context switch. It doesn't go to lunch.
When Shopify CEO Toby look tried the same pattern on internal company data, he got a 19% performance gain from 37 experiments in 8 hours. When Sky Pilot pointed it at a 16-GPU Kubernetes cluster, the agent ran 910 experiments in 8 hours, discovered that scaling model width mattered more than any single parameter, and spontaneously taught itself to use faster GPUs for validation. Total compute cost to figure that out: under $300. And that detail matters, and we're going to come back to it because so much of what we're talking about in 2026 is, can you do this kind of AI-ic work efficiently?
Now, auto-research will optimize training code for you, and that's important, but it's a very narrow domain. The thing that happened more recently, in the first week of April, is much more consequential. Kevin Goo's auto-agent took the same loop—the edit, run, measure, keep, or discard loop—and applied it to harness engineering. So instead of optimizing a model's weights or its hyperparameters, this loop optimizes the scaffolding around the model: the system prompt, the tool definitions, the routing logic, the orchestration strategy. The meta-agent reads failure traces from the task agent and diagnoses what went wrong and modifies the harness and runs that benchmark again. Essentially, they've turned the problem of harness engineering into just trying to optimize for a single score.
And funnily enough, the claimed results are striking. Goo says auto-agent hit 96.5% on spreadsheet bench and 55.1% on terminal bench. First place on both. Now, I want to be very transparent. As of this writing, those scores haven't appeared on the official leaderboards. And it's just a claim, right? The highest verified spreadsheet bench entry is Opus 4.6 at 34-something percent. So the gap between the claimed score and the verified state-of-the-art is quite big. And it's possible the claims are real but unverified. But I want to emphasize, instead of looking at a particular benchmark score, which I talk a lot about not taking benchmarks too seriously, look at the direction you can go when you kick off an auto-agent work loop like this. Those are the findings that matter more.
So, let's dig into what are the principles here that help us to understand how to scale your agent. First, the meta-agent/task-agent split. Goo's team tried having a single agent improve itself, and it didn't work very well. Being good at a domain and being good at improving at that domain are actually very different capabilities. That separation lets each agent specialize. So, the meta-agent becomes a harness engineer, and the task-agent becomes a domain specialist. This is a really interesting design insight that we can transfer directly into a bunch of business applications.
The second piece is model empathy. So, same-model pairings dramatically outperform cross-model pairings. In other words, a Claude meta-agent writes better harnesses for a Claude task-agent than a ChatGPT task-agent, and vice versa. Goo's team explains this as the meta-agent having implicit understanding of how the inner model reasons, its tendencies, its failure modes, and its preferences. So the meta-agent shares the same weights. So when it reads a failure trace showing the task-agent lost direction at step 14, it kind of understands that failure from the inside. This is a non-obvious constraint that matters for anyone who's designing these kind of self-improvement systems.
Third, there were emergent behaviors that the team did not program that we should pay attention to. The meta-agent independently invented spot-checking, running individual tasks instead of the full benchmark suite for small edits, and saving compute. It built forced verification loops and formatting validators. It steered the task-agent to write its own unit tests. It invented progressive disclosure, dumping long context of files when results overflowed the context window. And it built task-specific sub-agents and handoff logic when the domain required it. None of this was specified in the directive. The meta-agent discovered these strategies by analyzing its own failure traces and reasoning about what would help.
This is the escalation that matters for business. Optimizing training code—that's kind of useful, but it's also, to be honest, kind of niche. But if we're getting into a world where we are optimizing the harness, the prompts, the tools, the routing, the orchestration that determine how an agent behaves, well, now we're talking. That's universal. Every company deploying agents has to have a harness. And most of those harnesses right now, they're designed by we humans who are doing our very best with the knowledge we have. The auto-agent pattern suggests those harnesses can be systematically optimized by a meta-agent that understands the inner model better than a human engineer would.
And this is just from the open-source community. Inside the frontier labs, the ambition is even bigger. Anthropic has publicly stated that it wants Claude N to build Claude N+1. In other words, it wants to start a fully recursive loop. Auto-research. OpenAI has announced it's aiming for a fully automated AI researcher by 2028 with an AI research intern by 2026. At Davos in January, Deise Hosabi said the self-improvement loop is something all major labs are pursuing. The difference between what the labs are doing and what auto-research demonstrates is just in scale and scope. It's not in kind. This is the same kind of thing. The loop is the same: you propose a change, you run an experiment, you evaluate, and you keep it or you discard it. The open-source versions just operate on smaller systems with much narrower objectives. The labs are running the exact same loop I just described with big ambition. So that pattern has been validated. Now the question to me is how far it scales and how fast.
And that leads us to a new concept: the concept of a local hard takeoff. I want to be very careful with my language here because "hard takeoff" is a very loaded term in AI safety circles. It usually refers to a hypothetical intelligence explosion—an AI that improves itself so rapidly that it surpasses human control. That is not what I am describing. What I am describing is something much more mundane and, in a lot of ways, more immediately useful. A local hard takeoff is what happens when an optimization loop closes on a specific business system and compounds improvements faster than the surrounding organization can necessarily track it. So your pricing engine spends the weekend rewriting its own heuristics and comes back 30% more accurate. Your fraud detection model discovers patterns that a human analyst wouldn't try. Your customer service agent autonomously builds verification loops and escalation logic that cut resolution time in half. Each of these is a hard takeoff in the sense that the improvement trajectory is steep, sudden, compounding, and largely autonomous. But it's also local. It's bounded to a very specific domain. It's a specific metric. It's a specific sandbox. It doesn't escape. It doesn't generalize. We're not talking about Terminator here. It just gets really, really good at one thing really fast.
And this is what that auto-research pattern looks like when applied to business. And I think that this is why it's so important to think about it deliberately. And yes, it's happening in miniature already. A Reddit post from a couple of weeks ago about adapting auto-research for agentic coding skills described exactly the loop I just laid out: You analyze the current skill configuration. You apply a scope change. You run deterministic test cases. You evaluate on correctness. And you commit or revert. It's the same pattern, it's just applied to a different domain.
There's a subtlety here that the auto-agent work surfaces, and I think that this is very underappreciated but critical when we understand it for business applications. Traces are everything. When Goo's team only gave the meta-agent scores without reasoning trajectories, the improvement rate dropped really fast. Understanding why something improved seems to matter as much as knowing that it improved. Traces give the meta-agent interpretability over the task-agent's reasoning. And that interpretability is what makes targeted edits possible rather than just random mutations.
This has a direct business analog. An optimization loop that only sees outcomes like "revenue go up," "churn go down," it's going to produce somewhat random improvements. An optimization loop that sees the full reasoning chain, like "here's why the agent recommended the pricing tier," that's going to make much more surgical, logical edits. So the quality of your trace infrastructure as a business determines the quality of your auto-improvement. And this is another reason why basic agent deployment infrastructure must come first. If you're not capturing detailed traces from your agents, you have literally nothing for a meta-agent to work on.
Companies that can define scorable metrics, that can build eval harnesses, that can run optimization loops will compound improvements much faster. And companies that cannot do so are stuck at human speed and quarterly planning cycles with monthly reviews. We've all been there. So, the gap between these two rates of improvement is what local hard takeoff feels like, and it will create asymmetric competitive advantage for the organizations that figure it out first.
Here's where I have to be honest about something that's somewhat uncomfortable to a lot of people. Everything that I've just spent a few minutes describing here—the overnight optimization loops, the emergent agent behaviors, the compounding improvement trajectories—all of that supposes your organization can deploy agents at all. And if you've been listening to this channel, you know how I feel about how most orgs handle that. I have spent the last year documenting why so many org agent deployments fail, and I've cataloged multiple distinct failure patterns. Right? If you listen to this channel, you understand many different reasons like scale, endless loops, bad context layers, specific issues that orgs run into when they fail to scale agents. AI does not create those problems. It just makes them expensive faster. And auto-improvement amplifies all of the failure modes that I've spent a lot of time discussing on this channel previously.
Let's just look briefly at some of the failure patterns that matter the most because I don't want someone to hear this and say, "I can do auto-agents" if they haven't done agents in the first place. The context layer problem is the most foundational. I've talked at length about why agents fail when they lack structured external memory, persistent representation of goals, of state, of constraints that survive across sessions. Without domain memory, every agent session ends up reinventing a definition of done. Every session just sort of guesses at what happened before. Every session discovers a different sense of what success means. The research on context windows just makes this worse. The work I've done talking about lost-in-the-middle context windows is also a big issue because context rot is not solved by auto-research. Instead, auto-research just would introduce new failure modes to a bad context rot or memory problem. Imagine if you layered auto-improvement on top of a bad set of memory architectures. A meta-agent optimizing a task-agent would have no persistent memory. It would have no structured state. It would have no idea what it's tried before. It would be optimizing in the dark, and it would not be able to distinguish between "this change improved the harness" and "this change happened to work on three tasks that ran before the context window got polluted."
So really, the optimization loop is only as good as the infrastructure that runs underneath it. And for most orgs, that infrastructure is held together with conversation history and hope. At least right now, the technical know-how gap is severe. Auto-improvement requires us to get serious about eval harnesses, about sandbox environments where hundreds of experiments can run without a human, about scoring functions that accurately reflect business value. Most teams that I talk to, they have trouble writing a reliable eval suite today, even with their current agent deployment. They're measuring activity instead of outcome, sort of by default, or they're using metrics that don't actually correlate with the business result they care about. Or maybe they haven't even invested in the testing infrastructure to begin with.
On top of all of that, the governance vacuum remains a massive issue with auto-improvement. Who owns the output of an auto-improvement loop? How do we make it useful? Who reviews the 47th experiment at 3:00 a.m.? Who decides which optimizations get promoted to production? The organizations that struggle with who gets fired if AI makes a bad decision aren't going to suddenly develop clear ownership structures just because agents can now edit their own code. Auto-improvement is like a graduate-level capability when most orgs are struggling with agents 101. It requires that you've already solved agent deployment. And I don't think most orgs recognize that when Andre makes a cute tweet. You need to have your context layer figured out, your domain figured out, the eval infrastructure, the governance, the technical depth to build and maintain these systems. And most orgs are still stuck trying to figure out what a context layer means for them.
And so the flip side of this enterprise readiness gap that I'm seeing as I look at where these small orgs are going is that small, agile teams are able to take advantage of these kinds of improvements at the frontier very quickly. So, I mean, Karpathy is an example, right? Karpathy's auto-research was built by one person, Andre Karpathy. Auto-agent was built by a tiny YC startup. Sky Pilot scaled auto-research for under $300 in compute. And it's a tiny startup as well. Basically, a three-person team with $500 in compute can now run the same optimization loop that would take a 20-person enterprise team months to spec and approve and procure infrastructure for and then execute. The iteration speed advantage when you get this right is not marginal. It's multiple orders of magnitude.
And all of that value is leaning on the side of small teams right now because small teams are able to process the speed at which the space is moving today. They're able to experiment rapidly. They're able to wrap in these new changes without heartburn and many, many quarterly meetings. An enterprise team running the same experiments would take months to process and make this change normally. Now, this doesn't mean small teams will beat enterprises at everything. It means that on the specific dimension of rapid iterative optimization, small teams have a really clear structural advantage that enterprise scale can't overcome by default. You would have to deal with approval gates, procurement cycles. The only way enterprise gets through this is if you are cutting the red tape really intentionally as a leader and saying, "I will get literally everything out of the way to empower small teams within my enterprise to move quickly to get this done" because the failure patterns I've talked about with agents in the past are largely diseases of organizational complexity, and auto-research rewards teams that have simplicity at core, not complexity. Just define the metric, build the harness, and let the loop run. This is really a practitioner's domain, right? The compute is cheap. The pattern is learnable. It's something you can do very, very quickly.
Now, I would be remiss if I didn't address safety scenarios at this point because everyone thinks of doomsday when they think of auto-research. But I think the relevant safety concerns when we're talking about auto-improving agents in the business context are not really intelligence explosions. They're about failure modes that are quiet and specific and very easy to miss. The auto-agent team observed the most important one very directly: agents overfit. The meta-agent gets lazy, Goo writes, and inserts rubric-specific prompting so the task-agent can game the metrics. In a benchmark context, that means inflated scores that don't reflect real capability. In a business context, it means your agent is optimizing your fraud model to look really good in tests, and it's going to miss actual fraud, or a pricing agent will maximize the metric you defined while kind of killing customer trust in ways the metric won't capture, or the compliance agent will technically satisfy your test cases while finding creative interpretations of the rules that a human reviewer would obviously flag as terrible.
So, the broader safety landscape needs to take account of this. For businesses, the practical safety concerns break into several categories. Metric gaming is obviously the most immediate: an agent optimizing a proxy metric that diverges from actual business value. But there's also other issues. Silent degradation is the most insidious: you have subtle policy drifts, you have quality erosion that persists undetected because your monitoring infrastructure wasn't designed for autonomous edits. Contamination is another issue where the agent's optimization loop can influence the data it's evaluated against, and the entire mechanism becomes unreliable. Compounding errors also occur because a bad optimization in one system can cascade into a bunch of interconnected business processes.
The auto-research pattern's own designs provide the best mitigation framework we have. We need tight loops, clear baselines, version control, and the ability to revert any change. The agent should only be able to touch one file. The metric is fixed. The evaluation function is locked, and a human does need to inspect the results. These constraints take the optimization loop from being something that you can write a Reddit post about to something that you might actually use in business because the truth is, it is very, very difficult to bind these results to production business systems if you're not thinking systematically about how to plug in a fancy auto-research loop to your actual business processes.
Let's cut to the chase here. I don't think auto-improving agents are optional in H2 of 2026. They're coming. I think they're the next major capability jump in how AI creates business value. And I think the organizations that figure them out in the second half of this year, in 2027, will build advantages that are genuinely difficult to reverse. But I also really believe the jump from basic agent deployment to agents that optimize themselves is so big that most orgs will try to skip the prerequisites and fail spectacularly.
So, let's start with a more honest path to deployment. Let's start with picking one business system, your most measurable one, and try to define what I'm calling the Karpathy triplet: one editable surface (the thing the agent would modify), one metric (what you'd optimize for), and one time budget (how long the experiment will run). If you can't define those three clearly, well, that's the first project you have. You've just got to get to that point. And then once you have that, think about how you build the infrastructure to enable the optimization loop to run, and then how you plug in the results in a way that works for your business.
I know that building the agent pipeline and the eval harness is not the most exciting part to most people, but it's absolutely foundational. You need a scoring function that accurately reflects your business value. You need a test suite that covers the failure modes you care about, and you need a sandboxed execution environment where experiments can run without killing production. Most orgs will underinvest in evaluation infrastructure because it does not produce visible outputs. But for auto-improvement, it is the entire ball game. You cannot automate what you cannot score. So please focus on writing excellent evaluations for your systems.
In this case, I would recommend not starting with customer-facing systems or compliance workflows. Earn the right to auto-optimize by proving the loop works on systems where failure is cheap. You want to design for auditability from day one. You want to make sure that you're logging your experiments carefully. You want to make sure that you're logging all of the edits you make, and you're logging the metric trajectory over time. Make sure that you can revert if something goes wrong. When an auto-improvement loop produces a gain, you want to be able to explain why, right? Not just that it worked, but why. This is both a governance requirement. It's also key to actually taking what the agent is learning and transferring it into your actual human heads. So we all learn as an organization. The experiment log ends up being how your organization builds institutional knowledge about what kinds of optimizations work in your domain.
So we should heavily invest in the human judgment layer if we want auto-optimization to work. People who tell you the Karpathy loop eliminates the need for human judgment are flat wrong. It actually concentrates the need for human judgment. The human's job shifts from executing experiments to designing the experimental framework, writing the program.md file that sets direction and constraints, and you end up deciding ultimately what is worth going to production with and what to investigate further. That is a much higher leverage role than manually optimizing something. It's not lower skill at all. It requires deep domain knowledge, clear thinking about metrics, and the ability to spot when an agent is going off the rails and just gaming the system.
To do that well, as I've been calling out, you need to take small-team agility seriously. I don't care what the size of the overall company is. You need to think about a team of three to five people that can do this. That's not aspirational. That's a concrete statement about how hard it is to set this up if you are trying to share context across a wide team. Auto-research and auto-agent are available, MIT-licensed, open-source. They're all something you can roll out today. The issue is you need to have your eye on internal metrics and targets that you can genuinely auto-optimize against, and you need to have people in your org who know how to read what that is and implement the open-source architecture in a way that actually works for your organization.
You will probably not be able to run overnight optimization loops on production systems right out of the gate. But you can start building the infrastructure that makes it possible. You can build eval harnesses. You can sandbox execution environments. You can have metric definitions that reflect real business value. And you can have the organizational muscle to version, audit, and review the changes. These investments critically will pay off regardless of whether you ever run the full auto-improvement loop because they make your basic agent deployments better, too.
Karpathy was right. This auto-improve pattern will be everywhere. It's just too simple, too cheap, and too effective to stay confined to machine learning training. And that's why auto-agent already extended it to agent harness engineering. And we're going to see lots more extensions in the coming months. The next extensions feel really obvious to me, right? Business process automation, workflow automation, operational systems. It's a matter of when, not if. But the distance between "this works on a single-file training script" and "this works on our enterprise pricing engine" is not a straight line. It's a series of hard infrastructure problems, governance decisions, and organizational design challenges that an overnight optimization loop will not solve for you.
The organizations that win this transition will not be the ones that move the fastest. They'll be the ones that build the foundations—eval harnesses, sandboxed execution, clear metrics, auditability, human oversight—that make the auto-improvement worthwhile. So yes, speed really matters, but speed without infrastructure is running your Ferrari into a ditch. Auto-improving agents are the hard takeoff that we're not watching because it's happening locally and quietly inside really individuals' computers and small teams. But it matters enormously, and the orgs that can get ahead and start to build the hard infrastructure that takes advantage of auto-improvement have a tremendous advantage in the next six months.
The question is not whether auto-research is coming. It's whether your organization can define what "better" means clearly enough to hand it to a machine. And that's really a question we're all going to have to face. Because if you think that this is just about what orgs can optimize for, I want you to imagine how you apply this to your individual role. Within six months, we are going to have harnesses and open-source kits that allow you to take this exact auto-research loop, define a metric within your individual role, and auto-optimize that. What does that look like? Be ready to build the pipeline that enables a machine to auto-optimize pieces of your role, and be ready to recognize that that requires more human talent and more judgment from you, not less. You are being given the keys to a Ferrari, and you need to learn the rules of the road.
If you're not sure where to get started, I built a bunch of stuff to make this easier, including an audit plugin that helps you to understand your current situation, whether you're in an org or whether you're an individual, and also a harness skill that you can run that helps you to understand how your existing harness architecture works for an agent. I want you to have the tools so that if you want to kick off auto-research, you know you're prepared. That's really the most critical thing because the last thing you want to do is start something and not know how it works, not know how to plug it in, and really run a complex process without a clear outcome or piece of business value at the end of it. Best of luck. We're all running into this future together. Cheers.