Transcription
The agents are getting better. The people deploying them are not. And that is the whole story right now in agent deployments at enterprise, at small, medium, business level, and even with individuals.
Because the truth is AI agents are getting really, really good at doing their work. They can write code. They can generate designs. They can close tickets. We've all heard that story. I've told that story a lot. So the capability trajectory is real and it's getting faster.
But but but there is a memory wall. They still have short-term memories, especially when you think about the arc of a real job. Look, software jobs in tech are as short as they've ever been on average. They're somewhere between 18 months and two years if you average it out across tiny startups and big companies as well. The problem is AI agents are measured in weeks at best. And for most runs, it's like an hour or two. And so when you compare the time, even if we lament that it's short and it's like, wow, this is a 20-month job, it's not even comparable. Let alone the people who really hold institutional context and keep a business going, who often stay four, 5, 6, 7, 8 years, maybe longer. And I am convinced that that gap is one of the hardest problems in tech. And it's leading a lot of overoptimistic people astray on AI right now because AI still lacks common sense in very weird and unpredictable ways.
Yes, AI is getting better. Yes, the frontier is getting smoother in the sense that AI capabilities are starting to even out some. But the combination of AI skills with AI context, with AI tools, with AI resources, with AI workloads, with AI prompts, is still brittle. It still is somewhat difficult to predict what changing one thing will do. And this really matters because AI tools are getting more powerful even as they're still brittle. And they're getting more powerful very very quickly. And so that means net net if you improperly deploy them, they are getting more destructive, not less, especially when they aren't managed well. A mediocre tool that fails really obviously is just annoying. A power tool that fails silently is very dangerous. And that is a world we are headed to.
The best tools we have for managing that danger. I want to say this very very plainly are human brains and human brains crafting evaluations. Not better prompts, not bigger context windows per se, but human judgment about what matters, what's fragile, and what the AI doesn't know it doesn't know. And yes, eval to encode that judgment. But in the vast majority of cases, the correct solution is really, really good human judgment plus really, really good evals. And I don't see enough of that investment. And this video is about the enormous risk that real experienced developers run into when they don't pay attention to that. Yes, I have a killer story for you on this. And they also have a paper that nobody is paying attention to that actually measures the ability of agents to do long long long-term work like maintain a codebase for most of a year. And we have results on that and yes, I'll be telling you all about it.
But first, two weeks ago, an AI coding agent wiped out a production database. 1.9 million rows of student data gone in seconds. The backups disappeared, too. The agent never ever made a technical error. Every action was logically correct. It simply had no idea it was demolishing a live system because the knowledge that distinguished real infrastructure from temporary copies existed only in the engineer's head. And new studies that have come out show that's not a fluke. It's actually a really reliable pattern with the way agents work today. And the thing that enables you to deploy an agent safely in that context is a very very thoughtful human brain that is very very good at giving correct instructions against a correct eval. I sound like a parrot talking about eval but it's what's stopping us from having dangerous agent deployments. So I'm going to keep talking about it and telling the horror stories.
So Alexa Gregorov runs the data talks.comcl club course platform. It's a system that manages homework submissions, projects, and leaderboard entries across multiple courses spanning 2 and 1/2 years. He was migrating a separate website to the cloud and decided to reuse his existing infrastructure setup to save a few bucks a month. Sure, sounds good. His AI coding agent was running the deployment for him. The first warning sign is that the agent started creating a long list of cloud resources that should not have existed. Now, Alexe had recently moved to a new computer and hadn't transferred his infrastructure configuration. We all do that. The agent looked at the cloud, saw nothing it recognized, and assumed it was building from scratch. Pretty logical. So, Alexe stopped the process, but some duplicate resources along the way had already been created. Next step, he asked the agent to identify the duplicate files and remove them. very very reasonable ask. But the agent decided on its own that instead of removing resources one at a time, it would be quote cleaner and simpler, the agent's words, to demolish everything it had created in one shot. Also reasonable in isolation. But what Alexi didn't realize was that the agent had quietly unpacked an archived configuration file from his old computer. Inside that archive were the definitions of his real production infrastructure. So when the agent ran a demolition command, it wasn't clearing out temporary duplicates. It was destroying the production database, the networking layer, the application cluster, the load balancers, the host, everything.
So eventually, the story does have a good ending. It took 24 hours, an emergency support upgrade to Amazon, and a significant amount of luck to recover the data. Alexa immediately stripped the agent of all execution permissions. I would have done that, too. And he now reviews every infrastructure change personally. The agent was competent. The agent was confident. And I also want to call out that Alexi made a lot of reasonable asks. This is not a piece where I'm saying Alexe is not making good engineering decisions. He made a lot of asks that a lot of us would have made in this situation and he just got unlucky. And that's the point because the agent was wrong about which world it was operating in, production or not production, and it did not have the self-awareness to ask. The only thing here that could have prevented this disaster was a human who understood the organizational context or an evaluation that encoded that context into a guardrail before the agent ever got as far as running that command. But neither existed in this case and Alexa had a really rough 24 hours as a result. This, by the way, is the reason why 11 Labs is pushing AI insurance. Their agents are insured. We are going to see a lot more of that.
Now, Alex's story is vivid, but you might be like, "Nate, you're cherry-picking. A lot of agents do good stuff." Yeah, they absolutely do. I spend a lot of other videos talking about them, but I want to tell you about some data that we aren't paying attention to that underlines this idea that we have a persistent agent memory wall issue that is undercutting our ability to manage longrunning agentic tasks well without an extraordinary amount of human judgment that helps us build that infrastructure competently.
So, study number one, the remote labor index. So scale AI in the center for AI safety tested frontier AI agents on 240 real freelance projects from Upwork. So this is like video production work, architecture work, 3D modeling work, game development work, data analysis work and this was end to end, right? The average project cost about $630. The average human completion time was 29 hours, which is, I hasten to call out, something that is not out of the realm of possibility for today's longrunning agent harnesses. When you have companies like Cursor bragging about multi-week agent deployments that can build extremely complicated pieces of software, it does seem reasonable to say, "Well, the human took 29 hours. The agent might be able to do it in many fewer hours than that. We should give it a shot, guys." The best agent completed 2.5% of projects at a quality a paying client would accept. 97.5% failure rate on real work. 97.5.
Now, here's the confusing part. A different benchmark, GDP val built by OpenAI, shows the exact same class of models approaching expert level quality and completing tasks a hundred times faster than humans. Both numbers are real. Both studies are real. The difference is that GDP valves the model all the context it needs on purpose. Here is the brief. Here is the deliverable format. Here's what good looks like. The remote labor index, which is the one that studied Upwork, gives the model a client brief and some files and says, "Figure it out." That sounds more like a real job to me. The gap between these two benchmarks is the gap between can AI do this task and can AI do a job. Tasks come with context provided. AI is pretty good at them now. Jobs require you to bring your own. AI is not super good at that yet. And if you think by the way that we started talking about AI agents a memory wall and we ended up in jobs and you're wondering how do we get to jobs? It's pretty simple. If AI agents cannot figure out an Upwork task with any degree of reliability, there is no way that we are rationally putting them in charge of entire jobs. There will be entire classes of human jobs that are invested in building infrastructure to support longrunning agents. That's happening now. You can do it. There are examples I have discussed in previous videos like the dark factory example where it is possible. You get tremendous speedups. But that only happens when humans with really good brains spend a lot of time very thoughtfully figuring out how to do it. It does not happen magically when the CEO finds a LinkedIn post.
Second study, SUCCI or SWECI, a team out of Alibaba built the first benchmark that measures what happens when AI maintains software over time instead of writing it fresh. Yes, this is the long software study that I told you about at the top of the video. This one is amazing. 100 real code bases, each spanning an average of 233 days and 71 consecutive updates of actual development history. The agent has to evolve the code base forward, adding features, fixing bugs, adapting to new requirements. The way real software gets built over months and years. 75% of models tested break previously working features during maintenance. Three out of four Frontier models asked to maintain code over time actively make things worse. The benchmark punishes agents whose early decisions compound into technical debt later. and almost all of them do. Writing code and maintaining code are fundamentally different skills. AI is really good at the former now. AI is not very good at the latter. We only benchmark the first one right now. And the first one is the basis for the dramatic statements by Daario Amade and others basically saying jobs are over or half of jobs are gone. But I'm going to ask you if you got to have a person maintain the code, what are we doing here? And as much as I love AI agents and as much as I've seen incredible use cases for AI agents, what I see AI agents actually doing in production is depending on humans to build the infrastructure around them to build effective deployments. And humans have to also remember over time what the long-term goal of the project is so they can set this up. When the cursor team set up their AI agent to write these big projects like recreating Excel and writing a browser and this and that, they had to set that intent. They had to tell the AI agent, "This is what we're going to do." And they furthermore had to deliberately experiment on the AI agent harness so that the AI agent harness could work with the agent and effectively deploy the feature. A harness is just the context, the tools, the sub aents, the agent reporting structure, all of the stuff that goes into a longunning task. All of that was human- designed at Cursor. And yeah, the agent gets the credit for doing great work, but the humans should get some credit for setting it up. And the people reading the posts should remember that smart humans have to deploy and maintain this stuff.
Last study, the Harvard seniority paper, Hoseni Maum and Lickinger studied 62 million American workers across 285,000 firms from 2015 to 2025. Companies that adopted generative AI saw junior employment drop roughly 8% relative to non-adopters within a year and a half. Senior employment kept rising. The decline hit hardest in AI exposed occupation. And critically, despite the headlines, was driven by slower hiring, not more firing. The conventional read here is that AI replaces junior workers. We've all seen a thousand headlines to say that. The better read is that AI replaces task execution. Juniors are hired for tasks, debugging, document reviews, first drafts. At least they used to be. They shouldn't be anymore. You should find better jobs for juniors and I have suggested a number of them in recent videos. But AI does these tasks adequately in isolation. Seniors survive because they provide something different. They hold the mental model of the system. And we actually need that more than ever. They know which parts are loadbearing. They know the decision history. They know the things nobody wrote down. The Harvard data is showing a labor market that is learning in real time that context is the scarce resource not agentic coding execution and we should talk about that.
But here's the part where it gets really really spicy. This whole story so far has been about engineering but this story is not an engineering story per se. It just happens to start that way just as the AI revolution has started with engineering frankly because engineering is easy to verify. Yes, maybe the agent can't maintain code for 8 months, but it can pass a few soft evals and submit a PR. And it can pass hard ones, too, to be fair if you write them. Now, make the jump with me from engineering to a wider view. Alex's disaster where he had a technically capable agent that was blind to the larger context of the problem is not a software problem. It's a pattern that's about to repeat in every single knowledge work domain where agents get deployed and agents are getting deployed everywhere. There is no place that agents are not getting deployed in 2026. It is coming. If it's not come to your work yet, it's going to come soon.
Picture a legal team that gives an agent the job of reviewing contracts. The agent can parse clauses, could flag risks, can compare against templates. What it cannot know is that this particular vendor has an informal understanding about payment terms that was negotiated over dinner 3 years ago. It cannot know the company is in quiet acquisition talks and certain intellectual property clauses are suddenly existential. The agent will review the contract competently and miss what matters because the thing that matters lives in the general counsel's head, not in a document.
Or think about a marketing team running agents on campaign operations. The agents can build audiences, they can draft copy, they can allocate budget. They cannot know the brand had a crisis in that market segment eight months ago and the tone needs to be completely different there. They cannot know the CMO made a promise to the CEO about a positioning shift that hasn't been written down anywhere. The agents will execute a technically strong campaign that reopens a wound the organization spent months sealing.
Or finance agents can build technically perfect projections. They cannot know that certain numbers are politically dangerous internally, even if they are technically correct. They can't read the room. They don't know what the board cares about this quarter versus last unless you tell them. In every case, the agent does the task well. It's not the issue. In every case, the agent cannot know whether this is the right task done. the right way at this moment in this organizational context unless a whole lot more is being given to it. And in every case, the human who holds that context is the difference between the agent creating value and the agent creating damage. In some cases, existential damage.
The market is really confused about this. On the one hand, we have the SAS apocalypse. On the other hand, we have Gartner predicting in February that by 2027, half the companies that cut staff for AI will rehire workers to perform similar functions, often under different job titles. Their survey of over 300 customer service leaders found only 20% had actually reduced headcount because of AI. And Forers's data is even sharper. 55% of employers say they regret AIdriven layoffs. Do you see that talked about? We should probably talk about it. And yes, there are very public case studies where companies have regretted laying off their employees and rehiring them subsequently. I've done whole videos on that one as well.
So, where does this leave all of us? I think this leaves us in a world where we are all playing a game of highstakes telephone. Right now, CEOs are hearing that AI can do a lot of things and it can and that is true. It is absolutely transformative to the business. What they are not hearing soon enough is that really good humans are needed to make that happen and that AI being incredibly great doesn't mean that you don't need good people in the enterprise in those job functions. And there's a lot of evidence now starting to mount that there are some regrets about that. In fact, there are CEOs who are publicly stating they have regrets about that. I've made videos about that recently. And this keeps coming back to the same agent problem. If you want to know why is this thing, it is because AI agents can write code but they can't sustain code for 8 months. It's the memory wall all over again. But now you extend it across the enterprise to customer service into marketing into legal into product and you have the same memory wall issue. The AI is really really good at doing a thing, a task, a specific thing like it does on GDP val and really really bad at doing the complete job like supporting software like the Upwork study. That is a common pattern. It is not a solved problem and it is a big big deal that we should pay more attention to.
So now let's circle around to one of the biggest jobs we humans have besides maintaining the longunning context and that is giving the AI an idea at the end of the job of whether it did a good job. And the fancy word for that is just an eval. Right now human judgment is our critical safeguard. And yes, we use it to prompt. I've talked about prompting a ton on this channel and that's very important. It's also important at the end of the process to figure out whether it was good or not. You can sniff check it. You can just look at it, check it. But if you want to do automated work, you also have to put some rules in place to see if the agent can pass the definition of quality that you give it. And the industry is really, really struggling with this. I'll be really honest with you. Whenever I make a video about evals, it does badly. And I think that is so, so concerning. And I'm not going to stop talking about it. I'm not here for that. I am here to tell the truth about what is going on in AI. And the truth is that eval core are a way of encoding human judgment into a test that runs before, during, and after an agent acts. It's the bridge between what the human knows and what the machine does. And a good EVEL would have caught Alexis's disaster. Something as simple as before destroying any cloud resource, verify it is not tagged as production. or maybe before any bulk infrastructure change, compare the current state file against the known production manifest. These are important to prioritize, even if they feel like they're not glamorous. They're the kind of thing a senior engineer knows how to check, and an AI agent will never think to check on its own, at least not at current intelligence levels.
But here's the problem. Most companies deploying AI agents don't write EVLs at all. And if they do write EVLs, they're vibes-based evals. They have someone junior in the organization whose job it is to sit there in front of an Excel spreadsheet and write a bunch of things that in a list together the junior person thinks are a good test set. And nobody ever asks about methodology. Nobody ever asks if they're actually good eval. And nobody ever asks if they can actually prevent real world use cases from occurring until it's too late. Because most eval don't test whether an output is safe for a specific environment. They don't test if it's appropriate for this organizational context. They don't test if you're aligned with really important decisions that were made 6 months ago. The ones who do write evals are usually only testing surface level correctness. And they mistakenly think that this is formal correctness, complete correctness. And I think it's not an accident that we often think of evals as a chore and devolve them to junior team members. Junior team members don't have the context here. You actually need the senior people writing the evals. So don't get an eval into a production environment where it's testing agentic work and the eval says did the code compile like ask something better like does this change break something downstream that the test suite didn't cover here's 16 examples or here's three examples two counter examples and here's a repo to go look at the skill of writing great evaluations is the exact same skill that makes senior people valuable it is not a chore you have to know what right looks like in your situation not just in general you have to understand the system well enough to anticipate where an agent will go wrong in ways the agent can't anticipate for itself. That is contextual judgment seniors have encoded into infrastructure in ways that enable agents to succeed.
And I can hear, in fact, I've talked to engineers, I've talked to other senior people who say, "I do not want to do this work because if I do this work, I think they will fire me." I have got news for you. I want you to go back to the number of CEOs in the Forester surveys regretting their AI firings. Yes, there are leaders who make stupid decisions. It's true. And maybe your leader is one of them. I hope not, but maybe they are. But any leader, and you can show this video to a leader, by the way, if you would like to show this video to a leader and say, "Hey, Nate says this." That's fine. I'll take it. Make me the bad guy. Any leader who is worth their salt needs to understand that the ability to write evals is an evolving skill based on evolving context. If you think for a second as a senior that you can write an eval for an agent in marketing, in product, in customer success, in engineering and it's done and then someone could get rid of you and then it would magically work. It's going to magically burn up and it's going to be a complete disaster and you're going to get something like what happened to Alexa except it's going to be even more impactful because it's be at like corporation scale. It's gonna be really bad. The companies that win the next few years will be the ones that treat eval design as a core competency for seniors. Not as a developer task, not as an afterthought, not as a chore, not as something to throw at juniors, not as something to do before you fire people, but as a primary expression of ongoing institutional knowledge.
If you're deploying agents without investing equally in evaluation infrastructure, which is basically a way of giving them a kind of memory that works, you're handing a powerful tool to a system that has no idea what it's not supposed to destroy. So, it's going to destroy stuff.
Ultimately, the human role in an agentic world is what I've started calling contextual stewardship. maintaining the mental model of your system, representing what you know in ways machines can use, and exercising judgment about when technically correct output is organizationally wrong. This is not a technical skill. It's actually not about learning to code at all or even mastering a particular AI tool. It's about becoming the person in your organization who holds the context that keeps the machines from going sideways. And yes, that's a skill that can be developed deliberately. You can start documenting decisions, not just outcomes. Most organizations track what happened and they never capture why very well. They don't capture the constraints, the trade-offs, the context that made one choice better than another at that specific moment, let alone capturing that in a useful way that's repeatable. The decision context is the raw material that makes agents effective. Its absence is also what makes them dangerous. When Alex's agent destroyed his database, the critical missing piece wasn't a fancier model. It was a record of which infrastructure was production and why. Develop system level thinking not just for engineers for everyone operating in a complex environment which is increasingly all of us at work. Understand how the pieces of your organization connect. Know second order consequences. It's a really big deal now. Build a mental model that lets you evaluate whether an agent's output is appropriate for this specific moment. Not just correct in isolation. The marketing lead who knows the brand's wound history is doing system level thinking. The general counsel who knows the unwritten relationship terms is doing system level thinking. This is the senior skill the Harvard data says the market is paying for. So we should probably be using it. And invest in your ability to write evaluations even if it feels like a chore at first. This is the highest leveraged thing that most people are not doing. You don't need to be an engineer. You need to know your domain well enough to articulate here are the things that must be true for this output to be safe and useful in our world. And if you think to yourself, I don't have agents. Why do I need to do this? If you have Claude in your browser, you have an agent. If you have Claude in Excel, you have an agent. If you have chat GPT and it's using your computer, you have an agent. You got agents. But you don't need to be an engineer. You just need to think what are the checks that might catch a bad disaster before it happens. How can I communicate that context to the agent in a way that matters? And the ability to write an eval is an ability to scale your judgment across every agent your organization deploys. And yes, they will get fancier and yes, there will be central ones and this and that, but that skill will scale.
Remember, yes, AI capabilities are advancing quickly. Yes, the trajectory is real. Don't walk away from this and think it's not cuz it is. But the capabilities are advancing in ways that don't reflect long-term memory. So task execution, writing code, generating content, building models, that's improving at a pace that is really, really, really terrifying sometimes. Contextual understanding, the kind that prevents an agent from obliterating your database, is improving much more slowly. And that is part of why I made a whole video about OpenAI making a bet on this with the Frontier system with AWS. But that is a bet. that is not solved. It is not clear how it will be solved and when it is solved, it is not clear everyone's going to want to give this kind of long-running context to OpenAI or really to any private company because what we are talking about here is the story of your company inside people's heads. Do you want to give it away?
So, the asymmetry here is the story. It's not AI is overhyped. It's not. It might be underhyped to be honest. It's not AI will replace everyone because I think what you're starting to see is that there's some real gaps here. The story is that the gap between what agents can do and what agents understand is actually getting wider because agents are getting more intelligent without getting better at memory. And the humans who have to stretch to close that gap through judgment, through context, through evals will become the most valuable people in their organizations. The humans who don't who choose to say nope not for me will find themselves competing with machines on the only dimensions where machines are improving the fastest. Gardner's rehiring prediction is not about AI failing. It's about organizations discovering too late what their humans were actually providing. The task execution was super visible. The CEO could see it. The contextual stewardship was invisible. You don't realize invisible infrastructure is loadbearing until you remove it and something collapses.
So yeah, the agents are here. They work. They're improving all the time. And that's what makes it scary because every single one of them that keeps getting better is not automatically going to be aware of the context that keeps your business alive. In fact, it's going to be automatically unaware, automatically blind to the context that keeps your organization going. Your job, whatever your title, whatever your domain, is to be the one who sees what they can't and then to write an eval that makes sure they never have to. And if you think that's worked for the agent, I hope you heard that last 30 seconds and replay it again because you know what? This is about making your contextual stewardship visible. And if you make it visible through evals and you point out, hey, this is ongoing contextual stewardship. I added three more evals and this is why and this is what we prevented happening this week, etc., etc., you are reminding leadership that there is a lot of value in good context to make sure all of this fancy AI execution doesn't go off the rails like what happened to poor Alexa and I'm so glad Alexe that you got your database back I have been the person that horrifically deleted something in prod it was earlier in my career I deleted half an Oracle eyesore instance thanks to terrible UX inside Oracle at that time several several several years ago and the feeling in the pit of my stomach at that time is one I will never ever forget So, I'm so glad you got your database back, Alexe, and let's all make better choices when it comes to AI agents and how we assign them tasks and how we make sure that their memory constraints are not going to be actively destructive to our business, to our product, to our lives. Cheers.