📱

Get Our Mobile App

Take your business learning on the go!

Download on the App StoreGet it on Google Play

Your AI Agent Is Confidently Lying to You. Here's Proof and How to Get it to Stop

AI News & Strategy Daily | Nate B Jones13:40

Transcription

The hottest thing in coding right now is a little plugin for Claude Code named after a Simpsons character. That's right, we're talking about Ralph Wiggum, the annoyingly stupid Simpsons character who just says, "I'm helping." When he doesn't really.

Jeffrey Huntley is an Australian developer and he developed Ralph as a way of addressing what he found as one of Claude Code's most annoying features, which is that it says it's done when it's not. It says I'm helping when it's not. And so the technique he developed is alarmingly simple. All he does is he does not let the model stop and he keeps feeding the model the prompt over and over and over and over again. He force feeds the prompt to the model and doesn't let it stop until it actually fully completes a defined task.

Now this isn't perfect. It's not a universal hack. I don't want you to walk away and say, "Oh, we should have been refeeding the prompt all the time. This is just going to work perfectly for everything." This works well when you define done in a technically precise way that is very binary. It's either done or it's not. It does not work as well when it's like make the deck professional, right? Like that's harder to get right.

But I think it points to a larger thing I want to have a conversation around, which is that at the end of the day, we have been calling models smart or not smart based on whether or not they get done with tasks. And we've been implicitly assuming that it's up to the models to decide when they get done. And if they're smart, they'll figure it out. And what Ralph suggests is it might not be that hard. Maybe we need to decide when the models are done by being much more aggressive with our evaluation layers. Instead of making evaluation a test that you run at the end, Ralph suggests that we should make our evaluations the steering wheel for the entire process. We should basically force feed evaluations throughout every single iteration and not accept initial outputs and push until we get what we want.

Traditionally, eval meant grading a model's output, right? You give it a question, you score the answer, and you move on. But as agents operate autonomously more and more as they write code, as they modify files, a singleshot grade doesn't tell you a lot. What matters is whether the agent converges toward correctness when it's forced to confront reality. And all Ralph does is it forces the model to confront reality every single iteration until it actually finishes the task.

Now, technically, this plug-in mechanism is extremely simple, and that's part of why it works well. Ralph Wiggum is just a stop hook powered loop. In other words, whenever Claude thinks it's done, the Ralph Wiggum hook triggers, prevents the stop of the task, and reinjects the original prompt. So, every iteration is going to see modified files and history from previous runs along with the original prompt and continue to work against the original prompt with that updated history until the work is finished.

Ralph doesn't make the model smarter. It makes the evaluator more autonomous and more powerful in the system, which is why it feels like such a strong hack. It's essentially a simple harness extension over the top of claw code that feels like it gives the model some degree of external authority, not just at the end of the process when the model says it's done, but all the way through.

One of the things that makes Ralph especially powerful is that it confronts the tendency that models have to say they've done the thing when you ask them to do the thing when they really haven't. Models love exporting done when they haven't finished because they're wired to emit helpful responses and done seems helpful in the moment and the model's not thinking past that moment. And that's why Ralph is wired with a lot of framing to remind the model that it cannot escape by just writing done. So the plug-in prompt that goes with your system prompt when it stops claim from stopping contains extremely explicit anti-ying instructions like this statement must be completely and unequivocally true. In this case, it's your statement, your goal statement. Do not output false statements. Do not lie even if you think you should exit. Please trust the process. Do not force the end of the process by lying about the dness.

Look, these aren't magic words. The point is that this simple trick is confronting one of the alignment problems we see in models, which is that models like to seem aligned to your task when they are not aligned in practice. And this is why we need to move from the idea of evaluations at the end of the process to what I'm calling workflowshaped evaluations. Things that help us steer workflows in the middle of the process like Ralph.

Ralph works because software can be judged by machines if we have a clear sense of what done looks like and if you can keep smashing the agent and telling it not to lie. This is an inversion of the usual AI coding workflow. define the success criteria up front. You let the agent iterate toward that criteria and then you treat the failure as data. Right now what you have is a more of a recipe for a continuous run until the model converges on the correct solution. And once you accept that some of the per the the most public metrics that we have on AI agents start to look different. Your headline metric isn't what can the model do on the first pass. It's something closer to how accurately does the model converge over time or how efficiently does the model converge on the correct solution given a particular budget like how many iterations to green state would be a good example of that.

So why is this strategically important in 2026 specifically? Because it's suggesting to us that the real bottleneck in agent performance is moving pretty rapidly away from model capability and toward the way we harness our agentic models. If you can buy iteration, you can buy correctness, but only if correctness is anchored to something you can actually verify. And so if you're just saying make it professional, make it good and you're doing one shot, that feels like a very 2025 approach to development. Whereas if you actually are using something like Ralph where you continuously remind the agent, this is what a quality job looks like. These are the tests you have to pass, then do it again until you get it. Now you're starting to look at a 2026 pattern where you're iterating until you converge on the correct solution.

This has implications way beyond engineering even though we talk about it as an engineering problem. Yes, Ralph is framed as an engineering solve today. But I think what we're seeing is that Ralph like steering of iterating models is going to start happening to non-coding use cases in 2026 as well. Because as soon as we start to admit that what we really want is correctness, we can define correctness and we can converge toward correctness if we give the model multiple iterations. Well, then at that point, the thing that matters most is being able to construct something like Ralph that lets you say, "This is what's correct. This is a failure mode, and I'm going to stop you and not let you finish until you get it done." And then be the human at the end that ensures that indeed the model did finish.

More and more and more we are looking at a world where non- tech and tech workflows are converging toward these technical design patterns where you take software engineering principles and you push them into non-technical spaces. I think we're desperately in need of a dictionary for everyone that translates some of these concepts that are hard to believe and understand for non- tech folks or folks traditionally considered non tech. I think we're all considered tech now but but here we are. things like eval. Ralph is essentially an eval, but if you talk about it as an eval, you're kind of missing the point because we've traditionally put evals at the end of the process. And Ralph is really designed to work in the middle of what's considered a long multi-iterative process to force the agent to finish in a direction that's clear and coherent.

Now, there are other folks that set up these loops that work similarly, and I don't want to pretend that Ralph is the only way to do this. There are folks that set up their agents to pass a whole series of six or seven evals and send the agent back into a loop until it does that. Most of the folks that do this today are engineers, but I think one of the most productive directions to go with software development in 2026 is to look at how that same pattern can persist across workflows we would not traditionally consider technical.

Let's say you're building a PowerPoint deck. Your PowerPoint deck should be able to converge on correctness in the same way as a piece of software as long as you have the right evaluations for brand consistency, for quality of work, maybe for briefness and conciseness, maybe for clarity toward underlying numbers. But we don't have eval. And when we are building our decks today, we are as knowledge workers having to do those checks manually.

I think what we're starting to see is that work in 2026 is going to shift in a Ralph Wiggumlike direction. We are going to work more toward I define what good looks like at the beginning. I have agentic harnesses around my LLMs that help them converge toward that definition of done and then they are doing that automatically while I make coffee and I am coming back at the end and I'm checking the work. And what this suggests, by the way, is that workers are going to have to get much better at defining out large pieces of work. If you ask someone today to say, "What is a two or three day piece of work or a two or three week piece of work that you know you're going to have to tackle that you could delegate?" Most people cannot define that for you off the top of their head, let alone define it clearly enough that they can build a Ralph Wiggum pattern to evaluate and iterate on that loop. But we're going to need to get there. We're going to need to get to the point where we can say, "Yeah, I actually have like a two-eek project every quarter where I have to build my quarterly reports, and if I don't do it, it's going to be bad. I would love to delegate that." Or, "I have to do competitive reviews every month. they'd love to delegate that. You get the idea.

There are many, many categories of repeated knowledge work that are begging for something like a Ralph Wigum iterative conversions flow to drive a quality result over time. And the thing that is missing is our ability to define what good looks like, our ability to define what done looks like, and frankly an agentic harness that is more friendly to people who are traditionally non- tech. It is really, really scary for someone who is trying to use the terminal for the first time in a long time and use cloud code and who then gets told, "Oh, and now you have to install a bash script that's going to send a web hook in and stop cloud code from working and make it work longer and harder until it finishes your prompt." You can get the idea, but the act of doing it in the terminal is scary.

Now, I think that there's two sides to every story. There's two sides to every bridge. I think folks who are non-technical, we're going to need to get more comfortable being technical, being at the terminal. Bash scripts aren't that scary. I've written them. Even as someone who didn't start out as an engineer, it's going to be okay. On the other side, I think we need to do a lot of work to make a lot of these engineering patterns more translatable. And that's something I spend a lot of time on in my videos because I think the idea is intuitive. Even if you're not an engineer, it just makes sense that if LLMs are trained to be helpful, they would be trained to be helpful, even if that means lying. And it makes sense that one of the ways to fix it is to remind them of the original expectation of success and not let them stop until they are sure they've met it and they have checked and checked and checked and checked. And you can do that multiple different ways. The Ralph Wiggum loop is just one. But it's it's the principle that scales. It's that principle that scales out of engineering land to the way all of us are going to.

Ralph exposes a weirdly optimistic truth for 2026. If you can build something that judges the game you are trying to play, the product you are trying to build, the deck you're trying to make, whatever project you're working on, you are going to be able to buy accuracy, correctness, and reliability with tokens, with retries. And that's the real thing that's exciting. The world is going to belong to people who can define what done looks like, who can tell Ralph Wiggum, this is what finished looks like and who can do so in a way that's so clear and so verifiable that you can't gain the system. And so, yeah, Ralph is just a hack, but Ralph is a hack with a thesis behind it that's really interesting. It's essentially the ecosystem saying out loud, we cannot trust the model self-report. That era is over. And in 2026, the core question isn't can the agent do it. It's can the agent harness force correctness over time. And so my challenge to you is how are you thinking about that correctness? How are you thinking about steering these models so that they get where you want to go? And frankly, can you define that task? Can you define the larger pieces of work you want done and what done looks like so clearly that you can make sure even Ralph Waggum gets it.