Transcription
How do you scale the reviewing process? Because now that is blocking your senior engineers. It burns them out. One answer is don't do any code reviews at all.
This is Florian Buetow, AI engineer at Xebia, and we discuss the biggest bottleneck in software engineering right now: code reviews. What do you think of spec driven development? Which of these guardrails gave you the most value? Does it matter which harness I use? GitHub Copilot, Claude Code, Codex. It matters immensely.
Basically, what employees are doing is they give people a hand grenade, which is AI, and then say don't blow up the hand trade, but use it. This is how the best engineers are solving the code review bottleneck right now. Here's the full episode.
As we generate more code, this is really in the build phase code review and becomes a bottleneck. I'm just come back from Google I o that was last week. I think when this episode comes out a couple of weeks ago, and even their Google and the people there acknowledged code review is a bottleneck and we don't know how to solve it yet.
Yeah, Google is an interesting case because in 2025, they already reported that 50% of the code is I, and then I'm pushing for 75%. I always like to look at the big players and see what they do, because on one side you see all of these crazy numbers like efficiency gains, or we automated our entire deployment layer with AI like Spotify did, for example. But very seldom do you actually see how they do that. Right? So they report all of these gains. But then how do you translate that to your own work? That part is often missing.
And for the big companies I think so if you look a little bit about how what they publish and what they write, you can kind of see that they, they have like a, an approach that prioritizes reviewing. No, more like it becomes more standardized because they've identified that now that you can generate code very cheaply, you know, the bottleneck is moving somewhere else. Now some people and this is another tank, but some people would argue that the writing code was never the bottleneck. It depends on kind of how do you see software engineering. But truth of the matter is you generate ten more code. So now you have more pressure on your reviewing, on your engineers, etc..
And I think what the big companies are doing right now is they are combating that with policies on reviews at Amazon, for example, there have been some instances where there were outages, revenue loss due to I generated code, and then they put in policies in place where they now require some parts of the code or some parts of the systems that they have to be reviewed by senior engineers before anything gets merged or deployed. So they are identifying that, you know, in some cases, AI generated code has to be scrutinized, well, stronger than in other cases. So you can see that. And at the same time, you see other companies who say, okay, we automated our entire PR review process. They applying like these. How would you say, well, I like to think about it as a little bit like there's a horizontal way of scaling AI engineering. And the vertical one, the horizontal one is where you automate like processes. You currently have like a PR review, like you can you can say any PR that is created automatically review it with Copilot, for example. And that's what a lot of companies do. But then they don't really talk about how that improves the quality. Right? So the automating part of the human pipeline that we have, what's then the vertical axis. The vertical axis is if you have a project and a small specialized team, you let them build their own tooling to make sure that the product ships and is developed in the way that they intend to. So they are more involved in building custom built environments for their coding agents, and for the to deliver the software like they don't have a blueprint that applies to the product automatically. They refine it.
Yeah. And this is the I see us stacking a lot of capabilities, right. We have the model on one hand. Then the agent kind of harness around the model specifically. And you're talking about an environment in which this agent harness operates. How should I view that.
Well, maybe maybe should just go back one step. So if we look at software engineering, I think it's a well-established process by now. Like if we think before the area of AI software development lifecycle, everybody knows it. And part of the review was done by humans, and it would work because code wasn't delivered faster than you could review it. That has changed now over there. And so the question that becomes, well, how do you scale the reviewing process? Right. Because now that is blocking your senior engineers. It burns them out. There are studies about that where cognitive depth is growing, where people don't understand or their own code base because they just don't have the time to look at everything, or they don't even want to look at the age narrated code, etc.. And so the question is, how do you mitigate that? And one answer is don't do any code reviews at all, okay. But then the next question is how do you do that?
So I think one way to do it is to engineer the environment in which the agents operate. With, well, with intention, so that you don't have to be the human in the loop to give the agent feedback on common things. And it starts with very simple things. So like just let's say code formatting, automatic code formatter, we already had that or a security check. Lots of teams has integrated tools like sonar cube or something that gives you feedback that then a human would take and fix the code with, with the feedback from those systems. Agents can do that automatically. So if you provide agents feedback on what they're doing wrong and hopefully as close as possible to where the agent actually generates the code, which means on the developer's laptop, not necessarily in GitHub. After you make your commit or your PR, then you're suddenly in this world of where you're trying to engineer feedback to the agents in a way that helps the agents to run for a long time without human intervention. So that's what I mean by building these types of environments.
Yeah. And I think it's very easy to get started with it. A lot of it will feel like, but we've been doing this before, but now you're giving the agent that feedback. You're not taking it as a human anymore, right? And some of the models are surprisingly good at self-correcting cells wants to pinpoint them what they did wrong and let the environment give that feedback.
Yeah. Does the harness in this equation, does it matter which harness I use? GitHub Copilot cloud code Codex.
Oh, yeah. Absolutely. Well, in my experience, the harness matters more than the model. Okay. I mean, the harness is what provides the tools, what provides some type of prompting, provides the memory layer and all of these things that then get submitted to the LM to generate as bronze, for example, where the LM wants to make a tool call, etc., and the harness providers and the capability to execute the tool for the LM, it matters immensely. I made an experiment and one of my projects, where I tried to implement a tool based on specifications and tests. So I had a full suite of specifications like specification driven development, like many people like to do. Notice the idea is that if you specify your product exactly enough, I will be able to implement it exactly according to specifications, which is unfortunately not true, but it's an attempt that I made and that failed. And then I thought, okay, what if I use the TDD idea and just generate all of the parallel tests upfront and then let that be the feedback to the agent who tries to follow my specifications? And depending on the harness that I used, it worked or didn't work. Even if I was using the top frontier model in both harnesses.
Okay, which which harness gave you the best results?
Well, that keeps changing all the time. So at that time it was a cloud code. Gotcha. Now, I would have said I would say that shifted to Codex for implementation work. But that's one of those things that's like a moving target, which is also one of the reasons why you can't really stop experimenting. There's this tendency for software engineers to try to systemize things and make it a process. And okay. Those are now the rules. I don't think that works. Things are still moving too quickly. Okay, maybe I should walk that back a little bit. It works at some level, but if you define in your policy, we must only use cloud code. Who knows what will happen when the next release comes out. It might be completely different because the models have different quote unquote personalities like some are very good at instruction following others are very good at filling in the gaps. If you don't provide enough context as a human. So you need to be careful which model use as well.
Yeah, I want to zoom in on kind of what you recommend for harnesses. But before we do kind of following your trailer, thought of spectrum and development and solving kind of the code review process in that way. You mentioned the TDD approach worked better than the spectrum development approach, with a model following instructions in that way.
Well, I would say, well, specs driven was my first attempt and that didn't go well. It was the typical okay, I create the perfect prompt and then the model does something different. I didn't intend, but when you give those the feedback in addition to tell the model when it's off track, that worked for me in that particular instance. And I would say those results might be transferable to other project, but it really depends on the project. Which again, brings us back to the to the to the need to be to be willing to experiment what works.
Yeah. And this feedback, just so I understand this is in an automated way. Right. So you have specs, then you have execution and the feedback is not you telling what it did wrong. But what does that feedback cycle look like.
Exactly. Yeah. So if when you're using like the CLI tools, you have the ability to define what is called, for example, a stop hook. So when the agent is done with its work, the harness triggers an event which is a stop hook. And you can wire that up to a shell script that you are in, for example, so you can automate running your test suite or your guardrails. I like to call them guardrails. And they then you need to engineer your guard right. So they actually output like natural language text. This is forbidden. Do it in this in that way. So you base the feedback encodes the prompt that you would write as a human. So then you trigger that and then that triggers the LM to continue working on the thing. And you can pair these feedbacks that you give with things like roof loop, you know, Codex and Claude, they now have the goal command, which really helps. So they will just keep running for long and longer until they fix the issue.
Yeah. So Ralph Loops specifically allow you to run whatever you want to do in a loop, right? So it goes over it. And this feedback mechanism would then be input in an iteration to get it right and to correct it. Goal is something similar. The idea is the same. The idea is the same. Yeah, I don't know exactly what the implementation of goal looks like, but it seems to be functional equivalent to a roof loop.
And which of these guardrails that you've put in place gave you the most value? You mentioned linting already. Some security controls. You have more examples.
Yeah. So I'm I didn't know before, but I've since become a great fan of semantic grep. Okay. Yeah. Because it allows you to put regex for things that it could catch and code for example a certain type of code construct. The example that always give is I don't want any default values in any of my methods. In Python, for example, you know you can set default values for parameters in Python methods and in the signature of the method. And this, in my experience, is one of the greatest sources of frustration when you have to review and debug the code later. So just prevent that by putting a rule that detects these code patterns and that then triggers error. You must not write it in that way. It's against policy. So semantic grep is really flexible. And whenever I interrogate the AI about the code that I wrote, instead of reviewing the code myself, you know, some of these issues come up. Why did you do it that way? It doesn't make any sense. That immediately triggers me. Okay, let's add another guardrail rule for this particular type of pattern. So what then happens over time is you're shaping your environment. You're iterating towards it to be tighter and tighter, to be aligned with your preferences as a human or how you think code should look like. And part of that is not only taste, but also you want to enforce that the AI generates code that is easy to understand, not for humans, even if you don't want to read it as a human, but also for AI. Write code is context. That's why code quality matters. If you just wipe code something, sooner or later the eye is going to confuse itself by the code.
Yeah, the sustainability of code base has always been needs to be simple and easy to change. And that perspective hasn't changed. With agents, we allow ourselves to generate more code in the code that is out. There might not be simple and might not be as easy to change, but we're like, well, it works, right? And then build it on top of that might be quite brittle. Yes. And well, as humans, we also allow ourselves this little exception where we say as long as it's modular, it doesn't matter if one of the modules is messy or something, as long as it's isolated, right? It's behind abstractions. Yes. Yeah. And that's also something that I found with AI. If you keep your things modular, that helps a lot. Like if you have very clear boundaries between modules, if you have well-defined interfaces that are not allowed to change, for example. And, and I would say that brings us to another type of Gadara which are architectural constraints. So a lot of languages, they have architectural unit tests. They're like unit tests to execute extremely quickly. They only look at what dependencies exist between different modules of your code base, so they can analyze it very quickly and you can say, okay, prevent let's say the let's say what would you say the, the, the UI from accessing the database directly. You could enforce that. It always has to go through the business logic layer or something like that. And because they are also tends to make this weird interconnection between modules that a human would never do. If you just let the I sign your system and then draw your plot of your system diagram, you will see the weirdest things that enrage you, that you encode then into additional architecture unit tests, which I would count as another form of guardrail.
Is there anything that you would say? Well, this is then still kind of part of the human responsibility that would not fit in automated guardrails.
Yes. So something I mean, if you look at software engineering and I'm going back to the traditional way of doing it, it always used to be, okay, what do we want to build? How do we build it. How do we keep it maintainable. Right. Keeping the code base clean and simple means we can go fast for a long time. And how do we do it? Well with architecture, right? We define the architecture upfront and then we start diving into the modules. How do we implement each other? I think that should still stay the same. The models are not there yet. Whether they can do that on their own. And my workload now includes first understanding exactly what I want to build exactly so that there's no doubt, and then sketching out how this might look as a software system. Right? Could be different services talking to each other. It could be on a service level, different modules talking to each other. Which functions do I need in the modules. So you specify pretty much the entire architecture except the implementation. And once you have that you can already encode that as rules.
Yeah. You know so this I think that's extremely important to also to combat, I think on your show, people already talked about the cognitive dissonance that comes from noteworthy in code, where you don't understand your code base anymore and you're not able as a human, to reason about your code base anymore. That comes when you don't understand the architecture anymore, and how components talk to each other within your software system. And yeah, you need to do everything to combat that as much as possible.
Yeah, but yeah, it's very hard for me to imagine where this is going to go because I agree with you. If I understand my software system, I feel like that's the skill that's really going to remain for engineers specifically. And I do believe that the implementation we're already automating, and I feel like with enough card rails, we can hone code reviewing to where only the essence, potentially the behavior. I see people reviewing specifications that will then also remain. But if I put myself in the seat of an engineer, maybe someone that's listening, who's in a bigger organization you might not already know your system, then it's quite difficult to be like, okay, this piece of the puzzle in advance. Do we know how to build it? Or how much investigation do I need to do? That's all work that would happen up front now and potentially before AI, you would do that kind of as you go and you would iterate and you would improve. That whole way of working is now different.
Yeah. It's very, very, very difficult to be messy in your approach now, which is also why people find that what I would say, why some people might resist to it because, you know, doing all the hard work up front, right? You're not just discovering, as you just said, as you go, you're doing all the hard work up front. You probably have to do a couple of iterations on your on what your implementation to get a better idea of how good actually looks like. So that's not uncommon. But you can do that with vibe coding. Yeah, right. And I think that is a that is going to be the new normal for, for, for some time to come before things change again. The but the interesting part is that, you know, I always ask people, well, how could you develop software if you didn't know what you wanted to build? Yeah, right. That's also not possible. So you're doing the same work now. It's just that you're doing it up front. Right. Which might feel more intense, more involved. But that also means that this is becoming its own discipline. And when people think about what should junior engineers do now that I can generate all the code, I would say, well, this is clearly something that can learn. That's an incredible skill. Yeah, right. I've seen the best software engineers kind of fall into two categories. The ones where, like you mentioned, someone had to build something and I saw them stare at their screen. This was still before Covid where we I worked at the office five days a week, and I saw them with kind of pencil and paper, and he was like writing stuff and drawing things out before even doing any hands on. And I was like, that's a very interesting skill. And then I saw other people where they're like, it's very hard for me to Maginn and I have to start typing, but then they're still fascinating and extremely effective in the way they operate. But it's completely like the inverse of what the first engineer would do. Absolutely. And the cool thing is that if you allow yourself to do that, like to do the prototyping, to go in this discovery before you put your specifications not into stone, but before you kind of say, okay, this is what we want to build. Is that it's incredibly rewarding. I really enjoy that because you really feel the speed that AI gives you when you iterate through ideas. I oftentimes find myself talking to an eye for an hour just to explore this idea, and then maybe a tangent idea and that maybe I should have gone on. But it teaches you so much about how you can think about systems, and it also elevates you as an engineer, because now you're not only thinking about, okay, how do I implement this? But you also think about what do we actually need, right? What does the customer need? So the. You can see that I think engineers get much more elevated. They start to think at that product level much more now because of that. And that's just going to continue versus before they were thinking about okay, but how do we write this in code.
How has it been for you. Because a lot of engineers I talked to when they parallelize work is there's a lot of stuff going up here in terms of thinking and execution power all of a sudden, and at the end of the day, they're exhausted.
Yeah. Well, the burnout, because you're constantly exposed to stimuli and you're constantly contract switching because the eye takes 20 minutes to respond to you. That's real. That's absolutely real. Talk to a developer last week and said on Wednesday, spread out. Just just his brains just on standby because he's constantly reacting all the time. I think that's 100% real. As with anything, I think the way to combat that is a discipline. Okay. The first step is you have to be aware of what you're doing. But you also learn how to interleave tasks, maybe on the same project. They do different things, right. So the work on on the environment that guides the agent, that's its own project, besides the actual project that you're trying to build. So and I keep iterating between the two. So when I'm waiting for an agent to complete something because I want to see what it what it does or how it did it, I will just switch to something else. I might start another session with a different agent on the same code base, and start interrogating the agent about the code base. And so I stay kind of in the same context, but I don't switch between project as hard. And if I don't do that, what ends up happening is that I have to type in the session out. Please tell me, what did we do in the last half hour? So I just it has to remind me of what I did. And. It's weird sometimes 100%. Yeah, yeah, I saw Cloud Code does that automatically. Now, if you're out of a session for too long, it gives you like a summary of what you've done so you don't have to ask anymore.
Yeah, yeah, there might be some UX problem there solving the I think ownership and we already mentioned cognitive debt is going to be a very crucial topic that distinguishes kind of engineers that don't really maybe care about this craft or about their work. And also I spoke with Alias Mani last week and he mentioned this newsroom, which I hadn't heard of, which he mentions cognitive surrender. People are like, the agent just takes the wheel, and if it's the problem, it's the agent's fall. And if it works, it's also the agent. But what then? It's a little bit me, but they really just let go of what they are responsible for. And I feel like that's quite risky. And the way we are evolving, I think conversations like you and I are having are incredibly important to give people perspective and to also challenge what goes away and what remains as part of your accountability.
Absolutely. And the funny thing is, basically what employees are doing is they give people a hand grenade, which is AI, and then say, don't blow up the hand grenade, but use it. Right. So there's a lot of risk with AI, but all of that is going to be standardized. They're going to be policies on what you are allowed to do and what not with AI. And the example I gave with Amazon, right. They already made a policy distinction between your code that is more critical than other code and how that should be reviewed. So we're just going to see the same thing. And it's just going to be another, I guess, form of the part of the process for software engineers, right, where you have to define stuff where if you just want to YOLO it, okay, let's not touch the billing system, please. I think that's going to be normal pretty soon. And yeah, and also a lot of the review work that we don't want to do is we can outsource to AI as well. So the idea of not doing any code reviews comes. I mean, it's kind of like a loaded idea, right? But when you attempt to do that, there are a lot of realizations come with it right there. Cheap wins, like guardrails that are deterministic, that execute cheaply and quickly. And then the conversation shifts towards architecture and specifications and more validating the specifications upfront. Then then the code later. And likewise, even when you have to review the architecture later as a human, you can still use AI for that, right? You have a conversation with AI. Maybe if a specialized agent that you particularly set up to review for any violations of a specific type of architecture that you're trying to implement in your project. So you can also do an automatic scan over all of your changes, code changes with AI, and then use that as the point to start looking into it as a human right. So you can still automate a lot. And it's really just about, okay, where should I look? Where don't I have to look? Yeah. Otherwise I don't see how. How are we going to scale the review part? You know, the code generation is ten times faster or 100 times faster. The pressure is going to be on all of the systems that come downstream from that. Absolutely. And part of that is the review with the human in the loop. So you have to try to minimize that as much as we can.
Yeah. The I feel like the input we can likely scale faster, right? We can just execute on more ideas and validate things in production with different tests. The reviewing process is really the challenge. And some of the things that you mentioned, I do see those as ingredients to kind of solving this puzzle. That is the code reviewing process and not having a human in the loop at all. If that's the, let's say, the big goal that we're working towards, even though it might not be feasible now with the models we have, I do see guardrails in there. I do see capturing as much as you can behavior and functionality in tests. I don't think there should be any reason to not write a test anymore, because you can just generate the test according to the behavior that you want, right? If you solve a bug, definitely make a test out of that. That was already a good practice, but there's no reason to not do that anymore nowadays. So that's a trend. Yes. And what also people don't what people also often forget is that generating a test, a small piece of code, the chances of the LM not doing it right are much smaller than when you tell the LM generate me microservice. Yeah, right. So that's also another point that I think helps when you write guardrails, because even if you generate them automatically, which is what I do, the chances of failure are relatively slim.
I have seen and spoken to other teams. I'm trying to get one on the podcast that have fully adopted spectrum and development. As part of their workflow, they use one of the specification frameworks and they're very happy with their approach. That's why I want to have them on the podcast, but they mentioned that they have not built. I think it's my assumption. Got the guardrails, like you and I are discussing within their code review process. What they have done is the review process now very much revolves around the specification and that is what reviewed. That is what is reviewed. And I do think they have some guardrails in place to make sure that whatever's in the specification is also translated in code in a certain way. For me, there's only kind of the risky part, which is the non-deterministic behavior that can be in there. But I think that's an interesting approach.
Yes. And actually, good that you mentioned that because Gatorade can't also mean just a prompt. Right. That's I think, how guardrails started to be defined, if I'm not mistaken. But there can be the terminus thing as well, the specification of development approach, I really like that, but mostly because it gives a human clarity of what is being built. If you give that to an LLM and you observe what it does, you'll be surprised that after five minutes it starts to deviate from your attentions because you didn't specify something clear enough, or it there was some room for interpretation anywhere. So you can iterate now on your specification trying to get it perfect. Then you switch the model and it doesn't work anymore. So does this mean from your perspective, what do you think of spectrum development? Is it something that works now but it's going to go for gone in the future? Or how do you see that?
Well, I think that is part of the discovery product process of what we're trying to build. And I think it's extremely important to have that, because that can also be used for automated tests later, like checking. Do we, did we actually do this? How did we do this. Did we do this in accordance to our coding principles? Right. Those are questions you can then give to an AI to investigate so the specs will stay there. People like to go in-depth with the specifications like they treat it as if it was code. It's not right. It's just a document of shared understanding. I would say that specifies. Part of the behavior. Part of what? What is being built. I think a fine grained behavioral specification of whatever you're building is much more important. There was this idea of TDD where if you have all of your tests and your software gets deleted, but you still have your test, you can rebuild your software. That is one of the ideas of TDD. And well, I've seen it work for the very first time when I had behavioral tests implemented that would give the agent feedback, and I had a specification as a prompt to start the implementation. That was the very first time in my life I had seen this actually work well in my in my project. That's still quite incredible. I wonder if with existing code bases, right, if the test coverage. In theory, a lot of this is in theory, which is why it's quite the challenge. But I wonder if indeed if you would throw out the code if people have been doing TDD in that code base for quite a while, if an AI indeed can then generate the code exactly how it is according to the behavior that was captured.
Yeah, well, if you have some tokens to burn, you can try that over the weekend. That seems like an interesting experiment. What I have noticed is that also the bigger companies previously, before AI and a genetic software engineering, I feel like they were trailblazing and a lot of smaller organizations were looking up to good practices, best practices, conventions and blog posts of what is out there and what's happening right now. I feel like the playing field is quite equal because everything is very quickly evolving. You already mentioned over a few months harnesses and their capabilities might change and what is effective, and all of a sudden what is now possible, which gives us a very interesting playing field to experiment with.
How do you typically experiment and educate yourself?
Yeah, it's a concept trying to catch up, feeling behind type of thing, but I think that's normal for everybody who's trying to keep up with things. So I like to use projects. I have a couple of different projects that on a different scale of one project that is purely vibe coded, purely. I've never looked at the code, I've only prompted it to do stuff. I have one project that is a silly produce that is, that's the one that I use for the TDD approach with the behavioral specification, behavioral tests, Scott rails where test that type of approach. I have another project that consists of multiple microservices, which is much more complex, which is mostly driven, and I keep adding features to them using these different methodologies that are set up for this project and just observe what does the AI do, what does it do wrong, etc. and more often than not, I see they either something weird and then I make that immediately part of my default guardrails. So I kind of have a repository. We have a default project setups for different languages, and they're preloaded with a certain type of guardrails that I found useful in the project that I do. So the experimentation involves a lot of communicating with the LM. I really found it useful to ask the LM to explain itself. So if I give it a complex task, typically what I ask is please tell me what your understanding is of what we are trying to do. Very simple question. So I'm trying to find out what the model understood, what I said versus what I said. And you can see how models interpret things differently. And that gives you an idea of, you know, personalities is too far reach, but it's sort of type, you know, what type of personality does this model exhibit like? What type of behavior does it exhibit? Probably the better term to use. Yeah. So I do that a lot. I also try out the latest tools that I give us. Like when sub agents came, I tried that out. I concluded that I have zero visibility about what the agents are doing. Like what are they communicating with? I have no introspection. So then what I would do is I try to have introspection to that, and I build my own kind of system where I just told it, please spawn a sub agent, but do it in a different terminal so I can actually see what you sent to that different agent. And yeah, well, you will see the weirdest things that the models tell each other, right? You can already see how the models start to deviate, like at the first step when the handoff tasks, etc.. So trying to look under the hood as much as possible, what information gets exchanged, monitoring what the agents do when they talk to each other. I think those are incredibly important things for the orchestration itself, and I learned a lot the most from looking at those types of things, of what the models do and how they talk to each other. So I think that is how I experience most of the time to refine my understanding of how the models to and also trying to apply what they give us in terms of tools.
If someone's listening and they want to apply genetic software engineering to their own way of work, way of working some piece of code that is already live in production, and likely the team has already adopted some practices or not at all, and they want to kickstart something. What is step one that you would recommend for an individual developer, or for somebody in a. They could be within a team?
I think that's the most profound case. I would say the simplest thing is start with guardrails, static checks. Right. You should have of course a code formatter. You should have a linter. Try to write some SEM grep rules that enforce some of the best practices that you see in your code base. You can ask the AI, for example, what are some anti patterns that we use in the code base. Write a simple test that flags them. If you work in a team, you have a team discussion. You know if you get resistance, well then try the next next best thing. It's really difficult to convert people who are not buying into the ideas of trying to improve the code base by streamlining streamlining it with tests or with guardrails. But you can run all of these things on your local machine. You don't have to enforce everybody to use it, but you can use it for a code that you write yourself. You know, you compare that with a smart script that checks whether you created that file or somebody else, and where you enforce your guardrail rules. And the key point is you should measure or at least get an impression of how it works with those guidelines that you put in place versus without them. Because once you see that, they actually help you do more work and be less in the loop, you will not go back. I don't think it's it becomes really convenient to not have to worry about that and to babysit the model all the time.
I can see that right this term, babysitting. I'm trying to get rid of in the conversations that I have with people, and I'm trying to figure out what the best way is. It could be indeed kind of experimenting and going into this cycle of continuous improvement. And it could be you capturing behavior as you want, as guardrails for the agent to then pick up and do something with. And it can also be feedback that you get from your team, right? If you and your agent create a pull request and then there's feedback from your team, you can also then capture that in more guardrails. And there's also another idea. So what you can do is you can just say, hey, analyze my session logs on this project. It'll go into your home folder dot cloud, and we'll start looking at all of your conversations. And you can ask it, do you see any patterns where I had to repeatedly remind you of a certain thing so you can data mine your session logs to see where the model goes wrong, where you always have to correct it. Turn that into a static check. Right? This feels like a product that I would want to have. Yeah. Which is actually not a bad idea. Well, actually it's a skill you can just, well, use a skill to write you that skill. It's very easy. It takes like 15 minutes and then you can use it.
Absolutely. Earlier we mentioned different harnesses might have different results. And in some organizations some people just get one harness and that's it. Right. We've gone all in on GitHub copilot because has all the models you know, but it's only one harness. What would you recommend to either those people or those organizations?
That's a really difficult one if you can choose your tools as a developer. I don't think any developer is really going to be happy with that. I think education would help in that case. You know, I mean, there's studies out there that compare harnesses, and we have enough evidence to suggest that relying on one particular tool is not going to keep us at the edge. So there are lots of arguments to be made. I understand that organization. I have this, this, this, this desire tenancy just to standardize things. But it's not like you select a product and then you run with it for five years. That's not how AI works. And that just needs understanding of what AI actually means.
Yeah I feel like that's that missing. Right. Because the we are now beyond model specifically. And the harness might matter more than the models itself. On paper I think it's easy to understand models because that's we've been talking about that for a long, long time. Right. It's been 2 or 3 years right now. We've gone continuous iterations on models and now I feel like more so this year, maybe even end of last year, the conversation about harness has really started arising.
Yes. Well, there's another thing you can do as a solo developer. Try to find out at what things you particular harness is good at. Maybe it's it is at, you know, writing pull request documentation. Maybe it's really good at debugging. So try to find those use cases where you can still use it and then you use it for that.
Yeah. Yeah, absolutely. For the people listening I'm trying to do this new thing because I want your advice on what experiment you want people to run if they have a weak time and they can do one experiment as kind of their budget of time, they have, what would you want them to do an experiment with?
Well, that depends on where you are in the organization, I would say. But for developers, I mean, there are a couple of options and it depends on what you want. Do you want to get better at a genetic engineering like working with I? Do you want to get better at building your guide rails? Do you want to get better at understanding how the models behave? So that's the first question. What do you want? There's the infinite many different options of what to do. I would recommend if you want to get started with guardrails, start looking at semantic grip. Let's try to encode some of your human feedback that you would give in a PR as a rule, and the example that I have earlier with no default values and method parameters, there could be one example. Another one could be okay, let's never swallow any errors. There was any error must always be propagated something like that. Just start with experimenting with that and see how your agent reacts in that environment. So setting up I think the guardrails so that you can start to improve them and iterate on them. I think that's very worthwhile. And you will see, I would argue that you would see wins very quickly, even though you might ask for things. But we had static code checks first, but now you're doing it in a way that you encode your human preference as rules, right? So custom for your particular project.
Florian, thanks so much for coming on. This is for me. This is really good. Thank you. If you're still with us. Let me know in the comment section what you thought of this episode, and we'll see you in the next one.