Transcription
If you review every single line of code, you become the bottleneck. So I don't reveal this first pass code from the agent. Eventually I got to a point where I find myself never catching anything the agents don't catch. I typically have like at least five different sessions actively running. On average, there's like 20 to 30 agents running. Most of the time, uh, it's like 20 to 40 kind of PRs every day. Our workflows and how our teams work were built at a time when we spend most of our time coding. But when you start to write like 10 times more PRs, we are not ready for that to really scale up how much we can get from the agents. We have to move ourselves out of the loop as much as possible.
Hey everyone, today I'm really excited to welcome my friend uh, an L8 engineer from Meta at Microsoft who's now a solo AI builder. is going to show us exactly how he builds products using agents. I've been asking him a lot of dumb questions about all this. So, we're really excited for him to show us live. So, welcome, sir.
>> Thanks for having me here, Peter.
>> All right. So, uh let's get right into it. Maybe you can start uh by kind of walking through at a high level how you're building products with agents.
>> All right. Um that is my workflow. Um plan, code, and validates. Uh I don't think this is too different from uh what everybody does. Um, so I'll probably talk through the parts where I think I'm doing something unique. Um, so I think typically when we build something meaningful, we typically go through these phases, right? We plan what the requirements are. Um, and then we let the agent code and then we uh have to do some validation to make sure the agent actually did what we wanted them to do. Um, so this uh the high level workflow I think is pretty standard. Um, where I think I do something different uh is uh how much time I spend in each phase. Um so currently I think I spend more time in the planning phase. Uh so planning is like mostly me with assistance from uh the agents. The coding phase is pretty much entirely the agents. Um so I once the requirements are planned very clearly. Um I trust the agents to do most of the work. Um and then in validation phase uh I use agents a lot as well. Um and agents do most of the work with some judgment from me when things are ambiguous. And I think uh the the the part the part about this is that um if we actually uh start to delegate most of the coding to agents.
>> Mhm.
>> What I um the way I think u I can uh get agents to do more for me is to try to increase the amount amount of time agents spends in this phase because this is entirely agents right so if we can get the agents to do to go for longer uh then I'll get more done. So this is one area where I tried a lot of things to just scale up the amount of time I can let the agents run autonomously.
>> Yeah. It's almost like the code and the validation is a loop that the agent can run itself, right? And and so that it can actually code for a longer time period.
>> Yeah. Yeah. And also I think it depends on how much time we spend in the planning phase. So if I uh spend a lot of time crafting a very detailed plan then I can let the agents go for longer. um if I uh only write a very very short prompt then what I'll find is that uh very quickly the agents will get work done and then I'll need to go back and prompt them again. So uh like how much time we invest in the planning phase actually affects this a lot.
>> Okay, that's a really good point because I I've gone like super lazy with these agents. I don't actually like I just give them like one line prompts and yeah it never works for hours. So yeah would love to kind of see each face.
>> Yeah. Yeah. So yeah, I think the things that we we can do differently in the planning phase is like go from a short prompt uh to say what is the next action you should take to something more like a spec where um you you write down a more uh a more comprehensive set of details of the requirements and then go from spec to a goal. So if you can actually craft a measurable goal, you can let the agents do a lot of experimentation.
>> Okay. Okay. So can you show us how this works? like maybe we can start with the planning phase like
>> some some example plans that you write.
>> Yeah.
>> Yeah. Actually uh there's another uh dimension of how I optimize this flow as well. Uh which is like if you look at this uh this uh timeline, right? Um the parts that need me is only like this beginning and the end, right? Uh so what I do is like I I make sure I can paralyze a lot of sessions. Um so so that's I'm always spending my time productively um uh while the agents are doing the work. So I think increasing the the amount of concurrent parallel sessions that's also a very important aspect of how I get more done.
>> And do you parallelize sessions in the same uh project and product or like across products or both?
>> Uh both both. Uh so I have a hybrid of different projects. Uh but even within the same project I sometimes have multiple sessions doing different things.
>> Yeah. It's funny. It's funny because we used to uh like you know both of us used to work in big tech and um it used to be a lot of context switching between meetings but now you're context switching between different threads or [laughter] you know it's actually it's actually faster context switching in some ways.
>> Yeah. Yeah. Totally. I I think uh it's kind of like a um someone that's overseeing a very large scope, right? There's always different things happening and there are different things escalating to you and you need to jump into different things depending on what is the where you are needed the most. Uh so this is very much alike.
>> Okay, this episode is brought to you by linear. When engineers use tools like cursor, clock code and codeex, a lot of work happens invisibly. Someone can go from a bug report in Slack to a shipped fix without creating any record of what happened outside of the code editor. And that's fine for speed, but it makes coordination harder as you scale. Linear integrates with the very best agent coding tools directly like cursor and codeex. That way, anyone can see what an agent is working on and who assigned them to the task. You get the speed of agents without losing visibility across the team. Product teams at OpenAI, Ramp, and Block are all using Linear to collaborate with AI agents. And I use LIR myself to run my creator business. So, check it out at linear.app/ aents. That's linear.app/ aents. Now, back to our episode.
>> Can you show us your, you know, AI stack or agent decoding setup?
>> Yeah. Yeah, let's do it. Uh, so this is my terminal. Uh, this is where I I do like all of my work pretty much. uh occasionally I I switch to a GUI or a browser uh but most of the time uh I'm spending here. Uh so yeah, I'm using a project here as an example to walk through it. Um, so this is this is a project called hybits. Uh, this is the AI tutor I'm building for my son. Uh, it's an AI uh agentic uh harness uh for kids basically. And I just um built a new screen. Um, so let me let me show you what that looks like. Um, I revamped uh the um the main screen a little bit. Um, but this is very messy because I just did this this morning. Uh, and it's not looking good. Uh, this is like this is not how I want this to look like. Um, so I uh what I'll do uh like very typical workflow. I'll take a screenshot of this, right? Uh, take a screenshot and then I come to my agent. Um, I use open code a lot. Uh, so I'm going to just launch open code in here.
>> Mhm.
>> And you use it because you can use multiple models.
>> Yeah. Yeah. Exactly. So I I can very quickly try different models when the new models come out. Uh, that is the uh big benefits I get from these open source tools.
>> Makes sense.
>> Um, so yeah, so what I'll do here is I'll just say hey look at this uh this screen. I'll paste uh the image here. Um, and uh, I'll say uh the things we saw on the screen. The things that I'm I was not very happy about was there is uh too much technical details not uh that are not friendly for kids. Uh, also there is a big area of white space uh unused, right? Those were the problems that we saw on the screen that's were like clearly not uh uh ideal. Uh, so I'll I'll point out these problems um and I'll say hey uh can you propose uh some options for how we improve, right? So this is my uh the request I sent to the agent. So because I sent the screenshot uh the um the model is going to be able to see visually uh what is going on there and then it's going to uh look at uh the codebase as well. This so yeah, it's very quickly came up uh with this plan. So it says like best direction, option one, option two. The thing with this plan is that it's not very easy to read, right? Um, so like when you look at this long wall of text, I like this I I I I will spend so much time reading this text. Um, so what I do instead uh, let me just try a new session. Uh, what I actually do uh, is I use a visual editor to uh do the planning. So, uh, I'll say the same thing. uh look at this screen, there is too much uh technical details, same thing, right? Uh, I will just add one bit to say use lavish uh to discuss this with me uh along with any questions you have. Um, so lavish is a is a visual editor uh I built uh after I read the article about HTML uh over markdown. Have you seen?
>> Yeah. Yeah.
>> The from the
>> Yes.
>> Yeah. Yeah. Um, initially when I saw the article, I was not very sure about that because I I felt like HTML uh is going to be so token inefficient, right? Uh, the models will have to write a lot more than a simple markdown. Um, but when I tried it, it's actually super useful. Um, so I'll show you once uh once we uh have this result from here. Um, the HTML as an artifact can be a lot richer in terms of like supporting this collaboration between human and agent. Um, so it's not going to be a long wall of text I have to read through. It's going to be like very visually um things I can just interact with.
>> So Lavish is a is is like a app that you build to create the HTML in the format that you want. Is that?
>> Yeah. Yeah. It's a um it's a tool I built. Uh, so what I do is like I uh every time I encounter any kind of a friction in my workflow and I don't find anything that can solve the problem for me, I just build something myself.
>> Yeah, Lavish is a is a tool I built. Uh, it's a tool for
>> both generating the HTML artifact and also supporting the uh back and forth interactive experience between human and agents on that. Um, because what you um what we could do is I can just ask the agent to generate a HTML file, right? Uh, and I and then I can open up the HTML file in the browser and it works. Um, the problem with that approach is that once the HTML file is open and I I look at the HTML file and I see that there are some things I don't like, it's very hard for me to then tell the agent, hey, please change this part. U please iterate on this aspect. Right? So that back and forth is what um Lavage Editor is trying to solve.
>> Oh, awesome. Yeah. Really excited to see what what it is. Yeah.
>> Yeah. Yeah. So now it's writing uh the HTML. Uh, it'll probably take a little while because uh that's uh usually a lot of content to write. Um, so uh, let's see what I um maybe uh one thing I can show here um is that uh while the agents are working uh typically agents either coding or planning can spend quite some time doing this work. Um, so what I do is I'll just spin up another parallel uh terminal tab uh a window, right? I use t-mox, so this is a new t-mox window um and in this window I will do something else um and we can see it's in the same directory. The problem here is that uh if I spin up another agent to work in the same directory, they will run into each other, right? So what this agent does in this session will like step on toes of the other agents that were that's already doing the work.
>> Yeah.
>> So this is where people uh started using work trees. So typically people uh what people do is like get work tree ad and give another directory uh like high bits and spend like five minutes thinking about the name. Um, but I'm just going to say h high high bit too. Um, so the the thing the problem with this approach is that once I create a work tree like this, next time I come to this work tree, I have to think about what is hybrid 2 doing, right? What is this this work tree doing, right? Is it still being worked on? Is it like okay to like use for something else? It's very hard to keep track of.
>> Um, and the other problem is like when we create a new work tree, the dependencies are not installed in the in the work tree. So in this work tree we have things like node modules, right? Like these are dependencies downloaded on the fly, and these dependencies won't exist in the new work tree until you install all of them again. Um, so there were many problems like that.
>> And just for people who don't know, like like what's our definition of the work tree? Is it like a copy of the codebase, right? Or?
>> Yeah, yeah. So a work tree is basically like you can think of it as a clone of your current uh git repo um in another directory. So it's going to be a parallel directory and they don't directly interfere with each other. Um, so you can do um a different kind of work, different set of work in the work tree and it won't affect what you were doing in the main repo.
>> Okay. But but you're saying that there's like a many issues with the work tree. So what do you do instead?
>> Yeah.
>> Yeah. Basically there's a very heavy like cognitive load to maintain the work trees. You have to think about which work tree is which uh and which ones are okay to clean up, etc, etc. Um, so what I did was I have a tool called Treehouse. Uh, so Treehouse is basically like a a a no-brainer like uh a a very like dead simple way to manage work trees. Um, so every time I have to spin up a new work tree to do something new, right? I don't need to think about uh do I have another work tree I can use? Do I uh create a new one? I just type treehouse and treehouse will basically set up the work tree for me and drop me into the new work tree. Um, so now it you can see it's set up a work tree in this directory, right? And uh it dropped me into it. And the the good thing is that this directory um is a is from a pool of managed work trees. So um, so the dependencies are already installed here because I have used this work tree before. Um, so I don't have to like reinstall dependencies, rebuild the project every single time. It also saves on the efficiency aspect. So yeah, just reduce the mental load a lot. I don't need to think about anything. I just type treehouse every time I want to start a new session.
>> That makes sense. Okay. All right, dude. Well, let's go back to the other tab.
>> Yeah. So this is uh what's the HTML looks like. Um, so it's saying, hey, uh redesign discussion. Uh, it's basically there's a tiny icon here, not available. Not sure what happened there, but u basically it's uh it wrote the proposal in in a visual artifact, right? Um, so what's feeding off? The screen is doing like grown-up work in kids space. Exactly. Right. Um, and these things uh there's uh unused space. Um, yeah.
>> This is easier to scan and read for a human, basically.
>> Yeah.
>> Yeah. Yeah. And uh if if there's something I uh I look at the uh this artifact and I if I see something that doesn't feel right, I can just annotate. Um, so bit has no visible body. I can say I just click on this and say I don't care about this. Um, and give the feedback to the agents this way. Um,
>> Oh, I see. So this is your app. Okay. Got it. Okay, that makes sense.
>> Yeah. So this is a lot more difficult to do when it's a long wall of text, right? Uh, when it's a wall of text, you have to say to the agent, hey, I I I I don't I'm not happy about this part of the spec. Uh, and you sometimes have to copy paste a lot.
>> Got it.
>> Yeah. So it basically proposed a bunch of things. Uh, copy, clean up. Uh,
>> yeah, some of the layout things is not ideal, but yeah, I I get it. It's it's easier to read for sure. Yeah.
>> Yeah. And I I I think there's probably like something that went wrong in this uh page. Uh, let me let me let me just check. Uh, I can just ask the agent as well. Um, because uh when I look at this, I think the agent is trying to give me a visual representation of the layout. Um, but because of the CSS is not quite working or something. Um, it seems the CSS styles uh not working. Let me fix it. Um, so yeah, I can just send feedback back to the agent uh this way and um I don't have to keep switching between the HTML artifact and the agent uh in the terminal. Uh, I can just talk to the agent here. Um, and I can easily annotate everything uh and just point uh the pinpoints exactly where I mean.
>> Can you show folks where they can download this tool? It's it's open source, right?
>> Uh, so it's uh in my GitHub repo lavishi uh in this repo. Uh, and it has uh it's actually very simple uh to start using it. Just tell your agent use npx lavish axi to write the technical plan or do whatever you want.
>> Um, and the agent will go uh invoke this and everything goes on from there. And uh you have to do you have to hook up your own uh API key for the LM?
>> No, you you just use whatever agent you are already using. Um, this lavish editor itself does not uh run another agent.
>> Uh, it runs within your agent session. So actually
>> Okay, got it.
>> Yeah. So you can see here um the agent calling lavish axi uh to pull like this uh this artifact.
>> Okay, that makes sense.
>> So let's come back to it. Um, yeah. So, so now uh it fixed the CSS problem, right? This is what what is supposed to look like. Um, so you can see like this is a lot uh like more visual and easier to understand.
>> Looks a lot better. Yeah.
>> Yeah. So this is like pointing out the current layouts, current uh problems and then uh it's probably like proposed a new thing. Okay. So it proposed four directions for using the space better. Option A looks like this. This is like this is so much easier to see, right? Like than like the long wall of text we have in the uh in the uh terminal. Um, so here we can see okay it's uh moved the layout a little bit. Uh, now this is the chat, this is some other area. Okay, that's one option.
>> Um, and it even gave me buttons. So uh, if I like option A, I can just click this button and I get the option A. Um, got it. So option B looks like this. Uh, today's goal. Okay. Um, option C, uh, is this. Okay. Option C is very simple. I actually like this. Um, option D. Okay. Yeah. So, let's say I like option C. I can just click this
>> and it basically killed a uh a piece of feedback to the agent saying I like option C. Um, so it's just so easy to interact with. Um, I don't have to keep typing uh every time I want to tell the agent something. Everything can be done interactively.
>> Okay. Okay. So, and and this is uh the plan phase for like building a new feature on top of an existing app, right?
>> Yeah.
>> I'm curious and maybe not to show this, but I'm just curious like how you plan something from scratch initially. Like did you like spend a lot of time planning like the the milestones and the tech stack and that kind of stuff?
>> Yeah. Yeah. So, if it's something from scratch, uh I usually have to spend a little bit more time. Um, so, what I do is that I use the same lavish editor. Um, I tell the agent that I I want to brainstorm a new idea with you. Um, and uh, I'll probably like talk through some of my initial thinking for what things I think um, are the core parts of my idea. And then I'll ask the agent to um, criticize that and uh, come up with like areas of risks uh, or weaknesses I haven't uh, may uh, maybe I haven't thought through yet. Um, and then come back with it uh, its opinion. Um, and the agent will then come back with a uh HTML artifact like that, and I can look at the artifact to uh, basically like work with the agent to refine the idea to a point where it becomes a spec, basically.
>> Do you always like include some certain sections in your spec, like build it in three phases, or like here's the milestones, or like here's a tech tech stack I want you to use, like that kind of stuff?
>> Yeah. Yeah. So uh for some projects, for some ideas, I already have uh some uh opinions on things to use and things to do. Uh, in those cases, I'll just write them down and say these are my preferences. Um, but I always tell the agents that it's okay for you to push back if you see something that is not right. Um, because I want to give the agents the flexibility and I want to see more options as well. Um, so yeah, I I I basically like give my ideas to the agents uh, but let the agents give more back. So, so then do you have like a user level agent.mmd or something that like uh has some of these best practices, like you know, you can push back on me, or it's just more natural through the conversation?
>> Yeah. Uh, so I um I actually built a lot of those instructions into uh lavish editor.
>> Um, so whenever the agent is is using uh the lavish editor to work with me, the agent already knows uh a lot of those um like those best practices.
>> Got it. Okay. And how about how about like uh if you're building like a user-facing product, how do you think about the design? Do you have like another tool for design, or you just you have some skills?
>> For design uh you mean visual design?
>> Yes.
>> Yeah. So for visual design, I like cloud design a lot. Um, since it came out, I use that a lot. Uh, and uh, very often I I'll use a lot of the quota they have uh for me. So if you look at this uh this this bar where I track my quota, um cloud I mostly used up my weekly quota already, I'm waiting for the reset, and cloud design I used um like uh two-thirds of it. Um, okay, because I yeah, I just find it very useful to um, especially for new projects, I use this a lot to build a new design system. Um, because once I get the design system built, I can apply that to many many different components in my project uh very easily.
>> Yeah. Okay. May maybe you can show that later later, but why don't why don't we finish this work worker first?
>> Yeah.
>> Cool. So yeah, we basically we chose option C, right? Um, so now we can just say hey uh build option C now. Um, and uh, because we already have the plan uh written uh in the HTML artifacts, the agent already has the context on what that means and uh what's the choices uh were made. Uh, right. So, uh, the agent can just like go ahead and, uh, and implement that. Now,
>> how many like, uh, since you're just like building solo now at home, like how many of these agent building sessions do you have going? Like, like the agent actually building something for you at any given time, like?
>> Yeah. Yeah. Yeah. So, I I like closed as many sessions as I could before I uh started this session. Uh, but uh, I typically have like at least uh five different sessions actively running. Um, and in each session there are usually like a bunch of sub agents uh or different uh agents working. Um, so in total I never like really counted, but I I would guess on average there's like 20 to 30 agents running.
>> Okay, got it. Okay. So you mentioned you have sub agents running, like you actually specifically ask it to run sub agents, or like it just decides to like when when when do you actually need a sub agent versus just using one agent?
>> Yeah. Yeah. Great question. Uh, so I think the u most of the models today uh and the harnesses, they are not very great at proactively using sub agents. Uh, there are only a few cases where like cloud code or codeex will proactively use the sub agent. It's when like they have the their built-in agents like explore. Um, so when you uh ask a complex question, uh cloth code will often run a explore sub agent, right, to do some exploration in the codebase and come back with some investigation results. Um, those are the cases where the models will proactively use a sub agent. But in a lot of cases uh because the models I think they are not trained uh enough yet to use sub agents in various different kind of cases, you often have to prompt it to do so.
>> Got it. Okay. What are some cases where you actually want to prompt it to use sub agents, like to for like validation, or?
>> Yeah. So um, the reason uh I think the main reason I would use a sub agent is to avoid context um context window blowing up in the main agent's session.
>> Oh, I see.
>> Yeah. So uh what I uh what I do, what I uh I think the time when I choose to use sub agents is when I realize what I'm about to do uh is going to use a lot of context, and most of the context is going to be uh like investigation kind of exploration kind of uh scenario, and most of the exploration may be not meaningful for the main session. Um, so in those cases uh basically I like carve out those sub agents to do those investigations and only come back with their conclusion.
>> Okay. So it's like uh like hey, spin up a sub agent to look at this codebase or do some research on this topic and summarize it and give it back to the main agent, like that kind of stuff, right?
>> Yeah. Yeah. Or like there are cases where I have like 10 experiments ideas to run, and each experiment uh I each experiment can be done in isolation. Uh, right? So in those cases I also like just say uh, hey, like spin up 10 sub agents to do that. Um, if I do that all in the main agent, it's going to just like blow up the context window and take a lot of time and uh and tokens as well.
>> When you say experiment ideas, you mean like like AB testing stuff, or or or what, like like different ways to build things?
>> Yeah, so there are various kind of uh experiments I run. Uh, there's one example here I can show. Um, so this is one uh something I'm running. This is the one I didn't uh kill. Um, so this is a this is a benchmark I'm running to evaluate the effectiveness of different programming languages when given to agents.
>> And uh there was this benchmark that were that was published like two weeks ago called program bench. It's called program bench. Uh, it's built by the same people that built Swebench. Um, and it's their new thing. And program bench basically asks the agents to build uh a bunch of programs like ffmpeg, like these tools from scratch, and see whether the agent can actually get all the requirements done uh and pass all the test cases.
>> So that is the that was the benchmark. Um, but I thought the benchmark can be very useful for evaluating different harness uh harness techniques and also different uh programming languages. Uh, so right now what I'm evaluating here is I'm I'm running program bench on codeex and I I force codeex to use these programming languages like typescript, javascript, python, and see when they use different languages, do they get different results, right? Is there a programming language that will that will lead to the agent getting uh more requirements done and passing more tests and use less tokens, etc, etc. Um, so so this is a very large amount of uh experiments. Basically like there are um like 200 multiplied by uh eight, right? Uh, so there's that that's a lot of things to run, and in those cases I I basically like have sub agents uh running, and uh if I run all these in a single main agent, it's just going to keep running compaction and uh not going to be very efficient.
>> That makes sense. Okay, cool. Let's go back to the kit.
>> Yeah. So it looks like it's running a bunch of tests right now, right? So like is that just the model knows to run tests, or you actually you have some instructions to have it built unit test and stuff like that each time?
>> Yeah.
>> Uh, I typically um in my um agents MD in each project, I I will like have some instructions for how to uh perform tests. Uh, so uh here, for example, I can show the agents MD here. So in this is the agents MD for the high bit project uh we were looking at. Um, and in here we'll just have some like high level context on the structure of the project. Um, and then I'll have some uh testing instructions. This is actually super helpful. Um, so previously I didn't do this, and I let the agent decide what to do, and the agent will just do the like kind of do the minimum. Um, and uh, they they are trained to run some basic testing, uh, but they are not going to be comprehensive enough. Um, so I have here is like instructions for how to do end-to-end testing. This is important for like building uh front-end and UI kind of projects. Uh, right? We were looking at hybrids, which had a GUI. Um, so in this case I tell the agents, hey, uh, this is an electron app, you can drive this uh this app by running a browser and uh and blah blah blah, how to do this testing, how to actually test things end-to-end. So with that instruction here, the agent will uh will like just once it's done its work, it will actually validate things end-to-end for me. Um, so that can save me a lot of time from like running the app myself and visually validating is that actually what I want.
>> Okay. So it's basically like uh using browser use and checking out the app, see if it looks okay. Maybe checking some browser errors.
>> Yeah, exactly.
>> Yeah. And take screenshots as well. Take screenshots and look at these things visually and see whether it's actually aligned with what we talked about. I think if you use the codeex app, I I think it does it by default, but like let's say like I'm not very technical, like how do I even know to include this stuff? Should I just tell the agent to run a lot of tests, or?
>> Yeah. Yeah. Yeah. So typically um what I uh one thing I one thing that's really interesting I found is that
>> uh by default the agents like to write unit tests, like very uh purely code-based unit tests, and those unit tests often don't actually validate things end-to-end. So for example, uh even in codeex, I think codeex by default likes to use the built-in uh in-app browser, right?
>> Yeah.
>> Um, so when you work on some front-end changes, uh it will use the in-app browser to uh look at the change and uh have you look at that as well. Um, but this is an electron app. It's a desktop app. So it actually requires a different set of uh yeah facilities to validate that. Um, so the instructions here are basically how I would test this thing myself.
>> Um,
>> okay.
>> Yeah. So basically like the more um the more things uh that I find myself doing that I can delegate to an agent, I turn them into instructions and then let the agents do the work uh instead of me like operating the app myself manually.
>> Okay, got it. Okay. So so I guess like someone who maybe is not as knowledgeable as you can just like like I guess a general principle is like if you're doing something manually, like you're manually opening the app and looking at the screens, just ask the agent, hey, can you just automate this for me, right? Just just ask it and hopefully it can figure some something out too.
>> Yeah. Yeah. So yeah, if you are like not trying to dig into the technical details, uh then the principle the high level principle is like if you find yourself manually doing something, then try to turn that into something the agent does for you. Um, and you can very likely like with today's models, you can very likely just ask the agent um to like to do what you were trying to do. Um, and the agent will figure, oh, I should do this, I should do that.
>> All right. Well, it looks like it's done now.
>> Yeah, it's done now. So, uh, so now, good question, right? Like it's done. The agent says it's done, and we can look through what it did, right? It said it changed this, change that. How do we know this is actually uh a good change, right? How do we know there's no like bugs and everything? Um, so the validation phase is where um, like I see a lot of people spend a lot of their time. Um, so the default approach is like people will open up their IDE and start to review the code, like they will start to review the diff, right?
>> Yeah.
>> Um, but the the thing is that uh AI can write so much code. Um, so if you review every single line of code, you become the bottleneck.
>> Um, so what I do here is I I don't even review the code. Um,
>> I don't review this uh this first pass code from the agent. I use something I call no mistakes. Um, so no mistakes is another tool I built uh just to help uh make this part of the uh my life easier. Um, so what it does, I I'll show you. Um, I actually made a um alias uh so every time I got some change uh like some code changes done from the agent, I just nm and uh it will go through a few steps. First, it will uh ask the agent to create a branch for me. Um, so I don't even need to think about the branch name. Um, otherwise I need to think about the the branch name, the commit message, like I all those things just f it's just wasting time. Um, and I get the agent do that. The agent basically did that fix kit chat workspace, that's right, right?
>> Um, and the agent is now analyzing my session to understand my intent. Um, so the the agent here uh no mistakes is reading the session uh where we did the work to understand my intent. Um, so now it's understood what I I was trying to. It will do the all these steps for me. Um, so it will rebase my change on top of the latest main branch on the remote. So there's not going to be merge conflict later on. Uh, it's going to review my change. Um, so this is where uh I actually did a lot of um prompt engineering to get the agents to uh really scrutinize the change very, very hard. Um,
>> okay. So any kind of edge case or bugs uh like uh logical errors, things like that will get caught. Uh, so this is a very, very high recall um uh phase. I mo uh I when I initially built no mistakes, I did uh a lot of parallel testing where I let the agents review the change and I also review the change myself and see how often I catch something the agents uh don't, right? Um, and I use that phase to uh iterate on this uh the prompts and the uh the workflow within this phase. So eventually I got to a point where I find myself never catching anything the agents don't catch. Um, so in in this case, the agents act it actually didn't find any uh material problems. Um, so it's just passed. But if if it found some problems, uh it will uh categorize that into uh two categories.
>> One is obvious bugs. So if it's a just a obvious error, uh it will just autofix by itself. It won't even bother me. Another category is like when it realized there's an error, but fixing the error will have some product implications. Um, and then it will ask me instead of just autofixing that. Um, and in those cases it will escalate to me and it will basically pause at this phase and ask me to uh judge, do I actually want to make that fix or do I want something else?
>> This is like the PR review, basically, the agent doing PR review, right?
>> Yeah. Yeah. A PR review between the agent and the author.
>> And this no mistakes is like a whole new context window, right? It's like a new agent looking at your other conversation.
>> Yes. Uh, so this is a fresh context window, and uh, and actually did that deliberately. Um, I think that's an important thing to do, which is to use a fresh context window to review the change that was done. Uh, because uh, a lot of people what they do is like they will just ask, hey, can you review the change uh in the same session? Um, when you do that, the agent is very heavily biased by what was already done. Um, because it it saw all the context, uh, it saw every every step along the way, so it's biased into believing that what was done was correct. Um, and it will because of that, it will sometimes miss something. Um, so if you uh, I I I tested this a lot. Um, and when you use a fresh context window, you get just get a lot more edge cases caught.
>> I guess the only problem is like uh the no mistakes agent has to does it have to look at your whole codebase again to even understand what this app is about?
>> Uh, that's what this intent phase was doing. Uh, so it basically analyzed your session uh to understand what was your original intent and uh some of the surrounding context as well.
>> Um, but it's not copying the entire session uh into this new context window. It's like it's like you know like some senior engineer builds some feature, and then you're asking the principal engineer to come in with fresh fresh eyes to look look through everything, right?
>> Yeah, with fresh eyes, but you usually you will ask the senior engineer to explain a little bit of context to the principal, right?
>> That's right, yeah.
>> Yeah, so this intent phase is basically that. It's basically like explaining the basic context of what this change is trying to do.
>> Okay. And why don't we walk through the rest of the phases too, like um documenting, what is writing, what is observing?
>> Yeah. So, yeah. So, each phase, what it does is like review is just reviewing the code. Um, and test is running tests. Um, and the test phase is very different from what the agent does by default. Um, so the agent, what the agent does by default is running some tests. Um, and uh, validating locally, like uh, was the change um, was the change tested, and was that working? Um, but this test phase is a little bit different. It's more like CI. Um, it's validating, did this regress other things as well, etc, etc. And uh, this test phase will actually present some evidences, evidences of uh, the change actually working. It will paste screenshots or like sometimes a video to capture this thing is actually working. It's easier for me to review. I can just look at uh the artifact and see, oh, okay, it's actually working.
>> Oh, that that's actually really interesting. So yeah, because sometimes when I shift stuff with codeex, like the stuff I'm shipping works, but then it breaks some something else. It breaks like another core work workflow in the app.
>> So so this test base will actually look through all that and try.
>> Yeah, it look through all that, yeah. And uh, just like present very easily digestible artifacts for me to like have confidence it's actually working as I expected.
>> This is maybe a dumb question, but like for example, I'm I'm trying to build like a fitness app, right? And like and like there's like a few core workflows that I want to make sure that it tests each time, like creating a workout, tracking your workouts, you know, like so like do you do you have to manually define the stuff, or is the AI enough smart enough to figure it out?
>> To to test the stuff each time you make a change?
>> Yeah. Yeah. Yeah. I typically like try to get AI uh the agent to turn those things into an automated end-to-end test.
>> Okay.
>> Um, yeah. Because then it will be very easy to run that every single time, right?
>> And the automated end-to-end test is basically just like it uh lastly is like a browser app. So it it just kind of like actually beat the user and click click through stuff, right? And see if see if anything breaks.
>> Yes. Yes. Uh, so there are various kind of like end-to-end browser testing tools like playright. Um, so but but yeah, you can just ask the agent uh, you can say, hey, like write an end-to-end test uh for this scenario or this user work, this user flow, and make sure it's actually working end-to-end. It will typically be able to figure out what kind of frameworks or tools uh that needs to be used. I think the trade-off here, dude, is like it just takes a lot longer to actually ship a feature, right? Because [laughter] you're running all all these stages. But but I guess you have way more confidence that the feature you ship actually doesn't break anything. So so I guess if you're like if you have a lot of users, because a lot of stuff I work on doesn't have any new users, it's just me.
>> But if you have a lot of users that you ship the product, you want to make sure it actually works, right? It's like software engineering 101.
>> Yeah. So I I I would argue like even if it's only for yourself, uh, like probably you can make the trade-off, right? How much you want to u prefer just making changes very fast versus making sure things actually work. Um, because sometimes there's like a little bit of a cost to you as well.
If things broke. Um, yeah, so um, yeah, so this uh this phase taking uh longer time is actually okay because I never look at this like I I I never uh just stare at this screen and uh wait for every phase to pass, right? Every time I uh launch no mistakes, I just immediately switch to another session.
Uh, like I I don't even look at this. Um, what I uh have here um, I I'll show you now. I switch to another session, right? I can just look at the terminal screen here to see what phase uh is that no mistakes pipeline uh at. So I can see it's working on the linking pipeline and if it's like if it's uh waiting for me uh to like make a judgment or something, it will change the status here. So I can just like very easily see, do I need to jump back into that session?
Do you run no mistakes after like almost every change or or like if because if you do that, then why don't just automatically run it?
Ah, yeah, yeah. So uh, I run that on most changes but not not every single one because there are changes where uh, for example, I make a very simple documentation updates and I know like it doesn't need like so much validation. Uh, it's going to use a lot of my tokens as well. So I make some judgments on whether the change just justifies this kind of a heavy validation phase. Yeah. It's kind of like, yeah. When you work within a team and some of your changes don't like it's not that every PR will go to a QA team, right?
Yeah, only some like milestones, some meaningful things will go there.
Dude, do you think it feels weird like after spending, you know, your career in big tech? Because in big tech, when you push a change, you have like a teammate come and review your PR, right? And then you run some t tests and and now you're just by by yourself. So it's like [laughter] so so I guess like you have all these agents, but like like how do you feel like do you feel like unshackled or or do you feel like uh, you kind of miss the teammates?
I uh, so it's a bit of both. Uh, but I would say like uh, largely speaking, I feel liberated.
Liberated.
Yes. Uh, so I I think uh, teammates are great uh, especially in the brainstorming phase. Um, so when we are like thinking about an idea, if it's just me uh, it's a very like it's not a very diverse perspective, right? So I may not think through everything and I may not realize problems others can see. Um, AI can help to a degree, but I don't think AI is like quite there yet to replace uh, like a really smart team that uh, can ideate together. Um, so that is like one the part I miss. The part I don't quite miss is like everyone is busy and if I write like 20 PRs every day, no one's going to reveal that. Um, so yeah, that already happened before I uh, left my uh, last company. And what I uh, found myself doing was like I I have to write less PRs. Um,
Got it.
And spend my time elsewhere because the bottleneck is like really on on the rest of the team.
Yeah. Because the your teammates aren't actually reviewing the PRs. Like they don't have to have a lot of things going on, but like if you submit a PR to AI, it's always going to start working,
right?
Yeah. So this is something that I think uh, is going to like fundamentally change uh, as we progress on AI adoption. Um, so our workflows and how our teams work were built uh, at a time when we spent most of our time coding and uh, the average stats of like a average software team an engineer is like an engineer will write 10 to 15 PRs every month. That's like the velocity of an average software engineer uh, team. So um, when that's the case uh, you spend like it's okay for everyone else to do code reviews and all these uh, processes because the velocity is not that that massive.
But when you um, when you start to write like 10 times more PRs, we are not ready for that. Like our processes and our um, like human team composition and everything is not built with that assumption in mind. So what's going to happen is like uh, things are starting to break. A lot of teams are starting to um, change their practices in order to fight that. Um, so some teams es especially smaller teams in startups, they basically stopped doing PR reviews. Um, they still raise a PR but mostly for like a formality or for like leaving a record, they don't actually wait for another peer to review. Um, they sometimes just merge the PR and later on if there's a problem, they can go back to it. Uh, that's the kind of changes I'm starting to see.
Yeah, they they get the agents to review, right? I do think it does lead to like a little bit more unstable uh, products. Yeah. Um, but you know >> uh, that's because they are not using no mistakes. [laughter]
Yeah, it looks like it's done.
Yeah, this pipeline just completed. Uh, right. So uh, it went through all these steps. uh, and uh, there was actually one thing fixed in documentation phase. This is uh, yeah, this is something I uh, we can look at whether uh, it's actually a legit change, but um, this is something I find super useful and something both me and my agents often don't do automatically. Um, so it's like when you make a change, can you actually find all the places uh, in our documentation that can be affected by that change?
Okay, got it.
Yeah. So documentation linting uh, and push and create a PR. So I can just open up the the PR and uh, let's look at what it does. So it created this PR. Uh, the PR summarized uh, the intent uh, that was understood from my original session in open code. It summarized what changed. Uh, it did a risk assessment as well. Um, so like what this is very useful as well. uh, when I look at a low risk change, I spend less time. When the agent is flagging this is a medium risk or high risk change, I spend more time on this PR, right? So I can like decide where I spend my time more uh, intelligently um, so uh, and testing uh, yeah, it did some test. It had had a evidence uh, so let's see what this is uh, it renders the workspace. Okay, yeah, basically like the um, this evidence uh, is uh, is about presenting like actual results um, from the change. Uh, so uh, we can look at this and see, is that what we want?
Okay.
Um, and there is the pipeline and there is the documentation phase. Uh, what did it find? Uh, it found that the design system example copy was not updated. Okay. Yeah. So it actually it actually caught a inconsistency. Uh, so that's great. Yeah. So, so because it's a low risk change, I don't even go into the diff here.
I don't go there. Um, I I just merge it.
Um, okay.
And, uh, when it's a medium risk or high risk change, um, then I go into the diff and start to look at things myself.
Okay. But you you pretty much uh, always uh, somewhat look at the PR, skim the PR, and then you hit the button to merge, right?
Yeah. Yeah. I still look at this PR uh, because I think looking through the risk assessment and uh, what the agent actually did uh, what was the fix uh, those things are actually still useful.
That's the mistake that I'm making, dude. I I don't look at the PR some sometimes. I just I just tell it to merge. [laughter] So I just need to be a little bit more thorough.
Yeah. Also after the agent made some code changes, you just uh, just get it merge.
Well, I actually run some tests and stuff. I I don't use no mistakes and then I I I I get to merge and then um, yeah, inherently like you know like a day later I'll find something else broke. So [laughter] yeah, it's it's probably not the most efficient way to do it.
Yeah. Uh, yeah. So yeah, I think some some validation and then some uh, some like uh, review, but it's not a line by line code review. I think some review on what's changed and um, what um, kind of risks exist. That's still useful. And and you're probably submitting 10 15 PRs a day, right? Or like doing doing this.
Yeah. So I um, I I uh, actually do a lot. Uh, so I um, like [laughter] 26 uh, 14 27 30. Yeah, that's like the average. Uh, so it's uh, yeah, most of the time it's like 22 40 kind of PRs every day. Sometimes I do more. Um, like
I can I can tell I can tell when when uh, you became unemployed. It's like it's around March. very very very clear on this chart.
All right. So that so I guess we just walked through the whole plan build and validation process, right? Like that's basically it, right?
Yeah. So uh, yeah, we went through like uh, building a plan interactively uh, implementing that with the agents and then going through this validation pipeline. Um, this basically uh, if you think about it, I didn't spend much time in the uh, coding and validation phase at all. Right? Most of my time was actually on the HTML artifact iterating with the agent. Um, so that's kind of how I um, how I do these things now and as soon as I send the agent to do implementation, I just switch to something else uh, and work on that in parallel.
Okay, so I guess we we can provide the links to lavish the HTML PL planner and also no mistakes the validation uh, we'll provide it in the description of this episode. Uh, I guess let dude, let me ask you one last question.
Yeah.
I mean, you you know, you're like an LA engineer, you've been doing this for a while. There's there's like a lot more builders now, right? Like there's a lot more people trying to get into this stuff and learning how to build a AI.
Yeah.
How do you think do you have any advice for people to actually ramp up the technical skills and and also like what kind of technical skills do they actually need to learn? Like obviously like testing and validating everything. Uh, but also there's like there's like stuff like for example like if you don't set up your database properly in the beginning, like it's harder to change it later. Just like just stuff that you learn over time. So, so like do you have any thoughts on how people can actually scale up as they build more stuff?
Yeah. Yeah, good question. Um, I I think there's a few things come to mind. One is that uh, I think just play a lot. Um, build a lot of things. Uh, even if it's a throwaway toy, uh, build it and through that process, you will like often often discover things you can do better or things the agents didn't quite do very well and start to reflect on that. Uh, so do think do a lot of things. I think that's like probably the first uh, step. Um, some people I think they uh, what I see at least from some people is like they only they they spend a lot of time trying to decide what do they do and then uh, they only do one thing and that thing didn't work. They then they stop. Um, I think uh, the mindset I would encourage is to just like build every single idea you have. Um, whenever you have some idea um, like send the prompt to the agent and see what it does. Um, and um, whenever like you you have some uh, like inspiration or idea you you think might be interesting um, just give that to the agent and have it run for you. Um, I think through that like process uh, a lot of learnings can be uh, derived. Um, that's one. Uh, another I think is to um, like try to challenge yourself to use like more tokens and run more agents in parallel. Um, like I think that is a forcing function for people to like upgrade their workflow. Um, because when we by default work with one agent uh, at a time, we are still kind of like being a bottleneck. Um, we are putting ourselves into the loop too much. Um, and I think to really scale up um, how much we can get from the agents, we have to like move ourselves out of the loop as much as possible. Um, so that's like I think using more tokens and running more agents in parallel kind of forces us to do that. Um, that's probably like another uh, thing I can think of. Um,
Got it.
Yeah. Maybe like the last thing is to uh, like try to adopt AI in every part of your workflow, not only writing code. Um, so what we could see there like AI did a lot of validation and uh, documentation all those things for me, right? And raising the PR and everything. I don't need to do anything there. Um, I think um, like when uh, when we work through a project, whenever we find something manual, what we talked about earlier, like something we are spending time ourselves, just try to think about uh, can we delegate that to the agent as well uh, and through that uh, people I think we'll find a lot more useful like workflows that can uh, handle automation and reduce our workload.
Yeah, maybe there's like some sort of a skill or like some something we can build where because the AI remembers uh, it conversations with you. Like maybe the AI can actually proactively suggest like hey, you should auto you should automate this. It's like the second time we're talking about this.
Yeah, that exists that exists. So I can show you um, okay.
So uh, if I run cloud code >> cloud code has this slash command called insights. These insights will basically analyze your cloud code sessions and generate a report for what uh, what can be done better like what can what what kind of skills can you uh, add what kind of uh, things can you tweak in your like memory files etc etc to make cloud code work more efficiently for you.
All right.
Oh.
Yeah. Yeah. So this is super cool, but it's going to use a lot of tokens. I'm already out of tokens, so I'm not going to [laughter] demo that now.
Yeah.
Yeah. But this is something I definitely recommend people trying. This is a very cool thing.
Okay. Yeah. I'm going to write right now. Um, yeah, I I I think the token maxing thing is kind of like a meme. But I think basically like just summarize your advice. Number one is like putting the reps like try different things, try to build different things. Number two is like if you use multiple agents, you can put in more reps, right? Because you don't have to wait for one a agent to do anything.
Yeah.
And then and and then the third one is is u sorry, what was the third one again?
Part of your Yeah. Not only writing code. Yeah, I think the second one is especially hard, dude, because like I don't know like growing up as an Asian person, I have like a scarcity mindset. I try to save money and stuff and [laughter]
And like just trying to burn our tokens, it doesn't feel it doesn't feel right.
Uh, but there's a so most of us uh, like working as individuals, we have uh, the subscription, right?
So at least try to make the most out of the subscription and exhaust the quota.
Okay. Yeah. So, I guess it's kind of like going to a buffet and like trying to eat all the crab legs. [laughter] I guess I can.
Yeah. But I I I would say like uh, there's the token maxing thing. Uh, I I think uh, we shouldn't just use tokens for the sake of using tokens, right? We want to get actual work done. Uh, so I I think uh, it's more about pushing ourselves like my my point about number two was more about pushing ourselves to figure out ways to scale up um, and really like get more done with agents instead of uh, finding ourselves into the loop and only do one thing at a time.
That makes a lot of sense. That makes a lot of sense. All right, cool. Well, thanks, thanks so much, man. Uh, where can people find your like all the free stuff you've been shipping and also yourself?
Yeah. Yeah. So I'm very active on uh, X and YouTube. I'm I plan to share a lot of my workflows and tools and setups over there. Um, and I also uh, my GitHub uh, is also a good place uh, to uh, look at my projects.
Your your GitHub is just uh, slashkun, right?
Kungchan GUID. So I I have this uh, let me uh, let me move my window here.
Oh, there is. Yeah.
Uh, yeah, this is my uh, handle almost everywhere. Uh, so uh, YouTube, X and GitHub, LinkedIn, it's all this handle UID.
Yeah, I think it's like a blessing to all of us that you're shipping all the stuff for free and like we can all try it. So uh, yeah, I'm definitely going to try no mistakes and um, you know, every everything else that you built.
Cool. Cool. Thanks, Peter. Yeah, if you run into anything, let me know. I I I'm constantly trying to improve these tools as well.
Cool. All right, take care, man. Bye. Fitter.