Transcription
I wrote this paper called 12 factor agents, which is like don't use prompts for control flow. If you know the workflow, make the parts that are deterministic deterministic and have like smaller LLM steps baked in. And that's how you got more reliability, more performance. That's what all these legal tech and fintech and insure tech and compliance tech like AI companies were doing is they had these workflows with just AI sprinkled in at the right places.
Today I sat down with Dex Hory, the founder of Human Layer, and the guy who really put the phrase context engineering on the map. He's working now on what he calls a post-IDE IDE to solve the hardest problems in the most complex code bases using AI coding agents. He is significantly ahead of the curve, and I wanted him to show us how he builds software day-to-day. So, we watched him code, and he shared things that I believe you can apply to your workflow immediately, such as why you should not use prompts for control flow, the idea of tracer bullets, what to do if you hit a bug that the model can't solve, and how he uses tickets to drive his workflows, including how you have to strip out certain kinds of information to keep it from tainting the implementation intent. He also talked a lot about plans and specs and how to get the most leverage out of those. And he shared his vision for AI tinkerers in San Francisco, where he is the organizer. I think you're going to love this video. Hit like and subscribe, and let's get to it.
All right, welcome Dex to the show. It is an honor to have you here. I've known you for a very long time.
Sorry about that, by the way. No, I mean, just to give myself some credit, I feel like I do have, you've, you've greatly enhanced my reputation for spotting talent early because I've met you before you were famous.
Uh, yeah, that's true. Yeah. When we were, uh, I, we, I was like in the middle of like pivoting the startup to it. We were doing a non-AI thing, and then I was like, my co-founder left, and I was like, we're going to do an AI thing. And so I just like, every opportunity I had, I was doing AI stuff. And the first event I went to was like an AI tinkerers event in New York.
I went to like, there was like an agents takeover with the crew AI folks. And then, uh, I went to another one later, and I was like, oh, this is cool. There's like so much more energy here. Uh, and I was like, talked about some of the stuff I was hacking on, and everyone was just like, "Oh my god, that's amazing. Do you have that? Like, can I use that?" I'm like, "Well, it's just an idea." And it's kind of like we built it. It's like we were going to make a general purpose. And that was when I was like, "Okay, cool." Like, you know, we're trying to get information. We like, you know, this book, The Mom Test.
Sounds familiar.
It's like a short book about product discovery. It's like how to get startup ideas. And it's basically like these rules of like, if you tell someone your idea, uh, and they like you, they're going to tell you it's a great idea. And that's useless feedback. Uh, and what you need to do is just get really good at like asking people about their problems and like separate anything that you want to do, any solution, like just learn about people. And so the advice is like, okay, if you're looking for ideas or you think you want to solve something in a space, like just go to all the meetups and talk to people about what they're doing and try to get them to like tell you about their problem. So I was like, okay, cool. We'll fly around. There's like a Seattle AI tinkers I went to, and then that gave me like a lot more energy. Uh, yeah, and then I like, basically the first like version of Human Layer was demoed at like the first public demo was at that AI tinkers in San Francisco, like August 2024 or something.
So cool. I remember seeing you in Seattle where we started AI tinkerers, the first one at that point when you came. I think we were in probably like 30 cities or something. I don't know. But at that point, I think you were the first person that I was like, I haven't seen you before, you know? And it was like, I'm from Chicago. I flew in or something. And I was like, what? You flew for our meetup? This is incredible. I think the people, the quality is incredible. So, I could see why you did that now. But, so then I was, um, I was like, "Oh, we're debuting in Paris. I gotta go." Lo and behold, I'm going through the roster of like curated speakers with their demos, and it's Dex. I'm like, "That name is familiar."
So, I was like, "Hey, we're in Paris together. Oh, this is cool. Just you again." No, really.
I was at a wedding in like the south of France, and like the day I was supposed to fly home was the like kickoff for AI Tinkerers Paris. And I think you would like, you did like Paris, and then you went to Prague, and you kicked off Prague, and then you went to Dublin the next. And I saw you in Dublin. I'm like, okay, what is going on here? Who is this guy? But in Dublin, you were talking about different things at each one. I, you know, it was a little bit different. And I remember in Dublin, I was like, okay, this is the first person that I've met who can describe what an agent is or what an agent loop is like in a coherent way. This was, I don't know how long ago it was, but I was very impressed with, uh, that, you know, and I was like, okay, this is cool. We got to chat, you know, we're. And then you were like, I'm moving to the Bay Area. And I was like, oh, you got to get involved with AI Tinkerers fast forward, you're the leader of AI Tinkerers in San Francisco. So, very cool. Thank you very much.
Uh, and you're famous. It's very cool. That's, listen, all I did with like, I think the first like blow up was the 12 factor agents thing. And all all that happened there was like, I talked to, I built a tool for, it was like a small, sharp, like the Unixy, like small sharp tool. Plug this into your agent like soup of tools you're using for agents, and it lets you do human approvals and it has all this nice tech in it. Uh, and I talked to lots of founders. I built it for the AI agents ecosystem, you know, the like the LangChain, the Crew, the GripTape. There was a hundred that summer, right? Uh, and then I talked to a bunch of like founders and founding engineers who were like shipping real AI to actual like enterprise customers, and none of them were using any of that stuff. They were just like, hey, we, we built all the little like, all the loops ourselves. You structured output, like it's the whole BAML crew is like very like similar ideas there. And so I kind of felt a little bit like, oh, I thought the world worked one way. And now I realize it worked a different way.
Yeah. And, uh, and so I wrote about it. I was like, I don't want anyone else to get kind of like down this rabbit hole. Um, the thing with 12 factor agents though was like, it blew up on Hacker News, and then we did like a talk, AI Engineer about it. But the, the thing with a 12 factor agents was the most feedback I got was from people saying like, oh my god, you're saying all the things I've been thinking.
Yeah.
Um, and I think a, some people learned that like when you make content, you, you don't just want to make people feel seen. You want to like give them something new and useful and like, yeah, a framework to talk about it and a framework to think about it is valuable. But my biggest like takeaway from that was like, okay, this is cool, but everybody like that I look up to and I respect was already doing this. So how do we go even deeper and even harder into AI and agents and all this stuff and like I wanted to discover things that most people like didn't quite understand yet.
Yeah, I think you have a knack for that. That that thing that when you're bashing your head against the wall building stuff or trying to get stuff to work, you're running into problems and your subconscious is feeling things.
Yeah.
You, at that moment, you have a good, you have a good knack for like actually clarifying or or showing, you know, distilling it down to language. And you had a term last night on the phone that you said, I term is going to be everywhere six months from now.
Which one?
Remember exactly?
Yeah. Of the apparatus engine.
Yes. Yeah. That's a meme. So, yeah. So I was talking about like, okay, we have context engineering, and you use context engineering when you build an agent. But then like, and then there's like harness engineering, which actually I think is probably like, how do you build a harness? I mean, I think a harness and an agent are the same thing. People just wanted a new word for it because harness can also like describe a thing that you do reinforcement learning on. But, uh, I was posting about this. I was like, okay, cool. So what is the name for the type of engineering you do where you're not like crafting the individual context windows like in code, like the saying the every single token that goes in and out. And you're not like building the agent harness, like you're not, you're not building cloud code, you're using cloud code, right? But you're, these a lot of these agent harnesses and agent frameworks, they're, they're, I don't know. I worked with this guy Blake a long time ago, and we were doing like high performance Java for social media data, and he had this thing he would always say, which is like, Java is a battleship. Like there are a thousand knobs and dials on this thing that you need to understand if you're going to like steer it properly. And then cloud code is getting there. There's like hundreds of secret variables and all these flags, and it's like a very complex piece of software. And there's another type of engineering on top of it that I thought might be called harness engineering. I don't like this word because I think it sounds like, are you making a harness? But for me, it was like, how do you use the harness? How do you engineer your MCPs and your skills and your slash commands to like, get the, get the piece of technology that you're using to get the best results?
It reminds me of that that viral Twitter post from Boris from Cloud Code where he was like, you know, I just, we don't really do all this like customization. We just do this very, it's very simple. And then he goes through like 14 very complicated, very, very good things, right?
Yeah. Where the community is right now clearly is, hey, I sit here and if I do a lot of work on one turn of the agent, it can do great work. Yeah. But when I sleep at night, I have FOMO that it's not running all night doing this over and over again. We feel so close to being able to orchestrate that, and that orchestration is harnesses. So, we're all looking for harnesses or we're building harnesses. Is this a good take on where we're at?
Ah, so yeah, you can say that the code you like, the the Ralph Wookham thing, like the the bash loop around coding agent, you could call that the harness. But then the agent itself is also kind of a harness. So, I posted this, someone posted a picture in response. I wish I could go find this tweet. I I'll find it and send it to you and we can put it in the screen share. But, uh, it's, uh, he basically had this drawing of like a horse, and like the horse is the agent, and the harness sits on top of it, and the thing the human holds is the reins. And so maybe the, this is a terrible word, but maybe the like engineering how you use cloud code is reins engineering, right? Um, but the word I've been using is apparatus. Like you, you built this complicated thing that uses all these little like more lower level structured code loops on top of cloud code to, uh, kind of orchestrate it to get, again, the answer is always like outcomes, right? Everyone. This is my biggest beef with, and people, I'm not the first person to say this. People go on Twitter and they talk, "Oh, I'm running all these agents and I spent all these tokens." And I'm like, you hyperengineering, whatever it is. Um, Alex Ryman, uh, who I love, is the founder of a company called Agent OB. Do you know Alex?
Of course. Of course. Yeah. I don't know him. I need to meet him.
Yeah, he's cool. He's around. You should hang out.
He's awesome.
If he's not doing a hyperengineering hacker house in Vietnam or something.
Right on. Right on.
Uh, but yeah, he has this, uh, he did this post a while ago called No XP Waste. Oh.
Did you ever play any of those like online like MMO RPG style games? Okay, so like a World of Warcraft or Diablo. Basically like, every minute you're playing the game, you're leveling up, you're getting experience, you're getting items, whatever it is. And if you're sleeping and that's not happening, then you're like wasting XP. XP that you could have been gaining.
FOMO right now.
Yeah. Is like, how, how do I get the agents to work while I'm while I'm sleeping? It feels very primitive right now that if I can do a good job of specifying something that I have to do that work right now, that should be an agent loop that does it for me, you know? And then when I kick off a coding session, it might get stuck three minutes in for a task that might otherwise run 30 minutes. That seems like a problem. But the, do you see what I'm saying?
What do you mean by get stuck?
Like if it says like, uh, it decided to go off to the actual web and log into somewhere, it's waiting for a two-factor off stuff like that. It's stuck. It needs me. And it's like, okay, uh, if I had specified, you know, don't do that work, your way around it first principles, then it wouldn't have got stuck. And I'm better off because I'd rather it do the work and I can do stuff in parallel. But the bigger thing I wanted to decouple is like, why can't I take my whole like backlog of features and run through the product, you know, requirements loop and then kick off all the agent loops and build three variants of each, spend a trillion tokens, Yep. And just have me look at deployed, tested product that graduated to the end.
Okay. So, getting at, let's, let's like, let's like go a level down. So, you have, let's say you have 40 features in your backlog.
Yeah.
And that you trust the agent to go from like the ticket to the PR, uh, without needing your help. It can write the code. It can run the test. Maybe it has some browser stuff to look at how it looks and things like this. At the end of the day, and if you have to do three implementations for each one that you want to review, you now have 120 PRs to look at.
And I could write an agent to look at some of those and bake it down from 120 to 20 or 10.
Yeah. I mean, yeah. So, I mean, I guess the idea is like, if if an agent's going to do it, then it's just it comes in the pre-loop. Like I don't, I don't really think like we're all using the same models. If the agent can write the code, the agent can review the code. I don't see a big difference between like, okay, Claude writes the code in your CI system, or we do a bunch of stuff out of like headless and GitHub actions, and then add another step to like do a different prompt to do the review before you make the PR. Like that feels no different to me than just having a bot, other than like a little bit of fancy like GitHub actions glue and maybe some like really honed prompts that like, you know, are the right verbosity for. But like, you can build that yourself. And like, so I'm always, I'm always surprised by how popular AI code review assistants have become when it's like,
Okay, if you know enough to like automate everything up until the PR, it's not that much more of a stretch to add one workflow step to have Claude review it and like fix it.
Right. Right. So why do you need the stuff on the other side of that fence, basically?
Or like, why is it any better? It's like, it's the same. It's all, it's all just like, take a frontier model, put it in a loop of some tools, let it go read a bunch of stuff, uh, and then like, present its findings, and you can hand that to an agent to go do the fixes if you need to.
All right. So we're going to get into stuff you're working on now, which is in that arena of like greater autonomy, right? And greater parallelized, uh, agent, you know, coding, right?
A little bit. Yeah. I mean, I think I think I wouldn't, I kind of stay away from the parallelism thing because again, with this hyperengineering thing, there's always people bragging about how much they wrote and how many lines of code and how many tokens, how much they like juiced their max plan for. And I see a lot less people talking about what they shipped and who's using it and how valuable it is. And that, I think, is something that probably won't change because the algorithms reward hype, and so that's what you'll keep seeing.
Yeah, I get it.
But every time I see one of those, I'm just like, cool. What did you ship?
I get it. Yeah.
So, I think our focus is more less on how do you paralyze better? How do you like grind out more code? But it's more like, I, I think the software, how do you get it to production quality? How do you get it to, how do you ship? I think Rouch said this, right? He's like, coding is writing the code. Shipping is like writing the code, testing it, deploying it, monitoring it, fixing it, improving it over time, keeping it like maintainable, whether it's maintainable by humans or agents. It has to be non-thrashy. So it's like, I think the software engineer's role maybe changes a little bit from writing code to producing code. But there's still a lot. You can think of like, you know, writers of a movie versus producers of a movie, right? And it's like, okay, cool, now the writing is auto, I mean, obviously like LLMs don't write good movies, that's not what I'm trying to say. It's like, there's that idea of the producer who's like overseeing and making sure everything is good and getting like going all the way down into the weeds to fix some really small, like if there's a movie and the main actor is not getting his coffee, the producer will find a way to get that actor the coffee because we need to shoot today because we need to make it happen. And like, there's still a lot of engineering in there. So I don't, people call the death of software engineering, like I think that's, I think that's a joke. Like there is still engineering to be done.
And, uh, as far as I can tell, I mean, maybe GPT6 will have other thoughts on this, but like, models can't ship. Models can code.
H. What are the things that fall into shipping then for you?
Um, so it's like, take, take your world, right? Where you have 120 PRs. Let's say it just did one feature, right? You get three PRs, three different versions.
Okay.
You got to go like, look at the code. I mean, we've had things where we kind of like decided we didn't care about the code for like a month or two, and then we ended up like rewriting those whole parts of the codebase because we realized that like, there are patterns that like, the more the model sees them, the more they leak in, the quality becomes hard. And then, um, especially like one thing we found, a, there's lots of examples of this, but like, Claude likes to use the use effect hook in React. Yeah. Which is very hard to reason about for Claude models. Like when you have a bug related to like use effect ordering or like, like this like distributed like network of hooks that affect all these different parts of your codebase, it becomes very hard for Claude to debug. And then you have this like, every every couple months, if you, if you kind of lean into this like, okay, we don't pay as much attention to the code, you will hit a moment where you will have a bug that the model cannot solve, and you have to go onboard into a codebase, which like, if you were writing all the code by hand, you just have a really strong like fresh mental model of understanding how, like, that tradeoff. I still take most of the time of like, okay, we shipped like three times faster for three months, and then at one point I had to take a week and go like, like really clean stuff up.
I, I have, we all have bugs like that. One of the situations I run into all the time is the order stuff. Like I'm working on a page that's existed for months, and I'm adding a new feature or whatever. And to the agent, it looks like perfect. To the unit test, it looks perfect. To the user on a certain mobile device, it looks like I can't even click that thing because it's behind another DOM layer or some edge case. And I could totally see how the agent, the coding, uh, environment, whether it's Codec or Claude or anything, can't determine that bug on its own. Yeah. But an end user can. Now, an agent that I spin up to pretend to be a user also could, right? So is the realm of like, deploying code, I want to automate all that. If, if I'm producing 10 times as much features, Yeah. And then I'm going to have 10 times as many of this kind of bug. I don't want to be the one on the hook to me have to find and handcraft the, Yeah. The the. So what is the solve in this area for that?
Well, I mean, I think the answer is like, the, the better you are at wielding the agent, like the more you can get it to solve these hard sticky problems. Of course, right? Prompting, context engineering, like multi-step workflows, all this kind of stuff.
Are you saying I need to go take this all the way down to the first step and be like, while you're building this feature, avoid that and test that?
I mean, obviously like when you see anti-patterns, like I Jake has a really good talk from AI Engineer. Uh, he's like the, the best cloud code engineer at Netflix, as far as I can tell. He's not there anymore, but that's what he was. Y. Uh, and he talked about like, you know, a bad pattern because you've debugged it at 3 in the morning. Yeah. And so it's like, you don't really know bad patterns until they burn you. And then yeah, you go put in your golden rules of like, you know, here's all the, and I don't know if it goes in cloud MD or somewhere else, like we're still figuring out the right like balance of like where that stuff goes and when you like pull it into the context window because when you're researching the codebase, like, not really relevant for the model to know the rules for writing code. In fact, if you tell it how to write code and you tell it to go research, it's going to get steered into like, okay, I want to make changes. Yeah. Maybe not like it's going to make the changes, but your research documentation is gonna have leaked into it like opinions about what to build, which is like a thing we found makes for bad research.
Yeah. Um, Steve talked about, I forget what he called it, but it was that that, uh, seven or 12 like phases of agentic coding, you know, skill or or experience.
Okay. Yeah. The skill, the like level for the skill.
Yeah. One of the levels just like autocomplete, right? And like, Yeah. So we're, you're obviously on the far end of that, uh, and you're building stuff in that world, I think, in in some dimensions, right? Like something like that.
Can you explain what we're talking about, just so people listening can understand? Because then I, I want to set that context so that my next question is, can you show us your current dev setup? Like, how do you build code? I'd love for you to share that.
Yeah, we can do that. So like in August, we talked a lot about this flow we called research, plan, implement, which was basically like, um, you would have these three steps and you would run them in separate sessions. And there's a spectrum here, right? If you're doing a really small feature, like just tell, if you're like, change the color of this button, and you know what file it's in, you're like, go change the color of the button in this file. No research needed. You already know where it is. Um, but as the feature ramps in complexity, we found planning to be a valuable thing. I mean, cloud code's been investing a lot in their plan mode as well. Um, we were doing this six, seven months ago. Um, or we talking about it seven months ago. We've been doing it for nine monthsish. Um, and so that's like, cool, work back and forth, go over design decisions, ask me some questions, like, let's look at one where we're going. And then we have this other phase I call structure outline, which is like, how do we get there? Um, and so there's a lot of skill in like, okay, designing which patterns you want to follow when you're coding something. And then also like, to make your life easy when you go to implement, you want to be a little bit, um, intentional about the order in which you do things. Models really like to make what I like call like horizontal plans, which are plans that are like, cool, we're going to do the database models, and then we're going to do the service layer, and then we're going to do the API, and then we're going to do the front end. And like, there's nothing for you to check until the end. You can't actually see if it's working or not. And so, um, pragmatic programmers, actually Matt PCO just like turned me on to this. He had this, they have this idea of tracer bullets.
Yeah.
Which is when you're building something, basically like wire it all the way end to end with no logic, very simple logic, but it's like, okay, make the endpoint, like wire up mock data on the back end. Make sure you can render something on the front end, like just a placeholder. And then once that's wired through, then you go through like, I call them vertical plans, where you go through all the layers of the stack to wire it. Then you do another vertical pass to add this logic, and then you do another vertical pass to add this. And that means that you don't end up with a thousand lines of code and it's not working, and you don't know where the heck to start.
Right. Steve Vay talked about tracer bullets too on my podcast episode, actually.
Oh, sick.
Yeah. Same exact thing. That's great. So it's coming back to like, what's the solve for how do I avoid getting in these big messy bugs? Is like, we're building tools that enable like people to do staff engineering, senior engineering work, where you're designing it, you care about the architecture, you care about the code, you care about keeping it maintainable. And, uh, right now, I think a lot of people in that category are still in the camp of like, it's, it's growing fast. The people who accept that like AI is going to do this for them or allow them to work in this way, but a lot of them are like, okay, for these parts of the codebase, I have to do it by hand or I have to do it with low autonomy like tactics.
Who is still there? Really?
Lots. Many, many, many, many. Uh, there's holdouts everywhere.
I'm, Yeah. I'm only hanging with the people who are trying to do, you know.
Yeah. So what we do is we go into like a 3,000 employee like fintech company, and we try to help them roll out these like, that's probably it. Like regulated industry, like we have standards. We've been burned before. People are going to sue us. Whatever. You cannot move fast and break things because millions of dollars will get lost if you ship a bug. That you'll get fired if a human didn't review the PR before it ships. It's like a process step. Yeah. And that process step today is becomes a bottleneck if, if your productivity on the other side is so high.
Yeah. Right.
So, and so yeah, so like what we're building is like, how do you maintain high code quality, but also move like, you still want to move two to 3x faster. And that's all about how do you focus human leverage on the important parts of the workflow that are not like, a wall of green text in GitHub ain't going to cut it.
Ain't. Yeah. Yeah. And so the workflow that we've built, I mean, so we did RPI for a while.
Interesting.
We rolled it out to a lot of people. Uh, we found a bunch of issues with it. Um, I can talk a little bit more in detail what they were. We've kind of refined that workflow even more. And so the workflow that we do today is like, take a ticket or a task or something.
Yeah.
Turn it into questions.
Cool.
Literally just, what questions would you ask to learn about the codebase?
That's cool.
So ticket becomes questions.
That's better than me trying to just tell the agent like the answer.
Look around.
Yeah. Like tell the agent, go figure it out yourself, ask the questions, then go answer them.
And so the problem we had with research is people would say, here's my ticket, go do research, and the intent would leak into the research doc. When when these models are designed to like solve our problems, to build things. And so you'd get a research doc, here's how the codebase works, and the second half of it would be like, here's an implementation plan, right? Which is like, you don't want that because then when you go to the further steps, like the decisions are already made without a human in the loop. And like, model spewing out, here's what I'm going to do, is the same problem you would have if you were just like, go fix this, and it goes and does it.
And so we, we take the, we take the ticket and make the questions. And then when we do the research, we actually do not include the ticket in that context window, just the questions, so the model stays super objective. Yeah. And then we go to what I call the design discussion, where you take the research, which is like, here's all the codebase that's relevant to this ticket, model doesn't know what ticket's about. But then you give it the ticket and the research, and it says, cool, design discussion. How are we going to do this? Where are we going? How do you want the end state to look?
Yeah.
Back and forth there in that part of the workflow. And then we go to structure outline, which is like, okay, what order? How are we going to get there? How am I going to organize this so I can? And then we write the big ass thousand-line plan that is all the changes to make. And then that goes to work tree implementation. Yeah. Pull request. But with just the planning prompt, it had all those like design structure and then write the plan like baked into the prompt. That prompt has 50 to 100 instructions in it, and the design and the structure would get skipped.
All the time.
Yeah.
And like what we found is we were coaching people like, okay, when you use the create plan prompt, you have to, you know, repeat the instructions at the beginning, and then you repeat the important ones at the end. And so you'd have to say, you'd have to say, create plan, here's the ticket, here's the research. Work back and forth with me. Start with your design questions before you go write the plan. And that's how you get good results from planning. And that just felt like a shitty experience from like a product perspective to tell people they have to sprinkle in magic words.
Yeah. Yeah. Right.
And so the biggest lesson that I think we woke up in November, December, like realizing is like, oh, okay, I wrote this paper called 12 factor agents, which is like, don't use prompts for control flow. If you know the workflow, make the parts that are deterministic deterministic and have like smaller LLM steps baked in.
And that's how you got more reliability, more performance. That's what all of these legal tech and fintech and insure tech and compliance tech like AI companies were doing is they had these workflows with just AI sprinkled in at the right places.
And so, no, we're not using prompts for control flow. We've taken this big like technically three-step process and broken it into separate context windows, separate prompts, separate goals.
Yeah. Interesting. Kind of forces it. I think what you're achieving, if I'm right, is like actually leverage the reasoning of these models. And then two is like forcing first principles solutioning.
Well, and we're forcing the human. Yes. And we're forcing the human into the parts of the workflow where thinking is required.
Because I, I say this all the time is like, you can't outsource the thinking. Like AI cannot replace thinking. It can only amplify your thinking or amplify your lack of thinking. Yeah. If you chose not to think, you're going to get a lot of non, not well thought out code.
Yeah.
Can we see your, how you work? I just, I would love for people to see.
Uh, sure.
And you're also unique in that you're building a, a what do you call it? This orchestrator, this harness, this solution. You're building a tool to help with this stuff too, right?
Yes. Uh, it has a code name. The code name is Rip Tide. That may not end up being the final name, but that is the code name for it. Um, it is, uh, so the original code layer was open source.
You won't see it on the billboard.
We'll get it on GitHub. You might, I don't know. There's some, we're working on some like interesting trademark stuff that we're looking into. But, um, yeah, I can pull up my screen and just kind of like walk through the workflow.
You are, this is so, this is like alpha or beta or something where,
It's early alpha, like,
You're working with design partner customers or something like that.
Design partners, you know, I have a handful of like, you were in Cairo. Remember when I was like, where are you in Cairo? But for work? Was that like, you're going to do like a week of like workshops and training? Part of it also was like, the the code layer product with basic RPI. It's hard to get good results unless you like sit with someone who knows how to do it for a day, and that's just like, doesn't scale to large teams. Like I, I love training people. Actually, I got some feedback from one of our mentors, and he was basically like, Dex, you are really good at like solutions engineering and the forward deployed engineering thing. You love teaching people stuff. That's actually your failure mode. Like you need to figure out how, how do you make the product so someone can learn it in an hour, a week.
Interesting. Yeah.
So I could see that. Sure. So we're flying out.
You kind of skip, you can't skip the step of learning, I guess. But,
Yeah, but we can, we can build into the product something that makes it a little bit easier to learn, right? We can walk you through it.
So wait, so what is Code Layer versus Rip Tide?
Um, so it looks exactly the same here. Like I'll show it to you. Like it's the same. Have you used, have you tried the OG Code Layer? Uh, maybe when you've had a first alpha. Can you recording your screen now so we can share this?
Okay.
Yep. This is like a cloud code session. It's just vanilla cloud code. We've got a bunch of stuff here. You can, I mean, it's, it's a nice UI on top of cloud code that has like a lot of hotkeys for parallelism and stuff.
Very cool.
Um, the first new thing we've added is like organizing things by tasks. Yeah. So I have this like linear ticket here. I'm sorry, you're not really going to be able to watch these. You're not going to put it on the screen.
Yeah. Yeah, I know. Mhm.
Um, so I have this ticket here that is, um, like surface, it's adding like autocomplete for slash commands in the UI. You know, Claude has slash commands. Um, we had it in the old, in the old code layer, and then we basically rebuilt all of code layer from scratch.
Yeah. Cool.
Um, new architecture, new tech stack, everything. It was like, basically like we had figured out what we wanted and we knew where we wanted to go, and we just spent like five weeks starting over.
What is Code Layer? Like, just give me the one sentence elevator pitch.
It looks a lot like, it looks a lot like, it's sort of a heads-up display for cloud code. Uh, yeah, which is, it's an orchestrator for cloud code. So if you use tools like Conductor, Vibe, Kanban, it's, it has fewer opinions about like work trees and stuff. It's like, if you want to do work trees, you manage it yourself. But it's a way to like, run different cloud code sessions. It's, I think of it as like superhuman for cloud code. I used to play a lot of like RTS games, and like I was really into like hotkeys. And so like I wanted a like keyboard-first workflow. I always had like, right when I was in my 20s, I had these like riceed out configs and things like that.
Um, so that was the core of it.
That's really cool. Well, that's a trend, too, is people are reaching for like, I need a little visualization, or I, yesterday I was talking with some guys about like building a, a full fidelity browser screen where you're getting live preview of what you're working on. You kind of need that, and you can't do that in a CLI.
Yep.
Right? People are, and people are also putting like rendering in the CLI. It's like directly with the ASI, like art, you know? Okay.
So, essentially what I'm going to do is I have this linear ticket, which is to add slash command functionality. Um, it's fairly well speced out. Um, but basically what I'm going to do is I'm going to create a task. Um, what is the number? Core 386 slash commands. Um, and so I talk about a lot about tickets. Um, so this is the ticket part of this. It's actually 368. Let's make sure that's not broken.
Yeah, the screen studio overlay like popped up and,
It's being annoying. Okay, let me export this and then we'll go.
Yeah, 10%. Okay, cool. Okay, cool. So, uh, I'm here. Like I said, we have this linear ticket, fairly speced out to add slash commands to Rip Tide. Um, I'm just going to import this from linear. We have a bunch of like deep opinionated linear integrations. They're technically optional, but the product works really nicely if you use linear.
Um, so this is going to clone down the ticket. If I pop,
Rip Tide's an app, I saw.
Yeah, this is a desktop app.
Um, there's a web component to it as well, and like we're doing stuff with like task sharing and stuff, but,
You got to have the multi-user service. Exactly. It's, uh, so it's, it's pulled down this ticket as markdown. Um, we have some stuff to like pull the comments as well.
Um, and then yeah, we'll launch this up. So this is going to immediately, like I said, there's five steps in the workflow. Yeah.
There's like, um, you know, we have the ticket to the research questions to the research to the design discussion to the structure outline, all this stuff. I don't know if we're going to get to. I'm, I'm going to skip the most important part of this, which is like thinking really deeply and giving the model a lot of feedback during the flow. Uh, I did this as a demo last night too, and the result was not good, which is proof that like, you have to, you have to, you can outsource the thinking.
That's fine. Um, if I, just for context, if I was a human coming into this tool for the first time, Yep. Would it onboard me to this flow and guide me? Is it going to be opinionated on this multi-step flow?
So like, what we're working on right now is like getting the opinionated workflow. Like our goal is to, how, how do we help the best engineers in the world managing massive legacy or new code bases for me, like legacy for AI is like anything older than six months becomes hard for AI to be as productive in versus like greenfield. How do we help the best engineers in the world ship like 99% of their work with AI? And by 99% I mean like, not just the easy [ __ ] but everything part stuff too, and move two to 3x faster.
Okay, it's great.
Um, so this is written a research questions doc. I can pop this open in my editor really easily. So it's just generated a question.
Let me just see. So what was the, the ticket was to, what? What was the ticket?
So the ticket was to, so yeah, um, here's our ticket. The ticket was to add slash commands. So if I come in this prompt box now and I start typing, we don't load any of your slash commands.
Okay.
And then the research questions it came up with was basically, so it found, you know, you're using TipTap, so there's a question on that. There's something about Rip Tide UI chat input because you already have a component. So it's already discovered things in your codebase that are probably relevant. It's starting to ask questions to sort of learn more about how it should go about the solution. We're not solutioning yet. We're just learning.
We're just learning. And this is literally again, like because this one read the ticket, it's now tainted. It's now like steered towards doing something. Yeah.
And so this thing will like give you a next step. We've also built some cool like extraction features where we look at the assistant messages and we can say, oh, the, the assistant is telling you what to do next. We just give you a button to proceed to the next step.
Cool.
Um, I'm going to say ignore the TipTap. I'm just going to add one more cuz what went wrong last night doesn't go well. You can always just start over, right?
Uh, stuff in packages UI. It's not, it's not ready for prime time.
Okay. So the questions are now going to
A next skill which is create research.
Yep. So it's going to take the questions and create a plan for how to go answer them.
Yep. That's what we mean by research.
So research is is now like we're doing this thing that um I think I talked about this on let me see if I can pull this up. Um I have a video here. Um your videos. So we talked about like what how do we think about research and what do we want to do with it? um which is where is it? So this is basically like the um the animation of like what's happening during the research which you have your codebase and you point it over here like here's the kinds of things we want to look at right? um and so it's going to launch a bunch of sub agents to do these like vertical slices through the codebase so this is an alternative to having like all your context in cloud MD is like okay every time because it always changes right you have to keep it all up to date and stuff like this we talk about this a lot in the AI dynamically generating that context by doing research to code business is a more effective way for large code bases.
Yep. And sometimes we'll steer this to go research an open source repo. How does this open source repo handle this? Or sometimes you steer it to like go search the web. The idea is like you want to get objective context that can be used by the models. designed all that specific then you end up makes total sense and I've seen open source and entire startups to just that one narrow problem of just like code base understanding emerging.
Yeah, of course. um so this is like okay we we're building basically like you know the agentic rag thing which is basically you know quad cloud models plus grap is is pretty good sometimes a little slower but it's when you try to optimize it too. I'd rather have like slow and reliable. And I'll usually just run two of these in parallel, right? I'll kick off this research, this will run for like five or 10 minutes and I'll go get a cup of coffee or I'll go start looking at another thing or I'll review some PRs for hard stuff for like real features that like need to be good, whether it's like the UI is really intensive or the architecture is like flowing through three different services. I'll max out at like two of those things at once. I can't do five things in parallel. I mean for small little bug tickets, we do have a workflow that will basically um we have this like linear board.
Yeah. Uh that is uh this is a filter just things that I'm streaming about.
Um but this like will we'll like track this through the workflow and we have like one where like if you assign it to the assistant the assistant will just like do it in like serverless GitHub actions for you. And so for really small stuff we'll just be like full hands off but for things where I want to be in the loop we will we will use this one. So is it performing the research as well as designing what the research is?
So it's performing the research. It's just literally going to generate a document with a bunch of context about the current code base.
u with with regard to those questions.
with regard to and in this case I know I'm building a system. I'm building a new component that was already like well implemented in the old version of the product. So I'm just like give me a full like understanding of how this works in the old product and how it works in the new product. And then we're basically in the design discussion, we're gonna like lift and shift stuff over.
How I'm solving this in my personal workflow is for every major feature, component, page type, whatever, I'm diligently creating an MD file for just that system, which will also include a little bit about the componentry that uses and the standards I've set internally, the heuristics, whatever.
So, how many of those do you have?
A [ __ ] hundred of them.
How do you keep them up to date? What happens when one of the system changes?
If if I'm going back to something that I know I haven't worked on in in weeks,
Yeah. Then I'll say, "Please update that. Research and update, you know, and I'm like crossing my fingers that it does a good job."
But you have to remember to do that. Correct. And if you had 10 people on your team, I feel like you would end up like just foot guns everywhere.
I feel like a lot of agented coding favors the solo monorepo crowd.
Yes. Or the small startup team and then it dies. And I talk to guys who are in or gals that are in larger teams and larger companies. So what's the best, you know, productivity gain you've gotten as a as a team? They measure this stuff and it's it's a KPI 20%.
Yeah. 20 30% if you're just like mostly doing Yeah. Because of all of the churn and all of the like
Yeah. So yeah, the opportunity is like the this 10x over 10x for solo monorepo people. Yeah. Bringing it to the actual make it work for real systems that can't break um and like are maybe 10 years old and are very complex.
There's a lot of value in that. We billions of dollars flowing through the software today already. you always want to make, you know,
it's it's crazy to me. I'm also like we look at competitors like I look at those every single coding agent companies. You look at like cognition and factory and all this stuff and like I know they're all working on solving these problems and making the models better like hill climbing evals and all this stuff but it's it's surprising to me how proud people are of this is going to get a little spicy like of like hey all of our tiny little bug tickets go in a backlog and the agent handles them and it's like that's not impressive to me like I want to solve the problems that no one can solve with AI yet. We want to and I want to find ways to help everybody be able to do it.
Yeah. Where is Microsoft and GitHub and all this? They've historically been they've sort of been the dev tools leader in for 20 plus years uh in compilers and systems and idees and just where are they in this?
I mean I heard Copilot is is good now. I don't know.
Everyone left it and no one's come back to try it.
It was the best. It was best in class when it first came out because there was nothing.
Yeah. It like built on GPT3, right?
Yeah. Yeah. Interesting.
Okay. Okay, so this research is done and uh now it's going to write the research document. Um and so this is a single asset and again like as we go through this flow uh at every one of these stages, we're compacting context. We're turning the ticket into questions, we're turning the questions into research, turning the research into the questions and turning it into like here's just the patterns in the codebase to follow and here's the and we'll look at some of these docs that are uh much more exciting in a sec.
Amazing. One thing you're giving me with this, this has been really enlightening is uh versus the the Ralph Wiggum approach is literally just like let it run, cross your fingers.
Well, so the Ralph Wiggum too is like the specs have to be good. Yeah.
Specs have to be good and the problem has to be verifiable.
Well, no. I mean, if the specs aren't good, it'll still run diligently until it finishes. It just
Yeah, but you won't like the thing you get like it won't be useful. You'll spend the tokens, but like Yeah. I this is what I found is like if you know exactly what you and you can use AI to build the specs but a certain point like it's a leverage thing right the specs have way more impact than the plan has way more impact than the actual coding loop cycles and so like the reason why Jeff Huntley was able to build a programming language with AI and you know six weeks for 10 bucks an hour was because one programming language are very very verifiable the model can write a program in the language and then see if it runs and then go fix it back and forth and back and forth like it's very easy to run in a headless loop.
Yeah. Yeah. And also like he speced it out really freaking well. Like if you go to the cursing repo and you read the specs, it's like grammar, structure, like garbage collection. It's all it's all explained there. And maybe he wrote it with AI, but he owned that [ __ ] and he put his stamp on it.
Makes sense.
Um, cool. So now we have our research doc. Um, I can pop this up in my editor. This is just a long tour of everything of the code base. I rarely read these anymore. And like I would if this was a real thing I would probably read it uh a little more closely but since we're rushing through a demo it's good.
Yeah that's super use that's the kind of context that like if you're handcrafting at that level you're going to get good results. So
yeah the other cool thing we get here is um this will automatically write a permalink to our private GitHub repo with all of our research artifacts all of our artifacts. So for this ticket, I have all my artifacts here and then our linear integration will also
this seems to be an emerging best practice which is all these docs whether you're creating them or the AI that they should go in the repo somewhere or a repo.
Are you putting them main on repo like your your base repo you create a separate repo?
So we have a separate repo that gets simlinked in. So cloth sits writing to a folder but under the hood we have a write hook that autosyncs it to GitHub on it. Everyone just pushes and pulls to master. We kind of treat it as like an S3 bucket or a doc server, right? We don't care about like real VCS. We just want revision history.
And so while this is running, it's also attaching all these docs to the ticket.
That feels like a very important part of the solution for large organizations.
Yep. So anyways, I'm going to take it. Sorry, I know. I'm going to hit this button. I use all the hotkeys, so I'll just hit command shift enter and we'll proceed to the next. At any point, you can like tweak the prompt. If you're like, oh, there was a thing that wasn't in the ticket and I want to make sure we do it this way or whatever. Like we want to keep you in the loop, but it's kind of like the the downhill path is to keep running through the workflow. how fast this is the UI.
Oh, the Yeah, the UI is snappy. That's because Kyle rebuilt it properly instead of me like trying to vibe out a bunch of React code.
very cool.
And like
what do you what what technology are you using for the UI here?
So, this is just Vit. Um, it's built on React uh Tanstack DB tanit stack query. We use electric SQL for syncing. Um, it runs through a Postgress instance that we manage in the cloud.
You have a Windows version. You have a Windows version.
We don't have a Windows version yet. Uh if we ship anything next it'll be a Linux version.
Interesting.
Um that's part of our like salute to like we want to build real tools that are useful to like really good engineers and like so that's uh part of part of our ethos is like okay keyboard shortcuts you can use J and K to scroll through this. So if you if you love Vim JK L L and H will like expand these things for you.
Wow.
So this one read the research and then it's going to go do a little bit more of its own research because the research didn't have the ticket. Sometimes this is also like kind of we've kind of like accidentally implemented a little bit of something that feels like deep research where the research document ends up at the end we'll have like some open questions about like how does this work? How does this thing happen? What does this all do? Um but this thing basically and so as we're building up this directory at each subsequent phase we're going to take all the docs that exist and like basically I mean cloud uses a recall we're basically deterministically allocating those into the context window to provide the next step and so you have like
okay all of this knowledge big context window compact but it's very intentional how it's compacted the output formats are very specific
so we started with a linear ticket with something we want to build we generated questions questions and we generated research. What's next?
Uh so now we're generating a design discussion document.
Okay. And then at each of these steps, is the human guaranteed to be pulled in or are there sometimes where it can slide through or is that a setting?
So the design discussion um is going to come here. I can show you actually what the template looks like.
um you know what I'm getting at? I'm getting at like there's a mode where I'm like I'm okay with rolling the device spending the tokens and I'll come back on Monday to see what happened.
Yeah. So that's actually the next thing we want to we feel sometimes this feels like overkill for smaller stuff. So we'll basically have like actually probably four modes of autonomy. So what you're getting right now is level three which is like you are at each step you check the results you advance it. There's like a fourth mode which is like we just double the tokens on you do the research again do the design discussion again run it through a couple variations automat. Yeah. Automatically do it three or four times. And then there's like a slightly lower autonom higher autonomy one where you still want to be in the loop. And so what we'll do is we'll generate the questions, we'll generate the research, we'll generate the design discussion. You still play in the design discussion and the structure outline. What where are we going and how do we get there? But then once the structure outline is confirmed, then the writing of the plan, the creating the work tree, the doing the implementation, the PR is automated.
There's a full auto one where it's literally just like try to oneshot this and I accept all your recommendations from the design discussion. Just go figure it out.
You know that's the one I want, right?
Yeah, I know. I mean, the world of abundance is like I have probably 10 pretty large, you know, project things that I want to do.
Yep. And I'm not working on them at all right now. And I want to run them all through this. Why not? I can always hit delete on the result.
So, why why not run them all through? If it's just the cost of tokens, I'll pay.
Yeah. And I think I think the real answer is like unless you know it's going to be really hard, it's like we should give you the oneshot version. It should just go. If you don't like that, then it rewinds you to the design discussion. and you're like, "Okay, let me be in that would be killer."
And then if that still doesn't get good results, then you go and do it the like mode and you so like at each point you can kind of like, "Okay, roll this back and change the autonomy level and I want to start over." It's same same thing with like model overfitting, right? You commit commit commit commit commit and then you roll back to the one that's good.
So instead of the green square GitHub activity log, what I want is like a dashboard of each project where what stage it got to or a rewind button to go back.
Exactly. We so we've kind of like prototyped this on top of linear, right? That's what all these are is each of the stages. And I can briefly show like this is our board of active stories actually has a bunch of stuff in it.
Yeah. Um but yeah, this is uh this is yeah, that's where that's where we're going is like we need a a place to collaborate on the construction of software and not just like for you to solo build, but also like when this design discussion comes out
uh let's see. So it's now added it to the ticket. So I can come in here and open this file. Kyle has access to it. I can ping my coworker and just be like if I if I don't know the answer to a question. We collab here on the design thing which is what good engineers at big companies have been doing for a long time. But it's like how do we automate the the the parts of it that we can with AI while keeping the high leverage human work there.
So let's let's pull up this doc and we'll look at it. Um actually I'm going to open the GitHub one. So it's got a summary of my request. It's kind of built a mini spec at the top here. It's like here's where we are, here's where we're going.
Contrasting this with Gast Town, which was like feels chaotic.
I haven't tried it yet. I've watched people use it.
Yeah, this feels much more corporate in a good way. Like I'm in control. I can see where things are. I can go double click into something. I know exactly what it's guarantee for me is in the in the process.
Yep. And that feels more auditable,
uh, collaborative. I I like that that aspect like you're clearly targeting like the larger team.
Well, it's like and I think like people will solve the yoloy easy stuff however they want and I don't have opinions on that. I'm all about how do we push the frontier of what's possible.
Yeah. Very cool.
So yeah, so this is going to surface the patterns to follow. So this has actually pulled the full response editor from your own code. Okay, cool.
So if you have an old codebase and there's like six ways you do atomic inserts, it'll try to surface all of them or if it picks the wrong one, this is your chance to be like, "Oh, that's the old way. Okay, go find help in the new way."
This is killer. And and did you prompt that in to say please especially look out for that.
Um so it's prompted the form the output template has patterns to follow and basically like things that are relevant to the ticket in the code.
When it sees patterns to follow, it's going to naturally use reasoning to figure out like oh I found a couple. Would you please disambiguate?
Um so that's so patterns to follow gets surfaced at the top. We might actually move this to the bottom but this is all the code and then it we have design questions.
Okay. And so what I'm going to come here
just having that in the template is so useful obviously because the biggest one of the big fail modes is just like using an old pattern that I've moved on from still in my code base.
And what people used to have to do was with RPI like you had to go read this research document and check every pattern and make sure it was good and like verify that it didn't pull in anything bad and now but it was objective so it would pull every pattern it found and now what you get is you get this like here's the ones that are relevant to the ticket based on the research dock.
Yeah. Cool. Yeah, that makes sense. So, what's next after this?
So, we could come through and answer all these questions. Um, I'm going to just say I accept all your wrecks. Um, if I if I wanted to though, I would go through and answer the questions. It'll have like option A and option B. Um, you can also insert option C. Like, I don't like either of those, but it's going to say, okay, do we want to discover these or just have a flat array? How do we want to implement fuzzy search? Some of these might be actually bad. The point is like if you just blast through this, you probably won't get as good results.
Yeah, I yeah, my experience has been worth reading the questions if the CLI you're using is asking and y
it usually give two options. If you ask it to recommend one, it usually recommends the second one. I've noticed some patterns and I've gotten more comfortable with accepting defaults when I'm in a hurry, but I'm always getting better results when I'm participating.
Yep. Yeah.
So, I'm um I'm I'm just blasting through the workflow here for sake of timing, but I'll come back to this doc.
How many more steps do we have in the full process to getting the shippable code?
Um, so this is the structure outline and then from there it's kind of vanilla RPI. We've done a ton of a done a bunch of content on on research plan implement. So like once the plan is written I think it's pretty clear like okay you give this to coding agent I know what's going to happen next kind of thing.
Okay. And in the case of giving it to the coding agent are you you're building that into the ID must be
exactly yeah this all happens the work tree setup everything
and you're wrapping cloud code are you also able to switch to codec if you want to use codeex instead or something.
So right now we're doing uh cloud agent SDK. Uh we get a lot of requests for codeex for AMP for other for open code for other coding agents. I think Kyle is interested in rewriting a lot of stuff with open code
um
TBD when like we we found we're able to get really good results and almost everyone we talked to at a big company has cloud cloud max subscription. So like
it would be a cool thing to build and cool to have but it's like
you know we're a small team and we're solving the most important problem. This is more of a it's probably like a business strategy consideration that like you have to imagine that 12 months from now the kind of corporate clients that might use this are also going to be the ones that are like yeah actually around here we bundled like all the Gmail works work work place accounts with like you got to our developers have to use Gemini it's bundled you know I think that's how that's how the cloud guys are going to I think compete right to to kind of box claude out in a way like
maybe I mean we'll see it happen with Microsoft and Slack right
what
happen Microsoft Teams and Slack will happen and and you know if the if the agents are actually good the devs will accept it otherwise there's going to be some friction
but yeah
um people ask a lot about the ask user question tool
oh
um we don't use that I mean we we will add support for it because people like it and I think it's a good UX but I actually prefer this so if I go back on the history here and we can look at the previous while that other thing is running
um look at the history so the previous version of this file had uh a bunch of questions and so what I do is I would just come here and I would put on my like voice thing and I just be like, "Okay, for option for question one, do option A. For question two, like do neither of those. Go look how we do XYZ for question three." And like you just can't do that with the answer question tool. It's just not as fast. So like
my that's that that's that's why I like it flat like this. Um some people have other opinions.
Can we skip ahead just to like dip testing a little bit like
are you in your methodology that you've
you know you're opinionated, you're guiding people through and the tool supports.
Yeah. How are what are you doing? What is your best practice for testing?
Yeah, so let I can just pull up a different um let me just jump to a different like thing that's already been shipped. Um and so like the structure outline is going to have the testing strategy built into it.
Um and so here's our phases, right? So this is if you've seen an RPI plan, this is way shorter. Um, it's just like a summary of what we're going to do for the human alignment. The plans were really ended up being like really long for people to read. Um but yeah, so we have a bunch of test cases outlined. Um so this is like unit testing, integration testing. Some of these things will have like manual test cases in them. Yeah. Validation, setup, the config, and a directory. So like I want to align on the model not just on like what order we're going to do things in, but I want to verify that like it's ordered in a way that I'll be able to check it after each step or I'll be able to read an integration test and be like, "Yep, if that passes, I'm good."
Okay. So it's kind of part of the workflow that's going to account for all the kinds of testing that are appropriate.
Yep. And and baked in. Y do you can I ask a question sort of in the realm of like it's sort of a big picture
like
I have we are all much more productive with these techniques and we're this is going to take us even another level
and then we're also more ambitious like I am
I mean how many times have you been like I need to go do this little enhancement but while I'm in there I'm just going to like also like redesign the page and fix a bunch of other like just like when you're in the flow state you're just like it's so much easier just be like okay I'm just going to fix those three things that have been bothering at once. I'm much more ambitious about how much I take on when I dive into the code somewhere or the feature of the product.
I'm much more ambitious in general. I'm shipping a lot more and the surface area of
all this. You're always going to like no matter what you did in this testing that you showed me, there's always going to be production things that happen.
Yep. That are unaccounted for edge cases, confusing user experiences and like higher level problems. You're going to have more of those too. I'm still just one person though, you know, and then I sold my user base. They're going to discover those things.
Is there something pro what are the best practices emerging for for that like validation that the thing that I'm delivering testing it as a human would doesn't that need to be part of this in this world? Who's working on that? What are the solutions there?
Yeah, I mean you got the browser agent stuff that's like kind of coming along. It's good. I mean, I don't think we'll ever get to like I don't I don't know. I don't think it's worth trying to get to perfect, right? The other the other thing I talk to a lot of people about it, they're like, "Okay, cool. Like the agent can write the research and then the agent can write the tickets and then the agent can write the agent that writes the tickets." And it's like, "Okay, yeah, every like, okay, so planning gives you 3x leverage and research gives you 3x leverage and like that's a lot of leverage. That's nine times leverage. 10 times leverage. Like you should learn to be happy with 10 times leverage." And like we'll figure if you try to automate everything, you're going to get stuck in like the meta tooling soup of like just figure out how to like get three times leverage or nine times leverage and figure out the last keep doing the last mile until that becomes the bottleneck and then go fix that. But
what about like I mean these unit tests and these kinds of basic you know do they capture things like security and performance and usability?
Uh probably not. Like there's probably still always going to be a category of so I mean if you're building a programming language then like yes you can test it all the way end to end completely. If you're building a web app that users are going to use and that's hard. If you're building an AI chatbot then like yeah you have evals and stuff like there's all these different dimensions. If software is going to be more places and we're all more ambitious within there, more productivity, where is the solutions for that the security, the performance and the uh being on brand and and being uh good user experience like
it still takes taste. I mean yeah I think that I think the new mode is like engineering really good engineering for like hard to build systems like programming languages and like high scale cloud systems is still really important. AI can't do that that well in a way where I would trust it to do like high-risk stuff. But the new mode is like product and taste and knowing your customers and like the the thing that someone who builds a really good product has that like they're leaning on is they've spent hundreds or thousands of hours with customers understanding the problem. I don't care how much code you can ship. If you don't understand the problem, you're not going to build a good solution.
Yeah. Right.
Yeah. I find that, you know, if I'm narrowly asking to solve one incremental feature, after a few months, I'm going to have like a cluster [ __ ] of tools with too many buttons on them, you know, it's like, and where's the the process of stepping back from that and redesigning?
Yeah. Doesn't really necessarily fit in the world where I'm able to quickly 10x the number of features I build.
Yep. I Where's the agentic loop for user experience that's going to help me, you know, redesign, right? who's working on that layer of the emerging problem space. Does that make sense?
Yeah. I mean, I think I think like as if you are still a human in the loop, what you need to do and AI can help with this is like you need to build as much infrastructure as like when I want to test one of these things, I got to go pull down the branch. I got to go check out the work tree. I got to run it locally. I got to play with it. It's like, okay, if you're building a blog, you can use Versel previews or things like this. But like I think something that's really high leverage since the very first job I had was like have a really good way to preview stuff and poke around at it and send it to someone else to look at. Make it dead simple.
I get I guess what I'm saying is I want like now this is all done. I want to deploy in a private testable sandbox that can't actually send real emails etc. Like you know and I want to run a bunch of simulated humans on the website to give simulated feedback about using that or stuff like that. Like I who's building that product? I'm looking for, you know, what are the emerging startups that are trying to tackle this? Like this the amount of software is going to is exponentially increasing. We're just starting.
So, how do we get out of the the problems that come with deploying much more software?
Yeah. I mean, Jay calls it the infinite software crisis. We had this thing in the 60s and 70s, right? Was like, "Oh, yeah, we built lots of software, but we have no idea how to engineer it." And out of that was born software engineering. this idea of like here's how you design systems that actually scale and are maintainable. I think we're going to go through another cycle of that.
Yeah. Interesting.
I just realized I've been doing a bunch of these in the wrong directory. So, uh this is actually not going to work. But, um I can we can quickly the last thing I would show you is I can quickly show you the structure outline for the task we were working on.
Cool. Uh because that's the only other like novel thing that we're shipping over like vanilla RPI which it has the current state, desired state, end state. It has the patterns that we're going to follow, just like code snippets and like, hey, here's the files that we got them from. It has our design summary, and then it has our phases. So, it's like, okay, we're going to do this, and then we're going to add this. Okay, this this one's really simple. It ends up being two phases. If I had actually been in the loop here, I would say like, okay, we're going to add one line of sanitization. Like, just make that all one phase.
And then when we go to implement basically, um, okay, that's Renee. Anyways, that's that's the that's the high leverage stuff. Design design design discussion with the user. and um and and making sure that you order it in a way that you can validate it as you go. For hard stuff, you're going to want to know
Yeah. Um that is broken before you get to the end. And so this is this is the high this is the highest leverage way we found to use AI to ship complex features in large code bases.
Yeah. Super cool. Hey, we can we can wrap here. Um but I'd like to ask you like there are so much new stuff dropping all the time. We're all tinkering with things. What what's exciting to you right now? What is your side project if you have one? if you have any bandwidth for that or things you think are cool.
I'm working on uh automating the company.
Yeah. So I had a workflow that I would do where like I mean I I never did it which is why I'm like I have to automate this now cuz like we're ramping up like go to market a little bit again. We're onboarding lots of new customers and so there was this process where I would like read my calendar, find all the meetings that were onboarding us, add my co-founder to all of them.
What do you technology you use to implement that?
So uh and so the other thing we would do is it would read all the emails on the events. It would go like web search them and look them up, add them to our CRM that is all in markdown. Yeah. Uh I built a thing that basically like reads all the markdown files in front matter and then turns them into like an HTML readonly cananban board. So just like wow and then that runs every morning in so it's like GitHub actions plus clot code plus a bunch of custom CLIs to just automate random work.
That's really cool. And it's just running on your like dev instance of that project to make it happen. So you have like a snipper repo with all our CRM and stuff and it's got its own GitHub actions and it just runs cloud code headless.
Okay. Running in the cloud somewhere. But
it runs on a just runs like GitHub actions is a really good way to do like serverless on a cron without having to you just write a little YAML file and run script and it works.
Okay, cool. That'd be a fun thing to to dive into at some point.
We should use that to run AI tinkerers.
Yeah, that sounds really sending weekly updates in the channel.
Love to have like those tools exposed to all the city organizers. Yeah. Second, um, have you seen Paul Klein's CEO automation?
Yeah, I sat down with him, dude. I helped him. I was sitting next to him when he wrote like the snowflake tools for that, like CEO CLI. Yeah, it's really dope. I'm really glad he went in and did like a big twoy thing.
Yeah, super cool.
Um, this was great, man.
Yeah, this was super fun. Thanks a lot, Dex. I appreciate it.
Um, thank you.
Like and subscribe.
Like and subscribe to Joe.
Thanks.
Um, yeah, it was great. Um, the other thing that I'm really excited about is like what we're going to do with AI tinkerers in San Francisco in 2026.
Um, we done kind of like a vanilla just like the same events and format that we've always been doing and every other city. And I think we've like experimented a little bit last year and found some stuff that I really like. Um, so we're going to try to do more like smaller like less demo nights and more builder nights. Just come hack on something. There'll be a theme like building coding agent add-ons or whatever, but come hack. We'll do instead of hackathons with prizes and sponsors. I just want to get a bunch of good builders together to just like hang out. We'll get some food and like come work in a be somebody somebody's we'll get someone with a beautiful office to sponsor it and just like more code cafe kind of vibe.
Awesome. I'm excited for that.
Right on. If you're interested in sponsoring that, head to the website and head to the website an email. Thank you.
Nice. So cool.