Transcription
What's up, Engineers? Welcome back! Andy Deb Danan here. The 12 Days of OpenAI went out with a bang! They announced GPT-3 and the Next Generation Benchmark-breaking reasoning model, Google's Gemini 2.0. Flash is very clearly absolutely cracked, and it's still 100% free. The sleeping giant has awoken! I have no idea how Google is able to offer this at scale. Oh wait, I do—they're rich, and they want to get developers like you and me on their side. Not going to lie, it's working!
Meanwhile, we have Anthropic. To me, they seem like the cool guy in the corner at the party, having a good time, like they know something everyone else does not. I'm confident we'll see something absolutely insane out of Anthropic relatively soon. Last but not least, of course, we have Llama 4 right around the corner. It's going to have multiple releases, and I'm really, really excited for this. Tons and tons of models will be released or fine-tuned on top of the Llama series.
With these next-generation models right around the corner, the question for you and me as engineers and product builders remains the same: How can we use as much compute as possible with powerful models and the best AI coding assistance and AI tooling to build tons of software at higher rates than ever while maintaining high quality?
In this video, I want to showcase a powerful prompt chain built into my favorite AI coding assistant, AER. This prompt chain is called Architect Mode. Architect Mode is simple; it's a prompt chain of length two where you have one model that drafts your code, and the second model is the editor. The editor takes the draft from the architect and generates real working code. This workflow is the best way to understand what's coming next—a topic we'll discuss more in this video.
With all the hype around Gemini 2.0 Flash, I thought it would be cool to pit it up against the reigning undisputed champion, Claude 3.5. In this video, we're going to boot up two AI coding assistants and see if Gemini 2.0 Flash can code like the champion. Near the end of the video, I have a massive announcement and a new opportunity for you. I'm insanely excited to finally release it! If you write code with AI, stay tuned so you don't miss that.
What are we working on to show off this powerful prompt chaining technique using AER? Let's open up VS Code and break down what's happening here. On the left, I have an AER setup command that is going to start our AI coding assistant in Architect Mode, where we're running two models. The architect is going to be Gemini 2.0 Flash, and the editor is also going to be Gemini 2.0 Flash. The model is effectively talking to itself—one time it's thinking, and the other time it's editing.
On the right, we have that exact same setup with the reigning champion, the best LLM for AI coding, hands down. The only exception to this now is the brand new 01 series that's slowly rolling out. We're going to be covering that on the channel in the future. Yes, always! This is going to accept the architect's changes anytime they're suggested. We also have a brand new loading feature out of AER that lets you save and reload sets of context.
All I'm going to do here is copy this, paste it inside a terminal, and what you'll see here is AER will boot up in Architect Mode and add every one of these files in this .or file to the context. We're going to do the exact same thing on the left side, so we're going to boot up Gemini, and you can see we have the exact same context. If we type SL tokens, Gemini 2.0 Flash is completely free with a million tokens in the available context window. If we run tokens on the right side, Claude 3.5, of course, has a price tag to it. We're running at about 1 cent per prompt. This is, of course, well within the bounds of what we're willing to pay to get a lot done in a fraction of the time it used to take.
So here are the two sets of models we're running: two Geminis on the left and two Claude 3.5s on the right. This is great! We have our AI coding assistant, but what are we actually updating? What is this code? What are these files? I have this project that I've been building up as a personal knowledge base for AI agents. This is going to become more relevant, and we're going to talk about this code base more on this channel in 2025.
You can see here you can do a couple of very simple things: you can add arbitrary content, list all of your content, find similar content via embeddings, remove content, and back up the database. Everything here runs on a private local SQLite database. If we open up the terminal, we can easily run one of these commands. Let's go ahead and run our list command. We just have a couple of items.
So what are the changes we're going to make? We have this personal knowledge base system that we're building up. What are we going to add to this? We need to add a couple of new commands, and we're going to do that by firing a large spec prompt. So what is the spec prompt? It is essentially a plan for the work you want done. It's a specification document, except this specification document is built for your AI coding assistant.
You can see here we have four sections: the headline of the changes we want, the objective, context, and low-level tasks. The objective is, of course, what we want changed at a high level. You can see here I want to add these three new commands. I want the ability to quickly add YouTube scripts, so pull down YouTube videos, get a transcript for the YouTube video, and then save that into the knowledge base. This will help me run arbitrary queries and prompts on my existing YouTube content.
We also have ADD site, which basically scrapes down a website and then stores that in your personal knowledge base. Then we're adding a style command; this is just something that's missing from the knowledge base. If we open up main, you can see all of our current commands. If we collapse add, remove, list, similar, and backup, this is going to give us the ability to grab a knowledge base item by ID.
What's context? Context is, of course, every file you need to get the change done. You can see how this spec document is starting to drill into the changes and detail out more information for the AI coding assistant. If we go back to our VS Code window, you can see we've added all of those files into our AER instances on the left, and on the right, we're almost ready to run this prompt. The last thing we need to look at is, of course, the low-level tasks.
What I have written here is a list of prompts—a whole set of information-rich prompts. I'm not going to go into much detail here about how this works; I'll add some links to how some of this stuff works in the description. But effectively, what we have in our low-level tasks is a list of prompts that our AI coding assistants can execute top to bottom to get the work done. You can see there's more detail in here than you're probably used to. I'm writing very accurate, very precise AI coding prompts. We'll talk more about these prompts in the future on the channel.
So let's close everything and let's execute this. Spending all this time explaining this, let's go ahead and fire this off. We'll open up VS Code. I want to measure success in the most blunt, forward way. The only thing that really matters—of course, I'm being a bit reductive here—but there are only three things that really matter as a software engineer: First, did you accomplish the task? Second, how much time did it take you? And lastly, what did it cost?
We're going to judge our two AI coding assistants by the same metrics. So let's go ahead and paste this in on both sides here. After this prompt runs, we fully expect to be able to run these three commands on both code bases without flaw. So here we go! On the left, we're going to run Gemini, and on the right, we're going to run Sonet. We want to know if they built the three features we asked for—the three new commands—how quickly they did it, and what it cost. Obviously, Gemini wins automatically on the cost, but let's go ahead and fire these off and check out the results.
Okay, so right away, you can see Flash is off to the races! Right now, both AI assistants are in Architect Mode, so they're just drafting all the changes that need to happen here. You can see on the left here, Gemini Flash is quite a bit faster. The architect just finished, and now the editor is going. On the right, Sonet just finished the architect, and now the editor model is firing off. I think Sonet—no, no, I think Flash is coming up to the finishing line here. It's writing tests; we have a prompt for writing tests in there.
Let's see, where is Sonet right now? Sonet is—oh man, Sonet is updating the README. Okay, so Sonet actually finished a little bit faster. No, Flash has finished now! They both ran into linting errors. Yes, always! So they're going to both automatically fix the linting errors. Let's see how they perform here. So Sonet is—Sonet has finished first! I'm actually quite surprised here. And now Flash is finished.
Okay, so that was exciting! A lot of stuff happened there. If we just analyze the results here, this cost us a total of 12 cents for this session, and there's no cost for Flash. So on speed, surprisingly, Sonet won. I think it won because it received many fewer tokens. I think on a token-by-token basis, Flash would have won. But let's go ahead and actually see if our AI coding assistants accomplished the task, right? This is the most important thing. If they haven't accomplished the task we asked them to complete, then all the other side metrics like cost and speed don't matter at all.
So let's go ahead and see if we can run our new knowledge base command. What I'll do is I'll open up new terminals on both sides. If I type `git status`, you can see all the changes that the model made here. And `git status` on the left side here—same deal, right? Just clear this. Let's go ahead and start with Sonet. So if we open up main, we should be able to see all of our new commands. I'll collapse, and you can see we do have three brand new Typer methods.
Let's go ahead and run our `add YouTube script` method. So we also asked for usage docs, so we have this exact command that we can just copy out and run. What we expect here is a new knowledge base item with this YouTube script. If we just open this up, you know this YouTube script is going to be, "I want to share my AI engineering 2025 plan, max out compute." Super relevant! This is the video we want to pull the transcript from.
So I'm going to copy nearly all this command. I'm going to start from the Python since we already are running in a virtual environment. I'll paste this in. So let's see if our Sonet 3.5 prompt chain completed our `add YouTube script` command for us automatically. So this is good; it's thinking. We do get this bad recognized option, so we do get an issue here. All I'm going to do here is copy the entire output. Typer gives us a nice output block here. I'm going to come back to our AER coding assistant, and I'm just going to paste this in.
So I'm giving Sonet one shot to correctly resolve the issue. So I'm going to hit enter there, and then we're going to move over to the left side. So let's see if Gemini properly wrote that command. So if we do the same thing, we go to main while our Claude is running on the right. Let's go ahead and collapse and see our commands here. We do have `git` at the top, `add YouTube script`, and then we have `ADD site`. So this looks good! Let's open this up. We do have some nice usage docs from the Gemini Flash model.
Let's go ahead and see if this command will execute for us. So same deal, and let's see if Gemini created this command properly for us. Okay, so we ran into an issue right away. Definitely disappointed here. I'm going to copy all this and do the same thing. We're going to give both Gemini and Sonet a chance. I'm just going to come to the instance, paste this in, and execute.
Let's hop back over to our Claude AI coding assistant in Architect Mode and let's just rerun to see if it resolved the issues it had. So I'm just going to hit enter here, and let's see if it can get it right. Okay, fantastic! So you can see we have `add YouTube transcript 12`. Let's go ahead and test the new `git` method, right? So we should have this new `git` command here. Let's open this up and let's go ahead and fire it off.
So you can see here we created this new YouTube transcript with ID 2. Let's go ahead and just run `git` with ID 2. So if we hit enter here—wonderful! We can see our new `git` command gave us back that exact instance. If we open this up here, you can see we have this transcript. So this worked really, really well!
So we have one more command to test here for our Claude AER prompt chain running in Architect Mode, and that is going to be our `site` method. So let's go ahead and see `add site`. Just copy this from Python. So you can see here `python main.py add site`, and we're going to pull from the Anthropic research building effective agents blog post. If we click into this, we can see exactly what this looks like. There's a great rundown— a lot of the ideas discussed here we have discussed on the channel, of course: building blocks, workflows, agents, we can search prompt chains. You know, this is a really popular idea we have and quite literally are discussing right now. Great post!
It's weird because it kind of feels like, you know, on the channel we've discussed so many of these ideas long ago, frankly, but it's cool to see this collected by a big player in the agent space. So this is great! So anyway, let's go ahead and use this URL and let's see if our Sonet instance was able to build up this command end to end. Right? And you can see some of the details here, right? We download the site, generate embeddings, and then we run our `add KB row` command, which inserts it into our SQLite database.
So let's go ahead and fire this off and let's see if we can add this new site to our personal knowledge base. No issue so far—that's a good sign! So we're probably scraping right now, generating embeddings, and there it is! So added website content at ID 13. If we just type our list command here, `U main list`, and we actually need to drop `UV` there since we're already in a virtual environment, we can just fire this off and we should see both our website and our new site content.
So that looks good! Let me actually just go ahead and use the `git` command that we were just using, and this is going to be ID 3. Fantastic! So check this out! We have the markdown formatted version of Anthropic's agent post, right? So this is fantastic! It's working perfectly! We have this inside of our knowledge base, and we can use some quick similarity search. If I just highlight "workflow routing," and we have this similarity search command here, I was using it earlier. We just clear this out, paste this here, and it's going to be the top hit.
So we're looking for the similar items with this text, and you can see there—that is, of course, our top hit! So we got that knowledge base row returned immediately. So this is fantastic! We had one issue here with the Claude 3.5 Sonet prompt chain running in AER in Architect Mode, but overall the changes went through. You know, we can see we can do a `git diff` here, and if we want to, we can even run `git diff` and then write the diff to a file, so `def.text`, and then we can see exactly how much was changed here, right?
So, you know, about 2,000 tokens worth of file change. So this is fantastic! We should have a README update as well. Yeah, so we have this README update, which is a really cool part of our spec prompt that asked for changes to the README. So you can see here we have new documentation around adding YouTube content, adding website content, and getting content by ID. So you can see we have great usage documents here.
Let's switch back over to our Gemini Flash AI coding assistant and let's see if it was able to correct the mistake it had. Let's go ahead and just open this up a little bit more too. The issue happened on the `add YouTube script` call, so we'll just hit up and let's see if this issue was resolved. So let's run this. Okay, same deal. Let's give it even more grace. We'll copy this, and again, we'll let Gemini Flash attempt to fix its issue.
It runs really quickly, but it seems to get things wrong from time to time. To be fair, this is a three-feature spec prompt where we're asking for three changes, right? We want three concrete things modified, so there's room for confusion for a large language model. But it's pretty clear, even with this small AI coding sample here, you know, it's taking us an additional AI coding prompt to get Gemini Flash where Sonet 3.5 is.
So let's go ahead and just rerun this again. Let's see if Gemini Flash has resolved its issue. This is good; it's taking some time to load here. Added item 12—that's looking good so far! Let's go ahead and look for `git`. There it is! We have our new `git` command, so it did successfully create this. You can see we have that new `git knowledge base row` function call, and if we open that up, you can see here, diving into the layers of this code base, we also have that call there.
So that looks great! Let's go ahead and run this. So we'll copy this command from Python, and this is item 12. So let's see if we can get item 12 from our knowledge base. Fantastic! You can see we have that entire YouTube transcript from that video. We were able to add this to our knowledge base, right? So this looks great! We have one more command to run here on the Gemini Flash coding assistant code base.
Let's look at the `add site` method here. Everything looks relatively good. Let's copy this, paste it, and let it run. Right now, it's scraping the site, so that's good! It got to that point. So if we run the `git` for item 13, so let's just quick search `git`, update this to item 13, we should see that Anthropic building effective agents blog post. Nice! And we can go ahead and run that again and save this to a file. We can say, "AI docs building effective agents" in markdown, right?
So we can run that again, and now we have that loaded out of our knowledge base. We should have this new AI doc here, and you can see we have that post from OpenAI. Exactly! If we search "prompt chain," you can see that exact same search here from the site, right? So really, really cool to see these two models perform up against each other.
It's pretty clear that Flash is a great model. It's free; there's a lot of intelligence there. It definitely is not on par with Claude 3.5 Sonet, but it is still a fantastic model. So you'd be surprised how many times I've done this. I run a large prompt in a code base, and then I'll revert the code base, duplicate it a couple of times, and see if another set of AI coding assistants with different models can get to that same place in the exact same amount of time, speed, cost, and so on and so forth.
So, you know, jury's in. Claude ran the fastest; it made fewer mistakes, right? It made two fewer mistakes than Gemini, and it did cost. So we're going to give Claude two points. Gemini did not run the fastest; it had two additional errors, but it was free, right? And we did get there eventually, to be fair. So we're going to give Gemini one point.
This is just a simple comparison. These models are both great. We still need to see a little bit more juice coming out of Gemini in order for it to be on par with Sonet. For the longest time, Sonet has been on the top of the leaderboards. It is the most effective model for the most use cases, but that is definitely changing with the release of 01 through the API, which we're going to be looking at in upcoming videos.
And of course, we have 03 mini and 03, so it's going to be super, super wild to see, you know, model combinations and models like this roll out in the future. I wanted to share AER's Architect Mode and prompt chaining again here with you just because it's going to be a very important pattern as we get access to more powerful models and we start really building out our AI agents and agentic workflows in 2025.
I highly recommend you read through Anthropic's blog post here. It's a lot of what we've talked about in the past, but when a big company backs up everything we've been talking about on the channel and writes about it in detail, you know that tells you a thing. On the channel, we are on the right track; we have been for a while, and we're going to continue with that trend.
But also, there's a lot of really key ideas here, one of them being prompt chaining. That's going to be increasingly important as reasoning models allow us to plan and execute large amounts of work. You know, just to kind of call that out explicitly here again, this spec prompt detailed everything I wanted done in detail, right? That change pushed out tokens on both Gemini and Claude 3.5 Sonet in a very, very precise way across multiple files.
I mean, you saw that context window, right? Which is, you know, not something I normally suggest unless it's required by the change. We have over 10 files here, and you know, on both sides, we edited over 2,000 or 3,000 tokens changed. How were we able to do this? We were able to do this by scaling our efforts with a larger prompt, right? This is a theme that's going to be really important as we move into 2025.
The amount of work you can hand off to your AI tooling is going to increase as models improve. So that means we need better patterns like the spec prompt to allow us to pass off more work to our AI coding tools and our AI tooling in general. This technique and other AI coding techniques and the speed and pace in which AI tools and models are improving and increasing is all part of the reason why I created something I'm really excited to share with you right now.
This has been a long time coming! I want to introduce you to Principled AI Coding. Principled AI Coding is my take on the best way to learn AI coding for 2025 and beyond. The whole point, the whole theme with this course is to learn principles of AI coding that will help you, you know, not just stay relevant but excel with the AI coding tools of today and tomorrow.
Okay, so this is my official AI coding course. The link is going to be in the description for you. The theme here is really simple; you can read through this, but I'll just scroll down to kind of the key takeaway here, right? AI coding is the new standard. So 2024, I think, is the last year where if you are not using AI coding tools, it's fine. It's going to be okay. I can no longer say that with confidence. This is going to be the fastest way to fall behind and deprecate your career.
This is a course built for engineers. This is engineering-focused, and the entire goal of this is to really address the fact that we are hitting that threshold point where if you're not engineering with AI, you are not engineering—just plain and simple, full stop. So I created this course to help you and other engineers not just survive this transition but to thrive in a principled-based way.
This is not a course about a specific tool or a specific model, but this is really about the underlying principles of AI coding, of software engineering with language models and AI coding assistants. That's the focus for this entire course, and it centers around mastering the big three: the context, the prompt, and the model. If you've written code with AI, you likely already know this is what matters—your context, model, and prompt.
This is a theme for generative AI. You want to collect context, select the model, and create prompts, and then hand off all the hard work to your AI coding assistant. There's a lot of value here; it's available to you. Just to quickly go over what's in the course, we have eight lessons in here. So we're going to start from beginner, move to intermediate, and then end with advanced.
I can only do this in course form because I have enough time, I have the flexibility, and the control to really create a guided experience for you. So this is all here. I think the price is great. You know, we all want a good deal from someone we trust. Get in here early; the price will go up. I am putting this at the end of the video here for, you know, the true viewers of the channel—the die-hards.
You know, I want to give you guys a big thanks! We just crossed 30k subs last week, I think, and YouTube let me know that we hit 1 million views in total just the other day, which is absolutely insane! So again, I just want to say thank you. I want you to have this course at the best, cheapest price before it goes up. There are some pretty bleeding-edge ideas here that are likely going to become mainstream as people start taking the course and sharing some of the key ideas here.
So anyway, I don't want to preach on this too much. A lot of YouTubers, a lot of content creators, they go the sponsorship route, but you know, that doesn't feel real to me, and I don't want to give you something that I wouldn't want myself. By creating this course, by owning the IP, by owning the ideas behind this, and being able to share it with you, I've created content 100% relevant to the channel and to engineers and to you and me.
This is the course that I wish I had when I started AI coding two years ago. Now, once you sign up, the dashboard looks like this—super clean, super simple. You'll be able to see all of your courses, your course progress, you'll be able to leave a review, report any issues you see, ask questions, and so on and so forth. You'll start out with the first three lessons unlocked, and then once you cross the 50% threshold, you'll unlock the next lesson, and the next, and the next, and the next—all the way to the end.
This is Principled AI Coding. This is my take on teaching AI coding in an in-depth, principled way where we don't just focus on the tool; we focus on the principles underneath the tool—principles, not tools. I think if we look at the ecosystem at a high level, what we're seeing is a new series of models, new reasoning capabilities. We see GPT-3 completely smashing and just saturating benchmarks—just destroying benchmarks into ashes for us.
You know, I want to full circle back to the beginning here and talk about the question that I think about every single day now: How can I use as much compute as possible with powerful models and the best AI tooling, AI coding assistants, to build more software at higher quality, faster than ever? Right? That's where my head's at! You want to be solving more problems at a higher scale, higher rate with these powerful models, right?
We have the 01 series, we have the new Gemini 2 series, we're going to have Llama 4 soon—that's going to shatter the open-source ecosystem once again and kind of reset it. And then, of course, we have whatever Anthropic is cooking up. So our potential is limited by what we can do with these incredible tools. Language models are superpowers now. The question is, how can we access them? How can we tap into them?
And for AI coding specifically, that is a big part of why I built Principled AI Coding. Toward the end, toward the advanced lessons, the question is really about unlocking as much compute as possible, learning to build context the right way so that we can pass off as much work as possible to our AI coding assistant. This is a huge theme. As we say, AI agents roll out more and more, this theme will continue.
Long video today! I hope you enjoyed it. I hope you understand where things are going, and I hope this technique—prompt chaining, AER, Architect Mode—I hope it all makes sense to you. In upcoming videos, we're going to be talking about the 2025 predictions, and we're also going to be looking at the new 01 series that is getting rolled out right now. We have some interesting things happening on that same theme.
The creator of AER has rolled out a brand new benchmark, a polyglot leaderboard that is a much harder AI coding benchmark. 01 is already at the top of this, and you can see the incredible gap between Claude 3.5 Sonet—the current, kind of, or the previous, I guess, state-of-the-art model—and the new 01 coming out of the API. This is really incredible! We're going to be covering this as soon as we get API access. I'm hoping that rolls out over the next week. Really incredible stuff here!
I'm really excited to see some new, harder benchmarks. I'm going to link this blog post in the description as well. Paul, AER's creator, is one of the best engineers to be following right now. His writing and his benchmarks are something that I reference often in order to understand how the AI coding ecosystem is evolving.
We had to pick a tool in the Principled AI Coding course. For me, it was an absolute no-brainer to choose AER because it is foundational AI coding software. Thanks for watching! Next week, we're going to be looking at our predictions for 2025. It's going to be a big year! Like, comment, and subscribe to stay plugged into that. Stay focused and keep building!