Transcription
[music] How can we get cloud code works well as your codebase became more and more complicated without getting to circles and make you feel frustrated?
Once I learned that there are many things you can do to improve performance dramatically through good context engineering. Context engineering basically means how you can optimize what goes into conversation history thread for the cloud code or codeex so that only relevant and necessary information are included here to guide agents next actions.
As we know, coding agents like cloud code at default has a 200,000 token context limit. If you run `/contacts` in your cloud code, you will see a breakdown of token consumptions across different categories. This is a very basic example of what you can do with context engineering optimization because in general, the whole context consists of a few different things. The system prompt, assistant tools, is something you can't really change. It always comes with the agent. So the part you can optimize is the agent has any MCP tools or custom agents that you don't really use but consume a lot of tokens. And the memory file here is basically your cloud.md file. I know as your project became bigger and bigger, sometimes the cloud MD file just became so big that it leaves very little room for the actual message.
So in this very specific example, I can already see that if I remove all those MCPs from this agent, it will immediately give me 2% additional context token window. So this is one very basic example of what you can do to optimize the context window better.
The key part you want to optimize is the actual message, which including the user message you send as well as a list of tool call actions that the agent takes. The key thing here is how can you making sure all the information in the history here is relevant and reduce noise as much as possible. There are many things you can do here. For example, sub-agent is a great tool that you can use to reduce context. So when I look at what happened here, if you prompt agent "help me add a Google O," the majority of the actions agent takes here is actually not about making trends but doing the research to figure out the implementation plan. Those research steps generally took a whole bunch of context in the conversation thread. With features like sub-agents, instead of including all those research-related token consumption in the main conversation thread, you just delegate and offload the whole token consumption related to research to the sub-agent where it has an isolated conversation thread to do just this and return back a summary of the research, which means only absolute necessary information and tokens are included in the main conversation thread.
So one very basic thing you can do is that before you implement a big feature, you can just add a small text that says, "Let's use task or sub-agent to do the research first," so that it can trigger this specific behavior. This is why I often use a compact command much more proactively so that after the agent completes a certain set of fairly isolated tasks, I often just run the compact so that we can proactively clean up the conversation thread.
Methods that I often use is to set up your own documentation system for your codebase. There are many different ways you can do it.
And this is the part I want to dive a bit deeper today. But before we dive into that, one trend has been happening is that more and more people use cloud code to not only do coding tasks but day-to-day operation tasks from handling your emails, using different systems, automating tasks, different productivity apps, journaling. This tells us there's a massive layer to build. Look at what type of use cases people are trying to hack into cloud code then productionize it into AI products. That's why I want to introduce you to this research called "AI Agents Unleashed: A Pragmatic Report" about what kind of things and use cases people are hacking agents into, who those people are, what they are doing, and dive deeply into real-world examples where they interview hundreds of people from top startups and enterprises in the world across marketing, sales, operations, as well as real-world learnings of what actually worked and what didn't, the common pitfalls and challenges that people are facing today. So you can use those information to direct your agenda product roadmap much better.
One of the favorite parts is that they propose a really useful framework to think through if a task and use case can drive huge business value that people are currently paying for. So if you're trying to build a genius, I highly recommend you go check out this research. You can access this report for free in the description below. And thanks HubSpot for sponsoring this video.
Now let's talk about my documentation system for cloud code. The purpose of a documentation system is to create a summarized snapshot of your current codebase. So instead of the agent every time doing deep research across the whole codebase to find information that it needs, it can just read a summarized documentation that clearly pulls all the relevant information together. So it has less noise in the context window and also making sure all the relevant information are feeding into context instead of hoping the agent pulls all the relevant information together by itself.
The question is, what type of documentation is useful and how can you make it scalable as your codebase became larger and larger? So the common structure I have is something like this. I typically just have this dot agent folder that contains all the relevant and useful information that I need.
One is a task folder. This is where I store all the PRDs. If you don't know what PRD is, basically before I implement any feature, I always turn on plan mode and ask to generate an implementation plan. After finishing, I will always store the implementation plan in the task folder. So next time if we are implementing something similar, we can link to those implementation docs as a reference.
Second part is the system folder. So the system folder contains things like project structure, the database schema, your APIs, or some critical and complex parts of your codebase. Those things can go across different individual PRDs and are really useful for the agent to get an overall understanding. This is also the part that it can grow bigger and bigger.
Third is SOPs. This is where we will log the standard process for doing certain things or mistakes you saw the agent make. So after I get the agent to do something, I will ask it to generate an SOP for this specific process. It can be as generic as like adding a new database table, what are the list of actions you should take, to something more granular like how to integrate a new replicate model.
And last but not least, also a readme file. So as your codebase became more and more complicated, there'll be just so many docs. The readme here is almost like an index of all documentation files you have and when to read which documentation file. So the agent can quickly get an overview of all the relevant documentations and read injection history. So this is a common documentation system I often use.
Meanwhile, I also use the update doc command quite frequently. What it does is that it basically includes certain instructions to the agent about what to do when initializing the documentation structure as well as when I need to update documentation. So I will run this command after I implement certain features, or I can use this update doc command after we help the agent correct certain mistakes so we don't make the same mistakes again. Then it will create this SOP that clearly lists out all the step-by-step processes and also the related documentations so the agent can consume more relevant information. This SOP doc will also be included in the readme.md. So this is a pretty common structure I use for my projects and continuously maintain and update them.
Obviously, the best version of a documentation system is always something tailored to your codebase. For example, I talked to Simon from AI Builder Club where he showcased how he builds a whole documentation system with his team for their Lexi codebase. This is the pointer doc, as it were, that points to everything, how we do all sorts of things, and you know, best practices, etc. [laughter] And so this is very human-readable, obviously, as well, but it was designed more for our LLM friends. We document all the ways we do our like migrations, how we store the data. We got a whole class of things with methods for helpers, updators, utilities, all sorts of stuff like that. We've sort of split out all that, all this stuff into well-documented sections.
"The very first versions of these we did inside Claude. We got Claude to generate the MD files and then we started using Cursor and we've got our own little uh this thing here."
"All right."
"What, what are they, these things are called?"
"Now Cursor now has got the slash commands. I haven't yet experimented with turning this into a slash command because I've always found this rather frustrating that we've only got so [laughter] much room to see what's going on in there."
"But it's not complicated. We go through and then we say, 'Review this class' and upgrade the inline documentation. So we do that first so that then the class has got the best possible inline documentation and then we'll use a a prompt in the prompt interface to generate the class documentation in more detail. So there's this workflow that goes, generate or update the inline documentation and then output that into the bigger documentation. So I've just thrown all the LLMs at this to sort of get as best as we can. It makes the whole process easier to manage by exposing all of that."
"And let me give you a quick example of how do I do it. If I start a new project, let's firstly create cloud.md file. I just added one section for docs specifically explaining the doc structure we want the agent to follow and then give a rule that we always update the agent folder docs after we implement certain features and before we plan any implementation, always raise the readme first to get the full context. Meanwhile, I will also create a doc cloud folder commands update doc.md. This is a simple prompt that specifically telling the agent about the doc structure and what to do when we start initializing the doc and what to do when we ask to update doc, as well as some very specific rules when it creates new doc files."
"I'm going to show you how we use this setup to continuously update docs. So I firstly do `/update doc initialize`. This will race through my current project and try to set up a dot agent folder structure. And once it finishes, you will see that it creates this agent folder and sets up a project architecture as a first doc. And there's also a readme file under the dot agent folder which lists out all the docs that we have."
"Now let's just take you through an example of how do we use this. Let's say we want to build a basic app that can do text-to-image generation using a model hosted on replicate. I'm going to copy over a doc and give a prompt: 'How we build a text-to-image app using model above?' and I'm going to use a plan mode. And after I finish the plan, I'm going to prompt 'Save implementation plan in a do agent/task folder and start implementation.'"
"Great. Okay, now we have this text-to-image model working. Quite often in my experience, models will fail to integrate replicated models out of the box. That's where R can use this update doc command saying, 'Generate SOP integrating replicate model.' Then what it will do is that it will create this replicated model integration SOP. In this doc, it will explain specifically the step-by-step integration process, the directory structure should follow, and also there's a section called 'Related Documentations.' So for this one, it probably didn't make perfect sense, but you can imagine if your documentation became more and more complicated. This can be used as a directory to lead to all related documents. And in the end, it will also go update the readme as well to include more stuff."
"So now I can even clear the conversation without giving too much context and give a different model, which is text-to-video, and give a prompt: 'Help me add text-to-video capability using model above. Plan implementation and read do agent doc first for context.' Again, I'm using the plan mode. This time you can see it reading through different files to get the full picture. And we're going to do the same thing. Save the implementation plan to /task folder and start implementing."
"With this one, it was able to one-shot this video generation model, and it is fully functional without any errors. So this is how you can imagine this whole documentation system works. You can have higher confidence that I can just come in any time and start implementing something with pretty consistent performance."
"You can do the same thing for your existing codebase as well to ask cloud code to start generating the system architecture documentations, which can include text project architecture as well as database schema, API endpoints, and many more."
"If you're interested, we have a specific session in AI Builder Club where we include detailed instructions of the cloudd and update do commands that you can copy-paste as well as a 1-hour workshop where Simon dives deeper into his specific setup. So you can learn how others are generating documentation systems for big and complex codebases. I have put the link in the description below for you to join AI Builder Club. So feel free to click and join if you're interested."
"I hope you enjoy this video."