Transcription
You know that feeling? You have 1000 ideas in the back of your mind about what you could develop. Small app ideas, tools, little helpers, etc. But most of the time you lack the time to tackle these things because you are busy with your main job or your main task. Today I am showing you a way how I proceed to have my ideas implemented by an agent while I am working on completely different things or even sleeping. The next sessions last up to 45 hours. In addition, I want to show you how I can review the code that was generated so autonomously without going through every line. I personally use this to have two to three auto types, POCs, whatever, developed on the side each month and then look at them. As an example, I would like to extend an existing skill. A skill I like to use is the Watch Skill. With this skill you can make the LMS Video accessible, or rather, this cloud or Codex can watch the video and issue recommendations. I like to use it to have YouTube videos reviewed according to my preferences. Yes, the LNs know me based on memories, and for that reason it makes sense to ask whether the video is relevant to me or not. Let me give an example here. This YouTube video is about the use of AI to predict the World Cup, football, World Cup results, winners, and so on. You could e.g. say, is this a hype video or does it also have substance? If yes, please tell me at which timestamps it is interesting. Okay, and here we now see the result, which is basically a demo video from the sponsor Hostinger, but that the middle part explains the architecture, the agent architecture, relatively cleanly, and we have some timestamps that are worth it here. Okay, there we also have a conclusion. So in the conclusion we see, if the WMT does not interest me, then I just see timestamp 242 with the architecture here, then the Memory Pad, also the learning loop would be three and a half minutes that would actually be interesting for me. A fundamental efficiency problem is that the skill always downloads the entire video by default here, but that is not necessary in 99% of my cases, because most of the time it is sufficient to first check whether YouTube has already generated a transcript and then use that. And only when that is not available do you fetch the audio track and generate a transcript yourself. However, the entire video itself should only be downloaded if the LM wants to extract frames from the video to support my explanation, for example. And exactly this adjustment I would now like to have carried out autonomously. That means the transcript path is basically the standard, and yes, the other, the heavy video path, is then just a special case. And to implement long-running autonomous developments, I like to use Factory Droid. That has a feature called Mission Control. In principle, you hand over a specification. Droid asks a few more questions about the specification, and afterwards it generates a lot of documentation, verification dates, and so on. It splits all of that into features, does parallel research, and so on. When the initial plan is clear, it starts autonomously and then works until the goal is reached. Let's take a look at all of this here. I am here in Droid and can view my previous missions via the command Missions. Yes, these are my current missions, and I see e.g. the transcript first / watch mission and can then open it and arrive at this overview, where on the right I see the features that have all been processed. I see the log very abstractly here. I can go into each individual feature, can see which models were used, which workers, and so on. Control T, I can leave this view and scroll all the way up. Exactly. And then here you see the plan overview that was created, what it plans to change, all the milestones and so on, plus all the acceptance criteria. We have six acceptance criteria here that it must fulfill to complete the whole thing. I then only approved the mission here, and yes, after that it worked for three and a half hours as mentioned. We can also see that again; if I go back to the mission with Control T, we see on the top right it worked on it for three and a half hours. To review code that was generated after an autonomous run, I like to use Windsurf. Holdmaps offers me the possibility to gain insight into the code and the processes via a kind of abstract stack traces, right? One problem is that Code Maps has no way to access the git history. That's why I use a workaround here. So you see here several commits have been generated. I simply ask the model to summarize the changes of each individual commit in a file as git. You can see that on the right side. J C changes.d is then the file that contains all changes. Within the Code Map I can refer to this file Change.d and ask what has been implemented here. I am now in the current Code Map and you can already see what has been done. This fork inverts the default mode to Transcript only with opt-in for the frames for visual analysis. But the great thing is that you can now go into the individual use cases and drill down like a stack trace. For example, here we have the description of the transcript only mode, where it does not download any video. If I open that, I see here that it uses Main as the entry point and that when the transcript mode is activated, it calls this function here. If I click on the point here, we go directly to the code and I see a further description that Main calls this method when minus minus or dash dash Frames is not set. Yes, and that gives me a nice overview and I see that when the corresponding YouTube URL was passed, it downloads the captions from DJ. I see that it only performs the Whisper fallback when the boundaries. In that case it also only downloads the audio only files and at the end it generates its report. I can, for example, click on entry 2C and see directly with which parameters the tool was called. I can also ask questions. For example, when I was looking at the Frames mode, i.e. the one that downloads the video, which was already available as standard before, it wasn't entirely clear to me whether it overwrites the video when retrieving it or reuses it. That means, if the NLM wants to retrieve 3 frames, whether the video is then downloaded three times. You can ask a speaker question here by going over the line and then simply clicking on the speech bubble and asking a question here. I'll just open the corresponding chat for that now. You can see at the very top, if the Frames mode is called multiple times, the source is only downloaded once or overwritten. I scroll down to the conclusion and see that only if the parameter the dash out dir is set is the video shared. Otherwise it is downloaded again and again. That means in our case, it depends on how the LM uses this skill and how the parameter is documented in the skill whether the video is downloaded once or multiple times. For that reason, I had that no BMV video downloaded again from before, but I said that it should extract two images. One for the MCP overview and one for the VS Hosting architecture. It did that, and we see here that we have the two images here from the video. One of how hosting works, one image for the MCP overview. If you look at what errors it executed, we see that it uses the dash out dir parameter here and created a temp directory. And in this directory the video is then shared. The directory is called in this case Watch - VPS Instance Cloud - 1000. For the second frame we see here it uses the same directory. That means it really only downloads the video once. In principle, this is the workflow that I like to use to implement some prototypes, ideas, and so on in parallel with my main work, and also to understand at the end what the agent actually implemented. As for the economics, to say that Devin Desktop costs 20 € per month. Factory AI Droid, i.e. the Mission Control functionality, is also available for 20 € per month, but for long-running processes you actually have to go for the 200 € version. The HBJs are set similarly there to Cloud Code or Codex. That means you have a 5-hour window, you have a weekly window and a monthly window for the budget. When the budget for the respective window is used up, the agent interrupts and you have to manually nudge it again to continue working. That means the whole thing is only worthwhile if you really use it and then develop four or five POCs via these agents. So, I hope you found that exciting. I would say I am a better developer than a YouTuber, but I hope it was still useful for you. Please let me know what else you would like to hear, where I should perhaps go deeper. Otherwise, feel free to try it out and yes, see you next time.