Transcription
We built over 150 AI agents under the new "agents as a service" model in the last 9 months. In this video, I'll share 26 key takeaways we learned the hard way—costing us dissatisfied clients, time, and money—so you don't have to repeat our mistakes. Let's dive in.
Key takeaway number one: AI agents are not your employees. Today, everyone still seems to call AI agents either automations or employees. Unfortunately, agents are neither. The difference between agents and automations is that, in automations, every single step is hardcoded. This means you know the exact steps and their sequence in advance. The difference between agents and employees is that agents have less autonomy than employees, meaning you typically need way more agents than employees. Agents require training on exact instructions to perform a process manually; they can't just look at your SOP and learn by themselves using trial and error—unfortunately, not yet. Instead of thinking about agents in terms of roles, think of them in terms of SOPs. Typically, one agent can handle one standard operating procedure well, while one employee typically handles five or more SOPs.
Which brings us to key takeaway number two: Start from well-documented processes. SOPs are standard operating procedures—the processes employees perform in a business. In a good business, these processes are typically well-documented. By finding these well-documented processes first, you can make training an agent significantly simpler. Instead of collecting all the data manually or asking tons of questions from your client, you can simply take that SOP, which will most likely have everything you need to train the agent to reliably perform this process. Find these onboarding materials, find SOPs first, and then go from there.
Key takeaway number three: Business owners will never build their own agents. I believe that even when we have agents that build other agents from a single prompt, business owners will still never build their own agents. Just like no-code tools promised the end of software developers but instead started a wave of no-code developers, and just like automation platforms promised the end of backend engineers but instead started a wave of automation engineers, agent platforms and frameworks that can build other agents will only spike the demand for AI agent developers. As OpenAI and other labs keep releasing new agent capabilities like the Operator and the Deep Research agents, it's not building agents that will be the hardest; it's determining which agents to build—which is exactly where AI agent developers come in. So don't worry about agents building other agents; business owners will always prefer to entrust this to someone who knows how to work with AI agents best, even when the process becomes extremely streamlined.
Which brings us to key takeaway number four: Business owners have no idea which agents they need. Many of our clients, when they start their subscriptions, come to us with ideas for which agents they want to build. But in around 50% of cases, this is far from the most valuable agents we can build for that business. This is why consulting is a huge part of our service. To determine the best agents to build, we typically start from a customer journey. We ask our clients to help us map out their customer journeys on Figma, step by step. This gives us a great idea of which specific parts of that customer journey we can automate. If we see potential, we dive deeper into that specific part of the process and map out that process by itself. By doing this, you can typically find opportunities that are not only easier to build but also bring significantly more value to that business. Don't listen to your clients if they have some ideas for which agents they want to build; use it as feedback. Don't assume it's the best possible idea you can integrate into their business.
Which brings us to takeaway number five: You don't need 20+ agents. Right now, on YouTube and in our community, people seem to be building as many agents as possible. This only makes your systems more complex. By adding so many agents in a single system, you make it harder to maintain, increasingly more complex to debug, and increases the cost and time it takes for your agents to provide a response. Start with as few agents as possible—preferably one, the smallest agent you can deliver to your client as fast as possible. Once this agent is fully fine-tuned, deployed, and the client has tested it, then you can proceed to adding more agents as needed.
Which brings us to key takeaway number six: Data-driven decisions, but data with actions deliver results. There's a saying in data science: GIGO (garbage in, garbage out). This still holds true for AI agents. If you provide your agents with trash inputs, they're going to produce trash outputs. However, we recently discovered that the biggest impact doesn't come just from adding data to an agent; it comes from combining that data with relevant actions. By combining knowledge (like how to create effective Facebook marketing campaigns) with actions (that allow the agent to control the Facebook marketing API), you can achieve significantly higher results than by using either data or actions alone. By combining knowledge and data, the agent can provide suggestions on what to improve and how to perform the process best. Make sure to scrape both internal and external sources; this will significantly increase your agent's performance.
Which brings us to key takeaway number seven: Prompt engineering is an art. Honestly, prompt engineering is already a real job, even though many companies still don't recognize it as one. As these models become larger, smarter, and run for 10 minutes or more, prompt engineering is becoming more important than ever before. Right now, prompt engineering is like an art. You have to write your prompts like you would blog posts or essays; think carefully about every single word. As the models get smarter, prompt engineering will become more and more important. Here are some tips for writing effective prompts: Tip number one: Provide examples. One example is often worth a thousand words. Number two: Order matters. The order of your sentences or paragraphs makes a large difference. We often had agents where simply rearranging the prompt changed performance from completely unreliable to consistent and reasonable. Make sure the most important parts of your prompt are at the end of the instructions, not at the beginning, because large language models remember information closer to the latest message better. The last tip is to iterate and test constantly. Often, the only way to determine whether your prompt works is by iterating and testing it. Do not modify your prompt without testing how it affects the performance of the agent. Run your agent as often as possible and then adjust the prompt accordingly.
Which brings us to key takeaway number eight: Integrations are just as important as functionality. We tend to overfocus on agent capabilities; however, integration (where the agent is working) is often even more important. If it's not convenient for your users to use an agent, it doesn't matter how powerful it is; it still won't provide any value. Integrate your agents into the same systems your employees use daily. If you are building a customer support agent and your client currently uses Zendesk, the agent must also work in Zendesk.
Which brings us to key takeaway number nine: Agent reliability has been solved. Today, many startups are trying to solve agent reliability. In our agency, if the agent is not reliable, it's not the agent problem; it's the developer problem. The person responsible for solving agent reliability is Jason Leo. Back in 2023, he released a legendary video called "Pydantic is all you need," and he recently posted an update—and guess what? Pydantic is still all you need. He figured out that you can use Pydantic, a data validation library, to validate all agent inputs and outputs. This means that if you added all the necessary validation logic, the agent literally cannot screw anything up; it cannot take any action that would cause major consequences because it should be prevented by the developer. Make sure to check out his video and how it's implemented in his library and in my framework because, honestly, with Pydantic, you can now build agents for literally any use case.
Which brings us to key takeaway number 10: Tools are the most important component when building AI agents. The three most important components are instructions, knowledge, and actions. Around 70% of the work in our agency goes into building actions (the tools). Tools are how agents provide value. With standard chatbots or LLMs, value is generated through responses; agents generate value through actions. Agents must execute tasks; they shouldn't just tell you what to do or provide a response. They should actually do it for you. Actions are the most important component. If you know how to build and structure your tools well, you can build agents for any use case.
Which brings us to key takeaway number 11: No more than four to six tools per agent. This is our rule; we're not adding more than four to six tools per agent, depending on their complexity. This has proven to work best, although it depends on the complexity of your tools. With GPT-4, Claude 3.5, and the latest GPT models, you can't add more than six tools because the agent starts to hallucinate—confusing which tools to use or the proper sequence. If the agent starts to hallucinate, you need to split it into multiple agents.
Key takeaway number 12: Model costs don't matter. Recently, other companies made a lot of noise about costs, and it's impressive what they've achieved. But honestly, we stopped caring about model costs a long time ago. If your use case makes sense, you will almost always make a tremendous ROI from using an AI agent compared to performing the same process manually. If you focus on the right thing, you no longer have to worry about costs. Check out my previous video with five agent case studies; for example, for one agent, the process for filing questionnaires was reduced from $300 and 3 days manually to $1-$2 and about 20 minutes.
Which brings us to key takeaway number 13: Clients don't care about which model you use. Many are surprised when they hear that we never used an open-source model in our agency. Businesses don't care which model you use as long as your use case makes sense and you provide value without violating their customer policies. If our customers have strict data privacy policies, we use Azure OpenAI, which runs OpenAI models in your own private Azure instance without sharing data with OpenAI itself. OpenAI is still our provider of choice because of their developer experience; it saves us significant time developing agents because of how convenient their API is, which simplifies agent creation.
Which brings us to key takeaway number 14: Don't automate until value has been established. We primarily work with existing businesses; sometimes people want to automate a business that doesn't even exist. They want to build a process, not from establishing it manually first and then automating it, but from guessing that automating it will create more value. This is extremely risky because you don't know if the process will work and it requires significant investment (development costs, not model costs). You need to first establish value for a given process; maybe hire someone on Upwork to ensure the process actually works. Once you determine how the process should be executed and that it provides value, automate it with AI agents.
Which brings us to key takeaway number 15: Don't think about use cases; think about ROI. Our ROI formula is: (rate * amount of hours) - operational costs / development costs. (Rate * amount of hours) means the rate of an employee performing this process times the total number of hours all such employees take. Operational costs are typically just model and server costs, which are negligible. Development cost is how much it takes to develop your solution. For example, if an employee performs a process for $50/hour, spends 10 hours/week on it, and the development cost is $5,000, a year later, the ROI will be around 5.6—five times the return on investment. Focus only on what provides the most value for the business.
Which brings us to key takeaway number 16: Agent development is an iterative process. In data science competitions like Kaggle, the team that tests the most parameters and model architectures wins. The same applies to AI agents. I often get questions like, "How do you know how many tools to add per agent? How do you know how many agents you'll need?" Most of the time, you simply need to try it—try as many different architectures as possible and compare them side by side to see which one wins. Only after experience building agents will you start to see which architecture makes sense for a given solution. If you are confused or your agents are underperforming, build a couple of variations and test which one works better.
Which brings us to key takeaway number 17: Use a divide-and-conquer approach. This is breaking down a complex problem into manageable tasks. We use this approach for almost every agent. Deliver solutions incrementally. Instead of building an entire solution and realizing it's not what the client wanted, split it into manageable components and deliver each one by one. Find an agent in a whole system that can work by itself; build that agent first and deliver only that agent. Only after the client confirms it works, proceed to building the entire system. The same applies to what you automate in a given business. For example, automate by departments first; focus on one department and automate as many solutions as possible before transitioning to the next. This allows you to combine some agents in that department, making your system significantly more powerful.
Key takeaway number 18: Evals are a big deal, but only for big companies. Evals are evaluation metrics you set up for your agents to track KPIs and performance over time. Evals can be extremely effective to eliminate competition because they allow you to continuously improve your solutions. If you have set up correct evals, anytime you deliver a similar solution to a new client, all your previous solutions get better as well. Additionally, as AI models are heading towards self-improvement, evals will later allow your agents to possibly even self-improve over time. However, SMBs might not need evals because they don't have consistent traffic of requests; it doesn't matter if the solution is 5% worse. Evals only provide incremental results; you can get to 80% performance without evals, and the last 20% is where evals come in. For SMBs, it doesn't matter as much because the traffic of requests is much smaller. They might only use the marketing agent once or a few times per day, so they can easily get away without evals at the beginning. But if you are working with a large enterprise client, definitely recommend setting up evals from the very start.
Key takeaway number 19: There are two types of agents: agents and workflows. Yes, there are agentic workflows. I previously only focused on building agents for my framework, and currently, we don't even support workflows. But we found that sometimes there are processes defined by rules—processes where the steps are predetermined and the exact sequence must be followed every time. However, the steps themselves can be agentic. For example, if you're performing a lead research process, you might need to send three prompts to Google (the same prompts except for the company name), and the agent must perform the search on the exact same websites for these companies. In this case, you can combine standard workflows and automations with AI agents. You can have a workflow where each step is agentic—not the whole system, but specific steps requiring agentic capabilities. Crei, for example, is a workflow platform where you send tasks to an agent; you hardcode them and send them one by one. My framework is fully agentic; there's no way to send tasks. We did encounter a few clients (a very minor percentage) that want to perform the process in the exact same way, and the steps are identical. For this, we will be releasing a workflows feature very soon.
Which brings us to key takeaway number 20: Agents need to be adaptable on feedback. The whole point of building agents is that they need to be able to interact with their environment. If your agents interact with the environment but can't get consistent feedback, they're going to be confused. When building agents, don't just add tools that allow them to modify their environment; add tools that allow them to analyze their own results and how the impact of previous actions affected their environment. For example, don't just build an agent that can only update a database; add a tool that can read the database records to ensure the task has been completed successfully.
Which brings us to key takeaway number 21: Don't build around limitations. This is one of the biggest mistakes. Build your agents with the understanding that these models will get better and better. For example, in 2023, we built a complex system to avoid context token limits, but then OpenAI released the 128k context model, and two months later, this system became obsolete. The same might happen in the future. OpenAI is releasing agents like the Operator agent, Deep Research agent, and previously, they released RAG (file search). By releasing these agentic capabilities, they made a lot of startups obsolete. Before, there were tons of "chat with PDF" apps; now there are almost none. Don't build obvious general use cases; the next one might be software development. Don't spend a year building a software developer agent if it's such an obvious use case that OpenAI might develop it themselves.
Key takeaway number 22: Deploying agents is a lot harder than building them. We started building agents using my framework, but we figured out that it would take us 2-3 days to build an agent, but another 3 days to deploy it (integrate it into our client processes). This is why we decided to build our own platform. If you want to sign up for a waitlist, use the link below. It's the only platform we would personally use for deploying agents because no other platforms give us enough flexibility to do so in production. Many were built before agents were even a thing and were simply jumping on the hype train.
Key takeaway number 23: Waterfall projects don't work. We only work on a subscription basis because agentic projects are too agile. You cannot scope agentic projects for 3 months because they constantly evolve. When we start working with clients, we frequently find more opportunities we hadn't foreseen. You can start from one-time fees, but later transition to a more agile service where you work as their partner. The goal with agent development is not just to build an agent; it's to automate a business and provide value to your client. Have an agreement with your client where you have enough room to do so.
Key takeaway number 24: Include a human in the loop for mission-critical agents. Sometimes the margin for error is so low that if the agent makes one mistake, you can't reverse the results. For these agents, include a human in the loop. For example, you don't want to spend $110,000 on a less-than-ideal marketing campaign. In a previous video, we included a human-in-the-loop step where the client could review Facebook marketing campaigns in Notion first. Later, when the agent is fully fine-tuned and the client consistently approves every campaign, you can remove the step.
Key takeaway number 25: 2025 is the year of vertical AI agents. Vertical AI agents are specialized for a specific use case. Just like in B2B SaaS, vertical agents serve only a specific type of business customer. The same principles apply with B2B SaaS; it's much easier to scale, you can charge higher pricing, and you know your customer. You can fine-tune the agent to solve a specific problem for a specific business. However, do not start from vertical agents right away. It's fine to build a few horizontal agents for the same industry you plan to scale in. After you've built a few, you'll start to notice similarities—these similarities are what you can turn into a vertical AI agent that can be productized and adopted for many different businesses in this industry.
Key takeaway number 26: Agents don't replace people; they help businesses to scale. We've never seen a business owner fire people after their business became more efficient by automating. We help business owners think bigger; they scale faster, achieve higher revenues and profits, and their employees can focus on higher-level tasks they enjoy. Don't worry about agents replacing people; humanity will always find better things to do, leading us into a new age of abundance and prosperity. Thank you for watching. I'll leave all the videos I mentioned in the description, and don't forget to subscribe.