How to Build Effective AI Agents (without the hype)

The whole world is raging about AI agents right now. Yet, some of the biggest companies, like Apple and Amazon, still struggle to ship effective AI features within their products. Last week, Apple had to pull back Apple Intelligence because it was hallucinating in the new summarizations that the product was providing. And also, Amazon still struggles to put AI features into Amazon Alexa because of the hallucinations. Yet, if you look online, on YouTube, blog posts—everyone seems to be building these AI agents, and everyone has their own ideas, tools, and frameworks on how to do so. But here's the hard truth: building effective and reliable AI agents is really hard. And most of the examples that you will see online are really cool demos, but they are just that: they show what's possible; they show where the future is going with AI agents. But if you really put that into your product and let a lot of people use it, it will just simply break down.

Now, in this video, I want to share some practical tips and techniques for developers to build more effective and reliable agents. Now, while I definitely don't have all the answers here, I'm going to share some of the lessons that I've learned over the past two years building AI systems for our clients. And I'm also going to share insights from some of the leading companies working on these technologies.

Now, if you're new to the channel, my name is Dave Abar. I'm the founder of Data Lumina. I hold a bachelor's and a master's degree in artificial intelligence, and my journey just started about 10 years ago. And for the past six years, I've been building custom data and AI solutions for my clients. Next to that, I also run a community with over 100 freelance data AI developers. And I make this video to help you level up and become a better engineer, so perhaps eventually you might want to join us. All right, so let's first talk about what AI agents actually are, because before we can build them, we first need to all be on the same page about what they are, right? And depending on who you ask, you are going to get a very different answer most of the time. And that is because a lot of people have different ideas about what they are. And if right now you'll search for how to build AI agents online, you'll find some tutorials. Almost all of the tutorials that you'll find where they talk about AI agents, what they simply mean is you have some piece of software, you have some operations, where at some point you're going to make an API call to a large language model. But now, is it really fair to say that such a system can already be called an AI agent? Well, if you ask the experts, the answer is no. Yet, why is then everybody talking about AI agents like they are just the thing? Well, that is simply because there is a lot of hype and a lot of buzz around that particular word. Everyone wants to learn what AI agents are, how to build them, but really, in the end, what they're really after is they want to learn or they want to implement some kind of system that can take some process and automate it. They want an automation; that's essentially what AI can do for us. And right now, anytime that topic comes up, we say, "Oh, it's an AI agent." But for this video and for you as a developer, I want to dig a little bit deeper and show you some of the different tools and techniques that you can use to what we refer to at Data Lumina as AI systems rather than AI agents, where not all AI systems are necessarily AI agents.

And now, to make this more clear for you throughout the rest of this video, I want to use a definition, or distinction rather, as introduced by Entropic, because this makes total sense to us, and this is exactly also how we see the distinction between the different types of AI systems that you can make. So, so in this excellent blog post called "How to Build Effective Agents," under the section "What Are Agents," they talk about first how a lot of people have a lot of different ideas about it, but then at Entropic, they make a clear distinction between workflows and agents, where workflows, quoting, "are systems where LLMs and tools are orchestrated through predefined code paths." So this really aligns with what I was saying previously and what you often find online, where you have a certain system, some steps, some change, and at some point you make a call to an LLM. Now, agents, on the other hand, are systems where LLMs dynamically direct their own processes and tool usage, maintaining control over how they accomplish the task. So there's a clear distinction between the two, between workflows and agents. And as a developer, it's crucial to understand when to use which pattern. And because of all the tutorials and information out there right now where everyone is using the word agents, thinking that you need all kinds of tools and frameworks in order to build agents, there is a lot of confusion. If we then come back to Entropic's blog post and to the section "When and When Not to Use Agents," and I, in my opinion, this really hits the spot. So as a developer, consider the following when building applications with LLMs: We recommend finding the simplest solution possible—something you should always do as an engineer—and only increasing complexity when needed. This might mean not building agentic systems at all. And here's the thing: for many applications, however, optimizing single LLM calls with retrieval and in-context examples is usually enough. And we've been building AI systems for our clients across industry since the day the ChatGPT API came out, and I can definitely tell you this is true: for many applications, you don't need these agentic patterns. You can build predefined, simple workflows that solve a particular problem really well and then create a suite of tests and evaluations around it to really keep it in control and to really optimize it over time.

Okay, so then how do you actually build effective AI agents, or rather AI systems, I would say? So the first step for you as a developer is really deciding on what you are going to use to build your AI system. Now, if you have coding skills, you probably want to use something like Python or TypeScript or even JavaScript. And if you don't have coding skills, you probably want to look at something like Make.com, n8n, or Flowise. And it doesn't really matter what tool you use, like if you do full coding or use these uh workflow builders, which they essentially are—you, in both cases, you can build reliable systems. It's much more about the underlying patterns that you use to control the flow of your application and your data. So that's what I want to dive in right now, coming back again to the excellent blog post from Entropic, where they outline some of the different patterns that you can use to build applications around large language models. So here are some of the common building blocks that you can use when building AI systems, whether that's workflows or agents. And the basic building block that we start with, that you all start with, is what they call the augmented LLM. So we start with an LLM, a simple API call in the beginning, and we can augment that; we can enhance that by focusing on three things: the first one is retrieval; then we have tools; and then we have memory. Starting with retrieval, this is where your AI system pulls information from a different source, typically a database or a vector database, and makes that available within the context of the large language model. Now, in practice, typically this is done through retrieval augmented generation, or RAG for short, using a vector database where you perform a similarity search. People often compare this to giving your LLM application long-term memory, because you can essentially offload all of the relevant contexts that you need to affect your database, and whenever the LLM needs it, you can try to retrieve it. And I'm saying "try" because with retrieval augmented generation, or RAG, you never truly know what you're going to get back, especially as the scale of your uh database grows. So the second augmentation of LLMs is what they call tools, and tools are essentially little services or APIs that you can call within your application in order to get more information. This could be, for example, making an API call to get the current weather data or to get the latest shipping updates on your parcel using the uh tracking number. And then the last part here is memory, which in this context simply refers to the past interaction that you've had with the LLM system. So you can think of this, for example, when you're talking to ChatGPT; every time you send or ask something to the model, that is then a new record; the model will then respond; that is a new record; and the whole chain, the whole sequence of all of those interactions together is what you call the memory. So these are the three common concepts that you'll encounter when you're working with LLMs and creating these agents in order to make the API call to the LLM better. This is all about enhancing the context, providing more context in order to get better results. And when you combine these three and put them together in perfect harmony and in sync, they take your application beyond just being a simple OpenAI or ChatGPT wrapper and really taking your app or automation to the next level, just because it can get all of the right context at the right time.

All right, and then up next, let's look at a pattern that you can use to build very effective workflows, and often this is all you need, and this is called prompt chaining. So this is simply just chaining together multiple calls to an LLM, and typically using the previously generated information and then passing that to the next LLM call within the sequence. And in this way, you can break down a complex problem. And instead of asking the AI, "Write a blog post," you can break that down to: First, do some research and get clear on some ideas; then, dial in on a specific topic; then, provide an outline of what the um the essay or blog post could look like; then, give me all of the uh different chapters; now, write chapter one; now, write chapter two. And every one of those steps could be a a chain within the whole application, where at every step you now have more control because at every step there is data and a prompt that you control and can tweak, and this is guaranteed to improve your application and to make your system smarter overall.

All right, and then let's get into the second pattern: routing. So while prompt chaining can already get you great results if you focus on a single problem, if the scope of the problem that you're trying to solve grows bigger and there are multiple scenarios, multiple cases, multiple solutions, that is where routing comes in. And what you essentially do with routing is, given all the the data and context that is coming in, you let the LLM decide which way to go. And you could clearly instruct the LLM for the first step to categorize the incoming requests: So is it A or is it B? And we then capture that in a structured way, so then within our application, within our control flow, we can use the so-called routers, which in theory are practically just if statements or cases where we make a certain match, and if output equals A, we call that particular function with then, which then goes into a particular direction, and if the output was B, for example, we go in a different direction. So that is routing.

And then real quick, if you're a developer, you got some technical skills, and you've been thinking about starting as a freelancer, maybe taking on some side projects to make a little bit extra money or learn more, but you don't really know where to get started or struggle to land that first client, you might want to check out the first link in the description. It's a video of me going over how my company can help you with that. We have a community with over 100 freelance data AI developers, and we're all here to make more money, work on fun projects, and create freedom. So if that's something you're interested in, you might want to check it out.

All right, and the third workflow pattern is parallelization. And with parallelization, you also make—similar to a prompt chaining—you make multiple LLM calls, but rather than doing doing them sequentially where one output might depend on the other one, you do them in parallel. And this is ideal for where you can really split up a certain uh certain task, break it down, but they are independent of each other. And this can help you to speed up your application because the output would be the same if you sequence them sequentially, but then you need to make multiple API calls and wait for that to finish, then do the other one, do the other one. This is parallelization—is a way to do this async. So a typical example of this could be when you're implementing guardrails and you want to evaluate a certain output; you might have one prompt that evaluates accuracy or correctness; you might have one prompt that evaluates uh harmfulness; and you might have one prompt that is specifically targeted at uh capturing prompt injections. So this can all be done in parallel; it could then come together, and that could act as your guardrail system.

All right, and the next pattern is the orchestrator-worker. So this is still a workflow pattern, but this this already gets a little bit more agentic because it requires a little less explicit programming of steps, but it's still sequential and linear in nature and therefore also really predictable. So here's an overview of what that looks like visually. So another example of how you might want to use this: Let's take customer care again, and an email from uh a customer is coming in, and uh you let an LLM look at all of the contexts—so the customer question, you have the CRM data, and maybe you also have some some other order data, and you have the uh the the customer care guidelines—you assess all of that and then ask an LLM what's required in order to solve this particular problem. And the LLM might decide, "Okay, we need to look up and assess the right uh section within the customer care playbook; we need to do a look up of the particular order to check the status; and we also need to uh call the shipping API in order to get the latest shipping information." Through this pattern, we can make our application already more agentic and have less hardcoded uh pathways in there.

All right, and then the last workflow pattern is the evaluator-optimizer. And this is simply where you let an LLM create an output and then feed that into another LLM call to review it and give feedback and then pass that to another LLM to improve that. So you could say, "Write a blog post." You have the blog post; then you have another prompt that says, "Critically review this blog post and make sure it adheres to XYZ," a whole list of things that you would like to have there; then, give a detailed report of what we can improve. The LLM will do that; you now have a list of feedback points; and we feed that back to the LLM: "Here's the original output; here's the feedback; now process all of that."

All right, so that were the workflow patterns, but now let's look at what an agent pattern actually looks like, and here is that displayed visually, where we have a request from a human, in this case; we make an LLM call; that LLM is going to decide to take a certain action; it's going to check and assess the output of that action within a certain environment—so this is mainly just the data that it has access to and the outputs—and it's going to provide feedback back to the LLM, and it does that in a loop until it reaches a certain criteria, either uh completing the tasks or a certain stopping criteria or an intermediate step where it's going to ask feedback from a human in the loop. So as you can see, this is really agentic; there is almost no like hardcoded steps in between, just an environment and instructions and tools that an LLM can use, and it goes off in this loop. And that is one of the key distinctions compared to all of the other patterns that we've described, where a workflow has clearly a start point and an end point, whereas with a true agent system, we don't know all that; we know is that we have given it specific instructions to operate within a certain environment to reach a specific goal, but if it's going to do that on the first try, on the second try, on the 100th try, or never, because it just keeps iterating, stuck in a loop, we don't know yet. And with this specific pattern, agents can really handle sophisticated tasks, but the implementation is often very straightforward; we don't have to go to the drawing board and draw out a huge like diagram or workflow; we just give it a set of instructions and rules, and they are typically just LLMs using tools based on environmental feedback in a loop. And so while these agents, in this pattern, is actually pretty straightforward to implement, getting reliable results from them is really hard, and something that is everyone is trying to figure out right now. And for most problems—again, I want to stress this—for most problems, you don't need a pattern like this, and you don't want it; you want far more control; you want to start really small and work your way up, building confidence within the system. And I think the best example of this was was Devin, the AI software engineer that got a lot of hype, I think already half a year ago, and that you can now you can now rent it or buy it as like a junior engineer for your team; it's pretty pricey, and based on what I've seen so far and people that actually worked with it, the results are pretty so-so. I saw one example where out of 20 tasks that they would try, I think only four or so would work. And Devin is a perfect example of a true AI agent, where you as a human give it a task, you give it a goal—let's say, "Code this application" or "Solve this bug"—and it will go off on itself, and it will start to work in loops, and it's going to perform unit tests, try to run your code, try to build, see if it runs, looks at the errors, and then tries to correct those errors over and over again. So that's a true agentic system, but it doesn't work yet because it's really hard to pull off properly.

So those are the core patterns that you can use as a developer to build your AI systems, and it doesn't really matter which tool or framework or platform you're using, whether that's pure Python or Make.com; almost all of these patterns can be implemented. And really the key here is to start as simple as possible and build up the complexity over time, only when that's necessary.

And now, to conclude this video, I want to give you some final tips, next to these like core patterns that you should understand—that was really the main message of this this video—to dive a little bit deeper and understand that, understand the differences. And now the first one—one we already covered, but I want to reiterate it one more time—be very careful with agent frameworks; they can get you up and running really quickly, but make sure that you understand everything that's going on, and you probably don't need them, and it will make you a better engineer if you learn how to build these basic core components from the ground up within your own code base.

And now the second tip is to to prioritize deterministic workflows over complex agent patterns; start really simple and really isolate the problem and build it from the ground off. So you first focus on a particular workflow, particular problem where you can really optimize that and nail that so it works almost 100% of the time and not 80% of the time. And how you can do this: Look at the problem overall; look at all the data that you need to process; and as the first step within your system, create a categorization step where you essentially only select a very small portion of their problem. So let's take the customer care example; let's say out of all the tickets that you have, first focus on the "Where's my order?" question. So all the tickets that are coming in, you build a router: only if the LLM decides this is about an order, throw it into the AI workflow and solve that; all the other tickets, just send it to the human agents. And this can be extrapolated, this idea, to almost any problem that you solve: Take all of the data; get categorize it; focus on what you want to solve first; go vertical on that; then scale horizontal to other problems as well. Once you understand the full scope of the problem, you can start to think about introducing more agentic patterns, but you first need to understand how you as a human would break it down step by step before you can instruct an AI to do so.

And then tip number three is: Don't underestimate what happens when you scale your application. Going from the demo to, "Hey, it works! Here is cool!" to, "Now we're going to put this in front of hundreds or potentially thousands or millions of users," is such a big difference, and that chaos that we were talking about, if you like, scale chaos that is, is just going to get insane, and you will have hallucinations. So just don't underestimate that; it's really hard. So scaling RAG is also really hard; if you have your vector database and you increasingly add more and more and more data, that's also going to be a challenge on its own. So be very careful with scaling your application and putting it in front of a lot of users too soon, otherwise you'll end up with something that Apple had with Apple Intelligence. And the only way to do this systematically and properly is to have a proper testing and evaluation system in place, which is tip number four: Start with this from the beginning; don't neglect it. A lot of people simply like don't know how to do this, um, and that's why they don't have it in place. Again, ask yourself the question: If you were to change your system prompt right now, could you say for sure that it's going to improve your application beyond just a f check? This is something that you have to keep in mind. If the answer is no, you should look into that and figure out what it means for your application to set something up like this.

Okay, so the next one is to put proper guardrails within your application, and this is actually really simple to do, but a lot of people just skip this step, and it's really a quick win. So make sure before sending your data or the output back to the customer, to the application, you have at least another LLM do a quick check on whether we can actually send this. And while it may sound very straightforward, this is something that, for example, a company like Amazon didn't manage to put in one of their customer support chatbots, where if you would ask it whether it was an AI or a human, it would clearly say, "No, this is a human you're talking to," and it would even give you a name, but then someone tried to ask the customer support to give a code function back, so within that same chat first the said, "Yes, I'm human," and it was like, "Can you explain me how to build XYZ function in Python?" and it would give just the whole explanation and write out all the codes. And this is just like so embarrassing, and even the biggest companies out there right now um fail to do this properly. So really put those guardrails in place and perform various checks in order to protect your brand and to protect your reputation.

All right, and then the final optional tip that I want to give you—the other ones weren't optional; this is optional—is if you want to learn more about how we at Data Lumina structure our projects and build these AI systems, all the way from how we structure our code within the repository to the infrastructure and deployments that we use, then you can check out the link in the description to our Generative AI Launchpad. We wrapped this into a product and made it available in a GitHub repository, where this is essentially our entire company's IP that we worked on in the last two years, and we make that available for AI engineers who want to learn how to build and deploy generative AI apps. And now this is a paid product—just want to be transparent here—but if you're serious about AI engineering and you want to speed up your learning, join our community and get in our…

Discord with myself and the other engineers at data Luma to ask questions. Then you might want to check it out. All right, and that's it for this video now. Thank you very much for watching. If you found it helpful, please leave a like and consider subscribing.

Then next up, I recommend to check out this video where I go over 17 Python libraries that we use for our AI engineering projects.