AI Agents in 38 Minutes - Complete Course from Beginner to Pro

If you've been paying attention to AI in 2025, you've probably noticed that everyone is talking about agents. And for good reason. AI agents can handle everything from simple everyday tasks to complex multi-agent workflows at enterprise scale. And this is just the beginning. We're about to see a lot more innovation in this space.

If you're new here, I'm Marina. I'm a senior applied scientist at Amazon working on Gen AI. And today, I'm breaking down everything you need to know about building and working with AI agents. I've gone deep on my research for this topic. I took a bunch of different courses, read books, and of course, built my own agents. My research notes ended up being about 150 pages long, and I've distilled all of this down for you into one video.

Here's how we're going to break it down. First, the basics. What is an AI agent? What are the core concepts and where can you actually use them? We'll also cover some no-code options if you want to start experimenting without writing any code at all.

Then, intermediate level, we'll get into building and evaluating multi-agent systems that solve real problems. I'll do a demo of an agentic system I made that's currently saving me several hours of work a week.

Then advanced, what does it take to actually build reliable agent systems in production? And a bonus section for devs who want to go deep on the nitty-gritty of how tools like Claude Code work under the hood.

Whether you're a non-technical person just trying to automate parts of your own workflow or building production AI systems for your company, this video has something for you. Let's dive in.

All right, so let's start with the basics. What actually is an AI agent? Here's the simplest way to think about it. Imagine you need to write an essay. If you use a traditional LLM prompt, you'd basically say, "Hey, ChatGPT, write me an essay about how to get started in the gym." And it just writes the whole thing in one shot from start to finish. But that's not how you or I would actually write an essay, right? We don't just create a perfect first draft in one go. We plan, outline, do some research, write a messy draft, read it over, and revise again and again. It's a process.

That's what Agentic AI does. Instead of asking the AI to do everything in one linear pass, you let it work iteratively the way a human would. So, what does that actually look like? Let's stick with our essay example. Here's how an agent would tackle it.

First, it starts with an outline. It figures out the structure before diving into writing. What are the main points? What order makes sense? Then, it figures out what information it needs from the outline, and it actually gets that information. It might search the web, pull from APIs, or download relevant sources. And then, it uses this information to write a draft of the essay. But the neat part is that it doesn't stop there. The agent reflects on its own work and revises to do things like tighten up weak arguments, add missing information, or improve the flow.

This is what people call the react loop. The model reasons about what to do next, acts (often by calling a tool, which we'll talk about later), observes the result, then either gives you an answer or loops back to reason again. This works because each pass adds depth. You get stronger reasoning, fewer hallucinations, and better organization, which is all the stuff that gets lost when you try to do everything in one shot.

This approach works well anywhere you need careful, accurate work with proper sourcing. You can think about domains like legal research where you need to cite specific cases, healthcare documentation, or customer support systems that need to do things like look up account details before responding. Of course, the extra specialization and accuracy come with costs in terms of complexity.

So, this raises an obvious question. What kinds of tasks are actually worth building agents for? Some tasks make sense for agents and some don't. Let's take a look at some examples from the simplest to most complex.

A really simple example of an agentic system could be extracting key fields from invoices, then saving them to a database. Tasks with clear, repeatable processes like this are perfect for agents.

A mid-complexity task could be responding to customer emails. The agent looks up the order, checks the customer record, and drafts a response for human review.

One level up is a full customer service agent that handles questions like, "Do you have any blue jeans in stock?" or "How do I return this purchase?" For returns, the agent needs to verify the purchase, check the policy, confirm if a return is allowed, then walk through the whole return process, which has a lot of steps. The agent has to figure out what the steps are and not just follow a script.

A helpful way to think about which use cases make sense for agents is with a matrix that has two axes: complexity and precision. Some problems have both high complexity and a high need for precision, like filling out tax forms. Others are complex but don't need perfect accuracy. In this case, you could think about something like writing and checking summaries of lecture notes. The biggest value often comes from high complexity work and the fastest early wins tend to be on the lower precision side. That's why the high complexity, low precision quadrant is often a smart starting point. You get leverage from automating something tricky without being blocked by needing perfect output every time.

So to summarize, agents really shine when tasks need iteration, research, or multi-step processes. It often makes sense to start with complex tasks that can handle a little bit less accuracy.

So let's connect all of this to a real tool you can actually use today. Everything we've talked about today is exactly the kind of workload that Kimmy models from Moonshot AI are built for. Kimmy's latest model, Kimmy K2 and K2 Thinking, are what they call an agentic LLM. It's trained not just to answer once, but to plan, call tools, read files, search the web, and iterate over hundreds of steps when a task needs it.

Where it stands out is scale. Kimy's consumer app is designed around ultra-long context on the order of millions of characters, so it's very comfortable doing real agent workflows. Things like uploading 20 to 50 PDFs and having an agent read, cross-reference, and synthesize them in a single run instead of you babysitting the loop.

Kimy is a great alternative to the models you're probably used to working with. While GPT-5 and 5.1 are incredible frontier models, they're priced like frontier models, and you generally need a paid tier to use the best thinking models at scale. On the other hand, the Kimmy app itself is free to use with generous rate limits. So for personal workflows like reading papers, making slides, or running the okay computer, you're not constantly running up against quotas. And if you're a developer, the Kimmy K2 and K2 thinking models are much cheaper, roughly a five to eight times discount relative to GPT-5, which really matters if you're running agentic workflows that burn a lot of tokens. Plus, Kimmy K2 and K2 thinking are open-weight models, so you can do things like download the weights, run them on your own hardware, or plug them into your agent framework without being locked into a single vendor.

I'll leave a link to Kimmy in the description if you want to check it out or plug K2 into your own projects. Thanks to Kimmy for sponsoring this video.

Okay, now that you know what agents are good for, let's talk about how to actually build them. And the first big decision you need to make is how much autonomy you want to give your agent. Think of this as a spectrum.

On one end, you have scripted agents where you hardcode every step. Like for our essay writing example, that might be: first, generate search terms; then, call web search; fetch pages; then, write the essay. Done. It's deterministic, predictable, and really easy to control. The model's only job is generating the actual text because you've decided everything else.

On the other hand, you have highly autonomous agents. Now, the LLM decides whether to search Google, read sites, or research papers. It may figure out how many pages to fetch, whether to convert PDFs, and whether to reflect and revise. It might even write new functions and run them. This is more powerful, but it's also unpredictable and harder to control.

In practice, most real-world agents sit somewhere in the middle and are semi-autonomous. The agent picks from the tools you've defined and makes decisions within guardrails you set. But how does an agent know what tools are available or how to make decisions? This is something called context engineering, which is when you decide what information the agent has. This includes things like the background of the task, the agent's role, memory of past actions, and available tools. If you put all of this context together, this context steers non-deterministic models towards consistent, high-quality outputs. That's the practical foundation of intelligence in agents. It's not the model alone. It's how you engineer the context around it. We'll talk more about these components throughout the course.

Once the agent has its context, it's time to define the tasks it's supposed to do. Figuring out these tasks is arguably the most important thing you'll learn about building agents. Start with how you do the task. Then, for each step, ask, "Can an LLM do this?" Like, can it write a small bit of code? Maybe an API. If the answer is no, split it smaller until it is.

Let's stick with our example of building an agent to write an essay. Think about how you'd actually write, then figure out how an AI might do that task. It might go something like: outline using an LLM, generate search terms using an LLM, then call a search API, fetch pages using a tool, write a draft with an LLM using those sources, self-critique the draft using another LLM to reflect and list gaps, and revise using an LLM. Again, each step is small, checkable, and clear. When the output isn't good enough, you know exactly what step to improve.

Okay, I know I just covered a lot of theory, so let's actually build something. I'm going to walk through creating a simple agent using a no-code tool. We're building a weekly learning assistant. Here's what it does: You give it a topic you want to learn, like maybe vector databases, and it researches the topic, finds the best resources, and creates a personalized study plan for you.

Now, let's walk through how you can do this. I'm using Crew AI as a no-code, really simple option, but the principles apply to most tools. First, I'm giving a written description of what I want Crew AI to build. We can see Crew AI makes a plan and starts setting up the agents we will need. We end up with three agents to start: one to research the topic, one to curate resources, and one to write the plan. I noticed we're missing some way to visualize the final plan, so I asked Crew AI to add one more agent. We can then edit each agent and task Crew AI set up for us, including things like the prompts, output format, hyperparameters, reasoning mode, and the exact LLM we use.

Now, let's actually watch the flow in action. We can see what's happening at each step. The learning topic research specialist does an LLM call, searches the internet with a tool, does another LLM call, searches again, and so on. Then we can watch the educational resource curator work through a similar series of LLM calls and using web search tools. It also reads website content using a different tool. The personalized study plan designer does their thing. Next, the study plan formatter gives us back our final output, which is a nice markdown doc, but we could iterate further and ask for a UI or anything else that works for us.

All right, now that we have the fundamentals and have even built our first agent, we're moving into intermediate territory. And we're going to start with something really boring. But this is the stuff that separates hobbyists from professionals: how you measure agent systems performance.

Sometimes evaluations can be as simple as measuring the number of times your output is correct. If I ask my customer service chatbot if we have a certain item in stock, does it get that correct? Yes or no? But not everything is that clean-cut. Let's think about our essay writing agent. How do you measure if the essay is actually good?

One approach is to use a second LLM to judge the output. Have it rate each essay on a one-to-five scale for quality using a consistent grading rubric. You can evaluate your system at the component level to make sure each individual step is working and end-to-end to judge the final quality of the whole system.

If you find the system isn't working as well as you'd like, one first step is to examine each of those intermediate steps, which are called the trace. This includes things like the search queries the agent wrote, drafts, and thinking steps. If you read through all this, you may notice patterns like overly generic queries or that revision isn't actually getting passed to the critique properly. The observations become your next evals or your next fixes.

It's important that you start evaluating right away, but also that you don't worry about having a perfect evaluation system from the get-go. You can get something working quickly and iterate over time.

Now that we have a simple system set up and some way to measure performance, it's time to actually work on improving that performance. Memory is a really common way to do this. Memory is what lets an agent remember what worked, what failed, and what it needs to do differently next time, so it actually improves on each run.

You might have short-term memory that agents use to write down their work as they go. In multi-agent systems, other agents can read those notes. After the agent finishes a task, it can reflect on what it did, compare the result to what was expected, figure out what went well and what didn't, and store those lessons in long-term memory. Next time it runs, it loads up those lessons and applies them. This can be used to train agents similar to supervised learning. You can give the agent feedback on its work so that each run improves the quality. I'll show an example of that in the demo at the end of the section.

So, memory is dynamic and is updated on each run. Knowledge, on the other hand, is static. This is reference material that you load up front. Things like PDFs, CSVs, or documentation, or you could give it access to your database. You give it to the agent once, and it can pull from that library whenever it needs to cite something accurately.

Once we have an agent set up with its task, knowledge, and memory, we're ready to let it go bananas, right? Not quite. There's a really important step we haven't talked about. Because LLMs are non-deterministic, they can make mistakes. Maybe they write something that's factually wrong or in the wrong format. To prevent issues, we need to add guardrails to the system. Guardrails are basically a quality gate between what the agent says is done and the task that actually is being finalized.

There are three main approaches to guardrails, and most production systems use at least two. For deterministic stuff like output format and length, we can just use standard code snippets. These are fast and cheap and should be preferred whenever possible. Sometimes we're checking for more nuanced things like, "Is this response factually consistent with the sources?" or "Is the tone positive and professional?" In this case, we can use another LLM to judge the output. If the LLM judge says, "No, this fails," it explains why. That feedback gets sent to your agent, and your agent revises and tries again. Finally, sometimes you just need a human to check the work. Instead of the agent finishing and shipping the result automatically, you can make it stop and ask for approval first. You can give feedback and ask the agent to try again.

All right, so we've covered a lot about how to make the system function. Now, let's talk about how to make the system better quality. There are four core patterns that reliably boost quality and capability: reflection, tool use, planning, and multi-agent collaboration.

Let's start with the easiest and most effective: reflection. In a nutshell, reflection basically just means we don't stop at the first draft. When you use reflection, the model produces something, critiques it, then rewrites it if needed. That second pass, guided by a prompt that asks it to fix any problems, almost always makes things better. Let me show you a quick example with an email. This has the same content, but is cleaner, more specific, and more professional.

Reflection gets really powerful with code because you can add external feedback. You can write the code, have a critic agent review it, and then actually run it. This allows you to capture errors, test results, and outputs, and feed that back into the model. The model can use that concrete information to produce a much better V2.

Reflection is particularly useful when you have structured outputs like JSON, procedural instructions like the steps to brute force where the reflection can catch missing steps, creative work, and long-form writing. In particular, reflection works well when you can incorporate external feedback like running a schema validator against JSON or checking for missing citations in a research task. The drawback is that it adds latency and cost because you're doing multiple passes. So, make sure to test with and without reflection to ensure it's actually helping.

All right, let's talk about the second design pattern: tool use. Here's the core idea: You give the LLM a menu of functions that it can call. This can be things like web search, database queries, code execution, calendar access, or whatever your application needs. And then the model decides when and which tools to use.

This is important because an LLM by itself is just a text generator. It doesn't know what time it is right now or anything about your company's sales data. It can't execute code or compute exact answers. But if you give it tools, it can do things like web search, query a database, write to a CRM, or run code.

So, if I were to ask the agent, "What time is it?" The LLM calls `get_current_time`, gets back 3:20 p.m., and responds with that. Or you might ask it to search for local restaurants, query the database, or do a math calculation. In each case, the model recognizes it needs external information or computation, picks the right tool, and uses the result to answer.

When you give the model multiple tools, it can chain them together. For example, let's say you're building a calendar assistant. You've exposed three tools: `check_calendar`, `make_appointment`, and `delete_appointment`. The user asks, "Schedule a meeting with Alice for this week." The model thinks through the steps: check my calendar for availability, find an open slot, call `make_appointment` with Alice at that time, and confirm back to the user. The key thing here is that the LLM is choosing which tool to invoke next based on what it's learned from the previous tool's output. It's not a fixed pipeline. It's actually dynamic.

Okay, but LLMs only generate text. They don't execute code. So, how do they call a function? They actually don't. They request a function call. Here's the loop under the hood: The user sends a prompt. The LLM looks at its available tools and decides if it needs one. If it does, it outputs a special request like, "I want to call `get_current_time` with `time_zone: Pacific/Oakland`." Your code sees that request, actually runs the function, and gets the results. You feed that result back to the LLM as new context. The LLM uses it to finish its answer or to request another tool if needed. It's as simple as that. The LLM requests, but doesn't actually execute code.

In order for the LLM to be able to find tools and request tools, we need a consistent way to define them. Every tool has two parts: the interface for the agent. This includes a tool name, a plain English description of when to use it, and a typed input schema. For example, `read_website_content` with description "fetch and return the text of a web page" and one input `url` which is a string. And the implementation code. Whatever you need, like SQL queries, retries, throttling, and parsing. The agent only sees the interface. All of the messy implementation details are hidden.

Good tools also consider things like error handling, self-recovery, and rate limiting. They may use caching to memoize results for identical inputs to reduce latency, cost, and external API load. And they should have async support so the agent or other agents can keep working while long tool requests complete. Tools should be built like products with versioning, proper documentation, and sufficient tests. It's useful to maintain an internal registry of vetted tools with docs, versions, and ownership. Put all that together, and you now have given your agent a way to interact with the world, which is rad. But we need to make sure the agent knows what it needs to do in the real world.

Which leads us to the third design pattern: planning. Here's the idea with planning: Instead of hard-coding a fixed sequence of steps, you let the LLM decide what to do and in what order. Let's say you're building a customer service agent for a retail store. You could hard-code flows for every scenario. Like, if it's a pricing question, do X. If it's a return, do Y. If it's inventory, do Z. But what happens when someone asks something you didn't anticipate? Or when the same question needs different steps depending on context?

With planning, you give the agent a toolkit of functions like `get_item_descriptions`, `check_inventory`, `get_item_price`, `process_return`, and let it figure out which tools it needs to use and when. The basic loop looks like this: You give the agent access to tools. You prompt it to create a plan, list the step-by-step actions to answer this question. You execute the plan step-by-step. The LLM picks the right tool. You run it and feed the results back and repeat until you're done. It's basically plan, act, observe, continue, but with your tools.

Here's a concrete example using retail sunglasses. A user asks, "Are there any round sunglasses in stock for under $100?" The agent might plan: Step one, use `get_item_descriptions` to find round frames. Step two, run `check_inventory` on that list. Step three, call `get_item_price` on the in-stock items and filter for those under $100. Step four, compose the answer. You didn't predefine this exact recipe. The LLM chose it from the available tools.

Now, a different question comes in: "I want to return the gold frame sunglasses I bought, not the metal ones." The plan changes completely in this case: Step one, identify the user's purchase. Step two, match the gold frame product. Step three, process item return. And step four, confirm the outcome.

It can be helpful to ask the model to output a structured plan in JSON. Or you can let it actually write code, usually in Python, that encodes the entire plan. Planning increases autonomy, which means it also increases unpredictability. You need guardrails on things like permissions, validation on tool calls, and managing passing outputs of one step into the next step.

Today, the strongest use case for planning is in highly agentic coding systems. The model breaks down programming tasks into steps and works through them one by one. For other domains, planning absolutely works, but it's harder to control because you don't know in advance what plan the model will create. The tooling and guardrails are improving fast, though, so adoption is growing.

But what if you have a system where you need to do lots of different things, possibly simultaneously? That's where multi-agent collaboration comes in. Think about how you'd tackle a complex project in real life. You don't hire one super generalist to do everything. You build a team. You have specialists who are really good at their specific thing and they hand work off to each other. Multi-agent systems borrow that same mindset. Each agent has a clear role. Each one focuses on what it's good at. The output is better because you've got specialization at each step.

Besides specialization, there are some other advantages to multi-agent systems. It avoids any one agent having a huge context window. You can use multiple LLMs, meaning you can mix cheaper, faster models for high-volume and simple tasks and reserve larger, more capable models for more important tasks like strategy, delicate customer replies, or long-form writing. This gives you flexibility on both cost and performance. You can parallelize work and if you have really long-running operations, you can split up the work and see which agents are working on what to help users understand what's happening.

If you have a simple task, skip multi-agent systems. They can slow things down and make debugging more difficult. This is because multi-agent systems introduce a whole new layer of complexity. You can have resource conflicts if two agents try to modify the same file. There's communication overhead between agents and complex task dependencies. There are also issues like API rate limits and what to do if one agent fails. Do the others keep going or do you roll back? And how do you combine what multiple agents produced into one coherent output? This isn't impossible to manage, but you need to design for it. You need robust orchestration, good error handling, and clear protocols for how agents communicate.

So, let's talk more about designing these multi-agent systems. Let's use the example of creating a marketing brochure to illustrate our options. The first step is defining your agents by role. Each agent gets a clear job description and only the tools it needs to do that job.

A researcher agent who finds market trends and competitor moves. This agent might have access to tools like web search, retrieval, and maybe note-taking.

A graphic designer agent who creates charts and visual assets with tools for image generation, image manipulation, or code execution to plot charts.

And a writer agent who turns findings and assets into final copy. This agent could just be an LLM itself with no external tools needed.

You implement each agent by prompting it with a role like, "You are a research agent specialized in market analysis," and giving it only the tools it should have.

Once you've defined your agents, you now need to decide how they communicate. There are four main patterns, which we'll discuss from simplest to most complex.

Pattern one: sequential. This is the simplest and most predictable. Each agent finishes its work, then passes the output to the next agent in line. For our brochure, it could look like this: The researcher finishes, hands off to the designer; the designer finishes, hands off to the writer, and we're done. It's like an assembly line. It's easy to debug and has predictable timing and cost. This is where you should start. Depending on your use case, it might be sufficient.

Pattern two: parallel. Sequential isn't the only option. You could also run agents in parallel when the steps don't depend on each other, which is great for reducing latency. For example, maybe your researcher and designer could work simultaneously on independent parts of the brochure, and then the writer could combine their outputs. This speeds things up but adds coordination complexity.

If you start getting into more complex workflows, it can be helpful to add a manager agent that plans and coordinates. The specialist agents do their work and report back to the manager, not to each other. This keeps control tight while giving you flexibility. The manager can reorder steps, skip things that aren't needed, and ask agents to redo work. It's more adaptable than a linear flow without being super chaotic. This is probably the most common pattern in production today.

For even more complex workflows, you could have deeper hierarchies where some agents manage their own sub-agents, basically like a team. For example, your researcher agent might orchestrate a web researcher sub-agent and a fact-checker sub-agent. Your writer agent might have a style writer and a citation checker working under it. This is helpful for very complex tasks, but of course adds a bit more chaos.

Finally, we have the all-to-all model, which can be super chaotic. In this model, any agent can message any other agent at any time. This is rare in production because it's hard to predict and control. Outputs can vary wildly run-to-run, but it can work for more brainstorming, creative, or low-stakes tasks, like generating multiple variations of ad copy where one run produces garbage you can just try again.

We've talked about challenges with coordination a couple of times. Here are two of the most common pitfalls. First, redundant work. Multiple agents may redo the same searches or call the same tools. This can be addressed by tightening task scopes and having a clear division of labor between the agents. Second, unnecessary serialization. Chaining steps that could run concurrently slows everything down. To address this, identify truly independent tasks and run them asynchronously. Then route just the pieces of context the next step needs. In general, you want to start with the simplest coordination method you can and only add complexity as needed.

Regardless of which pattern you choose, here are four key best practices to keep in mind when designing multi-agent systems.

One: Define interfaces, not vibes. Each agent needs a clear schema for inputs and outputs. It needs to know things like what fields, what types, what IDs or references get passed along. Handoffs break more often than your models do. If your researcher returns an unstructured blob and your designer doesn't know how to parse it, the whole system's going to fail.

Second: Scope tools per agent. Give each agent only the tools it actually needs with least privilege access. This helps with security and makes the system easier to reason about, easier to audit, and easier to debug.

Third: Log the trace. Keep per-step artifacts. What did each agent plan? What prompts did it use? What tool calls did it make? What results came back? When something breaks, this trace makes error analysis really fast. You can see exactly where things went wrong.

Number four: Evaluate components and end-to-end. We've already talked about this a little bit, but you need two types of evals. Component level: Is this research relevant? Are the images high quality? Is the copy tone appropriate? And end-to-end: Is the final brochure good? Did it meet requirements? If your end-to-end eval system shows problems, but your components all look fine, then you know it's a handoff or integration issue. If a specific component eval fails, you know which agent needs to improve.

Okay, so now let's look at a multi-agent system I built to help automate one of the most time-consuming tasks in my week: writing scripts for YouTube Shorts and Instagram Reels for my second channel. I'm in the process of writing a book on mindset and productivity. So I have literally hundreds of pages of notes on ideas for content. But putting it into the right format takes me a long time and it can be tedious.

Here's how the system works: You upload PDFs of notes, ideas, or brand documents. The system uses a semantic search with Chroma DB in this case to understand the concepts, not just keywords. Three AI agents work together. Agent one searches the notes and generates video ideas. Agent two writes a 30 to 60-second script with a hook, value, and CTA. Agent three checks it against my brand voice using living brand guidelines.

At this point, we have a human-in-the-loop stage where I edit, approve, and reject each script through a stream interface. But this is where it gets cool. Agent four analyzes my feedback and automatically updates those living brand guidelines. This system literally evolves to learn my voice better and better. I was able to code this with an AI coding assistant in an afternoon. It's nowhere near production-ready, but this simple tool is already helping me save hours every week.

So, now we're ready to talk about what separates these small personal projects from the kinds of systems running in production at scale. So, welcome to the advanced section. If you've made it this far, you're serious about building real agent systems that could work in the real world. The techniques that got you from zero to prototype won't take you from prototype to production. You need different tools, mindsets, and more discipline. Let's get into those now.

We've already talked about task decomposition, but this gets increasingly complex when you're working with multi-agent systems. There are four main patterns you can use to guide you to do this well.

Pattern one is called functional decomposition. In this pattern, we split the tasks by technical domain or expertise. This is what we've been using in our examples so far. Basically, we're breaking down tasks by the kind of work that needs to be done. So, you could think about a full-stack feature development, for example. You've got front-end work, back-end logic, database changes, and maybe API updates. Each of these requires different knowledge and different tools, so you create specialized agents in each domain.

Pattern two is spatial decomposition. You can also split by file directory or structure. This is especially powerful when you're working with large codebases with many files that could be processed independently. Let's say you're doing a large-scale refactoring, maybe updating all your API endpoints to a new authentication system, and you've got dozens of files across different services. You can decompose spatially like this. In this case, you minimize conflicts by ensuring agents work on separate parts of the codebase. They can work in parallel, but if your files have complex dependencies on each other, spatial decomposition breaks down.

Pattern three is about breaking tasks into sequential stages where the later stages depend on the earlier ones being complete. Let's use a product launch as an example. You can't just wake up one day and start sending promotional emails. There's a logical sequence to get there. So, phase one might be market research. You analyze competitors, survey target customers, identify positioning opportunities, and so on. Phase two is launch planning. You define messaging, set pricing, create timelines, and identify channels. In phase three, we create assets. We write copy, design graphics, build landing pages, and prepare email sequences. And finally, in phase four, we launch and monitor. This is when we actually execute the campaign, track metrics, respond to feedback, and adjust in real time. Each phase gets its own agent or team of agents. Phase two doesn't start until phase one is done and reviewed.

And pattern four is data-driven decomposition. Finally, we can split by data partitions. This one's less common, but can be really powerful for certain use cases, especially tasks involving large datasets where you can partition the data and process chunks independently. Let's say you're analyzing application logs to identify performance issues. You've got gigabytes of logs from the last month. You partition by time or by service. Agent one processes week 1 logs. Agent two processes week 2 logs. Agent 3 processes week 3 logs, and so on. Each agent runs analysis independently. Then you aggregate results at the end.

You could also mix these patterns. So, for example, a full-stack feature might use functional decomposition for the main structure: front-end, back-end, and database. But the back-end agent could use temporal decomposition internally.

All right, so at this point, let's say we have a working system. We've done a comprehensive evaluation to find errors, and we're still just not happy with performance. Here's what to do. The first thing to understand is that you're working with two fundamentally different types of components, and they need different improvement strategies.

First, we have non-LLM components. These are things like web search, RAG retrieval, code execution, speech recognition or vision models, and PDF parsers. These can be improved in two main ways. First is you just kind of like tune the knobs. Fiddle with things like the web search you're using, date ranges, top K results, RAG chunk size, similarity thresholds, and so on. All of this AI engineering stuff. Or you could also consider swapping providers in some cases. Try alternative web search APIs, different OCR, or vision models, and so on.

Then we have LLM components. These are used for generation, extraction, and reasoning anywhere you're using the language model itself. There's a lot we can do to improve this part. First, we could prompt better. We can add explicit instructions, constraints, and schemas. Use few-shot input-output pairs to show the model what you want. You could try another model. Some models are better at following instructions. Some are better at code or factual recall. Don't assume one model is best for everything. We can decompose hard tasks into smaller pieces, and fine-tuning is a last resort. Fine-tuning is powerful, but it's really costly. So, save it for mature systems where you need those last few percentage points of quality and you've exhausted everything else.

Nailing the output quality should be your first step. After that, let's talk about reducing latency. First thing you need to do is get a baseline. The first step is to time each step in your workflow. You might find something like, "It takes seven seconds for the LLM to generate search terms, web search takes 5 seconds, drafting the essay takes 11 seconds," and so on. This gives you a baseline so you know what you should be optimizing.

Next, run anything in parallel that you can. Examples might be web fetches, multiple web searches, or parsing multiple documents. This is often a really easy win. You can rightsize the model, so use a smaller, faster LLM where tasks are simple, like keyword generation, and reserve the heavyweight model for synthesis and reasoning. Try faster providers. Throughput and token streaming speeds vary a lot. A provider with optimized serving can cut seconds without any prompt changes. And finally, trim context. Shorter prompts and contexts means faster decoding. So try to keep only what you truly need.

With high quality and latency under control, you're ready to look at cost. To kick off, you'll want to measure the cost of each step just like you did with latency. Agent systems have several cost sources. First, of course, the LLM calls. This is determined by the input tokens and output tokens. These are usually priced separately. Input tokens are cheaper, and output tokens cost more. API calls for things like web search, PDF conversion, image generation, speech-to-text. These often have per-call or unit pricing. Infrastructure. If you're running your own retrieval systems, vector databases, or compute for code execution. Say you're building a research agent that writes essays. Here's what one run might cost. If you're running this 1,000 times a day, that's $80 a day or $2,400 a month.

Once you know how much each step costs, here's what you can do to optimize. First, attack the big buckets first. If web search costs 2 cents per call and you're calling it 10 times per run, that's 20 cents right there. Do your best to reduce the calls, cache results, or batch queries. Tear your models. Use cheap models for easy tasks and frontier models only where it really matters. Cache aggressively. Deterministic results like search responses, embeddings, chunk retrievals, or intermediate summaries shouldn't be recomputed every time. Constrain outputs. Ask for structured, concise results with instructions like, "Return JSON with these required fields," "Give me five bullets max." Fewer tokens lowers the bill. And batch. If you're processing many similar items, bundle operations when possible. On AWS, batch processing is 50% of the cost of on-demand, for example.

So, you have a system with quality, latency, and cost that you're happy with. Now, you need to make sure it continues to behave as expected once it scales. This is where monitoring and observability come in. Observability covers debugging, quality monitoring, and hallucination tracking. Basically, anything that helps you watch the agent's behavior and performance.

The tricky bit is that observability for AI systems is fundamentally different from traditional software. With traditional software, you can trace a clear execution path. Function A calls function B, which queries the database, returns data, renders a page, that kind of thing. But AI systems don't work that way for many different reasons. They're non-deterministic, like we've talked about. The same input can produce different output based on model responses. You can't just replay the exact request and expect the same result. They have distributed execution and tools running in parallel with agents spawning sub-agents and things like that. Lots of external dependencies and potential failure points that are outside of your control, and more.

To manage all of this, we need two kinds of visibility. Zoom-in metrics help you debug single runs. This is your full trace: so prompts, tool calls, token usage, retry attempts, and every decision point. Basically, everything required to reproduce an error and see exactly where it went wrong. Zoom-out metrics tell you how the whole system is doing over many runs. This includes automated quality checks, often with an LLM judge, hallucination rates, success and ROI measures, and trend lines that show whether changes are helping or hurting.

You'll want to log not only what an agent did, but why it did it. For example, you might log things like, "Agent chose to use web search instead of RAG because query contained recent information," or "Reflection past identified three issues: missing citation, vague date, or wrong tone."

When you're running thousands of agents at once, you can't manually watch each trace. This is where quality sampling comes in. Instead of deeply inspecting every single execution, you define a sampling rate, say like a certain percentage of total runs to be evaluated for quality and hallucination. The system then uses that subset of executions to compute an overall quality score and a hallucination score for your agents. This lets you prioritize fixes and areas for improvement.

Beyond technical metrics, you also need to understand user behavior. What are people actually asking for? Are they using your agent as you intended, or have they found creative workarounds? Where do they get stuck? Do they rephrase and retry? That's a signal the first attempt didn't work. What did they do with the output? If they immediately ask for revisions, the initial quality wasn't good enough. How long are sessions? Very short sessions might mean quick success or immediate failure. Very long sessions might mean the agent is capable but inefficient. This qualitative data gives you a product roadmap as much as your technical metrics do.

Finally, we need to talk about one of the least exciting and most important pieces of building a robust system: security. Just like observability, security for AI agents isn't like traditional application security. You're not just protecting against external attackers. You actually have to protect against your own system making dangerous decisions or being manipulated into harmful actions. These are the kinds of things to watch out for.

First, prompt injection. Malicious content in user input or external data that hijacks your agent's instructions. Unsafe code generation. Agents writing code that accesses sensitive data or executes dangerous operations. Data leakage. PII or proprietary information could be exposed through agent outputs or tool calls. Resource exhaustion. Agents could spin up expensive operations or infinite loops.

Let's dive into that one in particular. Code execution is the ultimate tool for agents. It's incredibly powerful because agents can write code to generate charts, create markdown files, and process data. Really, anything they want within the boundaries you give them, which is a double-edged sword. Many tasks can be covered by well-defined custom tools. So, your system doesn't always need to fall back to free-form coding. But when you do enable it, you need guardrails. Here's how to do code execution safely.

Sandbox execution. Use Docker or a restricted runner environment. Isolate code execution completely from your main application. Resource limits. Set timeouts, memory caps, CPU limits, that kind of thing. Block dangerous imports, network access unless explicitly needed, and file system rights outside of a designated temp directory. Whitelist libraries only. So allow specific safe libraries like pandas, NumPy, datetime, that kind of thing. But don't allow arbitrary installs. If an agent needs a library, you add it to the whitelist explicitly. Validation plus reflection loop. If code execution errors, capture the traceback and let the model fix the code. Give it one or two attempts, and make sure you have a circuit breaker in place. Deterministic input and output. Have code return a small structured result like a number, a list, or a JSON object. Then you format that for the user. Don't let the code directly output to the user or write to files they can access. And input and output sanitation. So all inputs are validated before they reach the agent, and all the outputs are scanned for sensitive data like API keys or PII.

So, that wraps up the advanced section. With all of this, you're ready to build real systems that scale and serve real users in production. But, as promised, we have one more bonus for the super-advanced folks. Most of what we talked about today assumes you're using a framework like LangGraph or Crew AI. But if you're a developer interested in understanding the internal workings of agentic tools like Claude Code, I highly recommend this blog on agentic system design. It walks through things like the three core layers: the terminal UI, LLM intelligence layer, and tool layer; how to structure a reactive command loop with async generators; patterns for streaming and tool calling; and even a parallel execution engine and smart tool scheduling, so read versus write, that looks a lot like what powers Claude Code and Cursor under the hood.

If you enjoyed this course, check out my full AI Engineering course that's up next. Thank you so much for watching, and I'll see you next time.