Hey everyone, today we're diving into the book "AI Engineering" by Chip Huyen—800 pages of really great content about this in-demand field that's offering salaries of $300,000 or more.

In this video, I'm summarizing everything from the book to help you get a high-level overview of the field. We'll talk about foundation models, prompt engineering, RAG, fine-tuning, agents, how to build a system, improving inference, and more.

I also want to mention this is a super high-level overview of a very detailed technical book. Don't expect to learn all the details just from watching this video. I really recommend using this as a way to get an overview of what the field looks like and use it as a jumping-off point for your own research and exploration.

So what exactly is AI engineering, and how is it different from traditional machine learning? Let's break it down. AI engineering has exploded recently for two simple reasons: AI models have gotten dramatically better at solving real problems, while the barrier to building with them has gotten much lower. This perfect storm has created one of the fastest-growing engineering disciplines today.

At its core, AI engineering is about building applications on top of foundation models—those massive AI systems trained by companies like OpenAI or Google. Unlike traditional machine learning engineers who build models from scratch, AI engineers leverage existing ones, focusing less on training and more on adaptation.

These foundation models work through a process called self-supervision. Instead of requiring humans to painstakingly label data, these models can learn by predicting parts of their input data. This breakthrough solved the data labeling bottleneck that held back AI for years. As these models scaled up with more data and computing power, they evolved from simple language models to what we now call large language models, or LLMs, and they didn't stop there. They've expanded to handle multiple types of data, including images and video, often becoming large multimodal models. Nowadays, we're seeing foundation models power everything from coding assistance like GitHub Copilot to image generation tools, writing aids, customer support bots, and sophisticated data analysis systems.

Now that we've covered what AI engineering is, let's dig deeper into foundation models themselves—how they're trained, how they work, and why understanding their architecture matters for AI engineers. Foundation models, at their core, can only know what they've been trained on. This might seem obvious, but it has profound implications. If a model hasn't seen examples of a specific language or concept during training, it simply won't have that knowledge.

Most large foundation models are trained on web-crawled data, which brings some inherent problems. This data often contains clickbait, misinformation, toxic content, and fake news. To combat this, teams use various filtering techniques; for instance, OpenAI only used Reddit links with at least three upvotes when training GPT-2. The language distribution in training data is also heavily skewed—about half of all crawled data is in English, which means languages with millions of speakers are often underrepresented. This is why specialized models for specific languages and domains are becoming increasingly important. Also, the distribution of domains in one of the main training data sets leans heavily towards business, tech news, and art.

In terms of model architecture, most foundation models use Transformer architectures based on the attention mechanism. But to understand why Transformers were such a breakthrough, we need to look at what came before. Transformers were invented to solve the problems of sequence-to-sequence models, which used recurrent neural networks for tasks like translation. These had two main components: an encoder that processes inputs and a decoder that generates outputs. Both worked sequentially, token by token. The problem is that the decoder only has access to a compressed representation of the entire input. Imagine trying to answer detailed questions about a book when all you have is a brief summary. Also, input processing and output generation are done sequentially, so it's slow for long sequences.

Transformers solved this with the attention mechanism, which allows the model to weigh the importance of different input tokens when generating each output token. It's like being able to reference any page in the book while answering questions. Plus, Transformers can process input tokens in parallel, making them much faster during inference. Transformers work in two steps: first, pre-fill—process all the input tokens in parallel to create the intermediate state; and second, decode—generate one output token at a time.

The attention mechanism uses three types of vectors: first, query vectors—these represent what information the model is looking for; next, key vectors—like indices of previous tokens; and finally, value vectors—the actual content of the previous tokens. The model computes how much attention to give each input token by comparing the Q and K vectors. A high similarity score means that the token's content V will heavily influence the output. This is why longer context windows are computationally expensive—more tokens mean more K and V vectors to compute and store. Attention is almost always multi-headed, allowing the model to focus on different groups of tokens simultaneously. In Llama 2 7B, there are 32 attention heads, for example.

A complete Transformer consists of multiple Transformer blocks, each containing an attention module and a neural network module. The number of blocks is often called the number of layers. Before and after each block, there's an embedding module that converts tokens and their positions into vectors, and finally, an unembedded layer that maps output vectors to token probabilities.

So that's a super high-level look at this. I would really recommend either reading the book or checking out StatQuest for an awesome overview of Transformers and the attention mechanism. I'll link that in the description—that's really how I learned.

While Transformers dominate, they're not the only architecture. Models like RWKV, which combines RNN-based approaches with parallelization capabilities, are gaining traction for certain applications. In general, larger models with more parameters have greater capacity to learn and perform better. The number of parameters helps us estimate the compute resources needed for training and inference as well. However, note the parameter count can be misleading with sparse models—so those with many zeros—which can be more efficient. A large sparse model might require less compute than a smaller dense one. When designing models, compute is often the limiting factor. The Chinchilla scaling law helps calculate the optimal model size and data size for a given compute budget. It suggests that the number of training tokens should be about 20 times the model size—so a 3 billion parameter model needs about 60 billion training tokens.

While the cost for achieving the same model performance is decreasing over time, the cost for improvements remains high. Going from a 3% to a 2% error rate might require an order of magnitude more data, compute, or energy. But even small performance improvements can make a huge difference for downstream applications. As we keep scaling models, we're approaching two significant bottlenecks: first, training data—there's concern we'll run out of high-quality internet data in the next few years, forcing models to train on AI-generated content, potentially causing performance degradation or requiring access to proprietary data like copyrighted books and medical records; second, electricity—data centers already consume 1 to 2% of global electricity, limiting how much larger they can grow without significant energy breakthroughs.

Pre-trained foundation models face two main issues: they're optimized for text completion, not conversation, and their outputs can be factually incorrect or ethically problematic. Post-training aims to address these issues through two main steps: first, supervised fine-tuning—supervised fine-tuning optimizes the model for conversations instead of completion. This requires high-quality instruction data showing the kinds of requests the model should handle and how it should respond. It's essentially teaching the model what good responses look like; second, preference fine-tuning—preference fine-tuning aligns the model with human values using reinforcement learning, often called reinforcement learning from human feedback. This involves training a reward model that scores outputs based on human preferences and optimizing the foundation model to generate responses that maximize these scores. While reinforcement learning from human feedback has been the standard approach, newer methods like direct preference optimization (DPO) are gaining traction. Some companies even skip the reinforcement learning step entirely, instead generating multiple outputs and selecting those with high reward model scores. This is a strategy called "best-of-N."

Foundation models don't just produce a single definitive answer; they generate probabilities for possible outputs. How we sample from these probabilities dramatically affects the model's responses. The simplest approach is greedy sampling—always picking the highest probability token—but this leads to repetitive, predictable text. To introduce creativity, we use sampling techniques. Temperature controls how confident the model is in its predictions. Higher temperature values (like 0.7 to 1) make outputs more creative but potentially less accurate, while lower temperatures (close to zero) make outputs more deterministic and focused. Top-K sampling restricts the model to choosing from only the K most likely next tokens (typically between 50 and 500, depending on how diverse you want the responses to be). Top-P sampling selects the smallest set of tokens whose cumulative probability exceeds a threshold P. A value of 0.9 means the model will only consider tokens that together make up 90% of the probability mass. This probabilistic nature explains many of the behaviors we see in foundation models, like inconsistency with minor input changes and hallucinations, where models confidently state incorrect information.

Now that we understand foundation models a little more, let's talk about one of the most crucial yet underappreciated aspects of AI engineering: evaluation. For some applications, figuring out evaluation can consume the majority of your development effort. It's how you mitigate risks, uncover opportunities, and gain visibility into where your system is failing. Evaluating AI systems is significantly harder than traditional ML models for several reasons: first, the problems these models solve are often inherently complex—evaluating a mathematical proof or the quality of a summary requires deep expertise; you might need to read an entire book just to judge if a summary captures the key points correctly; second, tasks are typically open-ended, with many possible correct responses—unlike classification, where there's one right answer, a question like "write me a poem about resilience" has countless valid responses; third, foundation models are black boxes—you can only evaluate them by observing their outputs, not by understanding their internal workings; fourth, publicly available evaluation benchmarks quickly become saturated—which is when the model achieves perfect scores; what was a challenging test yesterday becomes an easy exercise today; and finally, for general-purpose models, you need to evaluate not just known tasks but discover new capabilities that might extend beyond human abilities. All of this is made worse by a general underinvestment in evaluation compared to model development.

So let's start with some fundamental metrics used to evaluate language models during training. Most autoregressive language models are trained using cross-entropy or its relative, perplexity. These metrics essentially measure how well the model predicts the next token in a sequence. Entropy measures how much information, on average, a token carries. The higher the entropy, the more information-dense each token is and the more unpredictable the language. If you can perfectly predict what I'll say next, what I say carries no new information. Language models learn the distribution of their training data; the better a model learns this distribution, the better it becomes at predicting what comes next, resulting in lower cross-entropy. A perfectly trained model would achieve cross-entropy equal to the entropy of the training data itself, and the KL divergence between the two will be zero. Perplexity is simply the exponential of cross-entropy; it measures the amount of uncertainty a model has when predicting the next token. Higher perplexity means that there are more possible options the model is considering. What counts as good perplexity depends entirely on the data; more structured data has lower expected perplexity because it's more predictable; the larger the vocabulary, the higher the perplexity, because there are more possible options; and the longer the context length, the lower the perplexity tends to be. While perplexity is useful for guiding training and serves as a proxy for a model's general capabilities, it becomes less reliable for models that have undergone significant post-training with SFT or RLHF. As models get better at completing tasks, they might actually get worse at predicting the next token in a statistical sense. Perplexity can also be used to detect if a text was in a model's training data because it would be unusually good at predicting those tokens, and to identify nonsensical text, which would have abnormally high perplexity.

For some tasks, we can perform exact evaluation, where there's no ambiguity about the correct answer, like multiple-choice questions. This is in contrast to subjective evaluation, like grading an essay. The gold standard here is functional correctness—evaluating whether the system performs its intended functionality. For example, if I ask a model to book a restaurant reservation, did it make the correct reservation? This is the ultimate metric for any application, though it's not always clear how to measure it. In coding tasks, functional correctness translates to execution accuracy—does the code run and produce the expected output? For gaming bots, we can measure objective performance metrics like win rates. When reference data is available, we can evaluate outputs by comparing their similarity to this ground truth. This approach is bottlenecked by how much and how fast reference data can be generated, either by humans or AI. There are three main ways to compare outputs to references: first, exact match—a binary measure that works for simple questions with definitive answers, like "Who was the first woman to win the Nobel Prize?"; second, lexical similarity—a continuous measure of how much the tokens overlap between the output and reference. This can use techniques like edit distance (how many changes are needed to transform one text into another) or n-gram overlap metrics like BLEU and ROUGE. The drawback is that you need a comprehensive set of reference responses, and the references themselves can be wrong. Plus, higher lexical similarity doesn't necessarily mean a better response; there are many ways to express the same idea; third, semantic similarity—this is a continuous measure of whether two texts have the same meaning, regardless of the specific words used. This is typically implemented by comparing text embeddings using metrics like cosine similarity. The advantage is that it doesn't require references, but it does depend on the quality of the underlying embedding algorithm.

One of the most powerful and common methods for evaluating AI models in production is using another AI model as a judge. These AI judges are fast, easy to use, and relatively cheap compared to human evaluators. They can work without reference data and can judge attributes like correctness, toxicity, hallucinations, and more. Studies have shown that AI judges can correlate strongly with human evaluators, sometimes showing higher agreement than between different human judges. They can also explain their decisions, which helps with transparency. You can use AI judges to score outputs, compare outputs to references, or pick the best of two responses. Since language models are generally better with text than numbers, AI judges tend to perform better with classification tasks than numerical scoring. When creating prompts for AI judges, you need to include the evaluation task, criteria, and scoring system. Few-shot examples generally work better than zero-shot (which we'll talk about later in the prompt engineering section), though longer prompts do increase costs. Interestingly, you don't always need your strongest model as the judge; specialized, smaller models can often perform evaluation tasks effectively, which helps reduce costs and latency. However, of course, AI judges have limitations—like all AI applications, they're probabilistic. The same judge, given the same input, can produce different scores if prompted differently or simply run twice. This makes evaluation results harder to reproduce or trust. Additionally, metrics aren't standardized across different systems; one system's definition of faithfulness might differ from another's. Models also exhibit biases; they might prefer responses from the same model (this is called self-bias), favor the first answer in a comparison (this is position bias), or prefer lengthier answers (verbosity bias). You can mitigate these biases through techniques like randomizing the order of responses, but this also increases costs.

Now that we understand evaluation, let's tackle one of the most crucial decisions in AI engineering: model selection. With the increasing number of readily available foundation models, the challenge isn't developing models but selecting the right one for your application. During application development, you'll go through model selection multiple times as you progress through different adaptation techniques. For instance, when doing prompt engineering, you might start with the strongest model to evaluate feasibility, then work backward to see if smaller models would suffice. If you decide to fine-tune, you might start with a small model to test your code before moving to a larger one. The selection process typically involves two key steps: first, finding the best achievable performance on the task; and then, second, mapping models along a cost-performance axis and choosing the model that gives the best performance for your budget. Your criteria for evaluating a model can be organized into four buckets: first, domain-specific capabilities—how well does the model understand your specific domain? For example, if you're summarizing legal documents, how well does it understand legal terminology? Second, general capabilities—how coherent, faithful, or factually consistent are the outputs? Third, instruction-following capabilities—does the model follow the format and structure you requested? And fourth, cost and latency—how expensive is the model to run, and how quickly does it respond? Sometimes, rather than evaluating absolute quality, you just need to determine which model is best for your use case. This can be done through point-wise evaluation (so you score each model independently) or comparative evaluation (where you directly compare outputs). When evaluating models, you also need to differentiate between hard attributes and soft attributes. Hard attributes are impossible or impractical to change; these include license restrictions, training data composition, model size, privacy requirements, and the level of control you need. These are often determined by the model providers or your own internal policies, and they can significantly limit your pool of options. Soft attributes, on the other hand, can be improved through adaptation techniques like prompt engineering or fine-tuning. These include things like accuracy, toxicity, and factual consistency.

A high-level workflow for model selection looks like this: first, filter out models whose hard attributes don't work for you; second, use publicly available information like benchmark performance to narrow down to the most promising candidates; third, run your own experiments to find the best model given all of your objectives; fourth, continually monitor your chosen model in production to detect failures and collect feedback.

Most companies won't build foundation models from scratch, so another question is whether to use commercial model APIs or host an open-source model yourself. Let's clarify some terminology. First, originally, open-source meant any model you could download and use, but some argue that a model should only be considered truly open-source if its training data is also publicly available. This allows for more flexible usage, like retraining from scratch with modifications. Models with open weights but closed training data are sometimes called open-weight models, while those with both open weights and open data are open models. So most so-called open-source models are actually just open-weight. These models also come with different licenses that may restrict commercial use or limit how you can use the model's outputs for training other models. For a model to be accessible to users, a machine needs to host and run it. The service that hosts the model and handles queries is often called the inference service, while the interface the users interact with is the model API. After creating a model, developers can choose to open-source it, make it accessible via an API, or both. Typically, model providers open-source their weaker models and keep their best ones behind paywalls.

Whether to host a model yourself or use a model API depends on several factors: first, data privacy—if your company has strict data privacy policies that prevent sending data outside the organization, externally hosted model APIs are not an option. There's also the risk that API providers might use your data to train their models; next, data lineage and copyright—most models aren't transparent about their training data, and intellectual property laws around AI are still evolving. It's unclear whether using a model trained on copyrighted data could create legal issues for your product; next, performance—the gap between open-sourced and proprietary models is closing, but the strongest models will likely remain proprietary. Commercial APIs often provide additional capabilities out of the box, like scalability, function calling (so accessing external tools, for example), structured outputs, and output guardrails. These can be challenging to implement yourself, so many companies turn to API providers. However, this means you'll be restricted to their functionality; you might not be able to fine-tune or access log probabilities, for example; typically, proprietary models are easy to start with and scale, but they can become expensive with heavy usage and offer less flexibility. It's wise to design your application with a standard internal API so you can easily swap between models if needed; control is another consideration—what happens if your API provider goes out of business, changes their terms of service, or is banned in certain regions? And if you want to run a model on device, third-party APIs aren't an option.

There are numerous benchmarks for different use cases, and a tool that helps you evaluate a model on multiple benchmarks is called an evaluation harness. For example, OpenAI Evals lets you run any of around 500 existing benchmarks to evaluate their models. When using public leaderboards, you need to consider which benchmarks to include in your aggregated ranking, how to weigh different benchmarks, and how to handle benchmarks that use different metrics (like accuracy, F1, BLEU, etc.). Keep in mind that the goal is to select a small subset of models for more rigorous testing with your own benchmarks and metrics. Public benchmarks rarely represent your application's needs perfectly, and they may suffer from data contamination—which is when the models were trained on the same data they're being evaluated on. To deal with contamination, you first need to detect it using heuristics like n-gram overlapping and perplexity. If perplexity on the evaluation data is unusually low, it's possible the model has seen this during training.

Once you've narrowed down your model candidates, you need a robust evaluation pipeline. Evaluate both the end-to-end output and each component (intermediate outputs) independently. You can use something called turn-based evaluation (where you assess the quality of each output) and task-based evaluation (where you measure whether the system completes a task and how many turns it takes). First, think about what makes a good response (factors like relevance, factual consistency, and safety). Then, create test queries and generate multiple responses to see how models perform. Develop detailed rubrics with examples for your scoring system. Whether you use binary scores, continuous scales, or something else depends on your data and your needs. The key is to make your rubric unambiguous so that human evaluators can follow it consistently. Most importantly, tie your evaluation metrics to business metrics. If your customer support chatbot's factual consistency is 80%, what does that mean for the business? Perhaps you can automate 30% of customer support requests at that level, but at 90% consistency, you could automate 50%. This lets you quantify the business impact of model improvements. You'll also need to establish a usefulness threshold; for instance, your chatbot must be 90% factually consistent to be viable in production. Different criteria might require different evaluation methods; you might use a specialized toxicity classifier, semantic similarity metrics to measure relevance, and an AI judge to assess factual consistency. You can even mix and match evaluation methods for the same criteria; for example, maybe use a cheap classifier on all your data and an expensive AI judge on just 1% for high-quality signals. While automated metrics are preferable for scale, don't hesitate to include human evaluation, even in production—just do it on a subset of data to keep costs manageable. It's also crucial to evaluate the application on different slices of data or users to ensure it performs well across segments and avoid biases. This helps you identify areas for improvement and prevent Simpson's Paradox (where a model performs better on aggregate but worse on each individual subset). How much evaluation data you need depends on your application and methods; generally, you want enough to be reliable but not so much that costs become prohibitive. A good way to test reliability is to create multiple bootstrap samples of your evaluation set and see if they yield similar results. If you get 90% on one bootstrap but 70% on another, your evaluation pipeline isn't trustworthy.

Finally, evaluate the reliability of your pipeline itself: first, is it getting signals right—do better responses indeed get higher scores? Next, do better evaluation metrics lead to better business outcomes? Third, how reliable is the pipeline—if you run it twice, do you get the same results? Fourth, how correlated are your metrics—you don't need two metrics if they're perfectly correlated, but completely uncorrelated metrics might indicate problems. And finally, what cost and latency does your evaluation pipeline add to your application? Model selection remains one of the hardest but most important topics in AI engineering. With the rapidly growing number of foundation models available, your challenge isn't developing models but selecting the right one for your specific needs, balancing performance, cost, privacy, and control.

Now let's dive into what might be the most accessible yet surprisingly nuanced aspect of AI engineering: prompt engineering. If you've ever used ChatGPT, you've already done some form of prompt engineering, but there's much more to it than just typing questions. Prompt engineering refers to the process of crafting instructions that guide a model to generate your desired outcome. It's the easiest and most common model adaptation technique because, unlike fine-tuning, it doesn't change the model's weights—you're just telling the model what you want it to do. While it's the most accessible entry point to AI engineering, don't be fooled into thinking that it's simplistic. Effective prompt engineering requires the same experimental rigor as any machine learning task. You should extract maximum value from prompting before moving to more...

Resource-intensive techniques like fine-tuning. That said, understanding prompt engineering alone isn't enough for production-ready systems. You'll still need knowledge of statistics, engineering, and classical ML for experiment tracking, evaluation, and data set curation.

Prompts typically consist of one or more of these components: first, the task description. This includes the model's role and expected output format. For example, "You are a helpful medical assistant. Analyze the following symptoms and suggest possible conditions, listing them in order of likelihood."

Next, examples. These show the model how to perform the task. For instance, if you want a model to classify text as toxic or non-toxic, you might include examples of each.

Third, the concrete task. This is the specific job you want the model to do, like answering a question or summarizing a book.

How much prompt engineering you need depends on the model's robustness to prompt perturbation. A robust model shouldn't produce dramatically different outputs if you write the number five versus write it out "FIV". This robustness is strongly correlated with a model's overall capability. It's also worth noting that different models have different preferred prompt structures. For example, GPT-4 typically performs better when the task description is at the beginning of the prompt, while Llama 3 does better when the task appears at the end.

Teaching models what to do via prompts is known as in-context learning. Each example in your prompt is called a shot, so we get the terms few-shot, zero-shot, and one-shot learning. How many examples you need depends on both the model and your application, so experimentation is necessary. The number of examples you can include is limited by the model's context length and, for API models, your cost constraints.

Many modern models distinguish between system and user prompts. The system prompt contains the task description, telling the model what role to play, its goals, and constraints. The user prompt contains the specific task or query. Almost all applications, like ChatGPT, have system prompts, usually created by the application developers rather than end-users. These system and user prompts are combined using a template that can vary between models and versions. If you use the wrong template, you might experience unexpected performance issues. Even small mistakes, like an extra new line, can cause problems when constructing inputs. Make sure to follow the model's chat template exactly. This is especially important if you're using third-party tools to construct prompts, as template mismatches often lead to silent failures.

Models typically understand instructions better when they appear at the beginning or end of the prompt rather than buried in the middle. Let's go through some key strategies for effective prompt engineering.

First, write clear and explicit instructions. If you want a model to score an essay, explain the scoring system you want it to use. Should it allow fractional scores? What should it do if it can't determine an answer? Be specific to reduce ambiguity.

Second, ask the model to adopt a persona. Asking a model to respond as a particular character or expert can significantly change its output style and focus. For example, "Respond as an experienced pediatrician" or "Answer as if you were explaining it to a 10-year-old."

Third, provide examples. Examples can dramatically shift a model's response style. For instance, asking "Will Santa bring me presents?" without examples might get a straight "No, Santa is fictional" response, but if you provide an example of a whimsical answer about the Tooth Fairy, the model is more likely to play along.

Fourth, specify the output format. Tell the model exactly how you want the response structured. This might mean requesting things like no preambles (so none of this "Based on the content of this essay, I'd give it a score of...") You can also ask for specific formats like JSON or Markdown and particular sections or headings.

Fifth, break complex tasks into simpler subtasks. This not only improves performance but also makes monitoring, debugging, and parallelization easier. However, it can increase the latency perceived by users if they don't see the intermediate outputs. You can also use cheaper models for simpler steps to reduce cost.

Sixth, give the model time to think. Several techniques can improve model reasoning: Chain of Thought prompting (so "think this through step by step"), process instructions (so something like "first analyze the key themes, second identify the author's perspective, and so on"), and self-critique (ask the model to check its own work). These approaches generally improve quality but increase latency and token usage.

Seventh, iterate systematically. This is so important. Different techniques work better for different models, so experimentation is crucial. Always version your prompts and use an experiment tracking tool with standardized evaluation metrics and data. Also, separate prompts from code; store them in configuration files rather than hardcoding them. This will make it way easier to update. Various tools aim to automate the prompt engineering workflow, including OpenPrompt and DSPI. These tools let you specify input and output formats, evaluation metrics, and evaluation data; then, essentially, they perform AutoML to find the optimal prompts. However, these tools can be expensive if they make many API calls under the hood. They also might produce prompts with typos or other issues, and they may not keep up with changing model requirements. For these reasons, it's best to start with manual prompt engineering before moving to automated tools. You can also use AI models themselves to write and refine prompts.

Once your application is available to users, it may face attacks from malicious actors trying to exploit it. Three main types of prompt attacks include: prompt extraction attacks (where attackers might try to extract your system prompt to either replicate or exploit your application), jailbreaking and prompt injection (the attacks attempt to subvert the model's safety features or get it to perform unauthorized actions, like providing instructions for harmful activities or executing dangerous code), and information extraction (these attacks try to get the model to reveal sensitive information from its training data or context).

To defend against these attacks, consider the following strategies: use benchmarks to evaluate safety against adversarial attacks, conduct security red teaming to proactively find weaknesses, be explicit in your prompts about what information the model should not return, repeat the system prompt before and after user inputs to remind the model of its constraints, design systems with safety boundaries (like running generated code only in isolated environments), require human approval for potentially impactful actions, define out-of-scope topics for your application, use anomaly detection to identify unusual prompts, and implement guardrails on both inputs and outputs. When evaluating your system security, track both the violation rate (so how often attacks succeed) and the false refusal rate (how often the model incorrectly refuses legitimate requests). You need to balance these metrics; perfect security with too many false refusals creates a really frustrating user experience.

By approaching prompt engineering with this combination of creativity and rigor, you can extract remarkable performance from foundation models without the complexity and expense of fine-tuning. Remember that small changes in your prompts can lead to significant improvements in output quality, so experiment widely and measure carefully.

Now that we've covered prompt engineering, let's explore how to give foundation models access to information beyond what they were trained on. To solve a task effectively, a model needs two things: instructions on how to perform the task and the necessary information to complete it. Two dominant patterns have emerged for providing models with the information they need: retrieval augmented generation (or RAG) and the agentic pattern. RAG allows models to retrieve relevant information from external data sources, while the agentic pattern enables models to use tools like web search and APIs to gather information actively. While RAG is primarily used for context construction, the agentic pattern can do much more. Let's start with RAG first.

So what is RAG? Retrieval augmented generation is a technique that enhances a model's generation capabilities by retrieving relevant information from external memory sources. These sources could be an internal database, a user's previous chat sessions, or even the internet. You can think of RAG as a technique to construct context specific to each query, connecting the model with information it wasn't trained on or might have forgotten. A RAG system consists of two main components: a retriever that fetches the information from the external memory source and a generator, the foundation model that produces a response based on the retrieved information. In today's RAG systems, these components are often trained separately, with many teams using off-the-shelf retrievers and models. However, fine-tuning the entire RAG system from end to end can significantly improve performance. The success of a RAG system heavily depends on its retriever. A retriever performs two main functions: indexing and querying. Indexing involves processing data so that it can quickly be retrieved later; this is the preparatory step where you organize your knowledge base. Querying is the process of sending a search query to retrieve data relevant to it. How you index your data determines how you retrieve it later.

Let's walk through a simple example. Imagine your external memory as a database of documents, like contracts or meeting notes. These documents can range from 10 tokens to a million tokens in length. Naively retrieving whole documents would make your context arbitrarily long, potentially exceeding the model's context window. To avoid this, you typically split each document into smaller chunks, which we'll discuss later. For each user query, your goal is to retrieve the data chunks most relevant to that query. Then, with some post-processing to join the retrieved chunks with the user's prompt, you get the final prompt that goes to the model.

Many existing retrieval algorithms can be used for RAG. Retrieval works by ranking documents based on their relevance to a given query, and algorithms differ in how they compute these relevant scores. First, term-based retrieval. This is also called lexical retrieval, and this approach finds relevant documents based on keywords. While this is straightforward, it has several limitations: many documents might contain a term without truly being about it, and queries can be long with many terms that aren't equally important, so TF-IDF can help address this. Also, simple tokenization can miss semantic relationships. Term-based retrieval is generally faster than embedding-based approaches during both indexing and querying; it also works well out of the box with existing systems like ElasticSearch.

Embedding-based retrieval is another option. This approach computes relevance at the semantic level rather than a lexical one, ranking documents based on how closely their meaning aligns with the query. The process works like this: convert your original data to embeddings using an embedding model, store these embeddings in a vector database. When a query comes in, convert it to an embedding using the same model, fetch the K data chunks whose embeddings are closest to the query embedding, and return them. Vector search is typically framed as a K-nearest neighbor search problem. This can be computationally expensive for large data sets, so approximate nearest neighbors algorithms are often used instead. In practice, most developers won't implement vector search themselves but will use existing vector databases. These databases organize vectors into buckets, trees, or graphs using various heuristics to increase the likelihood that similar vectors are stored close to each other. Embedding-based retrieval can significantly outperform term-based retrieval over time, especially if you fine-tune your embedding model and retriever, but it has its downsides: it can make it harder to search for specific names or error codes, and generating embeddings can be expensive and introduce latency.

A production retrieval system typically combines several approaches. For example, a cheaper, less precise retriever like term-based search might first fetch candidates, and then a more precise but expensive mechanism like KNN finds the best options among those candidates. Depending on your task, certain tactics can increase the chance of retrieving relevant documents. The simplest approach is to divide documents into chunks of equal length based on characters, words, sentences, or paragraphs. Overlapping chunks can ensure that important boundary information is included in at least one chunk. Smaller chunk sizes allow for more diverse information since you can fit more chunks into the model's context, but this can also result in the loss of important context. Smaller chunks also increase computational overhead, especially for embedding-based retrieval. There's no universal best chunk size or overlap percentage; you just need to experiment based on your specific data and task.

The initial document rankings generated by the retriever can be further refined to be more accurate. This is especially useful when you need to reduce the number of retrieved documents due to context window limitations. Documents could be reranked based on various factors such as recency (so maybe you give more weight to newer data) or additional relevant signals.

Next, let's talk about query rewriting, also known as query reformulation, normalization, or expansion. This technique involves rewriting queries to include necessary context. For example, if a user asks "What's its population?" after previously asking about Paris, the query might be expanded to "What's the population of Paris?" Each chunk can be augmented with relevant context to make it easier to retrieve. This might include metadata like tags and keywords, or for e-commerce products, it could be information like descriptions and reviews. You can also augment chunks with context from the full document to help them retain more of the original meaning; for example, maybe a summary of the entire document.

When choosing a retrieval solution, consider: what retrieval mechanisms it supports (term-based, embedding-based, and/or hybrid), for vector databases, what embedding models and vector search algorithms are supported, also consider scalability (both for data storage and query traffic), indexing speed and batch processing capabilities, query latency, pricing structure, and compliance requirements as well. It's also important to note that RAG isn't limited to just text; it can also be used with multimodal and tabular data. For instance, if a user asks "What's the color of the house in the Pixar movie Up?", a multimodal RAG system might first retrieve an image of the house to help the model answer. Similarly, RAG can work with tabular data using text-to-SQL conversations; the system can execute a query on a database and then generate a response based on the results. For complex database schemas, you might need an intermediate step to predict which table to use for each query, especially if there are too many tables to fit all the schemas in the context window.

In the next part, we'll explore the agentic pattern, which goes beyond passive retrieval to actively interact with external tools and APIs.

The agentic pattern is a more active approach to extending AI capabilities. This is a rapidly evolving field, so consider this section more experimental than the others we've covered. At its broadest definition, an agent is anything that can perceive its environment and act upon it. For AI systems, this means that a model can observe its environment, make decisions based on those observations, take actions that affect the environment, and learn from the outcomes of those actions. The environment is defined by the use case: for a game-playing agent, the game is the environment; for a web scraping agent, the internet is the environment. What makes agents powerful is the set of tools they have access to. For example, ChatGPT is an agent that can search the web, execute Python code, and generate images, among other capabilities. Remember our RAG example with tabular data? That was actually a simple agent with three actions: generating SQL queries, executing those queries, and producing a response. Let's see how this works in practice. If a user asks "Project the sales revenue over the next 3 months," the agent might first reason about how to accomplish the task, then generate a SQL query to fetch historical sales data. Next, it would execute that query against the database, analyze if the retrieved information is sufficient (possibly generate and execute additional queries), and then create a projection based on the gathered data. Finally, it would conclude that the task has been successfully completed.

Compared to simpler AI applications, agents require more powerful models because they often need to perform multiple steps to complete a task. The overall success rate decreases with each step because of compounding errors, and the stakes are higher since agents have access to potentially powerful tools.

Speaking of tools, agents can be equipped with various tools, which fall into several categories: first, knowledge augmentation tools (these could be things like text or image retrievers as in RAG, SQL executors for database access, web search capabilities, APIs for accessing inventory systems, email readers, etc., and web browsers for navigating online content, whether public or private); next, we have capability extension tools (like calculators, since AI models often struggle with complex math, time zone or unit converters, translation services, and code interpreters); we also have write-action tools (so tools that enable the agent not just to read but also write to systems; these can automate workflows but require strong security protocols).

Complex tasks require planning, and there are many possible ways to decompose a task. Not all approaches will be successful, and not all will be efficient. To help with debugging and to prevent cases where a model executes unnecessary API calls, planning should be decoupled from execution. The process typically works like this: first, ask the agent to generate a plan; then, validate the plan before execution; and then only execute once validated. Plans can be validated using heuristics (like removing plans with invalid actions or too many steps) or by using another AI model as a judge. You can even generate several plans in parallel and then ask an evaluator to pick the most promising one. For particularly important or sensitive tasks, you might want a human in the loop to review plans before execution. While foundation model agents use the model itself as the planner, reinforcement learning agents are trained using reinforcement learning algorithms. This approach uses more resources than foundation models but could offer performance improvements in the future.

The simplest way to turn a model into a plan generator is through prompt engineering. You tell the model what functionality it has available and the expected inputs and outputs for each tool. You can improve your prompts by writing better system prompts with more examples, providing clearer descriptions of tools and their parameters, simplifying functions as much as possible, using a stronger model, or fine-tuning a model specifically for plan generation. As a practical tip, always ask the system to report what parameter values it uses for each function call; this provides a sanity check that can catch many issues before execution. Another useful approach is to generate plans in natural language first, then translate them to the exact function calls in a second step. This helps if function names change over time or if you find a model specifically for plan creation; the translation can often be done by a smaller, cheaper model.

Agents can fail in various ways, so it's important to have robust evaluation methods. There are lots of different things that can go wrong: we could have planning failures (like using invalid tools, using valid tools but with invalid parameters, using valid tools with incorrect parameter values, or failing to achieve the goal or satisfy constraints). To evaluate planning capability, create a data set where each example is a tuple of task, available tools, and constraints. For each task, use the agent to generate multiple plans and compute metrics like: the percentage of generated plans that are valid, how many attempts it takes to get a valid plan, percentage of tools called that are valid, and how often invalid tools are called. You could also have tool failures (so that could include things like bad translation from high-level plans to specific function names, no access to the required tools, or tools giving incorrect outputs, like poorly generated SQL queries). For this, your efficiency metrics might be: how many steps does the agent need on average to complete a task, what's the cost to complete a task, how long does each action typically take, are there particularly slow or expensive actions, and how does the agent compare to baselines (which might be another agent or a human).

One of the key challenges for agents is remembering information over time. A memory system allows a model to retain and utilize information across interactions. A model typically has three main memory mechanisms: there's the internal knowledge embedded in the model itself through training, there's the context window (which is kind of your short-term memory for immediate session-specific information), and finally, external data sources like RAG systems (this is kind of like your long-term memory; information that is essential to all tasks should be incorporated via training; rarely needed information should reside in long-term memory, while short-term memory is for immediate context-specific information). Benefits of a well-designed memory management system include: storing information longer than the context window allows, persisting information between sessions, making a model more consistent in its responses and actions.

By combining RAG for information access, tools for capability extension, planning for complex tasks, and memory systems for continuity, agents can tackle increasingly sophisticated problems. While this field is still evolving rapidly, it represents one of the most promising frontiers in AI engineering. As with all powerful technologies, agent systems require careful consideration of safety, security, and ethical use. The more capable an agent becomes, the more critical it is to ensure it operates within appropriate boundaries and with proper oversight.

Now let's explore fine-tuning: the process of adapting a model to a specific task by further training it and adjusting its weights. While prompt engineering and RAG are relatively lightweight techniques, fine-tuning offers deeper customization but requires more resources and expertise.

So when to fine-tune? Fine-tuning can improve a model's performance in two ways: first, by enhancing domain-specific capabilities (like coding or answering medical questions), and second, improving instruction-following abilities (like adhering to specific output formats). However, fine-tuning requires significant upfront investment; it often needs more memory than what's available on a single GPU, making it expensive. This is why reducing memory requirements has become a primary motivation for many fine-tuning techniques that we'll discuss later. So you should consider fine-tuning when: you've already exhausted what you can achieve with prompt-based methods, you need to produce consistent, structured outputs, and you're working with smaller models that need to perform better on specific tasks. A common approach is model distillation: fine-tuning a small model to imitate a larger model's behavior using data generated by the large model on specific tasks. A small, fine-tuned model may outperform a larger, general-purpose model. On the other hand, you should avoid fine-tuning if: you need a general-purpose model (fine-tuning can improve performance on specific tasks but degrade performance on others), or if you're just starting to experiment with a project (many teams jump straight to fine-tuning before thoroughly exploring simpler approaches).

So what about fine-tuning versus RAG? After you've maximized performance gains from prompting, choosing between RAG and fine-tuning depends on whether your model's failures are information-based or behavior-based. If the model fails because it lacks information (like private company data or recent events), RAG gives the model better access to that information. If the model has behavioral issues (like outputs that are factually correct but irrelevant or are in the wrong format), fine-tuning might help more. If your model has both issues, start with RAG because it's easier; begin with a simple term-based solution and evolve from there. In many cases, combining RAG and fine-tuning will give you the biggest performance boost.

So the workflow to adapt a model to a task might be: first, design evaluation criteria and an evaluation pipeline; then, try to get the model to perform the task with prompting alone; add more examples to the prompt; from there, if the model continues to have information-based failures, try more advanced RAG (like embedding-based retrieval); if it continues to have behavioral issues, opt for fine-tuning; finally, combine RAG and fine-tuning for a bigger performance boost.

Because of the scale of foundation models, memory is a major bottleneck for both inference and fine-tuning. The memory requirements for fine-tuning are typically much higher than for inference due to how neural networks are trained. Neural networks are typically trained using backpropagation. Each training step consists of a forward pass (where we compute the output from the input) and a backward pass (where we update the model's weights using signals from the forward pass). During inference, only the forward pass is executed; during training, both passes are needed.

The key contributors to a model's memory footprint during fine-tuning are: the total number of parameters, the number of trainable parameters, and the numerical representation of these parameters. A trainable parameter is one that can be updated during fine-tuning; so during pre-training, all model parameters are updated; during inference, no parameters are updated; and during fine-tuning, some or all of the parameters may be updated. Parameters that remain unchanged are called frozen parameters.

One way to reduce training memory is through gradient checkpointing (also called activation recomputation), where activations aren't stored but recomputed as needed. This increases training time but reduces memory requirements. The key insight here is that the more trainable parameters we have, the higher the memory footprint; reducing the number of trainable parameters reduces memory requirements. This is the motivation behind parameter-efficient fine-tuning, which we'll talk about in a bit.

Another way to reduce the memory footprint is through quantization: converting a model from a format with more bits to one with fewer bits. For a 13-billion parameter model using 32-bit floating point, each parameter requires 4 bytes, resulting in 52 GB total. So if you reduce each value to 16 bits, the memory needed drops to 26 GB. Inference is typically done using as few bits as possible (16, 8, or even 4 bits). Training is more sensitive to numerical precision, so it's usually done in mixed precision, with some operations in higher precision (like 32-bit) and others in lower precision (like 16 or 8-bit). Different numerical formats balance range (the span of values that can be represented) and precision (how exactly a number can be represented). There are a few different formats. Reducing precision can cause values to change or result in errors, so it's important to load models in their intended format. For example, when Llama...

2 is released; its weights are optimized for bf16, causing significantly worse quality when loaded with fp16.

Now let's talk about PFT. In the early days of smaller models, full fine-tuning—so updating all the model parameters—was common. This required a lot of high-quality annotated data and substantial computational resources. As models grew, people started using partial fine-tuning, focusing on specific layers, like only the last layer. This reduces memory requirements, but it isn't very parameter efficient. Parameter-efficient fine-tuning techniques insert additional parameters into strategic IC locations in the model to achieve strong fine-tuning performance with a small number of trainable parameters. While this can increase inference latency slightly, as adapters add computational steps, PFT methods are generally not only parameter efficient but also sample efficient; they can work with just a few thousand examples compared to the millions potentially needed for full fine-tuning.

PFT methods fall into two categories: so we have adapter-based methods—this is also called additive methods—that add new model weights, and then we have soft prompt-based methods that introduce special trainable tokens. The most popular adapter-based method is LoRA (low-rank adaptation). Unlike traditional adapters, LoRA incorporates additional parameters without increasing inference latency. Instead of adding new layers, LoRA uses modules that can be merged back into the original layers. LoRA works by decomposing weight matrices into products of smaller matrices, then updating only these smaller matrices. For a weight matrix with dimensions n by m, LoRA first chooses a smaller dimension R (the rank), then creates two matrices: A, which is n by R, and B, which is R by m. During fine-tuning, only A and B are updated, while the original weights remain frozen. For inference, A and B can be multiplied together and added to the original weights. The efficiency of LoRA depends both on the chosen rank and which matrices it's applied to; it's primarily used for Transformer modules in the attention modules.

If you want to fine-tune a model for multiple tasks, you have several options. First, simultaneous fine-tuning: training on a data set with examples from all tasks at once. This is harder and requires more data. Or you could do sequential fine-tuning, where you first train on task A and then on task B, but this can cause catastrophic forgetting, where the model loses its ability on earlier tasks. Or you can try model merging; so there you fine-tune different tasks separately, then combine the resulting models. Model merging offers greater flexibility than fine-tuning alone. If you have two models that excel at different aspects of the same task, you can merge them into a single model that outperforms both. This approach can be done without GPUs; it can improve performance while reducing the memory footprint; it's an excellent option for on-deployment, and it can facilitate federated learning, where multiple devices train using separate data. Unlike ensembling, which combines the outputs of multiple models, merging combines the models themselves. This improves performance without the higher inference cost of running multiple models. Several merging approaches exist: so we have summing, where we just add the weight values of the constituent models together—this is the most common—we could have layer stacking, so we take different layers from different models and stack them—this is also called Franken merging—or concatenation, where we just combine the parameters—this is less recommended because it doesn't reduce memory compared to separate models.

Here's a practical fine-tuning approach and what a typical development path might look like. First, test your fine-tuning code using the cheapest, fastest model you have and ensure it works. Then test your data by fine-tuning a mid-size model; if training loss doesn't decrease with more data, something might be wrong. After that, run experiments with your target model to see how far you can push performance, and then map the price-performance frontier and select the model that makes the most sense for your use case. Alternatively, a distillation path looks like this: start with a small data set and the strongest model you can afford; then train the best possible model with this small data set; use this fine-tuned model to generate more training data; use the expanded data set to train a cheaper model.

When choosing fine-tuning methods, here are some things to consider. So, for beginners, start with adapter techniques like LoRA before attempting full fine-tuning. Understand that data volume matters; full fine-tuning typically requires thousands to millions of examples, while PFT can work with hundreds. Also, you'll need to know how many fine-tune models you need; adapter methods let you serve multiple variants that share a base model. There are also some key hyperparameters that you should know; these ones in particular significantly impact fine-tuning results: so we have the learning rate—just like in machine learning, if the loss curve fluctuates, the learning rate is likely too high; if it's stable but decreases very slowly, the rate's probably too low—generally start larger and decrease over time; we also have batch size—larger batches process training examples faster but require more memory; small batches lead to more unstable training, so to address instability, you can accumulate gradients across several batches; we also need to think about the number of epochs—smaller data sets typically need more epochs than larger ones; for millions of examples, one to two epochs might suffice; for thousands of examples, 4 to 10 may be needed; reduce epochs if you see overfitting; we also have prompt loss weight for instruction fine-tuning—this determines how much prompts should contribute to the loss compared to the responses; if it's set to 100%, prompts and responses contribute equally; if it's 0%, the model learns only from responses; the default is typically 10%.

While the technical process of fine-tuning has been simplified by frameworks that handle the training process and suggest sensible defaults, the strategic decisions around fine-tuning remain complex. The key is knowing when to fine-tune, which technique to use, and how to balance the trade-offs between performance, resources, and data requirements. While most companies can't afford to train foundation models from scratch, nearly all can differentiate themselves through high-quality data sets for adaptation. As the saying goes, garbage in, garbage out, and nowhere is this more true than in data set engineering. We're witnessing a shift from model-centric to data-centric approaches in AI development. Model-centric AI tries to improve performance by enhancing the models themselves—so designing new architectures, increasing model sizes, or developing new training techniques. Data-centric AI, on the other hand, focuses on improving performance by enhancing the data—developing better data processing techniques and creating high-quality data sets that allow superior models to be trained with fewer resources. For companies adapting foundation models rather than building them from scratch, the data-centric approach offers the greatest competitive advantage.

The type of data you need depends on your adaptation task. For self-supervised fine-tuning, you need sequences of relevant domain data. For instruction fine-tuning, you need data in instruction-response format. For preference fine-tuning, you need instruction-winning response-losing response format. For reward modeling, you need either preference data or examples with explicit scores. Your training data should exhibit the behaviors you want your model to learn. This can be particularly challenging for complex behaviors like chain of thought reasoning or tool use in agent workflows. When developing conversational applications, you need to consider whether you require single-turn data, multi-turn data, or both. Single-turn data helps train a model to respond to individual instructions, while multi-turn data teaches the model how to solve tasks through dialogue—like clarifying user intent before addressing the task or incorporating corrections. A small amount of high-quality data can outperform a large amount of noisy data—a principle confirmed by teams working on models like LLaMA 3. They found that human-generated data is often prone to errors and inconsistencies, particularly for nuanced policies, leading them to develop AI-assisted annotation tools to ensure high quality, which is interesting to me. But what makes data high quality? There are several factors to consider. First, relevance: the examples should be relevant to your target task; legal text from the 19th century might not be relevant for answering contemporary legal questions. You'll also need alignment with task requirements; if your task focuses on factual consistency, annotations need to be factually correct; if it demands creativity, annotations should be creative. We also need to think about consistency: annotations should be consistent across examples and annotators; they need to be correctly formatted, so data should adhere to the expected structure; they need to be sufficiently unique—you want minimal duplicates in your data set; they need to be compliant and follow internal and external policies; and you need coverage: your training data needs to cover the range of possible problems you want to solve, requiring sufficient diversity; missing coverage in important areas will result in poor performance for those cases, no matter how much data you have overall.

But how much data do you need? Asking how much data you need is kind of like asking how much money you need; the answer varies widely depending on your situation. Several factors influence data requirements. So, if you're fine-tuning, then the fine-tuning technique matters; full fine-tuning typically requires orders of magnitude more data than parameter-efficient methods like LoRA—with tens of thousands to millions of examples, full fine-tuning might be appropriate; with just hundreds to a few thousand examples, PFT methods will likely work better. It also depends on your task complexity; a simple sentiment classification task requires much less data than complex question answering about financial filings, for example. The base model performance also makes a difference; the closer the base model is to your desired performance, the fewer examples you'll need; larger, more capable base models generally require fewer examples to fine-tune effectively. OpenAI's fine-tuning guide demonstrates that with fewer examples (around 100), more advanced models give better fine-tuning results; however, after fine-tuning on a large data set (around 550,000 examples), all models perform similarly, regardless of their initial capabilities. So, in short, with limited data, use PFT methods on more advanced models; with abundant data, full fine-tuning on smaller models becomes viable. Before investing in a large data set, start with a small, well-crafted set of around 50 examples to see if fine-tuning improves your model. If you see clear improvements, more data will likely help further; if you see no improvement with a small data set, a larger one rarely solves the problem, though be careful to rule out other issues like poor hyperparameters or data quality first. In most cases, you should see improvements after fine-tuning with just 50 to 100 examples. You can also reduce the amount of high-quality data you need by first fine-tuning on more accessible data. So, one path might be self-supervised to supervised: first fine-tune on domain-specific documents, then on targeted question-answer pairs; or less relevant to more relevant data: first fine-tune on adjacent domains with abundant data, then on your specific domain; or synthetic to real data: first fine-tune on AI-generated examples, then on limited real examples. Experimenting with subsets of your current data set—so maybe 25%, 50%, and 100%—can help estimate how much more data you'll need. A steep performance gain with increasing data set size suggests significant improvement from doubling your data; a plateau indicates diminishing returns.

So let's say you need more data; how can you get it if you don't have enough for your use case? If possible, you'll want to create a data flywheel that leverages user interactions to continuously improve your product; this offers a significant competitive advantage. Or you could also just check available data sets; you can often mix and match different sources, though all data must be thoroughly verified for quality and appropriate licensing. When annotating your own data, the challenge isn't just the annotation process but creating clear guidelines; you need to explicitly define what makes a good response—can a response be correct but unhelpful? What distinguishes a score of 3 versus 4?—these guidelines are crucial both for human and AI-powered annotations. Trust me, one of the hard machine learning problems I've ever had to solve was an issue with human labelers. Data augmentation creates new examples from existing data, which is another option; so you could do things like flipping an image to create a new variant, or you could use data synthesis—this generates artificial data that mimics real data properties, like simulating mouse movements on a web page. The key difference between augmented data and synthetic data is that augmented data is derived from real data, while synthetic data is created from scratch. Data synthesis, therefore, is particularly valuable for addressing privacy concerns when working with sensitive information. Together, some combination of these techniques should allow you to produce data at scale, increase coverage across your problem space, and possibly improve quality with AI-generated data, since humans aren't always great at creating consistent data. But, of course, make sure to measure the quality of your AI-generated data just like you would for human-generated data.

Once you have your data, you need to process it. Data processing can be time-consuming, but it is critical for quality. Here are some best practices: start with filtering tasks and test scripts before big runs; avoid changing data in place—so you want to make sure to keep the originals; perform exploratory data analysis on distributions and outliers; examine inter-annotator disagreement and resolve conflicts; fact-check and manually inspect examples; de-duplicate data to prevent over-representation; clean formatting tokens like HTML and Markdown, which can improve performance and reduce input size; remove non-compliant data—so anything like PII, toxic material, or copyrighted content; filter out low-quality data identified during verification; if you have more data than your compute budget allows, use active learning to select the most helpful examples; and ensure data is in the right format for your model, using the appropriate tokenizer and chat template. While all these steps require a lot of effort, they're essential for creating data sets that will help your model to shine in the competitive landscape of AI applications. Well-engineered data sets often make the difference between mediocre and exceptional performance.

Now let's dive into one of the most practical aspects of AI engineering: inference optimization. After all, a model's real-world usefulness boils down to two crucial factors: how much it costs to run and how quickly it responds. These characteristics—inference cost and latency—ultimately determine which applications can practically use AI and at what scale. Let's start by understanding what we mean by inference. In the AI life cycle, there are two distinct phases in an AI model's journey: training and inference. Training builds the model, while inference uses the model to compute outputs for given inputs. In a production environment, the component responsible for running the model (inference) is called an inference server. This server hosts available models, allocates hardware resources to execute them, and returns responses to users. The inference server is part of a broader inference service that also handles receiving, routing, and pre-processing requests. So what does this mean for you? Well, if you're using a model API like those from OpenAI or Google, you're essentially outsourcing this inference service; but if you decide to host models yourself, you'll need to build, optimize, and maintain your own inference infrastructure. To optimize inference, we first need to understand what's slowing things down. Generally speaking, AI workloads face two types of bottlenecks: compute-bound bottlenecks occur when the limiting factor is the computational power available; tasks requiring intensive calculations, like image generation, are typically compute-bound; memory bandwidth-bound bottlenecks occur when the limiting factor is how quickly data can move between memory and processors; autoregressive language model inference is typically memory bandwidth-bound. Profiling tools like NVIDIA Nsight can help determine which bottleneck affects your workload through something called a roofline chart. What's important to understand is that different optimization techniques address different bottlenecks; a compute-bound workload might benefit from more powerful chips or distributing work across multiple chips; meanwhile, a memory bandwidth-bound workload might see better results from chips with higher memory bandwidth.

Now that we understand bottlenecks, let's look at how inference is actually served. Many providers offer two distinct types of inference APIs, each optimized for different use cases. Online APIs optimize for latency, processing requests as soon as they arrive; chatbots typically use online APIs since users expect quick responses. Batch APIs, on the other hand, optimize for cost, processing multiple requests together more efficiently but with higher latency; applications without strict response time requirements, like periodic report generation or synthetic data creation, can benefit from batch processing. The key is matching your inference type to your application's needs. So now, how do we measure if our inference is performing well? That brings us to our next section. Here are some key inference performance metrics. To optimize effectively, we need to know what we're measuring; several metrics help us evaluate inference performance. The first, and perhaps most notable metric, is latency: the time from when users send a query until they receive a complete response. For autoregressive models like LLMs, latency breaks down into two components: so we have the time to first token (TTFT), which is how quickly the first token is generated after receiving a query, and then we have time per output token (TPOT), how long it takes to generate each subsequent token. The total latency then equals TTFT plus TPOT times the number of output tokens. Some teams also measure time to publish (TTP) because the first generated token isn't always immediately shown to users, especially when the model first generates a plan or uses chain of thought reasoning. One important note about latency: since it varies across requests, looking at percentiles gives you much more meaningful information than simple averages. Beyond latency, we also care about throughput, which is the number of output tokens per second an inference service can generate across all requests; higher throughput typically means lower cost, which is why optimizing for it matters for production systems. It's worth mentioning that most AI applications face a fundamental latency-throughput tradeoff; techniques like batching can improve throughput but may increase latency for individual requests; your optimization strategy needs to balance these competing priorities based on your specific application needs. Finally, utilization metrics tell us how efficiently we're using our resources: we have model FLOPS per second utilization, which is the ratio of observed throughput relative to the theoretical maximum at peak computing power; model bandwidth utilization, which measures the percentage of available memory bandwidth being used.

Now that we know what to measure, let's look at the hardware that powers inference. At the heart of inference performance is specialized hardware. An accelerator is a chip designed to speed up specific types of computation. For AI workloads, the dominant accelerators are GPUs; those specialized AI chips are growing in popularity. You might be wondering about the difference between CPUs and GPUs; it comes down to their architecture. CPUs have a few powerful cores—typically up to 64 for high-end machines—which are optimized for general-purpose computing. GPUs, on the other hand, have thousands of smaller cores optimized for parallel processing; this makes them ideal for matrix multiplication operations that dominate ML workloads. Interestingly, training and inference have different hardware requirements; training demands more memory due to backpropagation and is generally more difficult to perform; lower-precision inference often emphasizes latency over throughput since users are typically waiting for responses. When evaluating hardware for inference, consider three key questions: can it run your workloads? How long does it take to do so? And how much does it cost? The specific hardware specifications to focus on include FLOPS (computing power), memory size, and memory bandwidth. For compute-bound workloads, prioritize chips with more FLOPS; for memory-bound workloads, focus on higher bandwidth and more memory.

With the hardware foundations covered, let's move on to techniques for optimizing at the model level. Now we're getting into the real tactics for speeding up inference. Let's start with model-level optimizations: techniques that make the models themselves more efficient. Model compression reduces a model's size, potentially making it faster; there are several approaches here: quantization, which we already discussed, reduces numerical precision; pruning removes less important parameters or sets them to zero; and distillation, which we also already discussed, trains a smaller model to mimic a larger one. Among these options, weight-only quantization is by far the most popular because it's relatively easy to implement, works well for many models out of the box, and delivers significant benefits without that much effort. Another challenge specific to language models is their autoregressive nature: they generate text one token at a time, which creates a sequential bottleneck. Several techniques address this limitation: speculative decoding uses a faster but less powerful model to generate candidate tokens, which are then verified by the target model; it's like having an assistant draft responses and a manager quickly review and approve; inference with reference copies tokens from the input when appropriate—for example, when answering questions about a document rather than generating them from scratch—this can significantly speed up responses for document-based queries; parallel decoding aims to generate multiple tokens simultaneously, breaking the sequential constraint; additionally, attention mechanism optimization improves the efficiency of Transformer models; attention calculations, which can be particularly memory-intensive. At an even lower level, kernels and compilers optimize how models run on specific hardware. Kernels are specialized code optimized for hardware accelerators; common optimization techniques include vectorization, parallelization, loop tiling, and operator fusion. Compilers bridge machine learning models and hardware, converting model operations into optimized code for specific accelerators. But optimization doesn't stop at the model level; let's look at how we can optimize the entire inference service. We can achieve significant performance gains by efficiently managing resources across an entire inference service. One of the most powerful techniques is batching, which combines multiple requests to process together. Batching can be implemented in different ways: so we have static batching, which groups a fixed number of inputs, but all requests must wait until the batch is full—this is simple but can lead to inconsistent latency; dynamic batching sets a maximum time window, processing the batch when either it's full or the time limit has been reached—this provides more consistent latency guarantees; finally, we have continuous batching, which allows responses to be returned as soon as they're completed, with new requests added to maintain batch size—this provides the best user experience but is more complex to implement. Another powerful technique is decoupled prefill and decode, which separates these two phases of LLM inference since they have different computational needs; handling them separately prevents resource competition and improves overall efficiency. For applications with repetitive patterns, prompt caching stores overlapping text segments—like system prompts or reference documents—to avoid reprocessing them with each query; this is particularly valuable for applications with long conversations or multiple queries about the same document. As models grow larger, a single machine may not be sufficient; this is where parallelism comes in. Distributing work across multiple machines: replica parallelism creates multiple copies of the model, each handling different requests—this is the simplest approach and works well for high-throughput scenarios; model parallelism splits a single model across machines—either through tensor parallelism (breaking operations into smaller pieces), pipeline parallelism (dividing the model into sequential stages), context parallelism (splitting input sequences across devices), or sequence parallelism (splitting different operations across machines). So what technique should you implement? We just talked about a lot; the optimal combination depends on your specific workloads and performance requirements. For applications prioritizing low latency, replica parallelism may be best, despite higher costs. For most use cases, the most impactful techniques are typically quantization, tensor parallelism, replica parallelism, and attention mechanism optimization. By thoughtfully applying these techniques, you can dramatically improve both the speed and cost-effectiveness of your AI applications, making them more responsive to users while keeping your infrastructure cost manageable.

In our next and final section, we'll see how all these components come together in a complete AI application architecture and how user feedback creates a virtuous cycle of continuous improvement. Now that we've explored all the individual components of AI engineering, it's time to pull everything together. Let's see how these pieces fit into a complete architecture and how user feedback creates a powerful loop that helps these systems improve over time. The simplest AI application architecture looks like this: your application receives a query, sends it to a model (either through a third-party API or self-hosted model), and returns the response to the user—no bells, no whistles, just direct input and output. But real-world applications rarely stay this simple. Let's walk through how these architectures typically evolve as your needs grow more sophisticated. The first enhancement most applications need is better context construction: giving the model access to information required to process useful outputs; this is essentially feature engineering for foundation models; so you might add RAG systems to search and retrieve information from your knowledge base, agent capabilities to gather information from external tools, document uploading functionality to analyze specific content, or more. These additions ensure the model has the necessary context to provide accurate, relevant responses. Step two: add guardrails for protection. As your application grows in capability, you'll need guardrails to protect both your system and your users. Input guardrails protect against leaking private information to external APIs and malicious prompts that could compromise your system. Output guardrails catch different types of failures: quality failures (like empty responses, incorrect formatting, or factually incorrect content) or security failures (like toxic content, PII exposure, or unauthorized actions). The key again is balancing protection with user experience; overly restrictive guardrails create frustrating experiences, while inadequate ones could leave you vulnerable. Stage three: implement model routing and gateways. As your application matures, you may discover that one model doesn't fit all your needs; different queries require different approaches, and this is where model routing comes into play. A model router typically includes an intent classifier that predicts what the user is trying to do and then directs the query to the appropriate model or pipeline. These routers should be fast and inexpensive, so you can use multiple of them without adding significant latency or cost. Along with routing, you'll need a model gateway to handle requests, manage resources, and provide a consistent interface to your applications. The gateway serves as a central point for managing all your AI models, improving scalability and maintainability. This ensures that your AI applications can gracefully handle increasing demand and adapt to evolving needs.

Gateway: This is an intermediate layer that provides a unified interface to different models, both self-hosted and commercial. Access control and cost management, fallback policies to handle rate limits or API failures, and load balancing, logging, and analytics. The Gateway approach makes your codebase much more maintainable; if a model API changes, you only need to update the Gateway, not every application that uses it. It's a classic example of separation of concerns in software engineering.

Next stage: Four. Optimize with caching. As your user base grows, performance and cost optimization become increasingly important. This is where caching enters the picture. Inference caching includes techniques like KV caching to optimize the attention mechanism and prompt caching to avoid reprocessing identical prompt components. Caching is particularly valuable for multi-step processes like Chain of Thought reasoning or queries requiring time-consuming actions like retrieval or web searches. For implementation, your options range from in-memory storage, which is fast but has limited capacity, to databases like PostgreSQL and Redis. You'll also need an eviction policy, like least recently used or least frequently used, to manage cache sizes as you scale.

Stage five: Add complex logic and write actions. This is where the most sophisticated AI applications go beyond simple question answering to incorporate complex multi-step reasoning flows, agentic patterns with loops and decision-making, and write actions that make changes to the environment. Write actions, like sending emails, placing orders, or initiating transfers, dramatically increase your system's capabilities but also introduce significant risks. These should be implemented with extreme caution and appropriate safeguards. As your architecture grows in complexity, keeping track of everything becomes increasingly challenging. This is where monitoring and observability become critical. While related, they serve slightly different purposes. Monitoring tracks external outputs to detect when something goes wrong but doesn't necessarily help identify the cause. It's like knowing your car broke down but not why. Observability, on the other hand, ensures that sufficient information about your system's internal state is collected so that when something goes wrong, you can diagnose the issue without deploying new code. It's like having sensors throughout your car that can pinpoint exactly what failed. There are three key metrics that can help you evaluate your observability: MTTD (mean time to detection), MTTR (mean time to response), and CFR (change failure rate), which is the percentage of deployments that result in failures. Each component in your pipeline should have its own metrics, and you should understand how these metrics correlate to your business's North Star metrics. Remember the golden rule of observability: just log everything. When metrics indicate a problem, detailed logs help you identify exactly what went wrong.

As your application evolves to include multiple models, data sources, and tools, managing these interactions can become increasingly complex. This is where an orchestrator becomes valuable, helping you specify how these components work together. AI orchestrator tools like LangChain, LlamaIndex, Flowise, Langflow, and Haystack help manage these complex pipelines. However, it's often wise to start building your application without an orchestrator first, to understand the core mechanics before adding another layer of abstraction.

Now let's talk about what might be the most valuable asset in AI engineering: user feedback. This feedback provides proprietary data that can give you a genuine competitive advantage. While everyone can access the same foundation models, only you have access to how your specific users interact with your system. User feedback comes in two main forms: explicit feedback is directly provided by users—this is things like thumbs up and down ratings, star ratings, or written comments; implicit feedback is inferred from user behavior—this could be things like early termination, error corrections, or question clarifications, complaint messages, sentiment, frequency of regenerating responses, and conversation length. When designing your feedback systems, consider carefully when to request input. You could ask for feedback at the beginning of the experience, like asking for skill level in a language learning app, or when something unexpected happens, like slow response time, or at natural decision points, like offering between two alternative responses. The goal is to gather valuable insights without disrupting the user experience. Remember that every request for feedback creates friction, so use these opportunities wisely.

While we've covered each component separately, a mature AI application integrates all these elements into a cohesive system. The architecture you choose should align with your specific use case, technical constraints, and business objectives. One important thing to remember is that complexity should serve a purpose; only add components that solve real problems for your application. Sometimes a simpler architecture with fewer moving parts is more reliable and easier to maintain than a complex one with every bell and whistle. The field of AI engineering is still rapidly evolving, with new techniques and best practices emerging daily. The most successful AI engineers maintain flexibility in their architecture, allowing them to incorporate new advances while providing stable, reliable experiences to their users.

And that wraps up our journey through AI engineering. We've covered an incredible amount of ground, from understanding foundation models and evaluation to mastering prompt engineering, RAG agents, fine-tuning, data set engineering, and optimization techniques. Of course, this was a super high-level overview of a very detailed book, so I really recommend using this as a starting point to check out the book on your own. I had a great time putting this together, and I plan to do more videos covering technical content like this in the future. So let me know in the comments which book you want me to summarize next, and don't forget to subscribe so you don't miss it when the next one comes out. Thanks so much for watching, and I'll see you next time.

Get Our Mobile App

AI Engineering in 76 Minutes (Complete Course/Speedrun!)

Transcription