Transcription
Hello, my name is Christine Yen. Thank you for having me. I'm excited to be here. While I am the co-founder and CEO of Honeycomb, I will say the statements and assertions I make should apply to any modern observability workflow. Let's get it.
Writing software today feels more magical than it ever has before. We've got LLMs everywhere. We have cheap API calls to foundation models. Not even going to touch the whole idea of vibe coding in this talk. There's a lot to be excited about here. It's very cool to be in the middle of a phase change in progress when everything is new to everyone at the same time. And most of us here today have been building software before this LLM boom. And we know the hard part about building software systems isn't just the writing of the code. It's the testing, it's the maintenance, the tuning, and the debugging that comes after.
So, while LLMs have a ton of implications for everything that we're doing, I want to spend this time with you today on this specific part of it: How we make sure that the code that we build on top of these magical black boxes still works the way that we expect after we've shipped it and it's in front of the user.
Now, we work with black boxes and software all the time, but LLMs take some of the ways that we're used to ensuring consistent, reliable behavior and make them a little more difficult. Take, for example, trying to make sure your code overall is testable, mockable, and debuggable. Well, unit tests rely on your being able to define a representative set of inputs. With LLMs, we're intentionally opening the door to a very long tail of possible inputs. As for mocks, LLMs are by nature non-deterministic. Swapping it out for a deterministic mock doesn't really help us much here. And when it comes to debugging your LLM behavior, we don't have simple logical paths to step through. The whole point of incorporating LLMs into our software is to wrap the full breadth of human expression into our code. Debugging LLM behaviors kind of just turns out to be trying something and seeing what happens.
So this turning upside down of our worldview is happening on a literal software engineering systems engineering level where these black boxes just aren't testable or debuggable the way that we're used to, which means there's no solid sense of "correct" to fall back to. It's also true at a meta level. There is no environment within which we can conduct our tests and feel confident in the results. Even normal product development or release practices have—have—turned inside out. Instead of starting with an alpha or beta and then feeling confident in a later broader release, all these early access programs tend to do is inherently fail to capture the full range of user behavior and edge cases. These boxes in the middle column may still be working for you and may still be a good idea. They just won't be enough for ensuring the correctness of your new LLM-backed functionality when you're inviting your end users to do a lot of things in your system that you may never have expected.
So, is it time to give up on everything we've learned and know how to do and just embrace the rise of prompt engineering as a specialized skill set of the future? No, obviously not. Or else I wouldn't be here today. Because a lot of the conversations we've been having as an industry over the years about how to build reliable and performant available systems in a chaotic cloud-native world are still relevant, probably more relevant with the sort of rapid iteration that building with LLMs demands. Adopting CI/CD let us ship code—let us ship code much more frequently, rapidly iterating on user experience. Talking about testing in production helped us all embrace the chaos of real user input versus artificially sterile test environments. High-cardinality metadata has become a must-have if software engineering teams are trying to understand complex systems, whether that complexity comes from your architecture or trying to understand the business impact or per-user experiences. High-dimensionality data—being able to break down aggregate data by a bunch of possible related fields, not just a small number of predefined ones—has become the default way to detangle the interaction of multiple contributing factors. And SLOs—service level objectives—borrowed from our SRE friends have let us leverage existing alerting workflows but anchoring them around fuzzy concepts like user experience or "are we delivering a good service to our end users?" These trends have already begun driving a distinct new approach to making sense of our software even in a pre-LLM world, and we already have a model for how to measure, debug, and move the needle on unpredictable qualitative or quantitative experiences: observability, where it's all about comparing expected behavior against what we're actually seeing in production in front of our live users.
Because these are some truths about building on LLMs: Something unpredictable will happen. User behavior will be chaotic. One fix will break something else. Tests won't be enough. And early access programs won't help you. These aren't only properties of extremely complex systems. They impact all of us once we've decided to take in natural language input or unpredictable input from users and pass it off to AI to make decisions about that path. It carries with it a level of chaos and complexity that will force us all to level up fast.
Now, observability helps embrace some unpredictability, enabling the sort of feedback loops that let you learn from what's really happening with your code. The same way that we've learned to work iteratively with tests, observability enables us all to ship sooner, observe those results in the wild, and wrap those observations back into the development process. We just said tests wouldn't be enough, right? That's where evals come in. As a quick sidebar for anyone who isn't familiar, these are tools that allow us to codify what "good" looks like in an LLM world that allow for a little bit more flexibility about what success or failure means for our applications. The pattern is that you develop your set of evals as you develop your application, and we use them to capture intended behavior or flag behavior as you work with your prompt.
Now, evals parallel observability in really useful ways. While observability is all about the unpredictability and chaos of production, evals capture the good and the bad as they happen. Although in both cases, whether you're talking about the instrumentation behind your observability or evals themselves, the intention is for them to evolve with your code. And what you find in your observability tools about how users are trying to use your software turns out to be the best source of input into defining those new evals. And so if you put these two things together, the same way that observability offers feedback loops into development, observability pairs with your evals to form these feedback loops, informing and improving your prompts. As you're uncovering the behavior of your LLMs in response to your prompts, you're defining evals, releasing quickly, watching that code in the wild, then closing the loop as you learn, and pulling those learnings back into your codebase to live on forever as evals.
Let's go one level deeper. What does it look like to ensure that you're setting yourself and your observability tooling up to support this kind of workflow? Well, because LLMs and Gen AI are these non-deterministic black boxes with unbounded variations on inputs they can receive, with a rapidly evolving set of paths used to refine those inputs, getting good observability into these systems is all about systematically tracking the inputs and outputs. Let's take a look at what this looks like for a standard web app. By instrumenting our application, we can capture what arguments were sent to it on any given HTTP request, some metadata about how the app was running, what was returned. All this lets us reason about the behavior that we expect for a given user endpoint or set of parameters, and it lets us isolate and debug the issue if the actual behavior deviates from that expectation. But what about this payment service? It's a third-party black box out of my control where, even if I wanted to, I couldn't go in and instrument it or look at the logs flowing through of what it's doing. What I do know is what requests my app is sending it, from where in the code, and on behalf of which user. And I know how long it took to respond and whether it was successful and probably some other metadata. By capturing all of that, I can start to reason about how the inputs impact the outputs of my black box, how my application and my business logic impacts all of that, and ultimately the impact on the experience the end user is having. Taking that and carrying that into an LLM world, there are a few more boxes, but the principles remain the same.
One tactical note: I like using traces for this in order to understand the relationships better between the overall end-user experience and sort of the subcomponents. If you want to use structured logs, go for it. You do you. Either way, we start with the end-user experience, the raw input, and eventually the output that we're returning. We can also capture metadata on the context that we're constructing along with our prompt and how long that took. We can keep track of the prompt itself that we ultimately pass to the LLM and useful metadata. Capture useful metadata like token usage as well as any parsing or validation of LLM outputs before returning to the user. By operating under the general principle that criteria for decision-making should be captured in a span, you can then go and isolate any of the interesting behaviors based on how a prompt is generated. And all of that ultimately lets us see all of the work that we're doing up to and including calling the LLM all in one place. Yes, of course, we can also get the aggregate graphs like the ones on the left in the middle. We can look at latency. We can reason about user satisfaction. But in a workflow where we're rapidly iterating on an LLM experience with tons of potential inputs that can impact whether your application looks like it's hallucinating, you need to be able to get from any aggregate graph to inspecting a given outlier. This blue row at the bottom—this is a span where ultimately we're actually calling the LLM. And being able to ask questions like, "Okay, what actually got passed to the LLM? What was it responding to?" relies on all the spans above it because that's all the work that we're doing to build the best prompt that we can hand to this commercial LLM. And when there are that many things that could result in an unsatisfying LLM response, we need all of the context we can get to iterate and investigate—investigate and iterate—towards a better prompt.
Now, there are a number of specialized tools out there that promise out-of-the-box answers for LLM observability. I will assert I don't want a siloed or specialized tool. I want something—especially not one that tries to tell me what to care about. I want my tools to reflect what I care about, what "good" looks like for my applications, and something that aligns with the workflows that my engineering teams are using for the overall application logic. Because everything I've talked about so far isn't some new skill set or mindset. So much of this shift towards embracing observability is already underway. In the last decade or so, we've seen a huge shift from developers aren't just here to write a lot of code—which AI code assistants are certainly helping with—to expanding our responsibilities to owning our services, being on call rotations, and testing in production. Ultimately, being responsible for what our end users see as a result of our code. Because, like it or not, when building for this new Gen AI world, nothing is predictable besides some level of chaos.
Over the years, as I've gone around talking about observability, I've communicated some version of this bottom statement: that software behaves in unpredictable, emergent ways. And the important part is observing your code as it's running in production while users are using it. This was true before LLMs, but now as we are intentionally embracing these non-deterministic, emergent black boxes with emergent behaviors, this statement is that much more true.
So, as we enter this age of AI, I'm weirdly optimistic because we have many of the tools and practices at our disposal to make sense of this brave new world, and we're just getting started. I, for one, am excited. We got this. Thank you for your time and attention today. If you'd like to learn more, I've got an O'Reilly report and a O'Reilly report and a book to recommend to you. Uh, I will be at the Honeycomb booth in the Expo Hall. My team and I would love to hear about what you're building. Thank you so much. Have a great CubeCon.