Transcription
Everyone, uh, welcome to another episode of AI that works. We are back and I think today one of the key things that I'm really, really excited to talk about is a concept that has come up a few times in our past streams, but it's really about how to build working AI pipelines.
Um, so for everyone that hasn't attended before, this is a show called AI that works. Every week we run a live conversation where our goal is to talk about how to build a pipeline that actually can make real productive examples. If you find this sort of stuff interesting, check out our link and we'll usually post new events there. And we run this every single Tuesday at 10 a.m. uh for about an hour and the goal is to get to coding as soon as possible.
My name is Vivof uh on the side. I work on BAML. Um and today I have with me special guest uh Dexter sadly can't make it today. So, for all of you that are hoping to see him, I'm really sorry you're stuck with just myself, but luckily we have Anders on um to make it a lot more interesting and fun.
>> Yeah.
>> Anders, do you want to give a quick intro about yourself? Uh I know you're a YC founder. I have gotten to know you uh for quite some time now.
>> Yeah.
>> And
>> uh first of all, is my mic working properly?
>> Okay. All right. Uh yeah. So yeah, it's been a it's been a while since we known each other now. Um but yeah, so we did the um YCS25 batch, but basically we're building uh SageKit, which is like a workflow automation platform. Uh it's kind of like chat GPT meets Zapier. Like you just sort of chat and it builds you like workflow automations. So yeah, very, you know, key thing to our product experience is just having like a super good like chat experience, right? So there's a lot of things involved in sort of making that kind of experience good, right? And uh like making sure that yeah, users always have sort of the right thing happen at the right time when they're on our platform.
So >> yeah, and I think the first time we met is actually when you were working on a thing called magnitude back at that time. Um, and I and for everyone else that we're going to talk about today is we're going to talk about event- driven actions. And why are we going to talk about event- driven actions? Well, because I think it's one of the most interesting ways to go model how to go build agents behind the scene. And we'll get to the whiteboards really fast before we get to coding in a second.
But the reason that we brought Anders on today is because I think Anders built their first thing called Magnitude as I mentioned before, which was an open source system. And I think you guys did pretty well in Hacker News. They got a bunch of GitHub stars around it. Um, and I think the way I described it was kind of like browser based but worked.
>> Yeah, that's that's a good way to describe it. Basically,
>> it's just like the I think you hit number one on all the benchmarks for a while.
>> Still are number one on the benchmarks. Um, I know a bunch of the YC companies that are in that space all use you guys. I know a lot of startups down there used you guys.
>> Um, and I happened to meet you because uh I guess you were using BAML at the time to go build it. But through that interaction, I think eventually you settled on this interesting really interesting product idea which was along the lines of I think we had a whiteboard conversations once about it uh while we were hanging out about how to build event- driven actions and I remember we were just like vibing and conver conversing about it but it sounds like you guys did it to like the nth degree.
>> Yeah. Before we get really into it, how about we just show people what it feels like and then we can talk about technical architecture and everything else once people get a sense of what's like what does it mean to have an event action because I think it can feel so ambiguous along the way,
>> right?
>> Yeah.
>> Um, should I go and share here?
>> Yeah, go for it. Screen share. Oh, shoot. I gota rejoin real quick.
>> Cool.
>> While everyone else is on here, uh, for everyone else that's joining, um, how many of you have actually built event- driven actions by chance? Has anyone built a kind of system that can be really almost feel natural? I think the closest that I have seen is probably quad code to where you can interact with it as it's going. And that was one of the first innovations that they had. I think that's what made quad code personally so good. And then obviously cursor copied it, everyone else copied it. And everyone had the very simple thing that you could do which is as you typed code you actually could cancel the previous message and make it go work. But Claude did something very interesting on launch which was it cued a message. And that behavior pattern I thought of queuing messages was foundationally different than everything else in a very subtle but massive way. I think very similar to a lot of the great UI innovations like Amazon's oneclick buy. So subtle the fact that you can just one click check out without having to do anything else. But it makes a huge difference. And I think in AI stuff event driven and like interruptions and queuing all these other behaviors are really the difference between like an okay app versus an amazing app.
We've got someone working on it right now. Um, that's that's great. Um, so Andrew, let's show the demo and show how it works and how it feels.
>> Yeah. So first of all, generally speaking, right? So this is like our our sort of chat experience, right? So let's say what I want to do is I want to go scrape Reddit uh subreddit for r/lang chain for everyone complaining about structured outputs or complaining about lang. Let's do that. Uh and the categor and categorize the types of complaints.
>> Okay, we'll ask it a question after. So, okay, let's let's make a modification.
>> So, that's interesting. What you do is you actually ban the inputs during certain time periods when you don't allow interruptions.
>> Yeah. Okay. So, let me extend the plan there. So, okay. So, just to be clear, yeah, we don't um basically over the past couple days, I was rethinking how sort of the events should work in the chat and like how we should best enable like stopping and interruption and queuing messages and stuff like that.
>> So, this is um not actually in our production chat app yet, but I have like another demo that's just like I basically used when I was prototyping the new
>> Yeah.
>> >> sort of architecture. Um, but this is just sort of a general idea of like the types of flows that you have in sort of a very interactive chat app. So it's like you have, you know, back and forth messages, you have, you know, plan approvals and stuff like that. You know, a lot of the same stuff that you have in like interfaces like cloud code and that kind of thing. And so if you weren't logged into like some of these apps, sometimes you would get like ooth flows happening in the middle here. So there's a lot of like um complex states that kind of come up that'll just sort of like happen in the chat. So it gets sort of like interesting to basically manage all that state and like figure out what should be happening at a given time like at a certain time like should the user be able to send a message? Should they be able to interrupt it? Should they be able to authorize with a certain like a connection? Um and yeah, so there we go. So it's it's basically just built a workflow for us. And this is sort of like a summary of what it's going to do. So it'll like use Reddit. um review them for complaints and it'll use like Gemini to analyze them and then it will format that. So yeah, we can just run that basically. So it just builds it all completely on its own and then we run it and it's basically like using the Reddit connector to list the posts and then it's making a bunch of AI calls to analyze them for complaints. Makes sense.
>> Yeah.
>> So, how do we um how so talk to me about how you what do where would you add interruptions? Because we've talked about this a bit and I think you have um like interruptions and queuing. So, it sounds like I can definitely send a bunch of messages, but this looks very linear,
>> right?
>> Yeah. So, what you really want to be able to do, right, and which we're adding like tomorrow or something, um is you want to be able to just queue up messages however much you want. Like right now if I like type another message,
>> >> it should just do stuff.
>> >> It should it should just do stuff, right?
>> >> Yeah.
>> >> And it's like I can send another message, but I can't send another message here. Right now it's just an invalid state.
>> >> Yeah.
>> >> Um but ideally like Yeah. You know, people want to be able to like send more messages like maybe they have a thought while it's building. So they want to keep be able to like send things. So um here's like right just a demo of sort of what the new uh architecture vaguely looks like that's sort of like very just event to the max.
>> Yeah. Let me zoom in. This is a
>> >> and then we'll take a look at the code pretty soon too. But I think this is this I think it's
>> >> uh what one thing I find really fascinating is always when we're able to go look into this in a deep way and just go check this out.
>> >> Yeah. So, if I, you know, I say hi, and there's not a lot of scrolling here, but you can see it stream back a response, and these are all just sort of like events like coming from, you know, the LLM that are eventually streaming into the chat. Um, but right, we might have other types of events. So, it's like tell me a long story. It's like another type of event is like interruption, right? So if we interrupt that happens sort of concurrently during this process of streaming and then the question is like okay how do you deal with that event in the context of like okay we're also streaming from the LLM um so like what's happening now based on those events that occurred right those like the sort of source of truth um in terms of like we started the LLM response it gave these chunks at this point in time the user pressed the stop button what should happen now right? So it's like basically react acts to that. It's like okay I need to stop you know listening to the LLM I need to you know present some sort of message that sort of reflects what the user did um and then you know put the user back in a valid state where they can send more messages. So yeah, the user can also just, you know, ceue up messages as well, right? And you sort of have to think about like what you want the behavior for queuing to be like, uh, we sort of followed like the cloud code model here where it's like if you send a message, it will start like responding to it and then if you like queue up more messages, it's like tell me a long story. uh if you like send a bunch of messages during this point, it'll just sort of like accumulate those and then once the LM is done with its story, uh these will basically all like resolve into the context window. So it will like see all of these at once and then respond again.
>> So I guess what I would say is like let's talk about this a little bit more. So what I'm going to do is I want to maybe we should jump to whiteboards because I think that often it might be easier way to go describe this kind of state because as we go see this
>> >> there's a lot of complicated things that are happening. One is I I have this chat thread this chat thread that's running all the time. I'm guessing in some backend API call because it can't run directly from the front end.
>> >> Yep. And then some it sounds like the most complicated thing here is what I'm really doing is I'm sending a bunch of commands from the front end and then I have to update the world view and somehow send a Wii view of the world view back to the front end.
>> The analogy that I've always personally found here is actually video games. If you ever play like a multiplayer video game of any kind,
>> >> that is, I think, the closest analogy to building these kinds of agentic systems because like for example, we might have totally different lags in the world. And what if I'm running and you like shoot me
>> >> in your in your scope? It might actually look like you actually landed the shot and you killed me, but from my perspective, I'm lagging in the world view, it didn't actually hit me.
>> >> So,
>> >> right. Like what video games actually do is they actually say we favor the person that is like killing the person because if it looks like they killed you,
>> >> it should actually kill you even if you can't tell. And they actually correct in that way because it feels really bad to feel like you did the right thing and then someone else uh because of their internet lag didn't and world state kind of lied based on what you thought.
>> >> Right.
>> >> Yeah. Yeah. It's basically like right you have all these events happening, right? And it's like what you show to the user is always some variation of a lie, right? It's like it's not the exact you're not showing them literally every single thing that happened, right? It's like if you showed them like the exact like function calling format that the LLM used and the exact tool results that the LM got back and all this kind of stuff like basically what you want the user to perceive is like different from the truth and it's different from what you want the LM to see also. So it's like everyone see needs a different version of the truth and so it's like how do you manage that and then how do you manage like the concurrency and stuff?
>> Yeah. So what we should do is um we should actually go ahead and um talk through what's it called? Let me open the whiteboard really fast so we can actually draw this out because I think it's going to be a lot better.
>> Um and I'll send you the link as well. Give me one second.
>> Share screen. Skelelet draw take over skele um invite. Sorry, I should invite you. Uh sorry about this. Dexter usually does this stuff live collaboration and then I'll send this to you uh only. So while we're actually going to go talk about this what we really want to do is we want to figure out a way to go draw this out from a system architecture. So what we have is we have a front end. And in our front end, what we're going to want to do is we're going to want to send a regular system of events to our back end. And once we send system of data to our back end, what we want to make sure is whatever this is triggering should not directly trigger the actual um it should not trigger the direct uh what's it called? The direct agent loop. because if it triggers a direct agent loop and it responds directly to to the user in a very linear way, we can no longer interrupt the system. So what we need to make sure that we do is we need to go ahead and somehow send data from the front end to the back end. And once the back end is gone, this needs to kick off some task in some form factor where this will go write information to some central database. get task and then somehow this database is or database like thing and somehow this front end will listen to state actions coming from the uh database and now what you can do at least and correct me if you're if I'm wrong on this Anders and you guys do something different because I think it'd be a very interesting conversation to have is as events come in as new events come in you can basically have your agent task somehow be loading these events from uh new events Oh, sorry. New can also be written to the database and then your agent task can also have a readwrite system from here somehow.
>> Great. Yeah.
>> Go ahead.
>> Yeah. So, yeah, the way I think about this, right, is that right, the event the event stream and the events produced, they're coming from a lot of different places actually, right? And it's like,
>> >> okay,
>> >> um, so it's like you have different sort of types of events that I that I think about, right? So this is like um Right. So this is what like the user over here. This is uh right. So I mean obviously we have you know an LLM in here.
>> >> This is a very different style. It's going to be an interesting graph. Uh
>> >> I can fix that. You can just keep drawing. I'll fix.
>> >> Okay. Right. But we have a lot of different things like contributing to the event stream. Right. So it's like we obviously have an LLM in here. Um, we have you know we have some kind of strategy for especially with streaming like we usually have some sort of way of like interpreting or like parsing um what the LM is doing. So usually it's actually like this. It's like we're doing that and then that's going into the event stream. Um, and then we sometimes have you know like handling for like doing different like tool calls or other things. So it's like we have some like tool calls and maybe sometimes the parser goes and does a tool call first and then that becomes sort of like an event, right? So it's like there's a lot of different things that can sort of contribute to that, right? And so sort of think what the the confusing part becomes is like okay, how do you um who's sort of in control? Like what's like what's in control of this whole like loop? Uh how do things like feed events into each other? um and and sort of what happens there. So yeah, the the way like there's a lot of different ways that you can structure that, right? So it's like we have sort of in this, you know, little diagram, we just have this event stream thing here, but it's like um there's different ways that you could do that, right? could be like one sort of continuous sort of event stream from like LLM to like parser to like um some sort of representation for both the user and for the LLM, right? So it's like you have different ways that you need to display the event stream. Um, and yeah, so I'd say like you basically yeah, you basically have things that add to the event stream in different ways and you have things that need to like perceive that event stream is like the main idea I would say.
>> So is is the right way to think about this? You basically have a function in your code somewhere else. So an event handler that takes in some event kind of some any of the many events that you have and then based on the event kind. Oh, this is center text. We did not want that. What you end up doing is you end up basically saying, hey, if it's an LLM stream, if it's an LLM event, I should say like a parsed event, then you do something else where like maybe it's like write send to UI of some kind. Otherwise, like if it's like a if it's a tool tool response of some kind, you say like do action where you do the next action over here,
>> >> right?
>> >> And you go fill this out. And then if it's like if user request,
>> >> you end up saying something like Q uh action of some kind where you like cue an action. And this is basically it's triggering a almost like a cron job that's running all the time where like every single system that you have like every agent task isn't actually a whole loop. It's actually a system that's just adding a new event through a massive like event log of what's happening.
>> Exactly. Yeah. I think basically the way I think about it is like Yeah. It's like a reactive system. So it's like it's not instead of like thinking about it as like a while loop um it's more like cues of events.
>> >> Yeah. So your worldview model so that's interesting. It's a it's actually a different worldview model than what I've used in the in the past where like my way of representing the worldview has actually been like I have a giant state object that represents the state of the world of some kind. It sounds like your world view is actually you have like an a ve or an array.
>> >> Yep. of actions that have happened in the past and this is what represents your state. It's not to say either one is good or bad but this is what you found is really easy because then what you say is for example the way that you pro tell me if this is wrong or not. The way that you implement queuing is that you say like a Q event is a thing that gets pushed into your array and once the next action finishes you go back and you see if there's any cute events that are not
>> >> done yet. Right. Yeah. It's it's very sort of native to that system, right? So yeah, so what we had sort of in the past was sort of more like a state based model where we just have a state and we mutate it and it's you know it's you know very reasonable kind of makes sense like you can just sort of change the state as needed react to that you know do think different things depending on what that state is at the current time uh and that kind of thing. But the problem is like what we did more and more over time is like we started uh sort of mixing these paradigms because it's like oh well we need like a whole like log of everything that happened in the chat so that we can like view it right in the chat and also the LLM needs to be able to perceive it in a certain way in terms of like converting that to something that goes in the context window. Um, and then it became very unwieldy to like maintain both like these events that are like the source of truth for like what happened and then also like a separate state which is like we have to keep it synchronized with like what happened and then so basically um what I wanted to do instead was just like okay what if everything is just basically a pure function of the events that actually occurred. Um, and so like you can have like representations of state in this model where you sort of centralize around events still, but they're just like functions of that event.
>> Describe I'm going to try and describe what a user might do and how someone else can view the code and then we'll take a look at the code really fast. So in this case, what I have is a user started off with some user event with some request. Maybe it's as simple as just like hello and your agent started responding with like hello how really fast and then right at that point the user said wait. So there and if they cued the event, the behavior ends up I'm guessing something like this where you get a bunch more stream text of some other text
>> >> and at some point your state engine recognizes that like obviously when you're streaming you can also say like done streaming and now your user invent uh uh your agent system can say as soon as it the event handler for the done event goes and checks the queue for this specific user ID or this specific conversation ID and says Q event. Well, let's give this event an ID of like ID of some kind. Sorry, let me make bigger and too small. ID equals 1. And then what I can do is I can say like handle uh handled event and now I can handle event id equals 1 where it'll actually treat this and transform this into another user event of like wait at this point. And now this has basically been handled. So now the system will start streaming and send the response into the user.
>> Right. One way to handle a queueing.
>> >> Yeah. But yeah, the main the main thing here is that um when this queue event happens, right, we have to immediately right interrupt whatever thread or is going on that's handling the streaming part, right? And ignore the rest of it right away. Well, no, not the interrupt. The queue event.
>> >> Oh,
>> >> yeah. Sorry.
>> >> You mean just literally?
>> >> Oh, yeah.
>> >> Oh, oh yeah. Oh, yeah. Yeah. My bad. Yeah. For queuing events. Yeah. Yeah. Yeah. So, yeah, we let it keep going. Um,
>> >> on the other hand, let's talk about the second part, which is interrupt event. Let's say the user sent this as an interrupt instead of Q. Oops. Interrupt. Well, interrupts are I think very very similar to queuing in that same in that same form factor where oh sorry one second. Um, I have to reload the page so my view length is better go back to the right area. Click on new. So interrupt is very very similar to queuing where instead of queuing the event though all these systems that come in what you can tell your back end is hey don't add these events to my I mean you can cancel it as well but you can also just say like just don't add these to the queue like basically don't add them into the queue and spin up a new event as if this happened. And you can do what's interesting here is you can choose how you handle these previous events. You can choose to maintain them in the record or you can actually say that actually you know what I am these are bad so I will just like delete them after the fact even though they technically exist in the record and my the my world view when I query and when I show in the UI how to represent this this exists but I don't render it or I can delete it and purge it from the event record in any way myself.
>> Yeah. Exactly. And I think that's the nice thing about this type of system, right, is that you can basically just look at what happens and sort of have however you handle events decide like what what is supposed to be viewed or perceived when like you know this type of situation happens where you have like okay these events were streamed but then the interrupt happened. It's like okay do I just ignore the ones that happened after that? Do I ignore all of them? And you can sort of you know implement that however you want.
>> Yeah. And like once you model the world in this way where it's just like a sequence of elements that are being collected and like drawn upon like writing code for this is actually very simple. I'm guessing like this code system is not that hard. And in terms of actually you like rendering in the UI well that's also very easy because you just say how do I render the sequence of events? And
>> >> you can do that in any which way that you want as a developer. You can render like and show the stream text. We know how to do that. You can render and show um show cute events. You can not show cute events. You can show interruptions. You can show middle states. You can show like the entire conversation history. You can go back and fork at any point because it's just copy this screenshot, copy this action array of everything up before this event and start copying from here.
>> Yeah. Yeah. I'd say there's like a little bit of like just like setup in terms of thinking about your agent as a sequence of events and not as like just oh I have state and I mutate it but like once you start thinking about it that way yeah you get a lot of
>> >> benefits from that like, you know, some of the things you mentioned
>> >> you know, what this really reminds me of this really reminds me of how like database transactions are made.
>> >> Yeah. um in some ways where like you basically have in a lot of ways you just write a bunch of logs to some event file and eventually the database becomes like consistent on its own by saying like these event files eventually have to be recorded in the database
>> >> with some yeah, it's kind of like that in a way yeah um right yeah so you can yeah like yeah, the key things that you can really get out of this are you have sort of right you have the one source of events instead of like state that you're mutating on the back end state that you're mutating on the front end those might get out of sync, right? You have all sorts of like weird things. It's like everything just driven by sort of this one engine that's handling the events and that kind of thing and it can just send like okay front end like you should look like this at this point in time based on the events.
>> I'm actually really curious to see what this code ends up looking like in practice. Someone asked a really important question before we go into that
>> >> um which is do you store the same events in the same table in a database? I my hunch says probably yes. I suspect your database is basically a union of events.
>> >> Yeah, it's literally just the way the way I've done this is it just it's just one array of events just in a you know database column. It's just um
>> >> I'm guessing you're you're
>> >> just like a discriminated union of different like event types. Um, that's it.
>> >> And I'm guessing it it's just basically like event and then you do a group by user ID and convo ID and now you have your entire event history for every single and like time stamp and obviously all the metadata.
>> >> Yep. Yeah. So that's basically what it is. Yeah. So you just an array of events and so the representation is like extremely simple, right? You can just load those events and do whatever you want with them.
>> Yeah. And you can load as many of the events you want. You can get the most recent 50. You can get the most, you can get all of them forever. You the first 10, the last 10. It's all up to you.
>> >> Yeah. And it's extremely like uh debugable and testable as well. Um, like when you think about like one one thing that was hard when we were sort of
>> >> building our our chat is just like it's hard to test it without like running the the back end and the front end and and like testing different scenarios like manually and that kind of thing like with sort of a traditional system where you have some sort of state that you're managing on the back end and on the front end that's trying to synchronize because you don't really know from the backend code alone what the front end is going to look like at any point in time. But if you have sort of everything driven by the events and like your back end is basically just telling what your front end and what it should look like exactly at any point in time based on those then it becomes very testable because you can just write tests that are like okay at this point in time I expect the interrupt button to be available or not based on these events that occurred.
>> It's it's another analogy for this for me is like very similar to like use state and react. When I write use state and react I can like how do I debug state that is bad? Well, I usually just hardcode the initial condition of the state. I say change the state to this like just show me what the rendering looks like and then I debug it really easily.
>> Um, because like React just says assume I have state of this shape and now render it however I want and it's a really easy transaction for me to go make as a developer. Um, we should uh we should get the code. Uh we got like two interesting questions while you set up the code and you go screen share for that. Take over my screen share while you go do that. Yeah,
>> >> which is um where do we persist the context for LLMs in this case? And I think this is a really interesting question that all people ask, but this goes back to how we we've talked about this in the episode quite a few times, which is what the heck is an agent? Well, let me see if I can pull that up. We talked about this in the 12 uh factors um selecting thousand tools from MCP uh tool. Let me pull this up really fast. I'm gonna take I'm gonna take over the screen share for one more second.
>> Okay, my bad.
>> >> It's okay. Um screen we talked about in this episode actually. And when you go think about an agent loop really fast, what I have personally found is u take it here really fast. Sorry. You can view every single agent as basically a thing a model as just a thing whose only job it is to do which is close up. You take some context window that you're building up over time and you ask the model to do something that is very stateless which is determine the next step. See the question that you're asking is how do you build your context window? Well, your context window is also really just a projection of this state. You take this state and you turn it into a chat prompt of some kind. And that's just a function that you can write which says or turn your action array or event array into a chat prompt. Now, I suspect Andrews will be showing this code when he talks through this. But let's get into it. All yours, Anders.
>> All righty.
>> >> And as people have questions, keep sending them. We'll try and uh answer them as fast as we can. Okay.
>> >> Yeah. All right. So, here's the code for um sort of that demo that I was showing that
>> >> bun. I love it.
>> >> Where do you see the bun at?
>> >> I thought bun.serve.
>> >> Oh, yeah. Bun. Um, so yeah, when I was playing around with sort of the ideas for doing this very, you know, event driven thing, um, one thing that I, you know, had, uh, heard about but not experimented enough with was effect TS. Um, and so I'm I'm an effect TS newbie. Um, but for people who don't know it's like you know a library that makes it
>> >> hard
>> >> gives gives gives a lot of tools um for for TypeScript in terms of doing a lot of different stuff like error handling but also like streams and just like piping things between other things and just like all sorts of different like transformations. And so um yeah, basically it became like obvious that like it was like a pretty good way to structured this type of system where you have sort of inherently like a lot of streams of events like feeding into each other uh and different stuff like that. So yeah, this is a very like this demo is like very eventt. So um yeah, basically like how this has been like structured here is just like effectively like the the the the back end here that's actually receiving the websocket connections from the front end and it's running the whole chat has basically these different like services that are at any time going to basically observe events and like maybe produce events also. Um, and so these are all sort of controlled by like an event bus which is like going to be getting different events from like the front end. Say a user sends a chat message that will go in here, right? So it's like let's take a look at a specific example. Um, yeah. So it's like basically when something wants to send an event back into sort of the engine that's has these different things perceiving the events uh it can use this event bus it can like publish the event and it will basically be subscribed to by the other things. Um, so
>> >> so let's take a look at some of the events if you're down. Why don't we look at
>> >> Yeah. Yeah. Let's start with um the events right. So yeah. So here's what some of the events looks like for example. Um, so basically these are all the types of events that can live in this event bus and can be like sent to it as sort of like from external source or like um sent from one of these event handlers back into the system to be you know transformed or something. Um, so like user message, you know, obviously that's like going to come from the front end. Um, and it's going to sort of go into the system and it'll be reacted to in some way or another depending on like what's currently going on. So like if we look at like what's Oh man, I've got like Hold on. Just delete this.
>> >> You can just filter forts files. Just hit the down arrow on the left or the dot dot dots. if you need
>> >> it or might be TSX if you're looking for your front end.
>> >> Uh
>> >> no, but it's like um yeah, so it's like this is like the message function that's just like on the server itself. So like when it gets a websocket message from the front end that's like I want to send a user message then it will basically like publish that to the event bus right and so we can see like what uses that right so we have basically a subscriber to that event bus right which is like this messages state service right and basically what this is just trying to do at any time is actually basically trying to like perceive like a current state out of what events are being streamed that have to do with that are like message related right so it's like user message or like different like LLM chunks or other things
>> >> so basically right
>> >> and this is interesting so it's not that you don't have a state object it's that you don't keep the state object as your representation of truth
>> >> you have an you have an event history
>> >> and you turn the event history into a state object in every single every single time you do something with it.
>> >> Exactly.
>> >> So then you're now you say now given the state object now what's the next action?
>> >> Right. Yeah. You could right. So you could imagine like in a sort of normal like um state based system where you just sort of like mutating state you would like you would just have an object like this and you would just initialize it to this and then like you would be like whenever you finish streaming a message you'd like append it to this list or something. Um, and like whenever you get a message and you look at everything and you're like, "Oh, I think this should be cued." You would add it to here instead. And then you'd like move it between these or something. So all we're really doing here is like we're saying, "Okay, we're still going to like this is still helpful to like perceive something as like
>> >> yeah,
>> >> different things in a certain context, but we can construct that purely from the events that are actually happening in a specific way. So that's what this is really doing here, right? this message state thing is it's just perceiving the event stream as like what's currently going on with the messages. Um, so like are we streaming are any of them cued? Um, like how many tokens are we using? Like all that kind of stuff
>> >> and this can change at any time because you're just it's just a simplified view of the event history because you're like I don't want to look at the act the time at which cute events happen. I just want to like pull out all the cute events and just have them available really fast.
>> >> Exactly. Yeah. you can just decide exactly what this needs to be like what what information is important to hold on to and that kind of stuff right so yeah and then so like an example of like something you'd actually do right I is like if we look at what's actually happening here right so it's like processing user messages so this is a specific stream for just the events that have that type user message these are basically coming from the front end through the websocket. Um, and basically we have the stream to handle those specifically. Um, and basically based on our current state, right? So we're looking at our current state like if we're streaming and that kind of stuff. So if we're currently streaming, we aren't going to send it right away. We're not aren't going to trigger the LM to start streaming again. We're just going to update it and we're going to put that in the ceued user messages, right? instead of the the normal messages and otherwise right we just add it to sort of the normal message array sort of the history here and in addition right we also have sort of an effect right so we also trigger the LLM right to respond right which is the sort of this utility here which again is checking some things and then it's producing another event right so it's like if we get a message and um you know it's not currently streaming then we're going to cue this event right lm response started and that's going to basically trigger an LM response. So we could we could trace that further here if we wanted.
>> And the reason and just to be very clear like the reason we add that LM response started is because we want to somehow show in our UI that we've we've handled the message in some way or the message is like addressed and we should show some UI indicator to the user or like we shouldn't cue we shouldn't just run another message right now because there's something else in progress at this moment.
>> >> Right? So it serves a dual purpose, right? So first of all, yeah, it could be represented in the UI if we wanted, but also it means that we can delegate sort of what it means to start an LLM response to like another part of the codebase, right? Which is like more dedicated for that purpose, right? So that's what like this is here, right? So it's like um that goes back into the event bus and now the LLM service picks it up, right? So this is not a thing responsible for maintaining chat state anymore, but this is um a thing responsible for basically managing a stream of chunks that come back from an LLM and deciding what to do with them. Right? So again, we're subscribing to the LLM response started event and then basically we're going to listen to that and then when we get one of those types of events we're going to do a few different things, right? So first of all, we reference actually a different state, right? And this is sort of a a state managed by another service, right? LM memory state which is sort of the perception of the messages as context which we can look at in a second but basically that gets like messages that are nice for the LLM um and they're sort of like in a a format that's ready to go for that um
>> >> and and the idea is that you can remove all the croft and state signal like you don't for getting messages in the current state you really only want the final LM response from every LM past LM interaction you want the chat thread with the most recent views. So interruptions and stream text and everything else is removed and you really want to simplify this.
>> >> You can decide like exactly what you want to go into the context window, right? It's like very explicit. Like it's not like a lot of the times like I feel like when you're building an agent loop, the easiest thing to do is just like you're just appending messages to like a list when it's like you know use like ro user this content, ro assistant this content and then everything is coupled together. like your control loop is like looking at those messages and like deciding what to do based on what those look like. You're feeding them to your LLM. You're trying to display them
On the UI, but like when your agent gets more complex, these sort of states diverge. Like what you want to show on your UI is no longer the same as what you want to show to the LLM. Um, and so it makes more sense to produce those states instead.
>> Projections.
>> Yeah. Different.
>> I think there's a really question that's happening here because I think
>> I I think I see what you're doing. You basically have this giant event catalog and based on different use cases, you're turning it into different subsets of that catalog based on how you want to go use it.
>> So case of a new user event comes in, what you want to ask yourself is the only thing you really care about is, is anything active happening right now? If something active is happening right now, the new user message is a cued message.
>> If nothing active is happening right now, the new user message is the next action that should happen next. And I think while and right over here, you did another projection where you said the LM is going to interact. So I want a subset of the messages that I want to send to the LM. But I think there's something that might be confusing a few folks really fast, which is a really important question.
>> When do they get persisted? Like I get that I could keep this thing in memory, but like do you save this into a database anywhere right now? And if you don't, where would you because how does that work?
>> Right? So in this like demo code, there's it's just all in memory, right? But it's it's very easy to
>> can you show the data structure that keeps everything in memory because I think that would also make it a lot more clear.
>> Yeah. I mean effect doing some magic.
>> So it's it's effect doing some magic. Okay.
>> So, it's it's all being basically it's all persisted in memory basically in the like the stream buffers and that kind of thing. Um, from my limited understanding of effect at least I think that I think that's where it lives.
>> Right. But
>> but if you were to save in a database, can you show the pointing code when you would do this? I suspect it's an event bus,
>> right? Yeah. So, you could do this. Um,
>> I suspect if someone publishes a new event, I suspect you do it on publish.
>> So, you could do it on publish, right? But it's like it depends on the frequency of persistence that you want and and sometimes, right, you don't want to persist everything right away because it's like you could have sort of invalid states like you imagine like
>> um if someone's just interrupted an LM in the middle of a bunch of stuff, like it's like, oh, do you want to persist it there or do you want to wait for like the LM to finish its turn? or like if an LM is like in the middle of streaming like it doesn't really make sense to persist it on every chunk right so it's like you have to decide like when does it make sense to persist it um but basically yeah you could have um one way to do it was just like you just have a subscriber to um all the events right and you just have like basically
>> and can you show let me just write some pseudo code really fast or maybe let's write let's write the code really fast I would show how it get persistent in effect. I think also for those of you that don't know, effect is actually a pretty good library. Um, if you're trying to build extremely resilient system, uh, effect will make it easier to go do that kind of thing. And in the world of LMS where you have a lot of error handling, it is easier. But that said, it is a long to learn. Uh, so
>> let's ask let's ask Claude code because I I'm not very good at writing effect, but Claude is pretty good at it.
>> I I love it. How would we persist uh all events in the event bus so we can reload the chat later?
>> Yeah, there you go. Um, it'll figure it out. Um, and it will go write the code and I think can you and we'll be able to see a git diff really fast as well. So that'll be able to just look at the diff. So that will give a little
>> while that's running. Thank you for queuing that up. Let's go continue reading the code.
>> Yep. Um, so yeah, maybe we could look at another example of sort of one of those projections.
>> Let's look at I think there I think the question there was actually really good, which was how does this actually go into an LM? I I think we should go look I want to see your projection of what events are you discarding when you create this giant event array from all the possible events to the LM memory state. And you can if you could put side by side the LM memory state that you're building and on the other side if you can put the um and on the other side you can put all the events. So we can actually just like see it side by side. I think it' be really cool.
>> And could you hit command? Yes, exactly. Thank you.
>> Oops. There we go. Um, right. So I mean in this example, right, the the events for the LM are we're just sort of rendering it out to just a simple message with,
>> you know, user assistant role and just string content, right? And then we have all these types of events that could be encountered sort of over the over the time where it processes events. Um and yeah, so what we do is we basically just we can listen to the specific events that we care about or not. But in this case, we're actually looking at the message state. So we're actually looking at another state in this state, right? So we're looking at the messages state, which we looked at earlier. um which again is basically just keeping track of a list of messages and some other information. And because that's really the only thing that we care about for showing the LLM in this case as well, it makes more sense to just um look at that than like redo all that logic of like trying to figure out when a chunk should be appended to the last message and like all this different stuff. So the LLM memory state here which is constructing the specific messages for the LLM is just going to tap into that state that we've already constructed. So it's just mapping those messages that are in that state to right these simple versions that can be passed to an LM. So uh in this demo right we have sort of text events which are just like you know from the user to the LM or the LM to the user right just sort of standard messages
>> or we have like function calls right so it's like basically we parsed the LLM response and we decided okay this is a function call
>> on that format function call so people can go see this I think it's going to help make it a lot more tangible
>> yeah so this is like basically the the function syntax that we told the len to use for context text here, right? So we we told it to use this sort of XML format, right? So when the LM responds and wants to call a function, it does something like this. And so when we and so we parse that out, right? And we can validate the parameters and that kind of stuff. But then we instead of storing it directly and sort of feeding it back since we want it to be sort of this clean um format as an event we have to just reconstruct what we want the LM to see that it's set right so basically we're we're reconstructing that format for the function call here. So that's just so you can keep everything be really in sync with what you're with what you wanted to do. And you could have done this and you could have done this in many which ways, but you're I'm guessing you just found that since you're doing codegen, I'm guessing XML works pretty well.
>> Yeah. So this is this is um the same uh XML format that like anthropic models use internally. So we just imitate that.
>> Good to mimic it. Exactly. Yes. Especially if you're using anthropic models, you should just mimic whatever they're doing.
>> Yep. So that's why we have that specific format there.
>> So it sounds like really I think the big takeaway for almost everyone that's on this and like what if you're down to you should commit this code to the AI that works repo in the folder for today's event. I think people might have fun running this code and playing out with them for themselves and just like understanding what's happening.
>> Yeah, sure.
>> Um, but just things like some final closing thoughts as we're out here. And also if people have questions, feel free to chime in. Uh, we've got a pretty hard stop today, so we'll be stopping right on time. Um, so what's the big takeaway here? How hard in your mind? How hard is this to build? Like on a scale of 1 to 10, 10 being like Linux.
>> So I think I think the the real question here is like how complicated is your agent? Like how many interactions and like possible states are there at any point in time, right? And like like if you're building a simple, you know, back and forth chat agent, like this is like completely overkill, you know? It's like you can just you can just store the messages and and and just show that and to the user, show that to the LM, like whatever.
>> Um, but yeah, we're I mean, the reason that I I started thinking about this in the first place is that um we were just sort of, you know, we have so many different things going on in our chat UI and our product because like you know, we have like um
>> You can screen share again and show show what you mean.
>> Right. So
>> we have a lot of different things. You can build you can build a full plan. You can run it could be running code and maybe you want to change the code. Maybe you want to change the pipeline while it's running the pipeline.
>> Like that's very complicated interaction to model.
>> Yeah. It's like I could be like um also email this to myself.
>> Yeah.
>> And it's like okay now it has to think about like
>> it has to think about like okay what's what's in the current workflow like how do I
>> oh yeah it's also seeing like errors. It's also perceiving errors like from the workflow runs like when we run it it will also like know about what happened during the run. So it's like making a fix based on that. the the thing I think that would really suck about that which is you said also emailed to myself maybe you've been running the pipeline for like 15 minutes already and it's been running and collecting a bunch of data and now it's basically just going to throw all that away and make a whole new pipeline and then say now let's run it again
>> and whereas if you did it with your event driven system you could say this is this is how I want to modify the pipeline you could run another event that says what's a diff of this new pipeline versus the old pipeline the diff is very small and now you and now it's basically just kind of working along the way.
>> Yeah. Yeah. But basically, right? Yeah. We we have like a lot of different types of interactions happening, right? It's like it's editing. So, under the under the hood, this is basically like a coding agent. Like the workflow is code and it's editing it.
>> Yeah.
>> Um, so it's it's like a coding agent, but then it's also like adding these connectors which require like a specific OOTH flow, which requires like human approval, and then it's also like proposing these plans and it's like, okay, the user has to approve or reject those. And it's like okay if they approve them it has to also ask for the ooth and then what happens if they reject one of those. It's like all these like different variations of state and it it gets like really unwieldy if you don't have sort of a system for like understanding it right and I think having sort of everything be a projection of the events allows you to like have very clear views of those events in the exact ways that you need to actually understand what's going on. I think it's kind of like React, right? Technically, use state is really really overkill. All you want to do is build a website,
>> but it turns out the minute you add interactivity, it's there's no simpler way. And that little bit of complexity just makes it a lot easier to build that website in the way that we want to.
>> But over here, you really, it's the same thing. Like yes, it's a little complex to think about everything as a projection from like your event history and your event array, but once you've done that, you really have a beautifully simple system where everything is really just a readonly interaction on the current view of the world.
>> Here's my current worldview. Go do something with it. Here's my current worldview and it keeps changing. And someone asked a really good question. It's like, could tools also write events to the stream? The answer is yes. They definitely could because there's just they're just adding events to this database that we have. Uh, if I were to share screen again, one second. Yep.
>> If we go back to what we were sharing over here, what everything is doing is fundamentally we're just adding data to this database which is user ID, convoy ID, and event. And you just keep pushing more and more data into that. And that is your current worldview. And the next time your system runs or handles on events, you just go do that. If any of you have ever built like an event handler or web hook service, that's basically what you're doing. An event comes in, handle the event, you get the world view, and you go do stuff with it.
>> It's funnily enough, this paradigm exists in so many elements software. The more and more I think about it,
>> yeah.
>> Um, like the more I'm thinking like this is just everywhere.
>> It's just that most of us haven't had to write code that has dealt with this, so it feels really foreign. But all of us have used code that has done this in some form factor or another.
>> Yeah. Yeah, it's just sort of like it's a product of right the the chat loop gets really complex and ugly.
>> The chat loop is so disgusting to maintain forever
>> and it's just like ew like what I'm I'm like holding these events like I have this chat history. So basically before I was like had this like, you know, history of events for the UI I had like event like or like messages for the UI messages for the LM some sort of like mutated, you know, mutated state that's like tracking other things and it's just like gez like I can't even think about like what's happening at any point in time. So it's just like very very non-productive at some point when the when the chat agent gets complex enough. So yeah.
>> Um, so I think we're gonna have to close it off here. I've got a sadly a hard stop today. Uh, normally we will stay on for questions and answers. Now if any of you um do have more questions about this definitely definitely uh, we'll send out an email update with this and send your questions there. You can definitely ask questions on the BAML Discord as well. We'll post a link on there in a second as well. Um, and you can find the link on there if you want to sign up for next week's episode. We're going to have another really fun episode. Uh, sign up on our link over here. This has been AI Networks. We've got Anders over here helping us go. actually wrote a whole demo, made it work, showed a ventor in action a really cool way. So, uh, thank you for making time for that. Follow them on Twitter, check out their demo, check out their product. It's really freaking cool. It gets really deep. And I'm super excited once you have it the event driven loop you show in the demo of live in the code. It's going to make it even more fun to play around with. Uh, but for everyone else, thank you for making time out of your day to come listen to us. Um, this has been fun.
>> Yeah,
>> another episode out the out the window. All right, we'll send an email update to everyone that's watching. Uh, thank you everyone.