Transcription
All right, cool. Welcome everyone. Um, yeah, so today we're going to be talking about building agentic applications, um, with a focus on generative UI. And, um, this is all within the context of Copilot Kits and AUI, both of which are frameworks for building agentic applications, as well as transporting and rendering and creating generative UI. So let's get right into it.
Um, I'm Tyler. I'm a founding engineer here at Copilot Kits. I am joined by Nathan, and I think Ulie is somewhere in the chat somewhere. Uh, Nathan, you want to introduce yourself?
>> Yeah. Yeah. So I'm the senior DevRel at Copilot Kit, and I spend most of my time working with the community and understanding, um, from a simple and deep perspective on how folks are building agentic systems, and then from there, how we can help them. And then that usually plays into being kind of in the middle between the user teams and customers and engineering. So that back and forth.
>> Yeah. Nice. And then, uh, Ulie is one of the co-founders here at Copilot Kit, and, uh, I think he's somewhere in the chat if he wants to say hi.
Cool. Uh, so let's go ahead and get started with what is Copilot Kit, right? Uh, Nathan and I both work at Copilot Kit. So what does that mean? Um, well, CopilotKit is a framework for building AI copilots, which are user-facing agentic applications. Um, it's open source. Um, it has about 30,000 GitHub stars and hundreds of thousands of weekly installs that grow weekly. Um, we work with the Fortune 100s, unicorns, Bay Area startups, and really everyone in between. And we're used by 10% of the Fortune 500s actually in production. We're actually the company behind the AUI protocol. And we'll be talking a little bit about the AUI protocol, um, as we look into demos and different types of generative UI.
So, let's take a look at the agenda. You're already one-if done. So, we've got, we're gonna have a lot of cool content to get into today. Uh, we're gonna talk about the state of Agentic UI. So, in that context, uh, what does agent UI look like today? What are people doing? What are some of the trends? And where does Copilot kit fit? And then we'll get into where does AUI fit into that. Then we'll get into some of the agentic UI patterns. And this is really the meat of the the presentation today, where we'll be talking about the different flavors of generative UI as we've noticed at Copilot Kits. And then finally, we'll end with, you know, kind of what we see as being on the horizon, um, as we enter the year of 2026. Um, so, yeah, let's get right into it.
Let's start with the state of Agentic UI. So, as we all know, as we're all here to learn more about every application is becoming a copilot kit. We've been really fortunate to be able to see the ecosystem grow, uh, kind of from the start. Copilot kit used to be a project dedicated towards serving models, and now it's really grown into serving agents, right, an abstraction of models. And as we know, like applications are becoming agentic. Uh, if anyone's used Claude, co-work in the chat here, that's a great example of something that is an agentic application. And, you know, I've been playing around with it. It's a great example of all the things that people are trying to build towards right now, both the UX and the DevX, as well.
But generally speaking, agentic apps fit into two buckets: SaaS copilots and productivity copilots. So, a SaaS copilot is going to be something like the HubSpot copilot, the Breeze assistant, where you have an existing SaaS application and you want to guide your user with an intelligent interface, like chat, through that complex SaaS application. So, think about HubSpot, right? I can ask the Breeze assistant, uh, and we will talk about it today. Um, so don't worry about that for now. Uh, SaaS copilots are basically these, uh, assistants that are able to navigate and interact with a SaaS application to simplify that process.
And then there are these productivity copilots. Uh, productivity copilots are things like Cursor, like Claude, co-work, and, uh, you know, these are where you're doing core work, and that is the interaction that is your application is automating and improving productivity for some task. And all the different types of generative UI really all fit into this. But there's kind of one key challenge when you're building these agentic applications, and it's that agentic applications are really quite complex. Well, why is that, right? Uh, one of the key challenges that you'll notice as you build an agent is that they really break the traditional request-response paradigm that we've operated on for the last 30 years. Uh, the biggest piece being that they're long-running tasks. They run for upwards of seconds to minutes, depending on the task, and they ultimately have streaming built in so that they seem faster. Right? At the end of the day, the task still takes the same amount of time, but since we're streaming this content, it makes it seem faster. And so as you build software, uh, around agents, you'll notice you have to work around streaming. That's a really fundamental, uh, paradigm when it comes to agents.
They also have structured and unstructured data, both as input and as outputs. This can be voice files. This can be text content. This can be tool calls. This can be state updates. Uh, state updates in the context of like structured pieces of state that the agent's generating. So, like a structured chat completion, for example. And they require composition. So, you need to be able to hand off control to sub-agents in most architectures that exist today to both control the context window as well as improve, kind of like, reliability and things like that. And then finally, and this is going to be a big piece of the content that we're talking about today, you can get non-deterministic UI, right? Things like AUI, things like MCP apps. These are UIs that you don't control that are instead being created by your agents, right? And these are all complex things that you need to kind of wrangle as you're building with agents today.
So, what can we do? Well, that is exactly what AUI is trying to solve. AUI is the agent user interaction protocol. So, let's dive into that a little bit. It's for connecting agentic backends, so your LangGraphs, your CrewAIs, your ADKs of the world, to your agentic front end. So, this is going to be things like, um, React and Angular and Flutter and Dart, and all the different ways that we build front ends today.
So, if we take a step back and we look at the agent protocol landscape today, there's really three. And there's going to be a little amendment to this slide that we'll make later, but I'll talk about now. So, there's MCP, which is for tools, context, and resources. With the most recent addition of MCP apps, this also expands to be able to send you back iframes that you can embed into content. And that content could be something like a chat. So, MCP apps can ship generative UI now. Um, and their approach to that is via iframes.
There's also agents through A2A. So, A2A is for agent-to-agent communication. And A2A can transport from it, from its stream, something called A2I. And what is A2I? A2I is essentially a declarative JSON spec that your agent is optimized for generating. So, you can generate structured, uh, components that are all like wired up to a component library, but then render that in a kind of deterministic way. So, the structure of the content is non-deterministic, but the actual rendering of it is deterministic.
And then finally, there's AUI, which transports all of this goodness to your front end. So, this is how you build a chat application, right? This is how you build a click-a-button to automate a task from an agent. I want to build agentic search and integrate that into my application. That is what ultimately AUI's concern is with. And then we leverage that at Copilot Kit to build the React implementation. We also have an Angular library as well.
So, looking at how all these things fit together. AUI is the transport by which A2A, MCP, MCP apps, and A2I all get into your front end. Your front end can be in React, it can be in Flutter, it can be in Dart, can be native, it can be anything that you need because AUI is a standardization layer, right? The standardization of about 16 events that agents are going to emit. And by using that, we are able to get data to our front ends.
And I'm just reading the chat as it comes through. Is AUI currently support streaming? In my case, it does not. One event is sent initially, then the rest events come later all at once. So, yes, we do definitely support streaming. Um, you need to be able to emit those streams typically via SSE or WebSockets. Um, that's a bit of a technical detail I won't get too deep into, but, uh, yes, we do support streaming. Absolutely. In fact, there's a bunch of key pieces of the spec that's implemented there. Maybe Nathan, you can link some documentation there.
>> Yeah, it it might also be the framework you're using as well.
>> Yeah, could be.
>> I've seen that.
>> Um, so let's take a look at the ecosystem and the adoption of AUI. So, first, when we launched AUI, we had all of the key open-source agent frameworks really step on board. So, these are things like LangGraph, like MRA, like CrewAI. Um, those were our initial launch partners. Um, as the protocol grew, we got some adoption from the Google ADK. So, now the Google ADK is able to work with AUI. Um, a couple months ago, we also launched support with Microsoft, and so now Microsoft's stack can emit AUI events. So, this is going to be Microsoft's, um, agent framework and Semantic Kernel and AutoGen and all of that good stuff. And not on this slide as well, uh, Strands and Agent Core are also supporting AUI as well, and that was a last month addition. So, we, when we launched, we had a really powerful reception. Um, AUI is currently sitting at something along 2 million weekly downloads. Um, it's supported across the entire agent stack, um, as we see it. And Copilot Kit has really leveraged a lot of this as well to build, kind of like, generic agent frontends as well. And all of this content, all of this context is really around generative UI. People are trying to surge to go and implement generative UI as we get things like MCP apps and AUI. And ultimately, that's going to be what a lot of what we talk about today is. But I want to ground us in the events of AUI as we're going and interacting with these different applications, just keeping in mind this is actually what's getting transported to our front end and what's being rendered, right? So, AUI is just a set of 16 standard events and it is for agent-user communication, right? So, if we look on the right-hand side, there is an example stream where we have a run that starts, we get some text message start events, which basically denotes the next content I will send is going to be a streamed piece of content, right? So, I will get a, in this case, delta event that says "hello," and then another delta that says "world," and then a text message end, right? So, that whole chunk was "hello world" in one message, and that's how streaming works is via deltas, right? And that's what AUI ultimately does for all of it, streaming. This can be state deltas, which is going to be things like, I want to generate a document, and so you send the deltas of that document over the wire. That is stateful, right? Um, messages are kind of a special type of state, but they're still state, and we just transmit the generations from the model itself. Um, it is transport list. So, we have some pre-built, uh, transports via HTTP, through, uh, HTTP streamable. Um, we also have WebSocket as an implementation and things like that. Uh, but you can ultimately transport this however you want, via WebRTC, via insert protocol here. Um, we are unopinionated in that sense. And if you want to learn more about how we do that, please reach out to me afterwards. Happy to talk more about it, but it's pretty spe, uh, pretty technical. Um, and then ultimately, AUI is just a client-server architecture, very similar to MCP, where your servers are your agents, and your clients are your front ends.
So, all right. So, let's actually see this in action. I'm just going to give off a quick demo here. Let's see if I can see this. Can everyone see my screen right now? Oh, I think I froze. Can everyone still see me? Nathan, can you see me?
>> Yes. Yes.
>> You can see the screen.
>> Yep. Yep.
>> Great. Um, so I have a couple of demos here on the left-hand side. Um, in the middle here, this is actually a template I've been building for the last, I'd say, week or so. We're going to be overhauling most of our templates with our different partner frameworks to be this. Um, but I just want to say, like, starting off really quickly, I can say, like, "Hi, how are you?" And the agent's going to respond to me. Uh, which I can see, I can see that content stream in. Um, on the left side here, there's this little button that allows me to actually see the events as they come through. I say "hi" again. And I'm getting some weird, uh, kind of like scaling in my UI here, but I can actually see all the different events come through. So, I said "hi," and then if I scroll over to, I'm actually going to make this be a popup so it's easier. Um, if I scroll over and I look at the text message content come through, we can see an actual real stream. This is connected to a LangGraph behind the scenes, for what it's worth, where I get a text message start event, the content including the deltas, and then I get a text message end event at the end. And then I get a state snapshot for, you know, relevant LangGraph states, and then I ultimately get a run finished. And all these raw events in between, these are kind of like break-glass solutions in case you want to get the raw underlying event from the graph stream. Um, LangGraph has like things like on-chain events, for example, that doesn't convert into AUI, so we transport that via the protocol via run events, that sort of thing. Um, but let's talk a little bit about generative UI, right?
So, I'm going to click on static generative UI, which we'll talk about in an upcoming demo. It's going to call to search the docs and it's going to generate a component for me here. It's searching the docs because it's trying to understand what is static generative UI. So, I've actually wired up the Copilot Kit MCP tool to be able to research exactly as we describe it on the the page. And this is a great example of, uh, static generative UI is instead of replying with text, you render a pre-built, developer-controlled UI component, like a card or a widget, in response to agent activity, often a tool call. It's static because the UI is predefined and consistent. This is a great example of something that you can build with Copilot Kit. Right? We got a weather component here. We searched the docs with our MCP server. We called, get the weather tool, which is a tool I defined on the backend. And then I've called a, um, I've given the agent a tool that's "show weather," essentially, which is this weather card where it plugs in all of this content for the humidity, the wind level, the feels like, clear skies, all that good stuff, and then it renders that that content. Um, so this is a great example of static generative UI. But, you know, I just mentioned that, what is that? What is static generative UI? Well, that's going to be what we're talking about for the next little bit, all the different kinds, right? So, design patterns. We are now in design patterns land. I'm just going to give a quick glance at our, um, chat here. Raymond, I'm glad you asked. How do we do that? How is that defined? That's actually the very next thing we're going to be talking about. So, great question.
So, generative UI, there's three kinds, three kinds. Uh, there is static generative UI, which we just saw. There's open-ended generative UI, and there's declarative generative UI. We're going to talk about each of these, but let's start with static. Static generative UI is essentially we're mapping generated data from the LLM to an existing component on the front end. So, AUI is going to transmit data, and we're going to take that data, that structured data, and convert it into component content, which we saw in that example here a moment ago, right? This is structured content, and really, this this content is being generated by the LLM. So, if we come back into here and we look at this weather component, um, it basically gives us deep control over the UI that's generated by agents. So, the agent's going to generate something like maybe a tool call, and then we map that tool call, and I'll show you the code here in just one more slide. Uh, we map that tool call to a front-end component, and all that's transferred via AUI. Um, so a great example of this is we maybe want to map tool calls to a component, or we want to map agent state to a component.
So, what does this code actually look like in action? Well, on the left-hand side is a simple agent. It's a nothing framework. I just invented it for the sake of example. I've mounted in a "get weather" tool. Uh, it's going to return a static temperature of 38 degrees and in a humidity of 50. And then I create and I instantiate an agent that doesn't exist, right? Looks kind of like create agent from LangGraph if you sequence. I give it a model and I give it some tools, and then I call agent.serve. Um, this is a pretty common pattern in the AGI ecosystem where our partner frameworks will, uh, give you a compatibility layer that emits AUI events. Um, on the right-hand side is the actual code, and yes, all of this code is available on GitHub. I will link that afterwards. Um, Nathan, for your reference, this is the with-LangGraph Python repo. Um, okay.
>> So, this is going to be on the right-hand side. Really minimal example of what I did in that demo over there where I have a Copilot Chat that is this whole component that we see here in the middle, and I have a `useRenderToolCall` which is basically going to map the "get weather" tool on the left to the component on the right. This is why it's static, right? We define the exact pixel-perfect accuracy of how we want this weather card to look, and we map that generated data to the tool component on the front end. This is static generative UI where we're rendering a tool. Um, and then, you know, if we dive into it a little bit further, um, just looking at it all together, you can see the chat, you can see the components. Um, this is the Copilot Chat experience that we saw on the right. Um, in Copilot Kit, you can also do this through, um, what we call headless UI, through our `useAgent` hook, which basically gives you raw, direct programmatic access to your agent framework of choice through a UI. So, in this case, we can see we have all the agent's messages. So, we just map them and we render the content, and we render the UI if it exists. Um, so in this case, `message` is going to be that component that we mounted before in this previous example with the `useRenderToolCall`.
So, there's some pros and some cons to this. The pros is that it's very simple, right? If you think about it, the mental model is a tool call maps to a component. State maps to a component. Structured data maps to a component. It's a component that I control. It is data that I can, that is generated by the LLM. So, I have a lot of ownership over how to render that, which gives you full control over your UI, which will make your designers really happy. Um, and it's great for your common trodden paths in your application. Maybe you want to generate a chart, and you want to be really specific about how that chart renders. Maybe you want to show the user a weather card, right? That's a pretty bespoke example at this point, but it's a great example because it is data that comes from either an API, and you can kind of envision how you can expand that, right?
Um, but there are some cons, and the biggest con is going to be coupling. And as we look at these different types of generative UI, I want you to think about the scale of coupling, right? And what, what do I mean by coupling, really? You know, maybe today your organization is set up into front-end and back-end teams, right? Agents make it a little bit tricky where your agent is going to be emitting tool calls, and those tool calls are going to be defined by your back-end team, the args, the output, the name of the tool, even. And on the front end, you need to be able to see what those tools are and you need to map those to components, and you get really good pixel-perfect accuracy, but it's like an API design, right? If the contract changes, things will break. And so that coupling is what you have to keep in mind here. Really high coupling. You're building a full-stack application. If you're a full-stack dev, really easy. But if you're separated into two separate teams, it becomes harder. Um, and ultimately, the codebase is going to grow linearly with your use cases, right? So, as you add more and more UI or more and more tools, and you want to have new UI for it, you're going to have to create on the other side components for each of them or map to pre-existing components that you've written. So, these are some of the problems that the different types of generative UI are trying to tackle: is really coupling and size, like complexity, essentially.
Um, so that leads us into the next kind of generative UI, which we we label as open-ended, but the best example on the market right now for this is MCP apps, right? So, this is where the agent is going to give you a fully open-ended piece of content. And ultimately, that's going to be either HTML content that you dangerously insert into your application. And I say "dangerously insert" because that's the literal function you call in the front end, uh, if you know what that means, which basically means you're editing the, uh, browser directly, or it's going to give you an iframe that you then embed inside of your application. Um, so a great example of this is, and this is a very out-of-date thing because it launched, I think this week actually, Nathan, is that right? That MCP apps launched?
>> Officially. Yes.
>> Yeah. So, we've been working, we're a day one, we're a day one partner with MCP apps. Um, so we've been working with them for quite a while. They used to be called, um, the terminology, right? They used to be called MCP UI, and then Anthropic and, uh, OpenAI got involved, and now it's MCP apps. And if you go and you build an OpenAI ChatGPT app and you embed it, like Zillow, that's going to be an MCP app. Same thing with Anthropic, uh, with Claude as well.
So, if we look on the right-hand side there, we have like this, uh, component essentially that we're embedding an iframe in, and, um, we're able to see different types of pizzas in San Francisco. And Tony's Pizza is the best, if anyone's ever been. Um, I actually have a demo of this with Copilot Kit on the right-hand side that we can take a look at. So, we have MCP apps here. Uh, this is the with-MCP-apps repo, Nathan, if you want to send that into the chat. Um, this is a template that we released as a part of our launch. We also have some guides on how this works. I think it's pretty cool. I think you're going to like this.
So, I have wired this up to an example MCP app server, uh, which serves me, uh, 3JS 3D rendered content. So, I'm going to ask this to, "Please generate me a cube with an outline." And what that's going to do is it's going to, if we look on the right-hand side here, generate a bunch of tool call arguments for generating that cube. And then I'm able to render that inside of the chat here. Pretty cool, right? I'm embedding an iframe from an MCP server and rendering it here. I can then ask it, "Hey, give it an outline." And it's going to go and edit that. Now, it's going to have an outline. Um, I guess it decided differently. Maybe I say, like, "Choose different colors on each face and make it spinable by me." So, then it's going to go generate, you know, like, basically what I'm doing is I'm iterating back and forth with an MCP server that's embedding an iframe into my application. Uh, so let's see what this actually comes up with. And again, this is generating tool call arguments. And so now I can actually 3D render this cube and spin it around. And then, just to drive the point really all the way home, I can ask it, "Make it a Rubik's cube." Now I can iterate back and forth with this agent to generate a 3D scene of whatever I choose. Um, and now imagine this for Zillow. Right now, I can embed my Zillow application. In this case, it gave us this pretty cool Rubik's cube. Um, I can embed my Zillow application. I can embed, um, my insert SaaS application here into users' applications. I can, as a, uh, agent builder, embed my partners into my chat application with an iframe, right? Like, that's the whole benefit here. Um, so if we come back into, um, the slides here, uh, there's some pros and there's some cons to this as well. Uh, biggest one is going to be that we have low coupling now between the front end and the back end, and we can ultimately render anything. Um, and there's a lot of cons, and I think a lot of people are getting into this in the into the chat here, and MCP apps handles quite a bit of this. If you're curious, their docs explain a lot of how they get around this. Um, but first thing is, the UI may UI may vary unpredictably, right? In the way that I set this up with a 3JS server, you saw it didn't give me that outline, right? It's unpredictable. Earlier it gave me the outline, now it didn't. Um, so your common use cases may suffer if you keep it too open-ended, right? Uh, it's really difficult to style. Um, I've done, I've given that demo five times, and the Rubik's cube looks slightly different every single time. Um, still a Rubik's cube, but it is slightly different, right? Um, and there are some security and performance concerns. Um, namely, you're embedding iframes into the application, and you run into some issues with XSS. As somebody else mentioned, MCP apps as a spec gets around this through, uh, the protocol, essentially. I would really advocate if you're curious to go take a look at it. Um, and yeah, somebody linked into it there. Great. Uh, thank you, Joan. But Copilot Kit has support for it as that spec continues to evolve and these things start to get kind of finalized as well. Um, but one last thing is that this is typically a web-first experience. Um, they are definitely looking at the different, like, native clients as well. Um, but through that iframe approach, it'll be a little bit tricky. So, it's pretty cool, pretty awesome. You can do a lot of really powerful things with it. I can embed my Zillow application into my ChatGPT. I can embed a, I can embed HubSpot into my application. Like, these sorts of things are pretty cool, and they're interactable widgets. Um, and the whole spec allows you for, allows you to control, like, clicking a button, which then renders the chat in a different way. Um, we have a couple of examples like that, Nathan, if you wanted to link it. Um, I think that's from one of our demo builders.
Uh, so the last one I want to show off is going to be declarative UI. And declarative UI is essentially a semi-open set of constrained UIs that are driven by a declarative spec. That's a lot of words. So, let's break it down. Semi-open. Basically, that means it's on rails, but you can still move in multiple directions. Um, it's constrained UI, so you decide what the UIs look like. So, imagine your agent being able to render arbitrary ShadCN components in whatever it sees fit. Uh, and it's ultimately driven by a declarative spec. So, I can, uh, have my agent generate some JSON, which then renders inside of my chat. Like, for example, maybe I want to have, I want to get the user's credit information, right? And so the user says, "Hey, I want to apply for a credit card." I didn't wire anything up for it to be able to apply to a credit card. And it whips up a component, and it has like their credit score and insert data here, right? That's just one example of something that you could do. So, it sits really between that fully open-ended example that we saw there and static generative UI. Uh, so static is, we have this maps to that. Open-ended generative UI is just render this thing that I gave you. Declarative generative UI is going to be somewhere in between where, here is what I want. Render it how you see fit. That is what the agent's going to be doing. It's typically, it's typically described by cards and widgets with like recurrent elements in between. So, like the, imagine a form, imagine a chart, that sort of thing.
Uh, who declares, who determines the UI components for declarative content? AUI. AUI is a transport, or A2I, which we're going to get into right now, actually. So, A2I is a declarative open spec, and this slide is very out of date. We need to update this to be A2I. Um, this was before A2I had actually launched. Again, we're a day one partner with them. And, you know, this is an early iteration, but now as a JSON spec, we're going to look at it actually here in a second, where you can envision it says, "I want a card," and in that card, I want an image with this text and this text. Right? That's all declarative. Like, we don't actually know what the card looks like from the agent's perspective. We just send that to the front ends, and then it renders it.
So, let's actually take a look at this in action. Um, this is a project we made with Google, uh, called the A2I Composer. So, A2I comes with really two pieces. One is going to be your spec, which the agent can generate, and another thing is going to be a renderer for that spec. So, they ship out of the box with Lit, which is a web component library that they have. Um, so if we look at the different components, these are all the web components that they have. You know, we see card, we can see text, content, we can see an image, text field, checkbox, all that good stuff. It also ships with some icons that we can see. These are the icons that it can use. And, um, we have a gallery of pre-built components using this spec. So, we can see a recipe card, we can see a track list, I can see a music player, I can see a user profile, all sorts of good stuff here.
What I'm actually going to do today is I'm going to take one of these components and ask the agent to make it funnier. Uh, that's essentially what I'm going to do. So, I'm going to take this flight card component. I'm going to open in the widget editor. On the right-hand side is Copilot Kit, and I'm just going to say "hi" to it, and it's going to say "hello." So, that's our sidebar component on the right-hand side there. That's a part of Copilot Kit. All of this content that we're talking about here is going to be transmitted via AUI. So, I just want to keep that in mind. Those standard events that I mentioned earlier are going to be transmitted via AUI.
So, I'm gonna ask this to make this component funnier. So, now what it's going to do is it's going to, I'm going to ask me some follow-up questions. I'm just going to say "choose everything." And now it's going to start generating that component. And so, as it kind of streams through, I can see all this content actually be generated. This is all JSON, right? This is JSON content. We can see on the left-hand side here. This is a declarative approach. So, now it's funnier, apparently. Uh, this is an LLM's idea of humor. "Adventure awaits. Buckle up for the fun ride." Departs. "Will we make it?" Arrives. So, I guess that's funny. I don't, no, I wouldn't say so, but...
>> It's pretty funny.
>> Yeah, I think it's funny. It thinks it's funny, but, um, this is a great example of, I just gave it something kind of arbitrary, said "make it funnier," and it came up with something. Maybe now I want to have a, a button. "Make a funny button on it," or "add a funny button to it." Now it's going to do another iteration. Now it's going to add a button, right? And as that's generating on the left-hand side here, you see what it actually did. So, it gave me a column, and in that column there was a header row, a funny subtitle, a route row, divider, times row. It gave me this little button called "Make me laugh," that when I click on it, you know, you can map the agent will, through A2I, map to some action. Um, and Copilot Kit is basically transporting this. Um, on the right-hand side is the generation. So, imagine the agent's actually generating it on the fly. Um, and in the middle is actually the rendering. Copilot Kit allows you to render that component inside of your chat. AI Composer is for actually taking this and giving this to your agent as some pre-built component that it can render. Um, that's the kind of idea is like, you can inside of here compose a bunch of different AUI components and then compose those into your application. That's the whole idea, right? So, I can then take this JSON and say, "Here's a component that I want you to render," that sort of thing.
So, yeah, if we just look through, this is the spec, right? You can have a component. That component can have text. In this case, the literal content is "Adventure awaits," right? And all this "on-time delayed boarding." These are different states that it can enter. You can see as I click through them, uh, this is the different states that the agent can choose to enter, right? Uh, this is the content that gets filled out either by you, the user, or by the agent itself. So, this is declarative, right? Now, I can have, kind of constrained but pretty open UI that the agent is generating, kind of on the fly. And I can give it concepts of things that I need it. Like, for example, uh, in this gallery, maybe I say, you know, "I want something that looks like a flight component," and it just goes and generates that. Maybe we start from scratch, because I think that's a pretty good example. Maybe I say something like, "Hi." It's going to take us into the chat here, and I say, "Make, um, I want to submit an article. Give me a form, something open-ended." So, it's going to start generating that. And in this case, I got a title, content, and submit article. Now, imagine this is in your application, right? The user says, "Hey, I want to message person Y, right?" You didn't set up anything to do that, but now then the agent, since it's wired up to Slack, can give you a form to insert that content and then go send it out. And when I say wired up to Slack, probably through like an MCP server or something like that. Now, imagine all this wired up, right? You can create some pretty interesting interactions where maybe you have a bunch of different MCP servers, and you have AUI on the front end, and your agent can generate it. Now you can kind of give your user any way to interact with any set of those MCP servers. Pretty interesting, right? Um, and one of the best parts is you fully control what each of these individual components look like, like the input, like the button. You can control the typography and all the pixel-perfect accuracy of it, but it's just the structure that changes.
All right, so with that in mind, let's come back to this and let's talk a little bit more about it. And Pavle had a quick question just regarding, um, >> can it generate mini apps like a calculator? Um, >> I've never tried that. Um, AUI ultimately doesn't have logic pre-built. So, you would have to kind of like work with the agent to iterate through that, but it's definitely possible. I would say that like apps is something that MCP apps is concerned with. A2I is going to keep expanding. Um, and right now, I think it's really dedicated towards forms. There's new versions of the spec coming out soon. Um, so keep in mind, I think that you can go check out the A2I repo if you're interested. Um, but they're starting to think about things like charts. This is like, how do I engage my user in a non-deterministic way, but I fully control what the UI looks like? That's really what this is trying to solve. That could be a calculator, but I think that's one of the more useful things is like a form for contents and that sort of thing.
All right. Um, so, right, declarative UI. There's some pros, there's some cons. Uh, now, one of the pros is you have very low coupling between your front end and back end. Your front-end team can pixel-perfect accuracy each component, and then your back-end team can decide exactly how they want these different AUI specs to be rendered and, or sorry, generated. But the rendering is managed by the front ends, the generation's managed by the back end. Um, you can accommodate many of your common use cases. So, now, you know, get inserting data from a form, that sort of stuff. You can have your agent generate those as it needs, and it looks like it fits into your application. That's one of the more important things, and it's really extensible to any rendering framework. So, React, um, Angular, Vue, Flutter, anything, right? It's just JSON data that maps to a renderer of some kind. Um, Copilot Kit is actually building the React renderer right now in the open source, um, outside of the Lit one, which is pretty cool. Um, we can also accommodate, uh, so some cons, sorry to get ahead of myself. Some cons is that we can't really accommodate fully custom UI needs. So, it's constrained by the spec. So, like, for example, that calculator thing you mentioned, uh, maybe I want to have a little display, right? That little display is going to have to be either custom or fit and be constrained by the spec. So, that's like one of the constraints with A2I. Um, and the UI UI may vary unpredictably again, right? So, now it's not the look, it's the structure, right? So, if we go back into that example with A2I Composer, and I ask it five different times to do the same thing, the structure might be slightly different. It might decide this is funnier than that, and sometime, uh, some instances, that sort of thing.
Awesome. So, let's just keep moving along. What does AUI support? So, this whole time I've been showing you everything through Copilot Kit's lens. Um, and I kept mentioning that all of this is supported by AUI. But what does that mean? Um, well, AUI's approach to the generative UI landscape is really to embrace it, to extend it, and to innovate on it. So, uh, as A2A was coming out, AUI supports embedding A2A meshes within any agent framework. So, AUI is really the, uh, protocol that's transporting all of these different agentic protocols to your front end. Um, as MCP apps came out, as MCP UI evolved into MCP apps and then, uh, released this week, we have day one support. So, now AUI can transport all that to your front end, and any AUI client is going to be able to receive and render MCP apps. Same thing with A2I. Um, so AUI is going to support everything that the generative UI landscape is coming out with. And we're also innovating, right? So, like, we're coming out with the React renderer for, um, A2I, and you'll be seeing that come out soon, where now you can map, like, ShadCN, for example, to A2I and everything in between. Um, I see a bunch of questions in the chat. Do not worry. I do see them. I'm going to be coming to them. Do not fret. Um, but I just want to get through this, given time. Nathan, how are we on time, by the way?
>> We're, we're doing pretty good. We've got 15ish minutes.
>> So, >> yeah. So, let me wrap up, guys. And I see all these, these, um, these questions. I really appreciate it. Um, I am going to answer them. I'm just going to stay focused on this presentation. So, I appreciate the, the questions, and we'll get back to it.
All right. So, agent state. This is another UI design pattern that we've seen. Um, and this is really for, think about like V0. I saw somebody mentioned V0. Think about Claude co-work. Think about ChatGPT, right? Right? If I ask ChatGPT or Claude co-work to give me a markdown rendering of a report that I want to write, and it starts streaming all that content out, and I see on the right-hand panel, um, a markdown file up here. Or if I'm in V0, and I ask it, "Make a website that looks like Uber but designed by Vercel," or something, I don't know, just coming up with examples, and then it goes and it writes something and looks great. These are pieces of state, right? So, if we look at it, agents are abstractions of LLMs. What are they abstractions of? Making LLMs stateful. Essentially, LLMs are stateless, right? If I give a content, it will not remember what I said. I have to give it the previous contents plus my new message to get new output, and have it remember that. And that's one of the fundamental things that agents are trying to solve is being a stateful abstraction of LLMs. So, they can, that state can be something like messages, but it can also be structured pieces of data, right? So, a message is stateful, right? Like sending all those messages, that's a stateful piece of data. Uh, but it can also be structured pieces of data. If you've used something like LangGraph before, um, as you traverse the, the graph within LangGraph, it has pieces of state that you can update, and you can pass that state either to your agent, or you can just have it be structured and stored within that interaction. Um, it could be generated, but it doesn't necessarily have to be, right? Uh, if I write a bunch of content, and I know exactly what I want it to look like as a developer, I don't have to have that be generated. I can have that still be structured. Um, and we can leverage all of these patterns to build some really interesting applications where we can actually share state between the front end and the back end.
So, AUI has a first-class concept of state. Um, so this is going to be one type of state is going to be messages. Another type of state is going to be just a state snapshot. Um, and a state snapshot is generic and intentionally so, so that any kind of state can be transported over the wire through AUI. So, on the left-hand side, again, it's that fake agent framework I mentioned where I'm creating an agent. I have GPT-51 as my model. I have some tools that I don't define or show here. But the important thing I want you to notice is the state schema, which is the name here. On the right-hand side, I can, inside of Copilot Kit, call `useAgent` and I can call `agentState`, and I can just read that state, and that's going to be reactively updated via AUI. And Copilot Kit, as the React renderer, is going to let you react to that on the front end. So, as agent state updates, your UI can also update. This works the other way, though. We can actually update the state from the front end to the back end through AUI. AUI allows you to send input to your agent, along with that input can be new states. So, you can get the state sharing, essentially. So, if you think back to Claude co-work or ChatGPT, you're able to also edit that document that it generated, right?
And then you can see those edits reflected and understood by the, uh, the agent of ChatGPT or the agent of Cloud, which is pretty cool, right? So how do they actually make that work? Well, they have their own ways, but one of the ways that Copilot Kit defines it is through state. Um, so we can update that through `setState` through AGI. So on the right-hand side, um, I've now, or sorry, on the left-hand side, I've now defined a state schema with a counter and on the right-hand side, I want to make a button that's going to increase that counter every time I click it. And so it's really just as simple as `agents.setState`. Um, this `useAgentReturn` is actually going to be directly the AUI client, uh, for TypeScript. Um, Copilot Kit manages all the security of that. Um, so it's a read-only version. Uh, read-only in the sense that your users can't edit the security configuration on your backend. Um, but it can edit things like state and add messages, that sort of thing. U, so in this case, when I click this button, the counter is going to increase and the agent's going to be aware of that, which is pretty cool.
But where it gets really powerful is doing both, both reading and writing. So now I can have a document on the left-hand side and I can have a tool for generating that document. On the right-hand side, I can have an agent. I can have some changes from the user and when the agent generates something, I can just render `agent.state.document`. So I think that we condensed that for the sake of brevity. That's going to be `agent.state.document` and then we can, when a user wants to commit some changes, we can do `agents.setState('document', changes)`. So this allows us to have like a bidirectional interaction where the agent generates something, user goes and edits it, submits that, the agent sees those changes and it's kind of collaborative, right? And that's one of the big pillars that Copilot gets built on is like collaboration with your agents.
Um, so we actually already looked at the demo. Um, but I am going to add one little piece to this which I think is pretty interesting, which is this. Um, if you see on the right-hand side, I have this little button called "App Mode", right? And when I click on "App Mode", um, what I have now are some sticky notes, right? This is the canvas type of interaction. I can now ask it to add a sticky note about pizza, right? And what it's going to do is it's going to call the `addNote` tool. And now I have a sticky note for pizza, essentially, right? And this is bidirectional. I can edit this and I can say like "hi" and the agent's going to be aware of that, right? This is one of the things that that bidirectional state syncing allows you to do, which is pretty cool.
Um, so we're getting towards the end now and I'll be able to answer some questions, but I just want to talk a little bit about what is on the horizon as we see it. Um, one of the biggest things is agent steering, right? So if you think about Claude, you think about Code Interpreter, you think about all these different ways that we interact with agents today. Um, they all give you surfaces by which you can steer that agent. Think about in Claude, one of their more recent releases, now in the interview, you can have it interview you and say, "Which of these options do you prefer?" and you select one, two, three, and that's like a UI that you click instead of Code Interpreter. That's an example of agent steering where you are working with the agent, you tell it to go do something, and then it comes back to you with questions. Copilot Kit supports this, but, uh, we call this "human in the loop," essentially, where you are human and you're in the loop of the agent's execution. Uh, so this is, if you're familiar with something like LangGraph, this is an interrupt. Copilot Kit also has a concept of human in the loop on the front end that you can define. Um, but as your agents become more autonomous, it's really important, and you'll get much better results if you can steer them mid-run, and we've seen that to be a really successful pattern and something that we're building in Copilot Kit as well.
Is actually the self-improving layer, and we call this Auto RLHF, right? So if we think about that agent steering a second ago, Um, that's user feedback, that's user input, that's human edits, right? If we annotate that into our content that we store, we can now train on that data. And this is actually how Cursor built their coding, uh, model. This is how Windsurf did the same thing. This is how you get GPT-5 codecs. This is how you get, um, Claude to improve over time. It's automatic human feedback of accepted or denied pieces of code content that they then pump into their model training, which is essentially a reinforcement layer, right? Um, so this is something that we're building into, uh, Copilot Kit. Um, which is going to be pretty interesting.
And, uh, what are some, some conclusions? I think some of the really big conclusions I want you to take away from this is agents are real. They're here. Uh, I think 2025 is taking them to production, and 2026 is going to be extending them with things like generative UI. I really want you to take away that agents change how we think about UI and UX, right? Um, we can have these kind of generative state patterns where I'm generating some content on the backend and that is updating my front end and the user's collaborating, right? It's a collaborative experience with an, uh, intelligent system, essentially. And generative UI is an essential pattern in that UI/UX toolbox, uh, where you're able to embed partnership applications directly into your app, or you're able to have your agent go and generate an experience that a user wanted, or you're able to map generated data to pixel-perfect components. The different types of generative UI that we talked about today are really essential patterns. I also just want you to think more about what you could do with something like Auto RLHF and how we could ultimately provide learning to agents at scale. I certainly know that the Bay Area is thinking about this a lot with some of the hackathons that are coming up.
So, um, that's the end of our presentation. I think now is time for questions. I'm going to leave these slides up as we get into this section of it, but please give us a star on GitHub. The AUI repo is great. I know Nathan, you just really, uh, recently launched the generative UI repo. Um, do you want to talk a little bit about that?
Yeah. So, we put together a resource that is actually pretty similar. It's like, you know, what these slides cover, really, uh, between the three different types of generative UI. Um, also, there's tons of resources out in the ether, so it's, it's open. And so we have a bunch of blogs and feel free to contribute to it. Um, we chose the ones that that were like high quality, and there's a ton of YouTube videos out there as well. And so, um, I don't want to take up too much time on it. I can put the link in the chat, but yeah, it's just a resource where you can check out generative UI examples, blogs, YouTube videos. It's anywhere from like Google to Microsoft to, you know, a bunch of different companies that are talking about it and how they think about it. So, it's good to take in like all perspectives.
>> Yeah, exactly. Well said. Um, cool. We got a lot of questions to get to, guys. Uh, I'm just going to start back from when I stopped looking. So, let's see.
>> There was a bunch on security. We should probably.
>> I couldn't talk too much on the MCP app side of this. I think that Joe posted a link where there's a lot of discussions. Um, if you're looking for the repo, it is the `model-context-apps`. That is where the spec is defined along with a bunch of examples if you want to take a look at it. But there's a lot of conversation about this actively happening right now. Um, and they do quite a bit of things. Um, one of the things that Joe mentioned that is good to mention is doing double iframe, excuse me, double iframe and sandboxing, which helps, um, that sort of thing. So, I can't talk too much on it. I'm not the expert there, but there is certainly active community behind this. It's backed by Anthropic, backed by OpenAI, so lots of people are thinking about that if anyone's curious.
Um, just going to start with Stephanie's question. "Who determines the UI components for declarative content? AUI question mark?" I think this is right before we got into A2UI. Um, but declarative UI is a concept, and A2UI is an implementation of that concept, right? Uh, somebody else mentioned, I think it's JSON. It's something from Vercel that came out recently. I'm trying to, we'll get to it, get to in a moment, but, um, there's lots of different, there's going to be lots of different types of generative or declarative UI specs where, uh, the JSON is optimized one way or the other for generating components, but ultimately, what those components map to is up to you. So AUI is just transporting them. Copilot Kit is just showing you whatever you've chosen, and, um, the spec that you choose can be swapped out. We fully support, um, the ones from Vercel, the ones from Google as well.
Um, so I'm just going to keep moving along here. Let's see. Uh, so how mature or standardized is HUI beyond Google? So HUI launched December of last year. So it's about a little bit over a month old. Um, they're rapidly approaching their 1.0. So if you are curious about that and the maturity of it, um, feel free to check out that repo. We're pretty close with them as well. We interact with them, I think, weekly at this point. And, uh, we're also contributing into that repo, um, the React renderer. So if you build a React application, you'll notice, um, similar things where you'll be able to render React, uh, HUI components. Um, so they're approaching their 1.0, uh, version of the spec, and you'll see that kind of mature over the next couple of months.
Um, some Keem, ATI, allegedly React is coming soon in Q1 2026 and the whole project is pre 1.0. Yep. Very accurate. We are building that React, uh, implementation. The Lit, Flutter versions of HUI, um, are already out, and we're building the React one, and then we're also, um, I think they're also moving towards that 1.0. Um, somebody linked the roadmap. Thank you for that. When you say AUI components, did you mean AUI components? Yes. If I said A2UI versus AGUI, it's because they're kind of similar in name. Um, so forgive me there. But AUI is not, like, creating components. HUI is from transporting the JSON content from your agents, which then gets rendered into components.
Um, I saw that mini calculator app thing. Um, I think you could do that. Um, for what it's worth, I think you could probably just oneshot it by asking that ATI composer. Um, maybe Nathan, you can link it in here and, uh, Pavle, you give it a try and let us know.
>> Um.
>> Yeah, for sure.
>> Hi, can these components from the form you generated be used to fill actual forms and hit our backend? Yes. And that's one of the coolest things about A2UI. Um, you can map all of those button clicks. You can basically give the agent the ability to map those button clicks to actions. Um, I won't get too into it because it gets a little bit techy, but there are ways to do it that are pretty interesting. Um, the A2UI docs explain this pretty well as well.
Can you compose them? This is all for everyone's context. I think it's all in the context of A2UI. Um, so next question was, "Can we compose a more complex view that consists of the component parts?" Um, you can ultimately, uh, HUI spec is really concerned about formatting. So there's a lot of ways to format content. Um, and that can be content that renders in a chat, but as you saw in the HUI composer example, it can also be just content that renders on a page, right? Um, you could expand that and say like, "Hey, you're generating a dashboard," and the formatting of HUI is able to support that.
Um, so I can start with a blank canvas and have MCTP servers for Slack, Discord, code editor, writing, and just let the agent choose on how to map it when the user is working on a particular thing. Exactly. That's one of the biggest benefits of something like A2UI is you can, you control what the theming of the components look like, but the agent decides what component it needs to service a user, which is pretty interesting, right?
Um, can we have audio elements in the apps like recording? Yes, you can. Um, they have a couple components for audio, for images, for video. Um, I think it's all built into the, the renderer as well. Backend agent will write different JSON for the front end to render, but ultimately must adhere to the schemas, which is a good thing. Validation. Yep. Create astute, uh, observation, Keem. What type of latency overhead does AUI/Copilot Kit introduce? Um, very minimal. AUI is, um, not a hosted thing. AUI is a translation of events or a standardization of events. So if you're someone like LangGraph, you already have, uh, an internal spec, but you want to ship that to a ton of different users, you convert that spec into AGUI. Sorry, you convert that spec into AGUI, and then now you can ship to all the different AGUI clients like Svelte, like Angular, like React, all that good stuff. Um, and the latency there is pretty minimal. It's as fast as code can run. Um, which I would say is very fast. And then for the latency of transporting that to the front end, I would say standard latency of talking to any kind of backend. Um, less than a couple, less than a millisecond to some degree, something like that, of added delay, I should say.
Um, >> there is like two minutes. I pretty much have a hard stop. I think I have a meeting in, in just a couple minutes. There's one question I wanted to get to, um, by, uh, Chong. I hope I pronounced that correctly. Uh, "Is AGUI ready for large scale deployment or production use?"
>> Yes.
>> And, and it is, it already is in production.
>> Yeah, we have a lot of, uh, people that either are working with us directly or working with us indirectly that are in production with AGUI today. Um, it's a very large ecosystem. Copilot Kit's one part of that ecosystem. You know, we are the core maintainers of the spec, but the spec has really outgrown us, like, just just us at this point. There's all sorts of people. We get different integrations every day. We get different UI frameworks every day. Um, tons of using it in, tons of people using it in production, in highly regulated industries like in finance, like in Medicare, like in, sorry, healthcare, like in, uh, uh, cyber security, like all sorts of different, uh, companies working with it today.
Um, there are a lot more questions. Um, I think we are at time, guys. I tried to get through as many as I could. Uh, if you do have any questions, please reach out to me on LinkedIn, reach out to me on email and Slack. We're on Discord as well. Um, I think that the Discord links are actually posted up here if anyone. Oh, they're not. Um, you can go to our docs page or our website and find our Discord link and feel free to reach out to me.
>> I'm, I'm pasting it in now.
>> Um.
>> So any questions that went unanswered? Um, you can join the AGUI Discord and we can continue the conversation there.
>> Yeah. Cool. All right. Thank you so much everyone. I appreciate you coming.
>> Yeah. Awesome. Thank you. Bye.