Transcription
[music] [music] Testing, testing. Hello. My name is Taras. I am a frontend developer at Aurora Center. And today I would like to tell you how an interface can adapt to user requests. As a developer, I always face such complex problems, complex interfaces. I need to create some huge table, a huge form, display some charts. And I understand that I cannot create a perfect interface for every user, because every user has their own requests, their own problems that they want to solve. Therefore, as a developer, I would like to have a system where I set some limitations, boundaries, tell it about the components I have, what my design is, what the API is, and for it to help the user create the interface they need. But here arises a problem. Not every user understands our system. They don't know its limitations. Here we can use a neural network or an LLM, which understands our system, which is configured for our limitations. It knows what components we have, and can provide the necessary interface upon the user's request. For example, the simplest case is when a user comes to our chat and says: "I want to see, for example, the device card number such and such." The LLM should understand what they mean, based on the context we have given it, go to some MCP server, call some tools, and give the user the exact result in the form of a widget. On this diagram, we can see that the user passes a prompt to the LLM. The LLM understands that it needs to go to MCP number one, call the Tool get device by serial, get information about this device. Then process it somehow and provide a structured output in the form of JSON to the user's UI, where the UI already understands that this JSON needs to be rendered in a specific widget. That's it. But we can go even further and not use pre-defined widgets, but create a set of small components, for example, the input of the same form. Here, by the way, we can see how using a switch we can determine which widget to render. That is, we see that we receive type data device, and a specific widget about this device is already rendered. Well, let's return to when the LLM can assemble the interface itself. The user asks: "Create a device with serial number 503 and the comment director's phone." The LLM understands that it needs to go to a specific tool, which will provide, based on this request, what fields our device contains, what fields are needed to create this device, what entity it is. The LLM forms a JSON, which we see here, the entity device, specific fields, pre-filled fields that the user has already requested to be pre-filled, and provides the user with a UI with a specific form where all the pre-filled fields they requested are already present, and there are additional fields that are mandatory in this case. This is the EMA. They fill it out and create a new device in the system. It seems interesting. We can go even deeper and entrust the system with the generation of the entire interface upon the user's request. That is, when a user comes to the system, they understand that there is, for example, no functionality to display graphs of devices, which ones were activated, which ones were not activated. They can say: "Generate a page for me with graphs of activated devices." Here, the LLM can generate a new microfrontend, connect it via the Federation module, and a new page will appear on the UI without reloading, so to speak, the frontend. Here we see that the LLM accesses the MCP server, which has the tool build microfrontend, which, in turn, calls the build agent, passes the code generated by the LLM to it, and this agent assembles it in some pipeline and deploys our new frontend, which is connected via Module Federation in the Module Federation Loader to our UI. Well, you might say: "And where did it come from? Maybe the LLM will make something up, write some code incorrectly, add some widgets not in our style." But here we need to approach it from the position that the LLM does not have root access to our system. And it must operate within the framework of our context, schemas, and design system API. This can be achieved by using input schemas and output schemas, in which we specify how our logic and our LLM work. We achieve type safety. We also use a Sandbox and Validation layer, in which we can run tests on this microfrontend, check something there, accordingly, against our design, system, and the like, test that specific APIs that we have already implemented are called. Here we can consider an example of describing a tool. For instance, we have a chart tool that returns information about connected devices. We describe in the description that this is obtaining the count of active devices by platform. This allows the LLM to understand that this tool can be called if it needs to get the number of devices by platform. There is an input schema. It allows the LLM to understand what needs to be passed to this tool to get a result. That is, here it is the platform by which data needs to be requested. There is an output schema, which also allows the LLM to understand what it will receive in response: the platform and the number of active devices. Using the data from this tool, it can then pass the information to the frontend. The frontend displays the required widget. In general, this approach, I think, is similar to a restaurant, you know, where we come and say: "Chef, make me a dish from the menu." The chef doesn't invent a menu that doesn't exist in this restaurant. He creates the dish you need from the necessary ingredients that are already in this restaurant. And he does not act, so to speak, outside the context of the restaurant. Similarly, in this case, we need to limit the LLM so that it does not hallucinate, does not fantasize, and acts within the context of our system. In general, at a high level, we can express and overview this architecture. The user forms their request in the form of some prompt. The LLM operates within the framework of MCP, API, and documents that it knows. Then the Validation layer checks that everything is fine and fits within our concept of our service. On the frontend, we understand what is required of us, and draw the necessary widgets. Let's look at the demo. Now I will show you how it roughly works. Here we see the demo application that I put together in just a couple of days. Okay, I'll put down the microphone now. >> Now I want to ask. Let's, thank you. Now I want to ask to display information about a specific device. Hello. Show me the device, please. Say, 600 6567. Now the LLM is processing my request. It has processed it, loaded the information about my device. And we see the card of this device. Then I think: "Well, I need to know where it is now. Where is it now?" I'll ask. Okay, now we're thinking, thinking. And here we see the location of this device on the map. Here is device 6567, which we requested earlier, it is in Moscow. Now I want to create a new device. Let's create a device with an email, for example. Well, now I'll make up some number and a comment. What comment do you want? Let's have some interaction. Beta tablet. >> Beta tablet. Now the data for the form is being processed and loaded. We get a pre-filled form with our comment, with our email, and an additional field that we need to enter. We enter it, click create device. The device has been successfully created. It is now in the system. Well, this is one case, there is also a case where we can request information about our Aurora Center. Now I will demonstrate. Tell me what are the rules in the Aurora Center policies. Okay. Well, since my demo is not working very well yet, we will have to repeat this request. It's running at home on my video card. [laughter] Okay. And we get that the Aurora Center policies have these specific rules that we can use. And this is the streamed response from the LLM. Well, there are a lot of rules. Well, while it's streaming, I can, for example, also ask. I'm going for a walk after the conference today, what's the weather like, and load, weather information. We are in Moscow, it turns out. Ah, let's find out about Kazan instead. And it's snowy in Kazan today. [laughter] >> We have Yekaterinburg tomorrow. Maybe there >> in Yekaterinburg? Yes, we can do Yekaterinburg too, but the weather will be exactly the same, [laughter] because the weather response here is mocked. Otherwise, everything worked dynamically. What we saw just now. Ah, in principle, we can return to the presentation, if we can do that. Yes. Well, thank you for your attention. If you have any questions, I will answer them. Here are some links to libraries that can be used to get started in this direction. >> One tricky request >> Any. >> Ah, you asked what's better: Android or Android? What [laughter] provocative questions? >> Well, well, look, I currently have GPT OSS LLM running with 20 billion, I think, parameters. And I don't know what it will answer, but let's try. >> Turn off the cameras. >> What's better? Well, now we'll find out the detailed answer. >> Aurora Center is better. [laughter] >> The tension has risen. >> Here >> if your goal is to ensure security, >> yes? Yes, yes. Then you should use Aurora Center. Android here is not for Well, by the way, we also manage Android, so everything is fine. Here's another interesting feature of using LLM in this context: we can write to it in any language, in principle, and it will understand and output the required widgets, for example, in English. Well, I'll write quickly. Show me device, blah blah blah. Oops. And here we get information about this device. Questions? >> Thank you. Yes. >> I am currently trying to recruit for Android development, and we currently have someone, well, we want to deploy a local LLM as well and train these models. I had a question, did you also deploy a ready-made LLM yourself and then write a separate API, like, for example, from scratch, or is it ready? >> Well, in this case, it's an R&D for us, where we are trying to use LLM to, well, improve the user experience. It's not for development, but for the end, so to speak, user, so that they can interact with it more effectively without being familiar with the system. And for this, you can use MCP servers, which have specific tools that can provide LLMs. You can also use some databases that it can access to understand, for example, how to answer user questions about Aurora Center documentation. And this is, in principle, all deployed. You can launch it through the same Ollama in a Docker container and try to use it. >> And another question, how does the context work? Because, well, how is it cleared, what parameters are used to >> Well, there are different optimization methods. First, it's context summarization. That is, if we talk to the LLM for a long time, the context expands, and then it starts to lag, so to speak. You need to perform context summarization, that is, compression. And starting new chats can help. >> Ah, well, if it's maximally automated, because not all users know how to, I don't know, create different chats. >> Well, yes, yes. That's what I'm saying, it's still at the R&D stage for us, we're not implementing it yet. We, uh, possibly in the near future, of course, will start implementing it, we'll see if our, so to speak, customers agree to deploy LLMs in their own environment, because it still requires resources. And since we usually deploy in closed environments, we cannot use external APIs. >> Thank you. >> Thank you. Your question. >> Yes, a short question. Speech text seems to be missing here. That is, voice recognition. >> Well, yes, that would have increased the quality of this user experience. Good idea. >> Let's have a question. >> Can you tell me more about the MCP servers you use? Are they custom? >> In this case, I use one MCP server for access to Aurora Center documentation. We already have a separate documentation website. I exported everything from there into Markdown format and wrote a server that, based on this Markdown, searches for the necessary things upon request. That is, roughly speaking, the LLM understands that it needs to expand its context by looking at information about Aurora Center. This is described in the tool. And it can call this tool. And based on specific keywords, articles are provided to it, and it uses them to answer the user. Very cool presentation. Great that it's interactive. >> Thank you >> to you. Thank you. And we had questions. You've gathered the whole combo from March for us. Except for the power bank. >> Well, give him the power bank already. [laughter] >> Okay, the power bank, of course, will be added. >> There was no bank. Here's your combo, >> Coca-combo.