Transcription
So I'm very happy to be here, um, at the conference, and thanks, uh, for inviting me. And today we will talk about, um, writing AI agents with NodeJS and architecting the system to work at scale and, um, like to work over time.
Uh, if you would like to, like, find me later on for questions, that's my Twitter. Uh, that's my, uh, LinkedIn profile, and that's my Gmail. You are, uh, welcome to contact me, and, uh, let's get to business.
So, um, at first, uh, we had the chatbots. Like, we had the ChatGPT, and, um, I mean, it was, uh, let's say, it was magical. It's kind of changed the world. We all were amazed and started to like use it a lot. Um, and, uh, it became like a central tool that we're working with. But, um, the chatbot had answered our questions, but it has like, um, it has like no conversation context in the beginning. Uh, it wasn't able to perform actions. Um, let's say it had like no memory. You had to ask a question like several times. Um, and then, um, you know, um, uh, let's say the, um, um, the intention was to develop, um, a tool or something that would be able to act as well and do activities and can have memory and to be more active. Not something like very stateless that you're asking a question and receiving an answer, like not like a Wikipedia, let's say, to have another teammate, um, that can help us, uh, like, uh, do our jobs. Um, so, um, then the chatbots have evolved from like stateless chatbots, as we said, which is just like input and prompts, to something which is more stateful, which is the AI agent that we are familiar with today.
So the AI agents are able to, um, let's say, when they receive a question, they're able to plan the next move and do activities and then also like repeat that circle, um, according to their like decision-making mechanism. So they have like memory, um, they're able to execute tools, they have a context, um, they're like, they have like a state which is long-living across tasks, and that's, let's say, the basic, uh, loop of an AI agent. It receives a goal, and while that goal is not done, um, it is like planning the next moves, um, accessing tools, and executing them, and, um, then it checks whether the goal has been achieved. If not, it's like doing another loop. And we will take, like, this, let's say, this loop and understand how to build a server which is doing that, and also try to understand how to, uh, write that server at scale.
All right. So the building blocks of the AI agents. First of all, we have the LLM and the prompt. That's the first thing that we have. And the LLM and the prompt, uh, are, um, let's say, they're like the brain. So to them, we can like ask a question, and they give us an answer. Um, then we have like an orchestrated sequence of activities, which is the chain, and from all of that, we get the agent, which is, uh, like the whole system, which is able to wrap around the prompt and perform the activities.
So let's first of all talk about the LLM and the prompt. Um, here we're seeing the prompt. So a prompt, uh, probably most of you know, is like a set of, um, let's say, instructions that, um, you give the LLM, and according to those, it gives you the response.
Then let's talk about the chains. So here you can see we're wrapping, um, the LLM and the prompt with a library called LangChain, and then we are able to get like a sequence which we're able to invoke. And then, uh, we can see the agent. And then at the agent itself, oops, sorry, at the agent code, um, as you can see here, this is like a code of an API, a simple API, but it operates an agent which at the end is inside, um, uh, composed from like sequences. It gets a message and it gives you the result.
So if we're looking at the evolution of prompts, that's like prompts and LLMs that were advanced to chains and agents. So at the end, prompts and LLMs were like, they were not context-aware. They weren't able to use any tools. They didn't have any flows. Um, and you can see here that, like, for example, the control flow has developed, as has developed that and evolved. That the agent has like an adaptive flow, and the chain has a memory which is short-lived. The agent is able to have like a memory which is much longer-lived.
So the agents are able to use a lot of tools, while the LLM itself weren't able to, and the chains had like limited ability. So all of this has evolved, um, to where we are today.
So we've talked about the agent loop a little bit. So the agent loop is, um, is receiving a question or a task, observing, let's say, go to the LLM, think about what it has to do, um, then according to the LLM response, it's, like, decides, let's call it, whether it has to call a tool or an API. Um, then from all of those, it's doing a reflection and understanding whether it has to do that loop again. So after we spoke about that, so let's look at this specific system. Let's understand, uh, what are the components that are needed.
All right. So first, if we want to build an agent, uh, we need, we need an LLM. Uh, here is a, a small code sample of an LLM. Um, here I'm using, uh, a library called LangChain in Node.js, and I'm using specifically the OpenAI model that they have.
So this is like our planning part, and, uh, that is like the reasoning engine that is doing like the reasoning and, um, like planning and everything. So the LLM is using prompt templates. Then it's like doing like reasoning and gets like, um, you know, conclusions and plans things. It generates the next actions, and it can analyze the outcome. It's like, did that succeed? Um, that component is the component that is revising the plan or doing the retries.
Okay. Um, then we're talking about the tools. Um, so the tool is there to complete information that the LLM doesn't have. Um, for example, I'll give you an example that I had from my real life. There is an athlete that I really like. She's, like, doing judo. Her name is Inbal Lanil, and she won an Olympic medal. So I asked the LLM at the beginning, did she win an Olympic medal? So, um, the knowledge that the LLM has was updated, I think, until 2023, to my opinion. So at that year, she didn't win the medal, but when I enabled the possibility to search on the internet, and searching on the internet is done via tools. So when I enabled that possibility, the LLM gave me the correct answer. So it, um, like, operates one tool, and then, um, it had like, it did like an internet search, and then it was, yeah, she won an Olympic medal, all is good. So those tools are enabling us to search the internet, to call APIs, to call DBs, to search in the file system. So, uh, the LLM has like this set of APIs, and when needed, um, it calls them.
And now, if we're talking about the tools layer, um, here, for example, you can see, um, that we have a snippet of working with a search tool. Actually, this is like an internet search tool, and that tool is doing like an internet search and, um, returning results.
Now, let's talk about, uh, memory. And here I will, uh, show you like, uh, um, like short-term memory with sequences. So now we will go to VS Code and look at, like, a tiny code snippet that I had preferred in advance. I, I had prepared in advance.
So here we can see, um, code of, um, I'll make the screen a little bit bigger, but here we can see code of, um, a sequence, uh, which is written with LangChain on JavaScript. And, uh, let's look at that, um, quickly at what we have here. We have all the imports. Then we're creating our model, chat, uh, chat GPT. I took that model and I use GPT-4 Mini. And in addition to that, I am creating, um, like a history context. So this is something that I'm creating synthetically, but this is actually demoing, um, um, the context of the conversation. Here you can see the prompt as well. Um, um, so you can see, um, like, uh, which, like, instruction I gave to that agent. And, um, there, here is like a map which at the end is, um, holding the history itself. Um, you can see that, um, like by session, we will, um, like, um, save the data.
Let's move forward. Um, here, let's look at the function of the demo after, like, all of this, like setup that we did here, creating the chain. And then I will start, like, ask my, um, um, I will start to ask questions. Um, so see, I'm saying my name, I really love judo, I see a lot of like competitions. And then, um, I would ask it, I would ask the model to plan a weekend, like, um, plan a program for the weekend for me. And let's see what it will give me.
So let's run that. Um, hey, um, so I'm saying, yeah, I left judo. And then, okay, now I've asked for the plan. So let's see what we have. On Saturday, start with a fun judo class or practice session. Maybe invite a friend to join. Lunch, have a picnic with some healthy snacks. Afternoon, spend some time practicing the violin. Uh, yeah, I also gave it the information that I'm, I'm practicing violin. That's also something about myself. And I have family movie night with a family friend. And, yeah, so it includes like, um, here it's also like includes like some judo classes on Sunday. I should tell him that I'm from Israel, and Israel is working on, is working on Sunday, and our weekend is like, uh, Friday and Saturday. But, yeah, he has like on the afternoon also a judo playdate with a friend and a violin practice according to the preference I gave it.
Um, so that is like a short-term memory. If you've seen it, it was everything like was in memory itself. But, um, if we want to have something which is like long-term memory, um, so when we're like saving something in the memory, it was, it's only like in the memory of the process, like when the process is, um, like, uh, is destroyed. So all of this memory is destroyed. If we want to have like a persistent memory, usually what is done is using like a vector DB.
Here is like an example of a memory vector store, but, um, you're able to use like, um, a DB called Chroma. And when you're doing that, um, process, you need to embed, um, the documents that, uh, you give to your model. What is embedding? Embedding is actually translating words to vectors. So at the end, you're storing vectors in a vector DB. That would be like your long-term memory and fetching data on every question.
As I said, for for things like that, I use Chroma.
All right. So now let's look at the code for, um, a one-process agent which will be simple and effective. We would have a front end, and we would have a back end that would have an API that is wrapping an agent. And, yeah, let's talk about the code. We have the model itself, the LLM.
Another thing that I would like to say, we're going to use a library that is called LangChain and LangGraph in order to build the agent. And on this library, you're defining some kind of a graph, and that graph represents the order of activities that the agent is doing.
So here, um, we're seeing like the search tool. We said that there is a set of tools for the agent to look in the internet, look at the file system. And here is the graph itself. We're creating, um, like the node. I'm saying the start would be a user input, and of the graph itself, that would be like the start node. And then we are having another node that would operate an internet search according to the flow, if needed for that question.
Then we're adding another node, and here on that node, we're adding, um, like a prompt to give instructions, what we want. And then here, um, the inputs from the internet would arrive, and, um, we want to have a summary of what we got. And that would be like the full graph, like structure with four nodes that are user input, web search, summarize, and response.
So now we will go to the code again, and let's, um, look at the code real quick. It's already running here. So that is the code that I showed you, sorry, um, before. So let's start from the beginning. Um, we are having like the graph itself. We have an Express endpoint. Um, um, we will explore a route, and you see that here I define the AI LLM, and here I define the tools, and I define the graph, and I'm adding the nodes as, as discussed here. I'm adding like, um, the user input node, the search node, the summary node, and the response node, and I'm adding the edges between all the nodes and compiling the graph. And here is the input itself.
So this thing is already running, and excuse me, it was tested before, but let's run that again. So if I will refresh this, I want to ask it, what is the difference between WebSockets and gRPC in communicating? So let's send the request, and, uh, it's going to take some time because it's doing like a web search. Here it's returned, and let's see. So the summary of the search results for WebSockets and gRPC. So, yeah, it gives us like, um, um, let's see. WebSocket is optimal for real-time applications like chat apps, while gRPC is favored for inter-service communication or microservices and IoT. WebSockets provide bidirectional communication. gRPC uses HTTP/2, RPC protocol. So it gives us, if I want to like write a system and, like, think what should I choose for communication, WebSockets or gRPC. It gives me, like, um, like knowledge and summarizes everything.
So, uh, yeah, that's, um, that's our agent, and that's how to write it. And I can ask it more questions.
All right. So, let's recap this demo. So we have one process agent, was simple, it was effective. We took that loop, we've defined it in a graph, as you saw with LangChain and LangGraph, and we're doing like the plan. We're executing the tools. We're reflecting the results. So we're taking decisions whether we need to continue or not. And, okay, so, um, we have like a myth. First of all, let's think about scale. So this is like the, like the main goal now.
So, um, there is a myth that, like, the agents will block the event loop, but in reality, they don't block the event loop, um, because they have like HTTP with the tools and DB operations, and all of those usually are designed as asynchronous operations that are not blocking, um, the event loop. What can block the event loop is, um, like CPU-bound work, for example, like parsing JSONs, and actually, like the LLM reasoning itself. But, yeah, for that, um, we want to scale. So we've seen like the, the agent itself is not blocking the event loop. So for, let's say, low and medium systems, the one process that we showed would work. But we do want to like progress the system to workers and queues, to like an architecture design pattern like that. So that would enable us, for example, to handle back pressure better and to do recovery for requests better. For example, if all the reasoning would be in one of the workers or as a consumer to a queue, if something is failing, we would be able to, um, like reprocess it, and we would be able to handle retries much better.
So if we're like thinking about the architecture for all of that, I think that an architecture that is an approach which is good to take is to have the API itself on one layer. As you can see here, you also can have like a load balancer between the client and the API, and then you can scale that layer. Then the tasks are inserted into a queue. You can take your RabbitMQ or Kafka. I think that for this, like RabbitMQ is better. And then, like the agents, would, um, like take the response. Here would be like the graph itself. It would be located here. Then would do the reasoning and return the response to, um, the API.
This, like, architecture, uh, needs to, like, get more detailed because there are some, like, scalability tweaks here. For example, you don't want all the APIs, all the, like, consumers that get, like, agent response to get all response, but you want to direct it into a specific consumer. That, for example, can be solved by using, if you're using Kafka, Kafka partitioning, for example. But I think this architecture gives us, like, advantages of, like, like better recovery from errors and better retries and possibility to scale each part of the system, in order to at the end, like, handle more load.
Here you can see, like, an example with RabbitMQ. Like, a quick example. This is the producer. This is like what would send, um, the messages to the queue. And here, we can send like the question that we received from the API. And here you can see the consumer itself, like, that one would fetch the message. And I think that, like, the graph code, the agent itself can be, um, like, um, like implemented there. And, yeah, I guess, um, that's it from my side.