Python Advanced AI Voice Assistant - Full Tutorial with Frontend & Backend

In this video, I'll show you how to build an AI Voice Assistant with Python. This assistant will have agent capabilities, meaning it can interact with a database, call Python functions, and really do anything you want.

After we build the AI Voice Assistant, which won't take long, I'll show you how to integrate it into your own custom front end so you can see how to interact with this in the real world. This is super cool! I guarantee you're going to learn a ton, and you'll see how simple it is to actually make these agents, even if you're not a complete AI expert.

With that in mind, let me give you a quick demo of what we're going to build, and then we'll dive into the tutorial, and I'll walk you through it step by step.

What I did is create a really simple landing page for something like a car call center, where you might need to call in to schedule a service appointment or get an oil change, or whatever it might be. Typically, for these types of places, you talk to someone on the phone, and you have to go through all the AI messaging, providing the VIN of your car, the date of the appointment, and all that kind of stuff.

So, I wanted to show you how you can automate that using an AI voice assistant. Let's have a look.

There's a button here that says "Talk to an agent." If I click this, I can enter my name and connect to this AI Voice Assistant system. Let me show you how it works.

Welcome to our Auto Service Center! For us to assist you further, could you please provide the VIN of your vehicle? If you don't have a profile with us yet, just let me know, and we can create one for you.

Okay, so here it's asking me for the VIN. Typically, when you go through these systems, it needs to look up some kind of profile, and you need to have the information on the car. So, I'll tell it to make a new profile as if I don't have one.

"Hey, can I make a new profile, please? I don't have a VIN on file."

"Of course! To create a new profile, I'll need some information about your vehicle. Could you please provide the make, model, and year of your car?"

"Sure! This is a Jeep Wrangler 2025, and the VIN is V123."

"Your new profile has been created for your 2025 Jeep Wrangler. How can I assist you further today?"

"Can I schedule a service appointment for this, please?"

"I'll transfer you to our scheduling department to book your service appointment for your Jeep Wrangler. Please hold for a moment while I connect you."

Perfect! That's exactly what I wanted it to do.

Now, there's obviously a lot more functionality that this has, but the idea is that it can interact with the database. When it asked for the car information, it was actually going to save this in the database. So, next time I come back, it would already have that profile saved based on my VIN. It would be able to look up the car information, and we could forward that over to the scheduling department or whatever department we need to go to. They would already have all the data they need to assist us with booking an appointment or whatever it is we wanted.

I didn't build out a ton of features here; I wanted to keep it minimalistic so you can extend it to what you actually need. But I promise you, in this video, you're going to learn a lot about how this works, and you'll understand how to change this to your own custom situation or whatever it is you want to build out. I know a lot of you probably aren't going to build this exact example.

With that in mind, let's get into the tutorial. Let me show you how to build something like this. The first half will be more about building the agent, and then the rest will be about integrating this into the front end.

Let's begin by learning a little bit about the framework and technology we're going to use to build this project. For this video, I'm using LiveKit. They are the sponsor of this video, and I've worked with them in the past. You guys really enjoyed that other video we did, which is why we teamed up again.

They are open source and completely free; you do not need to pay for them, at least if you're just using it in a tutorial setting. They are actually used by a ton of massive companies, including OpenAI. So, if you've ever used OpenAI's voice mode before, that's actually powered by LiveKit.

The reason they use LiveKit is that it's the best in the business for ultra-low latency transportation of voice, video, and audio data. With LiveKit, you can build real-time AI systems like we're doing here with the ultra-low latency Voice Assistant. You can also do video conferencing, audio conferencing apps, and all kinds of interesting stuff.

If you want to check them out, do it from the link in the description. In this video, I'm going to focus on the Voice Assistant agent, but of course, this has a lot of other functionality.

To get started with LiveKit, we're going to create a new account. Again, it's free; you do not need to pay for this. The nice thing about LiveKit is that if you don't want to use their self-hosted options, you can actually host the LiveKit server completely on your own.

If you build this out and decide to deploy it, you're not tied into their environment or pricing model; you can just deploy it by yourself. But obviously, it's easier if you just use the LiveKit self-hosted environment.

What I'm going to do is go to cloud.livekit.io. It's going to bring me into my account because I already have one. If you don't have an account, please go ahead and create one.

Then, we're going to create a new LiveKit project. From there, it'll be straightforward to get started. I'm going to go to the bottom here and press "Create New Project." You can see I already have some spun up here. I'll give this a name and just call it "Car Service Center" or something like that.

Okay, create project, and then I'll tell you where to go from there.

I'm here in the project, and honestly, there's nothing we need to do to set this up other than get access to a few of the API keys we'll use to connect to the LiveKit server.

Before we go through that, I will mention that if you want to mess around with this on your own, they have a bunch of great quick start guides and agent guides. That's what I followed to build out the code you'll see me write. If you want, you can read the documentation directly, but obviously, walking through the video, I'll explain some more concepts that aren't in there.

So, that's pretty much it. What we're going to do next is open up a code editor. In my case, I'm using VS Code, and I'm going to start by making an `.env` file where I'm going to paste in some of the keys I need to connect to my LiveKit project.

Let's get that out of the way right now, and then I'll talk about the architecture and what to expect when we build this out.

To get the keys for our project, we're just going to go to settings and then to keys. For keys, we'll press "Create New Key." I'm just going to give this a name, so I'll say this is my Windows machine. You can call it anything you want.

This will give you an API key, a secret, and a WebSocket URL. For now, we'll just copy all of them.

Let me split my terminal here for a second. Let me get this guy open over here first.

The first environment variable we're going to have is the LiveKit URL, so we're just going to copy this WebSocket URL here for the LiveKit URL.

Next, we're going to have our LiveKit API key, so we're going to say `LIVEKIT_API_KEY = ` and then copy the API key. Obviously, you don't want to leak these; I'll leak them to you now and just delete them after.

Then we're going to go `LIVEKIT_API_SECRET = ` and then this.

Make sure you copy those correctly into these variables because we'll need those to connect to LiveKit.

Now that we have those, the only other thing we need to continue with this project is an OpenAI API key. You can use any AI provider you want with LiveKit, or at least there are a lot of different options. For example, you could use LLaMA and run this locally on your own computer, but it's just a lot easier to use OpenAI.

So, I'm going to say `OPENAI_API_KEY = ` and I'm going to go and get my OpenAI API key.

To do that, we can go to platform.openai.com/api-keys. I will leave this link in the description. I believe you will need a credit card on file to do this, but it should be free or cost you just a few cents.

Again, if you don't want to use this, you can look in LiveKit, and there are a lot of other options for the AI models. OpenAI is just going to be the easiest and fastest in this case for us.

So, I'm going to create a new secret key and just call this "LiveKit Car Service" or something.

Okay, let's create that key, and then we will copy it into our project.

I'm going to copy it here in that environment variable file. Perfect! Let me get out of that, and now we will go back to VS Code and start writing some code.

But first, I want to explain the architecture and design of how LiveKit works.

I've got the LiveKit documentation opened up, and I just want to show you this diagram here and explain what's going to happen in terms of the flow of data. You don't have to understand it fully, but I think it's a good idea before we go much further.

What we're going to focus on building right now is this agent. You see where it says "Your Backend" and then "Agent"? This is what we're going to build.

This agent will connect to the LiveKit cloud, and that will be connected to our various clients. Essentially, what will happen is we'll have some client, whether it's a website or our phone, that will connect to the LiveKit cloud.

When they do that, they're going to open up a new room. Immediately when this room is created, our agent will join that room and start communicating with the client.

That communication will happen through WebRTC, which will be facilitated through the LiveKit cloud. This means we don't need to deal with all these transfer protocols and sending the data back and forth; LiveKit does that for us.

We essentially just define what we want the behavior of our agent to be, and then everything is taken care of by LiveKit.

Now, obviously, our agent will also connect to OpenAI. In this case, we're going to use something relatively new, which is the real-time API, allowing us to have extremely low latency in the responses.

As you saw, it was very fast—actually insane how fast it is because we're using this new real-time API feature.

There are other things you can build with LiveKit, but in our case, we're going to do the OpenAI real-time API.

This documentation I will link in the description, and you can read through it if you want to understand it more, but that's the basic architecture.

We design our agent, some client joins the LiveKit session, and then our agent will join the room with that client and start communicating with them. All of that happens through WebRTC, which is facilitated by LiveKit cloud, which is very fast.

One thing to note is that you can have multiple agents running at the same time and joining multiple rooms. LiveKit cloud will handle that concurrency for you, and you can have multiple rooms at the same time, which means your agent can be talking to multiple people and have different sessions or conversation chains going on.

Hopefully, that makes a little bit of sense.

With that in mind, let's start building this out.

To get started, we're going to create a `requirements.txt` file, and I'm just going to list the various requirements we need to work with LiveKit.

We're going to start listing them out, and all of this code will be available from the link in the description. There will be a GitHub link in case you just want to copy it in.

First, we're going to have `livekit-agents`, then we're going to have `livekit-plugins`, then `openai`, then `livekit-plugins`, then `solero`. This will be used for voice activity detection.

We're going to have `python-dotenv`, which will be used for loading our environment variables.

We're going to have the LiveKit API, which we'll use later in the video, and then we're going to have some dependencies related to Flask.

We're going to have `flask`, `flask-cors`, and `uvicorn` for running our Flask server.

The last five dependencies you see here will be used later in the video for handling authorization and issuing tokens to our front end to connect securely to the LiveKit cloud.

Again, you'll see how that works later, but for now, we're just going to focus on these. I just figured we'd put them all in one file so we can install them all at once.

Now that we have that, we're going to create a virtual environment and then install those dependencies.

You don't need to make a virtual environment, but I would recommend that you do just to separate out the dependencies.

To do that, I'm going to type `python -m venv AI`. If you're on Mac or Linux, you can change this command to `python3`.

When we run this, it should create the virtual environment for us, and then we can activate it.

This will separate the dependencies for us.

To activate the virtual environment, I'm going to type `.\AI\Scripts\activate`.

If you're on Mac or Linux, your command will be like this: `source ./AI/bin/activate`.

Now that you have the virtual environment activated, what we want to do is install the dependencies.

We're going to type `pip install -r requirements.txt`.

This should install all of the requirements we listed in that file in our virtual environment, so we can use them in the project, and then we can start writing some code.

All of that is finished, and now we're going to start building out the agent.

I'm going to make a few different files. I'm going to make a file called `agent.py`; this will be the entry point script.

I'm going to make a file called `api.py`; this will specify some of the tools that our agent will be able to use.

I'm going to make a file called `database_driver.py`, which will handle managing our database and connecting to that for things like our vehicle information.

Then I'm going to make one more called `prompts.py`, which will have some of the prompts that we'll use for our AI agent.

I'm not going to write all of this completely from scratch, but we will with the agent file at least.

For our agent file, this is going to handle creating the AI agent, the Voice Assistant, which will connect to LiveKit.

To do that, we're going to need a few different imports.

We're going to start here and say `from __future__ import annotations`.

Then we're going to say `from livekit.agents import Autosubscribe, JobContext, WorkerOptions, CLI, LLM`.

We're then going to say `from livekit.agents.multimodal import MultimodalAgent`.

Next, we're going to say `from livekit.plugins import OpenAI`.

Then we're going to say `from dotenv import load_dotenv`.

We're then going to say `import os`.

That's all we need for now.

Next, what we're going to do is load our environment variable file.

We're going to say `load_dotenv()`.

This will simply load the `.env` file in this current directory, and then we'll be able to access those various keys we need for LiveKit.

Perfect!

Now that we've done that, what we're going to do is start defining some functions that we'll need to use.

We're going to define an async function, and we're just going to call this the entry point.

We're going to take in some context, and this is going to be some job context from LiveKit.

The first thing we're going to do inside this file is wait to connect to LiveKit.

To do that, we're going to say `await ctx.doc.connect()`.

We're going to say `autosubscribe = Autosubscribe.subscribe_all()`.

If you hover over this, you can see exactly what it's doing, but this is going to connect to a LiveKit room, and we're going to subscribe to all of the various tracks we have.

That will be the video track, the audio track, and the text track in that room.

Again, LiveKit can handle multimodal, so not just voice; it can handle video and text as well.

You can see if I go over here, it says whether to automatically subscribe to tracks. The default is to subscribe to all, which is what we're doing.

The next thing we're going to do is say `await ctx.doc.wait_for_participant()`.

This means we're going to wait to get the identity of the current participant and just make sure they're in the room before we start talking to them.

Next, we're going to define the model we want to use.

We're going to say `model = OpenAI.realtime()`.

There are a lot of different models you can use here. In our case, we're just using the OpenAI one right now for the real-time model.

What we need to pass to it is some instructions. This is essentially what we want it to act as. We can give it some kind of system prompt or something so it knows how it should be communicating with the user.

We can then define the voice. There are a bunch of different options here. I'm going to go with "Shimmer," but you can choose whatever you want if you want a different style of voice.

Then we can define the temperature of the model. I'm going to go with something like 0.8.

We can define the modalities, so essentially what type of medium we want it to work with.

In this case, I'm going to provide audio and text so that it will give us voice and generate the text associated with that voice.

Now that we've done that, we can actually create an assistant that's attached to this model.

If you've ever worked with AI agents before, you know that essentially what you do is give an AI model access to various tools that it can call and utilize during its invocation.

So, I'm going to set up a class that will contain all of the different tools that this AI agent will be able to use.

These tools will be related to interacting with our database, looking up a car based on its VIN, creating a new profile, scheduling a meeting, calling an API—you can do anything you want with the tools here.

I'll show you how to get that started.

We're going to go into our `api.py`, and this is where we're going to define our tools.

I'm just going to stub out what the tool class will look like for now so we can actually test the AI agent, but then later we'll write them all out.

I'm going to say `from livekit.agent import LLM`.

Then I'm simply going to say `class Assistant(LLM)`.

Then I'm going to inherit from `LLM`.

All I'm going to do inside of here for now is define a simple initialization.

I'm going to take in `self`, and then I'm going to say `super().__init__()`.

Later on, we'll write the rest of this class body and define some functions that will act as tools that the AI agent can use, but for now, we just need this to invoke the AI agent.

Now that we have that, we're just going to import this assistant function.

I'm going to say `from api import Assistant`.

Now, beneath our model, we're just going to create a new instance of this.

We're going to say `assistant = Assistant()`.

Now, we can start the assistant and connect it to the room.

We can say `assistant.start(ctx.room)`.

What we did up here is we waited to connect to this room and for the participant to join.

We then created a new AI assistant, and we said that the AI assistant will now join the room.

Now the AI assistant joins the room, and it can start interacting with the participant.

Now, what we need to do is tell the AI assistant to do something, maybe to talk to the user or greet them.

To do that, we can say `session = model.sessions[0]`.

We'll just grab the first session we have, and then we're going to make a new conversation item.

I'll talk about how this works in a second, but let's type it out.

We're going to say `session.conversation.item.create()`, and then we're going to make a new `LLMChatMessage`.

This message is going to have a role equal to "assistant," and we are going to have the content equal to... and then we're going to provide the content here in one second.

All we're doing is adding a new chat message into the context or the current session.

After we add that chat message, we can tell the assistant to respond to that message.

So now I'm going to say `session.response.create()`.

What it will do is read this conversation item and then generate a response.

That response will be voice and text as well, and it will actually speak out to us and say what it said.

So, now that we have that, we actually have a functioning AI assistant.

I'm going to show you how we test it in just one second, but we need to call this entry point function.

To do that, we're just going to use the classic `if __name__ == "__main__":`.

Then we're going to say `asyncio.run(app())`, and we are going to use these worker options here and pass the entry point function as our entry point.

Don't worry too much about this; what this is going to do is run this entry point function asynchronously for us, and then we will automatically connect to our LiveKit cloud.

The reason why this will work is that we've provided these values in our environment variable file, and these are the values that are being looked for by LiveKit to make the connection.

Make sure that you name these environment variables exactly what I have here. You need all four of them for this code to work, and then they'll automatically be searched for by LiveKit.

We'll connect to the LiveKit cloud, and then we can handle any of the clients connecting here and add our AI Voice Assistant into the room.

You can see this is very straightforward.

What will happen now, once we run this code, is we'll wait for someone to connect. Once someone connects, we will run this entry point function, and then we will generate a response.

We'll just say one thing to the user.

The last thing we need to do is write these prompts. We need the instructions and the content.

Let's go inside our `prompts.py` here and copy in these prompts.

I don't want to spend too much time writing them, so I'm just going to copy them in here. You can change them to be anything you want, and you can copy them from that link in the description.

The first one is: "You're the manager of a call center. You're speaking to a customer..." and so on.

Then the welcome message: "Begin by welcoming the user and ask them to provide the VIN of their vehicle."

Again, adjust this to whatever your situation is.

Now we need to import these and use them.

I'm going to say `from prompts import welcome_message, instructions`.

Now, I'm just going to plug in the variables.

This will be `instructions`, and this here will be `welcome_message`.

Perfect! Now that we have that, we're good to go ahead and run this.

I'll show you that LiveKit actually has a playground we can use to test this agent before we start integrating it with our front end.

I'm going to bring up my terminal here and simply type `python agent.py`.

Then `dev` is how you run this in development mode.

I'm going to hit enter, and we're going to see if we get any errors.

It looks like we may have spelled something incorrectly.

It said, "Add LiveKit API Secret in your environment."

It's possible that I just spelled "secret" wrong here.

Let me just paste this here, and yes, I did.

It should be `LIVEKIT_API_SECRET`.

You can see it's telling me that error other than whatever I had there.

So let's change that to `LIVEKIT_API_SECRET`. Apologies about that, guys, and let's rerun this.

We can see now that this is running, and the next step is obviously to test this.

To test this without writing our own custom front end, LiveKit actually provides something called the agent's playground.

What I can do is just search in my web browser for "LiveKit agent playground."

I'll leave a link to this in the description.

If we click into this, it will allow us to connect to our project and start interacting with the agent.

You may need to sign in with LiveKit, but I believe my credentials are already here, so I'm able to get access to it.

Then I can just click on the "Car Service Center," which is my current project, and connect.

We'll be able to mess around and hear our agent ideally.

I'm going to be quiet now so we can hear this.

"Thank you for choosing our Auto Service Center. To assist you better, could you please provide the VIN of your vehicle? If you'd like to create a new profile, just say 'create profile.'"

"I don't have a VIN."

"No problem at all! Let's create a new profile for you. Could you please provide your name, phone number, and email address to get started?"

"Tim, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, tim1@example.com."

"Thank you, Tim! Your new profile is now created. How can I assist you with your vehicle?"

Okay, so that's that.

I'm just going to mute my mic and disconnect here.

You can see that this is working.

Now, obviously, it's just kind of making up replies after this point because we don't have any logic for what we wanted to do, but you see that it worked!

We connected, and it was able to respond to our messages.

It did keep going and give us some conversation, but just not in the format we would have liked.

So that's how we test it.

I'm going to leave this page open.

I can always reconnect by just pressing "Connect" here, and then you can see the agent will connect.

If you have issues with the agent connecting, just make sure your keys are all correct.

I know a lot of people were having issues with this in the previous video, and I believe it's because they weren't using the correct environment variables.

At this point, I'll assume that you were able to connect.

Really, the only thing that could go wrong is if your code has a mistake that doesn't match what I've written here.

Again, you can copy this from the link in the description, or you just don't have the correct environment variables.

So just make sure that you double-check those because I know personally I had some issues with them because I just had the wrong values in the wrong slots.

Silly mistake, but it happens.

Now that we have this, I want to start providing some tools that my agent can use.

I want it to be able to save VINs, look up VINs, or create a new profile.

To do that, I'm going to write this database driver file.

I'm just going to copy in the code here.

Again, you can find this from the GitHub link in the description.

The reason being, I don't want to waste your time writing a bunch of database-related code, but I will walk through it so you can understand what it's doing and how this works.

For my database driver, I'm just copying this in.

It's 60 lines of code, and this connects to a SQLite3 local database.

You can use any database you want; you can use MongoDB, a JSON file, an in-memory database—it doesn't matter.

I'm just doing a quick example here so you can see how we interact with data with our model.

You can even use something like retrieval-augmented generation if you wanted to.

I'm just not going that far in this video.

You can see that I bring in my SQLite3 package.

I have all of this other stuff: data class, context lib, all this stuff.

I create a data class with my VIN, make, model, and year.

This just represents the information I want to store in my database.

I create a new database driver class, take in the database path (where I want to save this), and then I just initialize the database.

Initializing the database is right here, where it creates a new table for me in SQLite with my VIN, make, model, and year if it doesn't already exist.

That's it!

Then I have this `get_connection`, which is a context manager that yields the current connection to our SQLite database so we don't constantly keep reconnecting to this if we don't have to.

Going down here, I then create two functions: `create_car` and `get_car_by_vin`.

This is really all I need.

I need the ability to make a new car and the ability to get one based on its VIN.

If you want to do something like delete a car or look up multiple cars or have more operations, of course, you could write them in here.

I just kept it nice and simple.

To create a car, we take in the VIN, make, model, and year, and then we return a new car.

We get the connection, get the cursor, insert this information into the database, commit that, and then return the car so we can use it.

Same thing here with `get_car_by_vin`: we get the connection, search for a particular VIN (which should be a unique field), fetch that value, and if it doesn't exist, we return None.

Otherwise, we return the information about the car.

Again, you can copy this from the link in the description if you want to use this exact code.

So there we go!

We have our database driver, and what I need to do is connect the database driver to my API file.

I have to make functions that my LLM is able to call inside of this class, and then in those functions, I can do literally anything I want.

I'm just going to start with some imports here.

I'm going to say `import enum`.

I'm going to import my typing, or I'm going to say `from typing import Annotated`.

Then I'm going to say `import logging` just so I can print some stuff out.

Then I'm going to say `from database_driver import DatabaseDriver`.

First thing first, I'm going to set up my logging.

I'm going to say `logger = logging.getLogger("user-data")`.

Then I'm going to say `logger.setLevel(logging.INFO)` so I can display some information in my terminal when we start using these functions.

You can see that they are indeed working.

I'm then going to make a new database instance.

I'm going to say `db = DatabaseDriver()`.

Then I'm going to make a simple car class that defines an enum for the different fields in my car class.

I'm going to say `class CarDetails(enum.Enum)`.

This is going to be an enum.

You'll see why we need this in one second.

I'm going to say `VIN = "vin"`, `MAKE = "make"`, `MODEL = "model"`, and `YEAR = "year"`.

You might be wondering why we need this.

Again, just wait one second.

It's so that we can annotate what should be passed to these functions by using this Python enum.

It's important that when you write these functions, you do all of the typing for them in Python so that the model knows what values to pass for the various parameters when it starts using it as a tool.

Now, what we're going to do inside this assistant function is store some context, and that context is going to be the car that the user is talking about.

The idea is that whenever you call something like a car service center, you first need to know what car you're talking about, right?

Unless you're buying something, it's either a car that already exists that you've been there before, or it's a new profile.

So we want to store what car the user is currently discussing so that the LLM has that context and information and can use it throughout the discussion.

I'm going to say `self._car_details = {}`.

The reason I start this with an underscore is that I make it a private member in Python.

It's not actually private because you can still access these values, but it's a convention you use if you don't want something to access or change this attribute.

Again, it's not truly private, but it's a convention in Python so that something outside of the class doesn't try to modify this attribute.

Now, what I'm going to do is simply say `car_details["vin"] = ""`.

Now, notice I'm using the enum, and the reason I'm using my enum is so that the model always knows what field it should be using or the name of the field when it's trying to access values in my class.

Again, you'll see this in a second, but just bear with me.

I'm going to say `car_details["make"] = ""`, `car_details["model"] = ""`, and `car_details["year"] = ""`.

These will all be filled in later.

This is some kind of general context I want to store here, and you can store anything you want.

Then beneath this, I'm going to make a few functions that will act as the tools that the LLM can use.

To do that, you need to decorate the function with `@llm.AICallable`, and then you need to provide a quick description of what this function does and when it should be called.

I'm just going to say "Look up a car by its VIN."

Then I'm going to define `lookup_car` as my function name.

It's going to take in `self`, and it's going to take in a parameter, and that parameter is going to be a VIN.

Now, for the VIN, I need to specify what the type of this will be.

I'm going to say this is `Annotated[str]`.

Then I'm going to say `description = "The VIN of the car to look up."`.

Okay, so let's spell "lookup" correctly.

Great!

Now, right away, I'm going to say `logger.info("Lookup car")`.

Then I'm going to say `vin = f"{vin}"`.

I'm just going to pass the VIN here so that we have some output and can see what the model is actually using to look up the car when it calls this function.

Just some basic debugging, essentially.

Then I'm going to use my database to look up the car.

I'm going to say `result = db.get_car_by_vin(vin)`.

I'm going to say if `result is None`, then we're just going to return "Car not found."

Otherwise, we're going to set our car details.

I'm going to say `self.car_details = {}`.

Now we're going to write our field, so I'm going to say `result["vin"]`, `result["make"]`, `result["model"]`, and `result["year"]`.

Obviously, we can store more information, but for now, that's totally fine.

So what we're doing is looking in the database for this car based on the VIN that was passed by the AI model.

If we don't find one, we say "not found."

Otherwise, we set the context, and then we're just going to convert this into a string and return it to the model.

So we're going to say `car_string = ""`.

We're going to say for `key, value in self.car_details.items()`, we're going to say `car_string += f"{key}: {value}\n"`.

Then we're going to return an f-string and say "The car details are: {car_string}".

The reason I'm doing this is I just want to give a string back to the model so I can tell it, "Here are the details that you found," and then just give it all this information rather than trying to return it in a Python dictionary, which could be problematic.

Now I'm just going to make a quick method that will do this for us because we will use this later on.

So up here, I'm just going to say `def get_car_string(self):`.

We will take in `self`, and then all we're going to do is copy this right here.

Okay, and we're going to do this, and then we're going to say `return car_string`.

Then I'm going to go here, and I'm going to use `self.get_car_string()`.

Now this is inside of here, and it's a little bit more reusable.

Great!

So we have this first method now that allows us to look up a car.

What I'm going to do is copy this, at least the function definition, and I'm going to write the same thing now for creating a car.

So rather than looking up a car, we're just going to go with `create_car`.

Now for the description, we're going to say "Create a new car."

That's really all we need.

For creating a car, we're going to need a VIN, a make, a model, and a year.

So let's make this a little bit easier to read.

Let's go back here and shift-tab this back and put our parentheses in the correct place.

Now for the make, this will be a string.

And rather than the VIN of the car, we're going to say the make of the car.

I can remove this two lookup thing.

Okay, and just keep changing these and adjusting them to be correct.

Now for the model, we'll say the model of the car, and then this will be the year of the car.

For the year, we're going to change this to be an int, not a string.

Okay, perfect!

Now we have our parameters: `self, vin, make, model, year`.

That's what we need.

Now we're going to log the information that was passed.

We're going to say `logger.info("Create car")`.

Then we're just going to pass the details so we can see what they are.

We're going to say `vin: {vin}, make: {make}, model: {model}, year: {year}`.

Now we just need to make a car.

So we're going to say `result = db.create_car(vin, make, model, year)`.

Then we're going to say if `result is None`, so if there's some kind of issue here, we're just going to return "Failed to create car."

Otherwise, we're going to set these details.

I'm just going to copy this here and paste it here because it's going to be the exact same thing.

So we're just going to grab the result and set it, and then we can just return a string that says "Car created."

Perfect!

Now we're just going to write two more methods.

One more method I'm going to have is just going to be an internal one, or I guess external, but it's not going to be one called by the LLM.

I'm going to say `def has_car(self):`.

All I'm going to do is say `return self.car_details["vin"] != ""`.

So if this field is not equal to an empty string, it means we currently have a car, which is something we want to know in order to change what the model is asking the user.

Okay, and then the last thing we're going to have is this `get_car_details` function.

So I'm just going to copy this again.

I'm going to say `@llm.AICallable`.

Rather than create a car, this is going to be `get_car_details`, which is a method that the LLM can use to get the details of the car it's currently talking about.

All we're going to do is say `return f"The car details are: {self.get_car_string()}"`.

For the description, I'm simply going to write "Get the details of the current car," meaning the car that we're currently discussing.

Then the last thing, we're just going to log a message so we can see if this is being called.

We're going to say `logger.info("Get car details")`.

Let me zoom out a little bit so we can read this a little bit easier, and we'll just go through what we wrote.

We imported all the stuff, created the logger, used the database driver, defined our car details enum, and then started writing things like the initialization where we're going to store some context about the current car we're looking at.

We have a method to get the current car string, so this just looks at these details and converts it to a string for us.

We then have this to look up the VIN; that's what we looked at.

We have another one just to get the current car details.

We have one to create a car, which takes in all of the different parameters we need, and then lastly, one just to tell us if we currently have car details.

So if we already have selected a car or we have a profile already pre-selected.

Great!

Now that we have that, this allows our agent to simply use this.

Because we provided the assistant function here to the multimodal agent, it can just go ahead and use any of this functionality.

So if we go and test this now, it should just be able to create a new entry in the database, for example, or look something up.

Let's rerun our code here, and let's go back to our agent playground and test this out.

So I'm going to go here, disconnect, and reconnect to see if it can look up this VIN for me.

Welcome to our Auto Service Center! Could you please provide the VIN of your vehicle or let me know if you'd like to create a profile for your car?

The VIN is V12345.

It seems that we don't have a profile for a vehicle with the VIN V12345. Would you like to create a profile for your car?

If so, please provide the make, model, and year of your vehicle.

Sure! My vehicle is a Jeep Wrangler 2021.

Your Jeep Wrangler 2021 has been successfully registered in our system. How can I assist you further with your vehicle today?

Can I schedule a service appointment?

I can help you with that! Please let me know your preferred date and time for the service, along with any specific concerns or services you'd like to address.

Okay, sweet!

So it seems like that's working.

I just want to look at the logs and see if it did create it with the correct VIN or not.

So I'm going to scroll through here and see if I can find this.

Okay, user data create car, and yes, perfect!

It used the correct VIN this time.

So what I'm going to do is disconnect and reconnect, and I'm going to see if it can look up this VIN for me.

I just want to test that feature.

So let's go back, disconnect, and rejoin to get the agent in the new state.

Welcome to our Auto Service Center! Could you please provide the VIN of your vehicle or let me know if you'd like to create a new profile?

Can you look up the VIN V12345?

It seems that we don't have a vehicle with that VIN in our records. Would you like to create a new profile for your car?

What is the VIN that you looked up?

The VIN I attempted to look up is 1HGCM82633A123456.

If that's not correct, please provide the correct VIN or let me know if you'd like to create a new profile.

Please look up exactly V12345.

Thank you for providing the VIN! I have found the details for your vehicle: make Jeep, model Wrangler, year 2021.

How can I assist you further with your Jeep Wrangler today?

Okay, sweet!

So that's what I wanted to test, guys.

Sorry for going back and forth there.

It looks like I just need to be specific, or because I'm not giving a valid VIN, it's attempting to shortcut it and look up a VIN that contains those characters.

Obviously, that's not what we want, so we could add some more specific instructions to the prompt, but for now, it's fine.

It's able to look it up, right?

And it found that information.

If we come back here, we should be able to see the log if we look at user data, look up car V12345, and then it gets that data.

Okay, sweet!

So that is it for the AI agent.

Now it's time to actually implement this on the front end.

Now we are moving on to the front end.

First things first, I just want to take all of this backend code and put it in a folder called "backend" just to clean things up a little bit.

So that's going to involve me stopping my agent from running right now.

Let's quit that and grab all of these files and throw them inside of "backend."

I'm going to throw my virtual environment inside of there as well.

Okay, sweet!

So that is inside "backend."

The next thing we need to do is create the front end folder.

I can delete this `.pyc` folder for right now.

Okay, let's get rid of that.

Now let's move to make our front end.

To make our front end, we're going to use React, specifically with Vite for the command.

I'm going to type `npm create vite@latest`.

This is going to be `front-end --template react`.

Now, in order for this to work, you do need npm installed on your system.

You don't need to be a pro in React here, but obviously, I'm not going to walk through all of the React code.

This is more just to show you how to get the voice assistant connected, the tokens, and all of that kind of stuff.

By the way, you can do this with pretty much any front end you want.

So I'm doing it here with React, but if we go back to the LiveKit documentation, you'll see that we have a lot of different options.

You can do it with Android, Swift, React, and there should be a lot of other options as well.

In fact, if you want to learn how to do this for a different front end, you can just go to this documentation and click into it.

I just followed along with the React documentation, and that's where you'll see a lot of this code comes from.

Obviously, I just made it a little bit more custom, so I'm going to walk you through that now.

Okay, sweet!

So let's go back here.

Now what we're going to do is `cd front-end`, and we're going to type `npm install`.

Then we can run the server, but we do need to install a few dependencies for LiveKit as well.

Okay, so let's wait for that to finish.

Now we're going to run this command here: `npm install @livekit/components react-livekit-components styles livekit-client`.

We are going to save that.

Again, this comes directly from the LiveKit documentation, so let's go ahead and run that to install those dependencies we need.

Let me close this file while we're at it.

Okay, so while that runs, I'm going to go into my front-end folder and start setting a few things up.

Specifically, I'm going to create a few components that we'll need, and I'm going to bring in some CSS styles that I've already written, which again you can find from that GitHub link in the description.

I just don't want to bore you by focusing too much on the front end when you guys care about the voice assistant integration.

In my `src` file here, I can go ahead and delete my assets.

I'm not going to need that.

I can delete the public folder as well; I'm not going to need that one.

Then we can continue from there.

Inside of my `app.jsx`, I'm just going to clear all of this out.

I'm going to remove the state, remove the import to the React logo and the Vite logo, and I'm going to make a new folder inside of `src` called `components`.

Now, inside of `components`, I'm going to have two.

We're going to keep this nice and simple.

One is going to be the LiveKit modal; this is the thing that's going to pop up when we want to connect to LiveKit.

I'm going to call this `LiveKitModal.jsx`.

Then I'm going to make the simple voice assistant, and this is going to be `SimpleVoiceAssistant.jsx`.

That's going to handle the logic for the voice assistant.

Then I'm going to do the same thing for CSS.

I'm going to say `SimpleVoiceAssistant.css`.

That's all we need for now.

Now what I'm going to do is copy in a bunch of CSS styles here.

Again, you can find this from the link in the description in the GitHub.

So just go there to the GitHub, look in the components folder, and you can see the `SimpleVoiceAssistant.css`.

Copy it into this file.

For the `app.css`, same thing; I have some custom CSS, so I'm going to copy it into this file.

Then same thing for the `index.css`; I have some as well, so I'm going to copy that into this file.

Okay, so now we have `index`, `app`, and `SimpleVoiceAssistant.css` completed.

Now what we can do is move on to writing some of the code.

What I'm going to start with is my `app.jsx`.

Inside `app.jsx`, I'm just going to write a very simple landing page.

You can make this really anything you want, and then we'll move on to actually implementing the LiveKit components.

Keep in mind that LiveKit has all kinds of documentation here.

If I go back, you can see that what we're using is some components from LiveKit, which will handle things like rendering the audio for us, rendering the video for us, multiple tracks, sharing participants.

If we actually go into the React components, they have a whole library of documentation here that explains how to do a bunch of stuff and shows all of the components they have, like bar visualizers, chat entry, focus layout.

So if you really want to make a more advanced front end, obviously check this out here, and you can use a lot of the pre-built stuff, which is kind of what we're doing here.

Okay, so let's go back, and what we're going to do is start by defining some state.

We're going to say `const [showSupport, setShowSupport] = useState(false)`.

This is what will show the support modal on screen.

We're then going to create a div here, so we're going to say `<div className="app">`.

Then inside of `app`, we're going to have a simple header.

This will be `<div className="header">`.

Inside of the header, we'll have a div.

We'll say this is `<div className="logo">`.

Then we're just going to call this "AutoZone."

That's the name of my website.

Then we're going to have a main section.

For the main section, we'll have a section.

This will be `<section className="hero">`, which will be like the top section on the page.

Then we can just put in some kind of placeholder text here.

So we'll just have an `<h1>` and say "Get the right parts."

Okay, and we'll say "Free next day delivery on eligible orders."

Okay, sweet!

Then we'll just have a simple search bar.

We'll say `<div className="search-bar">`.

Inside of here, we're just going to have an input field.

We're going to say `<input type="text" placeholder="Enter vehicle or part number" />`.

Okay, sweet!

Then we'll just have a simple button.

Then we're pretty much done here.

We're going to say `<button>Search</button>`.

Then outside of the section, but still in the main, I'm just going to make another button that will float in the bottom right.

This is going to say `<button className="support-btn">Talk to an agent!</button>`.

We're going to have an `onClick` handler here, which will allow us to talk to the agent.

What we're going to do here is just have a function, and we're going to say `const handleSupportClick = () => { setShowSupport(true); }`.

Then we're just going to use this, and we're going to say `handleSupportClick()`.

The idea here is that when you press this button, it's going to set this state to true, and then we're going to open up a modal that will get you to enter your details, like your name, and then you can connect to the LiveKit room.

What we're going to do now is design the LiveKit modal.

We're going to go into this component here and start writing that out.

We're going to say `import { useState, useCallback } from "react";`.

Then we are going to import some LiveKit components.

We're going to say `import { LiveKitRoom, RoomAudioRenderer } from "@livekit/components-react";`.

We need to spell this correctly.

Next, we're going to say `import "@livekit/components/styles.css";`.

I believe that's it, and then we'll have one other import, but we don't have the component written yet, so we don't need that now.

Okay, then we're going to say `const LiveKitModal = ({ setShowSupport }) => {`.

This will be a function, and then what we'll do here is start defining some of the information we need to render the live room.

First things first, we're going to say `const [isSubmittingName, setIsSubmittingName] = useState(false);`.

This will be false.

We then need to know what the user's name actually is, so we're going to say `const [name, setName] = useState("");`.

This will be an empty string.

After that, we're just going to do a little bit of CSS here to create a modal that allows the user to enter their name and then connect to the LiveKit room.

We're going to say `return (<div className="modal-overlay">`.

We're then going to make another div and say `<div className="modal-content">`.

We're going to have another div, and this is going to be `<div className="support-room">`.

Then we're going to say if `isSubmittingName`, we're going to have a form.

We're going to say `<form onSubmit={handleNameSubmit} className="name-form">`.

Inside of the form, we're going to have an `<h2>Enter your name to connect with support</h2>`.

We're then just going to have an input and submit buttons.

For the input, we're going to have `<input type="text" value={name} onChange={(e) => setName(e.target.value)} placeholder="Your name" required />`.

Then we're just going to define a button that says "Connect."

This will have the type equal to "submit," so when we press this, it will call this function here in the form.

We're then going to have another button.

We're going to say `<button type="button" className="cancel" onClick={() => setShowSupport(false)}>Cancel</button>`.

So this `setShowSupport` is the function that's in our `app.jsx`, so we'll no longer render the LiveKit modal, which we'll be rendering in just one second.

Actually, let's just add some text to the button so you can see it.

So it says "Cancel" now.

Okay, and then we will format again.

Alright, great!

Now, if we are not showing the form, that means we've already submitted the name.

Then what we're going to do is show the LiveKit room.

The LiveKit room is where we can render the audio and show all of the LiveKit-related components.

For that, we're going to say `<LiveKitRoom serverUrl={serverUrl} token={token} connect={true} />`.

We will fill in the `serverUrl` and `token` in a second.

This means that as soon as the room is rendered, we're by default going to connect to it.

Then we can specify if we want to display video or not.

In this case, I don't.

So, that's the basic structure of our front end.

Now, we can start integrating the AI Voice Assistant with the front end.

This will involve connecting the LiveKit components to the backend we built earlier.

We'll handle the user interactions and ensure that the voice assistant can respond to user queries effectively.

That's it for this part of the tutorial!

In the next part, we'll dive deeper into the integration process and finalize our AI Voice Assistant project.

I want video, so I'm going to set video to false. Then I want audio, so I'm going to set audio to true.

Next, I need to have an onDisconnected function here, which we'll call when we disconnect from the room. When we disconnect from the room, we're going to set showSupport to false, meaning we're no longer going to show the modal. We'll also set isSubmittingName to true. This means that when we come back into this component, we will be able to submit the user's name.

Now, inside of here, we're going to render the RoomAudioRenderer. This will actually display the audio or allow us to hear the audio. We will also need to render a few other components for controlling the audio, such as muting our microphone, showing the different input sources, showing the waveform of the agent's audio, and showing the text data. We'll write that component in a second, but this is kind of stubbing what we need for the LiveKit room.

Before we go any further, I want to test this. To do that, I'm going to export this component. So, I'll say export default, and then this is the LiveKit modal. Next, I'm going to go to app.jsx and render this modal at the bottom of the screen.

To do that, I'm going to go outside of this div and say if showSupport, then showSupport, and then we're going to show the LiveKit modal. We'll pass to this setShowSupport, which is equal to setShowSupport. The LiveKit modal was automatically imported for us from do/components.

To run this and see if it works, I'm going to type npm run dev. This should run on localhost, so let's open it up. You can see that we have our website, and if we press this button, it looks like there's an issue with the modal. It's probably just a problem with my CSS, or for some reason, it's not appearing correctly.

Let me see what that problem is, and I'll be right back.

So, silly mistake: we just need to set isSubmittingName to true because we want that to be the default state so that it shows us the form and not the LiveKit room. In this case, it's showing the LiveKit room, which is not going to work right now because we haven't connected it.

Now, if I press this, you can see that it shows me this. There's some CSS-related issue, but the cancel button is working. Let me try to fix that CSS. I probably just misspelled a class name or something. The modal was spelled as "model," not "modal." So, let's fix that by replacing the "E" with an "A."

If we come back and refresh, and we click on this, you can see that now we have the modal, and everything is working. If we press connect and fill in the name, you can see that it just quits out because the LiveKit room is not working right now, but you get the idea.

Now, if we want to actually connect to the LiveKit room, we need to get the LiveKit URL as well as a token. I'm going to discuss how the token works.

First things first, we're going to create a new environment variable file in our frontend. We'll call this .env. In .env, we're going to pass a variable. It's important; it's called exactly this: VITE_LIVEKIT_URL. This needs to be the same as what we had in our backend, so we can just copy it and take this WSS URL.

Let's take that and paste it here for the LiveKit URL. That's how we know what LiveKit project to connect to. But we also need to have a token. Now, we're going to discuss the token in a second, I promise.

For now, though, let's add the server URL to our LiveKit room. To do that, we can just do a set of curly braces, and then we are simply going to write the following: import.meta.env.VITE_LIVEKIT_URL. This is how you import an environment variable inside of React, specifically with Vite.

That will import the server URL, and then we need a token. The token is essentially our access credentials, right? It's the ability to connect to this room. We don't just want anyone to be able to connect to any of our rooms.

In order for us to connect from our frontend to the LiveKit server, we need to have some kind of access credential. If we're just testing this, we can get a set of credentials that only last about 15 minutes to test this out. However, in a more production environment, we want to have a backend server that issues us new credentials anytime we want to connect to a room. This way, we have complete control over who is able to connect.

The basic idea is that if you want to connect to LiveKit, you need an access token. The access token should be generated from a backend server. Typically, for that to work, you'd want the user to be authenticated. Maybe they would sign into the webpage, and then they'd be issued a token.

In our case, we just want anyone to be able to connect to the room, so we can issue a token to anyone. But that token will specify what room they're able to connect to. With LiveKit, you can have multiple different rooms and multiple sessions going on at the same time.

We might not want someone to be able to join room A, but maybe we want them to be able to join room B. So, we can issue them a token that only gives them access to that specific room. When they use that token to connect, it will bring them into that room. Hopefully, that makes a little bit of sense.

Again, it will be clearer in a minute, but for now, we're going to get a temporary token that will allow us access to the LiveKit room to test this on our frontend. Then, I'm going to show you the more permanent solution, which is writing a very slim backend server that will issue the tokens for us.

I'm going to go to my LiveKit service center and go to settings and keys, which is the page I'm already on. Go to your project, go to settings, go to keys in LiveKit Cloud. Then, you're just going to click on the key you're using. I believe mine is this Windows machine, and you're going to press generate token.

Now, you just need to give this a name. You can see here that we've specified the username of the person that this token will be used for and the room that we want them to join. Then we have the permissions. If they can participate and publish tracks, they can subscribe to tracks, and they can publish data.

For example, we could issue a token that would allow someone to join but not to talk. You can check all of these different boxes and change the permissions. I'm just going to generate this token with all of the permissions allowed. It's going to give it to me, and I'm going to copy it. This is what I'm going to use now in Vite or in React.

I'm just going to take the token and paste it here as my token to my LiveKit room. Now, this will actually allow me to join the room and hear the audio.

Now that we have the token, we could test this out, but I'm realizing that nothing's going to happen at this point because we're not displaying anything on the screen, and we're also not running our agent.

Before we go further, I don't want to jump the gun here. What I want to do is write this simple Voice Assistant component, which will display some stuff on the screen so we can actually see what's going on and know that this is indeed working.

I'm sorry that we're not able to test it right now; we just need some more components first. So bear with me, and let's get a few things on the page.

I'm going to say import useVoiceAssistant from livekit-components-react. We're going to import the BarVisualizer, which will allow us to visualize the waveform. We're going to import the VoiceAssistantControlBar, the useTrack, which is going to be transcription, and the useLocalParticipant.

Let me just format this so you guys are able to read it. These are some of the hooks and components that we're going to use. We're then going to import Track from the LiveKit client, and we're going to import useEffect and useState.

Next, we're going to import the CSS. We're going to say import './simpleVoiceAssistant.css'.

Great! Next thing, we're going to make our component. We're going to say const SimpleVoiceAssistant, and this will be equal to a component function. Inside of here, we're going to display all of the audio messages as text, right? All of the transcriptions.

But before we do all of that, let's simply just try to render the control bar to get something else on the page so that we can see something going on.

Let's render a div. We'll give this a class name equal to voice-assistant-container, and let's make sure this is returned from the component.

Then we're going to have another div. We'll say class name is equal to visualizer-container. Inside of here, we're going to put the BarVisualizer, but we'll do that in one second.

We're then going to have another div, and this is going to be class name equal to control-section. In the control section, we're going to display the VoiceAssistantControlBar, which is just a pre-made component that provides a bunch of controls for us.

We're then going to have a div. We'll say class name is equal to conversation. For the conversation, we're going to render all of the different messages.

Now, that's all we need for now. We're going to export this, so we'll say export default SimpleVoiceAssistant. For now, this should work; this should show us something on screen. I could be wrong, but let's try it out.

Let's go to our LiveKit modal and bring in the SimpleVoiceAssistant. We'll say import SimpleVoiceAssistant from './SimpleVoiceAssistant'. Let's put this beneath the RoomAudioRenderer.

Now, let's go back, refresh, and talk to an agent. Let's give a name and see if this works.

All right, so silly mistake, but we are forgetting to set isSubmittingName. I'm going to say setIsSubmittingName to false like that.

Now, hopefully, this will actually do something for us, and we'll be able to test this.

Okay, so let's refresh, talk to an agent, give a name, connect, and okay, so nothing's happening. But we were able to get to this page, and then I can disconnect, and it closes. That's what we're looking for.

The reason nothing's happening is that we're not running our agent. So, we're able to connect to the room, but we don't have any agent connecting to the room with us. That's going to be the next step: run our agent, and then hopefully, we'll get some audio output.

What I'm going to do is CD out of this, and now I'm going to activate my virtual environment again in my backend. I'm going to CD into backend, then say .\a\scripts\activate because I'm on Windows.

Now, I'm going to run python agent.py dev. Okay, now that's going to run our agent for us. The agent is running, and now what should happen is when I connect to my room, the agent will connect as well.

So, let's try this. Talk to an agent, let's go name Tim, and connect.

"Welcome to our Auto Service Center. How can I assist you today? If you have a specific question or need help, please provide the VIN of your vehicle. If you don't have a profile yet, just let me know, and we can create one for you."

"Hi, how are you doing today?"

"I'm doing well, thank you for asking."

Okay, sweet! You can see that that's working.

Now, what I did to get that to work, I just paused the video quickly. I just needed to restart my frontend server because we had added this value in the environment variable file. But because we hadn't restarted the server, it wasn't loaded by Vite yet.

So, when I first connected, I wasn't getting anything, and that's because we hadn't restarted the server. When I restarted the server, everything worked. You can see now the Voice Assistant is connecting, and if you look at the logs of the Voice Assistant, you can see everything that's happening here, like what the agent is saying and all this kind of stuff.

Okay, perfect! Now we want to continue building the frontend. I want to make this SimpleVoiceAssistant look a little bit better, and I want it to display specifically all of the text that is being said by both the person and the Voice Assistant.

So, what I'm going to do is make a new component called Message. This is going to take in the type and the text of the message and just display it on screen.

We'll say div class name is equal to message. We're going to have a strong tag here, and this is just going to be the person's name. We'll say class name is equal to message-dash, and this is going to be the dollar sign in brackets type.

We're just going to put this inside a set of backticks so that this works. Let's fix that, and then this needs to go inside a set of braces.

There we go, so that's our class name. Let's fix that, and then we're going to say type is equal to agent. If it's a question mark, then we're going to say agent; otherwise, we're going to say you: like that.

This is like the prefix of the message. Then we're going to have a span, and for the span, we're going to have class name equal to message-d-text, and then we're going to put the text of the message.

Okay, just a simple subcomponent here for displaying our messages.

Now we want to get the transcription from our participants. To do this, we're going to say const [state, audioTrack, agentTranscription] = useVoiceAssistant.

The useVoiceAssistant hook will look for the first agent participant in the room. You could have multiple agents, by the way, but we're just having one. Then it's going to get the state of the agent, which will tell you if it's connecting, if it's talking, if it's listening.

You can actually use this state variable if you want to see that state. You'll get the audio track from the agent itself and then the live transcription of what the agent is saying.

That's what useVoiceAssistant does. Now, we'll have the track from the agent, but we also need to get the track from the person or the participant, which is us.

To do that, we're going to say const localParticipant = useLocalParticipant. Same thing; this is going to get the local participant for us and give us all this information, like if this information is enabled, all that kind of stuff, and the local participant reference itself, which is what we want.

Next, we're going to say const [segments, userTranscription] = useTrackTranscription. Now, for useTrackTranscription, this will give us the transcription of any track. In this case, we want the local participant's track, so we're going to pass inside of here publication, and this is going to be the localParticipant.microphoneTrack.

Then we need to provide the source. The source of this is going to be track.source.microphone, and then we need to give the participant itself. So, we're going to say participant is equal to localParticipant.

I know it's a little bit confusing; you can find this in the docs, but this is how you get the local participant's transcription.

We're then going to have some state, and we're going to say const [messages, setMessages] = useState([]).

At this point, we will have the agent transcription and the user's transcription segments. What we're going to do is just have a simple useEffect hook that will look to see if any of these are changing and then update the messages with those transcriptions.

We're going to say useEffect(() => { ... }, [agentTranscription, userTranscription]).

In here, we're going to have our function, and then we're going to have our dependency array. The dependency array will be the agent transcription or the user transcription.

If either of these changes, then we want to use this useEffect hook. We're going to say const allMessages = [...].

We're going to unpack here, and we're going to say agentTranscription?.map(t => ({ ...t, type: 'agent' })) ?? [] and then do the same thing for userTranscription, where the type is going to be 'user'.

Then we are going to sort these. We're going to say allMessages.sort((a, b) => a.firstReceivedTime - b.firstReceivedTime).

What we're doing is we're getting all of the messages from the agent transcription and from the user transcription. This comes in an array, and it is like the sentences that they're saying.

You'll see the agent will say one sentence, then the next, and it's kind of streaming in these different replies. So, we're grabbing all of those and just mapping these and adding the type to all of the different messages.

Then we combine these together and sort them by the first received time so we know who talked when, and we have the correct order on screen.

Then we say setMessages(allMessages), and we just keep running this effect anytime the agent or the user says something. That's all that's happening.

Okay, so that is our useMessages hook. Now that we have that, we can just start displaying some stuff visually.

We can display the track of the agent, so I can say <BarVisualizer state={state} barCount={7} trackReference={audioTrack} />.

This will display the Voice Assistant audio. You can do the same thing for the participant. We're just going to do it with the Voice Assistant next in our conversation.

We're just going to display all of our messages. We can just say messages.map((message, index) => <Message key={message.id || index} type={message.type} text={message.text} />).

That should be it for the message, and now everything should be displaying properly, assuming our token is still valid. If not, we'll get a new token. I'll show you that in a second.

So, let's refresh this, talk to an agent named Tim, and see if it works.

We got some error there. I assume that's because our token is expired. Yes, we got a 401 issue.

This is what I was talking about before. This token that I generated only lasts for 900 seconds. Now, that's because this is meant to be used temporarily. If you want a more permanent solution, you're supposed to issue the token yourself.

So, what we'll do for testing now is generate a new token. I'm going to generate this token, and this is now a new fresh token that we can use. But again, it will expire in 900 seconds, so it's not a permanent solution, but for testing, that's a fine way to do it.

I'm just going to replace this token here with this new one, and if we go back here and refresh and talk to an agent, now ideally this should work.

Let's try it.

"Welcome to our Auto Service Center. Could you please provide the VIN of your car, the vehicle identification number of your car? If you don't have an existing profile with us, just let me know, and we can create one for you."

So, that is working. However, the messages are not showing up.

Let me disconnect here and see what we did wrong with displaying the messages.

All right, so silly mistake here, but I need to call this agentTranscriptions plural, and then this needs to be agentTranscriptions and then userTranscriptions here as well. I just forgot the "S," so that's why we weren't getting the transcriptions from the agent.

If I come back here, refresh, and try again, this should work now.

Let's go Tim and run this.

We got an error. Okay, so let's see what the error is.

There is an error occurring: agentTranscription is not defined. Okay, so where did we have agentTranscription? That's right here.

So, let's fix this to be agentTranscriptions and userTranscriptions. Apologies, guys, just a silly spelling mistake there.

Let's try again and see.

"Welcome to our Auto Service Center. How can I assist you today? If you could please provide the VIN of your vehicle, I'll be able to look up your profile and assist you further."

Okay, so a silly mistake here when it's not displaying the messages. I forgot to return the div from this Message component. Of course, that would be the issue.

Anyways, now that we're returning that, if we go back here and talk to an agent, and we go Tim and connect, let's see if this works.

"Welcome to our Auto Service Center. Could you please provide the VIN of your vehicle so I can look up?"

So, that's working now. The reason you were getting that zero popping up is I was just debugging to see if the messages were actually working properly, and they are.

You can see now the messages are appearing. So, this is working. We pretty much have finished the frontend.

The only thing that we need to do now is deal with our tokens. Like I said, what needs to happen is we need to write our own server that can issue the tokens to our frontend. This gives us more control over the tokens and allows people to join different rooms.

We're going to go ahead and do that now from our backend.

In our backend, I'm going to make a new file called server.py. Inside of here, we're just going to write a really simple Flask server that will allow us to issue the tokens to the frontend.

Just bear with me while I write this out. It's about 40 lines of code, and then I will show you how we call this and how we dynamically get tokens and how you can have some more control over the rooms that are being created.

What I'm going to do here is say import os. I'm going to say from livekit import API. I'm going to say from flask import Flask and request. I'm going to say from dotenv import load_dotenv.

I'm going to say from flask_cors import CORS. I'm going to say from livekit.api import LiveKitAPI. I'm going to say from livekit import list_rooms.

I'm going to say import uuid, and this needs to be in lowercase. This will allow us to create a unique identifier.

Now, all of these we already have installed in requirements, so as long as we're using our Python virtual environment when we run the server, then we'll be fine.

Now, what I'm going to do is just say load_dotenv, and I'm going to say app = Flask(__name__).

Then we're going to say CORS(app, resources={r"/*": {"origins": "*"}}).

What I'm doing is just specifying that we're going to allow any other domain to access this backend. You usually don't want to do this, but I'm just making it explicit here that we'll allow anything to access it so that we don't get any cross-origin request issues.

Now, all we need to do is run the server and make a simple route that will issue a new access token from LiveKit. It's very easy to do this, but just bear with me.

I'm going to say if __name__ == "__main__": app.run(host="0.0.0.0", port=5001, debug=True).

Now, we're going to define a single route in this backend, and this route is going to be /get_token.

All this is going to do is just issue a new access token, which will allow the user to connect to a new room.

You have a lot of things you can do here with the LiveKit API on the backend. You can list all of the rooms that exist, delete a room, create a new room, and list the different participants.

You can imagine that if you were creating a more complex system here with the AI agent and you wanted to have multiple people in the same room, that's where you would do this. You would set this up on your backend server, so you really have complete control.

What I'm showing you is a basic implementation; keep that in mind.

So, what I'm going to do here is define a function called get_token.

What this is going to do, and actually, let's call it get_or_token to stick with Python convention, is we're going to get the name and the room that the user wants to connect to.

We're going to say name = request.args.get("name") or "my_name" as the default if you don't pass anything.

Then we're going to say room = request.args.get("room") or None.

The idea here is that we're going to allow the user to pass as query parameters the things that come after the question mark in the URL: the name and the room.

The name is their name so we can identify the participant, and then the room is the room they want to connect to if they have a specific room they want to connect to.

Now, we're not going to do that right now, but that is something you could do.

We're going to say if not room, then room = await generate_room_name().

Anytime that you want to issue a new access token, you need to give a name or an identifier for the room that you want the user to join.

In our case, anytime someone presses that button, we want to put them in a new room, like a random room name, so that they're talking to a new AI agent.

You can have multiple people in the same room, but if you did that, then they would be in the same conversation chain as that AI agent that already existed in the room.

I understand it's a little bit confusing; you'll see what I mean in a second.

We're going to write this function in a sec, and then we're going to say token = API.access_token().

We're going to load into variables, so we're going to say os.getenv("LIVEKIT_API_KEY") and load the LiveKit API secret.

Then I'm going to go down to the next line and say .with_identity(name).

We'll go down to the next line and say .with_name(name).

We'll go down to the next line and say .with_grants({"video": {"room": {"join": True, "name": room}}}).

This is just the default way to generate an access token.

What this is going to do is use the credentials that we have for our project, our API key and our API secret. It's going to identify the user with this name and this identity.

You can change the identity and the name if you want, but that's what we'll use for now.

It's going to give them permission to just join this room. That's all we're doing.

There are more granular permissions that you can provide here, but that's what we're doing with this token.

Then we can return the token, which is a JWT token.

Now, we just need to write this generate_room_name function.

To do that, we're going to write two functions. We're going to say async def generate_room_name().

For the name, we're going to say name = "room_" + str(uuid.uuid4())[:8].

This is going to create a unique identifier for us. It's not always guaranteed to be unique if we just grab the first eight characters, but in our case, this is totally fine.

We're just saying room and then some random string associated with it.

Now, we're going to just make sure that this room is actually unique.

We're going to say rooms = await get_rooms(), which is a function we're going to write in one second that will get all of the rooms from LiveKit.

We're going to say while name in rooms, then we're just going to generate a new name.

Then we're going to return the name.

What we're doing is generating this name for the room. We're hoping it's going to be random.

We're going to look at all the rooms that currently exist. If this name does exist in the rooms, it means it's not random, so we're going to generate another one and keep going until we do get a random name.

Next, we're going to say async def get_rooms().

For this, we're going to get all of the rooms currently active.

You can do that by saying API = LiveKitAPI().

We can say rooms = await API.room.list_rooms().

Then we pass to this the list room request.

Then I can say await API.close() to shut it down, and I can return a list of these rooms.

I can say return [room.name for room in rooms].

Actually, we don't need to wrap this in a list because it's already going to be a list.

So, all we're doing here is using the LiveKit API from the LiveKit API. You have a bunch of different operations you can perform.

For example, one of them here is listing the rooms that currently exist. You can also delete a room, add a room, and do all that kind of stuff.

You can see the participants that are in rooms, so I'm just showing you one simple example with the API.

We grab all the room names, and then we'll use that in the generate_room_name function.

That should be it for our Flask backend server.

Essentially, what happens now is because we've used our credentials from LiveKit, when we run this, if we send a request to /get_token, it will generate a random room name for us unless we've provided a room that we'd like to connect to.

Then it will allow us to connect to that using the token from our frontend.

Now, all we need to do is send the request from the frontend to the backend.

First things first, let's run the backend. To run the backend, we'll need to make a new terminal.

Let me just split this again. Let me go into my backend, so CD backend.

Let me activate my virtual environment. I'm going to try to make this a little bit larger so you guys can see these commands.

So, .\a\scripts\activate.

Okay, let me just clear this, and then we're going to say python server.py.

Okay, great! So now the server is running, and we have our agent running, and we have our frontend running as well.

These are kind of the three things that need to be running locally for this to work.

Now, what we can do is go back to our frontend.

Let's close all of this, and we're just going to send a request to the backend.

From our frontend, we're going to go to vite.config.js, and we're going to set up an API proxy.

Here, under plugins, we're going to say server, and we're going to say proxy.

For the proxy, we're going to have /api, and we're going to say that if you're sending a request to anything that starts with /api, we are going to change the target to be http://localhost:5001 because that's the port of our Flask server.

We're going to say changeOrigin: true, and we're going to adjust the path.

We're going to say rewrite: (path) => path.replace(/^\/api/, '').

What is this doing? Essentially, all we're saying is, "Hey, if you try to go to any /api endpoint when you're sending a fetch request here, which you're going to see in one second, we want to send that request to 5001, which is our backend server."

We will change the origin so it's not whatever this is running on; it's just this server.

Then we're going to rewrite the path so that we'll remove the /api component of it and just send whatever comes after.

This is required in order to send the request without getting a CORS error.

Now, what we're going to do is go to our LiveKit modal, and in the LiveKit modal, we are going to get a token as soon as this modal is opened up.

We're going to say const getToken = useCallback(async (userName) => { ... }, []);

The async function is going to take in the user's name.

What it's going to do is send a request to the backend to retrieve the token.

It's going to say try { const response = await fetch('/api/get_token?name=' + encodeURIComponent(userName)); }

Now, after we do that, we're going to get the token from the response.

So, we're going to say const token = await response.text();

We're going to say setToken(token), and we're going to add some state for storing our token.

We're going to say const [token, setToken] = useState(null);

Then we're going to say setIsSubmittingName(false).

Otherwise, we're going to have a catch, and we can catch some error and log it in the terminal.

We can say console.error(error);

This is our getToken function, and what we're going to do now is under our handleNameSubmit, we're going to change this to call the getToken function.

We're going to say e.preventDefault() so that we don't refresh the page.

We're going to say if name.trim() { getToken(name); }

Now, this is a callback function, and what it's going to do is send a request. It's going to grab the token for us, and then we should be good to go.

Now, we just need to do one thing: we need to change the token here to be the token from our state.

That's coming from the backend.

So, let's fix that.

Now, token is there, and then we just need to make sure that we're only going to show this if the token exists.

We're going to say {token ? <YourComponent /> : null}.

So, we're just saying, "Hey, if we're submitting the name, show this. If the token exists, show this. If the token doesn't exist, then just show null."

It will take a second to render this because obviously, we need to grab the token first from this request.

So, that should be it.

Let's just make sure all this is running still. It looks like it is.

Let's go back now and see if we're able to get this to work without hardcoding our token.

Let me go talk to an agent named Tim and then connect.

We got some problem; something happened there.

Or at least it's not showing the page.

Let me look at what the issue is here.

All right, so two silly mistakes here. I forgot to add the "E" to my handleSubmit, and I forgot to add the dependency array to my useCallback.

Because I didn't have the "E," it was just refreshing the page and wasn't preventing the default behavior, so that's why this was not working.

If I go back here and refresh now, and I go Tim and connect, we can see it takes a second, and then we got some issue.

It says invalid authorization token.

Okay, interesting that that token is not correct.

However, it did look like it sent the request to the backend, and that's why we got an issue in our backend server.

So, it never actually gave us the correct token.

Let's go back to the backend, server.py, and we need to change this here to say await API.close() not simply close on line 24.

Now, let's go to talk to an agent.

Let's go back to Tim and connect and see if it works.

"Welcome to our Auto Service Center. Could you please provide me with the VIN, the vehicle identification number of your car? If you don't have a profile yet, just let me know, and we'll create one for you."

"Can you make a profile, please?"

"I'd be happy to help you create a profile. Could you please provide the following details about your..."

Okay, perfect!

I'm going to disconnect from that because you can see that is indeed working, and now we are getting that token from the backend.

Again, that is the advantage of using that backend server. You now have complete control, and you can do all kinds of fancy stuff if you really wanted to in terms of issuing the tokens and connecting people to the same room, all that kind of stuff.

Now, if multiple people press that button at the same time, they're all going to get different access tokens with access to different rooms.

They're all going to have their own instance of the AI agent running; they're not going to be using the same conversational chain.

So, that is something to keep in mind.

With that said, guys, that will pretty much wrap this up.

I will let you know that if you do want to actually make this like a real call center, you can do that with LiveKit.

If we go into the docs here, let's go to the real-time API. Let's go to the multimodal agent.

There is a way that you can actually get it so you can call a phone number using something like Twilio and have it respond to you on the phone.

You can see there's some stuff with the SIP trunking if you go to this page in the docs.

You can see kind of how this works, and you can make it so that if you call a phone number, then your agent actually connects to that room and starts talking to you.

You could literally make this work over the phone, not just on the frontend UI.

But obviously, that's what I wanted to show you here because I imagine that's what most of you are looking for.

All kinds of cool stuff you can do with LiveKit; this is obviously just scratching the surface.

If you want to see more tutorials on LiveKit, please do let me know by leaving a comment down below, and I'm sure we can team up again and do another cool tutorial for you guys and show you maybe even some more advanced behavior.

With that said, I am going to wrap it up here. I hope you guys enjoyed all of this.

Code will be available from the link in the description, and I look forward to seeing you in another video.