Transcription
Most AI voice agents sound great in testing and then fall apart the moment they hit the real world. Today, I'm going to show you exactly how we build voice agents that don't just work, but are capable of handling the many scenarios that they will eventually face. In my past videos, I've given you a general look at prompt engineering and configuration. But I've never shown you start to finish the full process that we use inside our agency to deliver production-ready AI voice agents.
Over the past months, we've been quietly building a piece of software that makes developing and maintaining these agents faster, easier, and dramatically more reliable. And today, in this video, we're launching it. It's called Reliable. It's an automated AI voice agent testing and monitoring tool. We've been using it internally for months alongside beta testers from within my community, and it has saved us countless hours while eliminating some of the biggest headaches in AI voice agent development. You can check it out at reliable.ai or via the link in the description.
But otherwise, I'm going to do a full rundown of the platform in this video, building out a complete voice agent from scratch. I'm going to try to break the prompt, fix it using Reliable, probably going to break it again, and just keep iterating on that prompt, so you really get an unmasked, behind-the-scenes look at exactly what it takes to build and ship an AI voice agent that's actually production-ready.
So, we're going to be using Vappy for our voice agent. This is obviously not going to be a beginner tutorial. Also, if you've never seen Vappy, you never used Vappy, I would recommend obviously watching a more in-depth beginner tutorial to get started on this, but obviously if you use a similar platform, you might be able to watch it and get along with it. But otherwise, this isn't a beginner tutorial. We're going to be using Vappy for the main agent. And for the first time, we're also going to be jumping into Reliable. So, Reliable is my own platform that we've built out and this is going to be directly for helping test and monitor your AI voice agents. And I'm going to give you the complete rundown as to how to use this platform, how to use it best to actually make these voice agents just far better. Here's a quick sneak peek as to what the platform actually looks like here on the website. But once again, we're going to be jumping into this, diving into everything about it, how to use it, why you should even use it, and what the benefit is much later in the video.
So, to get started, we're going to jump into Vappy, and I'm going to go ahead and start to build out a single prompt agent. So, we're going to hit "Create a System." I'm just going to create a brand new from blank agent. Not going to use a template. I don't have a prompt that I've already created. I'm just going to create this system from scratch. The use case that I think I'm going to start with is just going to be sort of a real estate example. We're going to try to build out an inbound system for a real estate company. Might have some property inquiries, lead qualification, lead capture steps, all that stuff we're going to be building out. And then we're going to use Reliable to stress test it, to break it as much as possible, and just see what happens and just make this a very transparent building process. So you can really see what it takes to obviously go from absolutely nothing to having a prompt that is far more production-ready and stress-tested with a lot more prompt engineering done to it.
So, once you've hit "Create Assistant," it's just going to ask you to choose a template. We're obviously not going to be doing a template here, so I'm just going to type "Real Estate Agent Demo." And I might just tag this as "Reliable," just for my own naming needs. Blank template. We're not going to be using any of these, obviously. Going to hit "Create Assistant." And it's going to load us into obviously the page for creating our brand new assistant. So, once we've done that, we're going to land on our page right here. I'm just going to go through all of the main configuration settings that we would obviously just configure when we're building out any new system for a client or for whatever reason. You know, normally we're going to configure all these settings the way that we like them before we jump directly into building out our prompt.
One big thing to note is that today, actually, GPT-4o just came out and you can see Vappy has been pretty quick on it. They've already released all of the GPT-4o brand new models. I'm not going to be using GPT-4o right away. For today, I'm just going to use the the trusty 4.0 models just for now because I know that they work. This literally came out a couple hours ago, so you know, I'm not exactly sure how good they are yet. Uh, but I will jump into that in a different video. But we're going to start off with GPT-4o. As for the other settings, we can jump up and look into the voice settings as well. At the moment, by default, Vappy sets their voice settings to their own provider. So, Vappy has their own provider if you haven't looked into this. They sound quite good. They're really well-tuned for, I'd say, phone call situations. So, they sound like real phone calls. You know, they sound super, I'd say they sound monotone, but it works quite well for a phone call call center situation. So, they sound quite realistic. Although, there's not a whole lot of options. We click into the different options provided here for Vappy. Looks like there's probably what, 15, 10 maybe voices here. Few different accents, although there isn't obviously a large library of voices compared to something like 11 Labs. So, overall, not a big deal as to which voice you choose. I think most of the Vappy ones have been pretty well looked into, so they're not like glitchy. Some of the 11 Labs voices are tuned for audiobooks and all that stuff like that. So, just to make it easy, I will just select one of the Vappy voices to use, but obviously you can choose any voice you'd like.
When it comes to obviously building out production-ready voice agents and building out a voice agent for a client, we'd obviously want to do pretty extensive testing on the voice. We do have some voices from 11 Labs that we know personally. We do have a bit of a list of 11 Labs voices that we know are quite good and work well in production. You know, some voices you you'll think sound quite great. They'll sound super realistic, but it's usually those ones that are sort of a Trojan horse and later on after a few hundred calls, they start to act up and start to mess up and glitch and start to screech. So, one key thing to note is just make sure that you're choosing a voice that is going to be reliable. But that's the whole point of this video. We're going to choose a voice and see how good it is in the long run of actually running these calls for a real production-ready system. So, we'll choose that voice for now from Vappy.
As for additional settings here, we can jump into our transcriber. Deep. Pretty solid. Don't really need to, you know, really change a lot of this stuff, you know, not unless not until we've actually started building it and testing it and calling it. Do we really have to change a lot of the transcriber settings here? Obviously, we've got our threshold, everything like that. This stuff we just do once we're going. So we don't have to do that right now. Tools, obviously. Once again, once we understand what kind of functions we want the assistant to have, then we can start to jump into all this stuff.
So, I'll go ahead and start to build our prompt. So, once again, I think I'm going to go ahead and build out a real estate sort of inbound real estate example. So this could be, you know, you're interested in selling your house, you're interested in buying a property that a particular company is offering. You're going to call up this number. It might be on their website, might be on advertisements, whatever. you're going to be calling in that number and you're going to be wanting to speak to somebody about a particular property or some sort of inquiry and we want to be able to facilitate that. Likely we're going to be capturing some lead details. We might want to run a qualification process. We might want to be capturing an address, an email. So, all of those steps we're going to build out here on this agent. And to make this super realistic and really aligned with how we actually build these systems for our clients in our agency, what we always do is start off by building out a call flow diagram. And so this is something that allows us to really map out start to finish, top to bottom, what is this agent actually going to include? And this is something that's going to be really helpful when you're building out a voice agent. If you're building it out yourself or you're going to be having somebody else build it out for you, what we typically do for our clients is we go ahead and scope it out on a sort of a diagram system like this. So this is using Whimsical if you've heard of Whimsical. It's pretty much the same as Miro. If you've seen Miro, there's plenty of platforms like this. Essentially just allows you to build out sort of a visual canvas, adding some blocks, connecting them together, adding some text and whatnot. Just allows us to really visually look at what is our voice agent going to do and then how can we translate that into an actual prompt.
So, getting started, what we're going to do is just type "inbound call initiated" or something like that. "Inbound call started." So, obviously we'll just keep this here for our own reference. It's an inbound call. Somebody's going to be calling into the phone number and the AI is going to pick up immediately. So, we'll go "inbound call started." We'll go ahead and type out a first message. So, that could be, you know, "Hi, this is Emily from Inflate Real Estate Services. How can we help you today?" So, super simple. Obviously nothing too crazy has to happen here. Just really thinking, you know, naturally what is somebody going to say, you know, when you call in and that's quite well. We can have that read out right away. Obviously, there's many different settings if we want it to not read out right away. We can obviously configure it to do whatever we want. And this is going to be something that we obviously want to to start to track. So, when we start to get real calls in the system, we want to see how well is it going to be doing this. And this is going to be one of the things that we're going to be testing for and we're going to be obviously assessing to make sure that this gets read out every single time cuz we obviously don't want to say anything too much outside of this. And one thing that I will add to this first message is that this is actually an AI system. So, "Hi, this is Emily." Put that there. "I'm an AI agent from Inflate Real Estate Services. How can we help you today?" So, "Hi, this is Emily. I'm an AI agent." So, the reason that we do this is that we really don't want to trick anybody into thinking that they're speaking to a human when they're not actually. Obviously, by displaying right away, this is an AI agent. This is an automated system. It's an AI phone. Whatever you want to say, however you want to phrase it, just good to do this. We haven't found any, you know, negative result by doing this. We've only found it to be beneficial. Obviously being transparent at the end of the day, especially for an inbound system. If somebody's calling in, they just want to get what they're looking for. They don't really care if it's an automated system or not. And so that's what we'd recommend.
So, next steps for this agent, we're going to probably classify this company's services into sort of three different buckets, which would be buying, selling, and renting. So, they're either looking to buy property, sell their property, or rent a property from from our Inflate Real Estate Services. So, I'm going to go ahead and add in three different blocks for each of those services. And once again, building out this diagram is just super helpful for obviously prompting. Being able to visually see the kind of structure that it would have when we start to prompt it is just super, super helpful. So, we're going to have three different blocks coming out of this block right here. And these are going to be for the three different services. We're going to have buying, going to have selling as well, and we're going to have renting as well. So, buying, renting, and selling. Obviously, this doesn't have to look super, super fancy. There we go. Buying, renting, selling. So, three different pathways is going to happen after this first message. So, right out the gate, "How can we help you today?" down one of these three paths.
So, for buying, what we could have is just a standard message for anybody that the AI detects is looking to buy one of our properties. So, we could just say, "Great. Which of our properties were you interested in?" Obviously, this doesn't have to be a finalized bit of messaging, but this is just some something to have an understanding of the flow. So, somebody's looking to sell. I'm actually going to start to do a little bit of data capture. So, "Great. Could you please provide us with your address?" We'll then start to go down the pathway here of just capturing information. Might just spread that out a little bit so we've got a little bit more room here. "Great. Could you please provide us with your address?" Then after that, we're going to say, "Thank you. Could you please provide us with your full name?" Capture their name after their address. "Thank you. What is your reason for selling?" Maybe. "Have you done any repairs recently?" Little bit of data capture. Just really anything to to obviously prove the point here. "Any repairs recently?" and then maybe "What is your timeline for getting this place sold?" Cool. So, we got a few questions capturing on the selling side and I'm just going to quickly do stuff on the buying and renting side as obviously you probably don't care too much about the real estate example here. So, I'm just going to speed through this and then just show you the result.
So, I've gone ahead and just sped through and just added a few more questions and flows to this obviously just to speed things up. So, we've got buying, renting, and selling. For renting: "Great. Which of our properties were you interested in?" "Thank you. Could you please provide with your full name?" "Thank you. What is your timeline for moving in?" And then, "No worries. Would you like to book a walkthrough?" So, a little bit of data capture. I think what we'll do is set up maybe a spreadsheet to start to capture this data. We sort of simulate a CRM system. Obviously, we can plug into all different types of systems as long as it has an API connection to do so using platforms like Make.com or Zapier, and even. We can obviously use as a middleman tool to do that. Otherwise, we can also do direct connections. But anyways, we can obviously ask those questions.
The buying flow: "Great. Which of our properties were you interested in?" "Thank you. Could you provide us with your full name?" Then we're going to capture. "Do you have another property you need to sell first?" So, obviously if they're buying, they might have a property they're selling. Going to have another sort of flow path change here. So, either yes or no. If it is yes, "No worries. Would you like our assistance in selling this property?" Once again, either no or yes. "No worries. Would you like to book a walkthrough?" Otherwise, "Great. Could you please provide us with your current address?" And then, "Thanks. Would you like to book a walkthrough for the new property?" So, few different pathways there. That's going to be really good to obviously test and push to see how well we're able to get it to to be consistent at doing that under all different types of scenarios. Um, chances are we're going to need to do a little bit of prompting if we really start to hammer this system if we are going to be doing this in sort of a one-shot prompt approach. Otherwise, here we go. Three different pathways, little bit of condition there, and we're going to start to build out a prompt to mimic this.
So, jumping back to Vappy, the first thing that I'm going to do is go back to our diagram here and just copy our first message. Obviously, we want it to be able to read that out. And in the first message here, we can obviously just paste that in. "Hi, this is Emily. I'm an AI agent from Inflate Real Estate Services. How can we help you today?" So, that's super easy to do. Message mode, assistant speaks first. Firstly, for the first message, I believe here on Vappy, it's going to read that out word for word. So, this is isn't a prompt. We're not going to have to worry too much about it failing to read that out cuz it's just going to read it out the same every single time.
Otherwise, jumping into our system prompt. This is where all of the magic happens and where all the failures could occur as well, of course. So, jumping into the actual structure of our prompt, I always like to start it off with a role. If you've seen any of my other prompting templates, role is just a good place to start to really shape the personality of the agent that we're building. At the end of the day, we're building a personal assistant. We're building a receptionist. This is sort of a real situation that we're dealing with and obviously giving it that deep understanding of exactly what its role is is a really good thing to do. So, for our role, "You are Emily. You are an AI agent from Inflate Real Estate Services. You will help answer our inbound phone calls, helping clients to sell their property, buy a property, or rent a property." So, super basic obviously to get started. Literally just mapping out what's off the top of my head. This is obviously not going to be the most optimized prompt ever. Just sort of one-shot it right there. Grammarly is is complaining about it. So, I'm just going to click this. "You are Emily, an AI agent from Inflate Real Estate Services. It will help answer our inbound phone calls, assisting clients with selling, buying, or renting a property." Awesome. Super clean, super simple.
The next stuff that we want to obviously add to this is really just drilling into everything that this agent is capable of. So, if we jump back to our Whimsical, we're obviously not just going down, you know, three different paths, but there's obviously information that needs to be captured. And a lot of this stuff we'll add into the next part of our prompt, which is going to be the task. That's more specifically what the agent has to handle. But other stuff that we could add into the role might be that you're friendly. So, "You are Emily, a friendly AI agent from Inflate Real Estate Services." And you'll be surprised as to just adding those two words there is probably going to change its personality quite a bit. Rather than being robotic and and strict, it's going to be maybe start to throw in some words that are going to be a bit more friendly or whatever. So, you know, you'll get an understanding as to when you call it and talk to it. So, what some of these changes can do.
So, what we can do right away is actually go ahead and just publish this agent. And if we'd like, we can just give it a call and just see how it sounds.
>> Hi, this is Emily. I'm an AI agent from Inflate Real Estate Services. How can we help you today?
>> Hey, I was just looking to purchase a property.
>> That's great. I'd be happy to help you with that.
So, there we go. Obviously, um, I was just trying to test out exactly how it would sound. We called it Emily, although we are actually using a male voice, so we probably want to change that one. Pretty easy fix. So, I'll just jump into the Vappy here. We'll just choose one of these voices. We'll choose Hannah here. That's a female voice. I'll hit publish. So, that's a good change to make obviously. Cool.
So, we got the role. We're going to be jumping into the task. We're likely going to be coming back to the role as we start to build this agent. Obviously, when we start to call it and talk to it, a lot of the personality stuff, a lot of the way that it talks and responds, the way in which it structures its conversations and its outputs is going to be quite deeply handled by the role section. So, revisit that later when we're actually talking to it. But the next section that I like to add in is the task. So, once again, this isn't a very strict prompt. You don't have to follow the role specifically or the task specifically. This is just a structure that we like to use. Uh, we found it to work quite well. But this is not a a must thing. This is not like a strict guideline that needs to be followed. Really just doing whatever makes the system work is obviously going to be the way to go about it. If it works, it works. If it doesn't, it doesn't. That's the theory that I like to follow. And obviously the task here is really going to be just fundamentally looking at the core aspects of our agent, which is going to be following through with some qualification questions.
So, right under the task, we're going to add in some subsections. So, obviously, we've got three different parts to this agent. So, our task isn't just going to be one task. Kind of got three different tasks going on. So, we're going to add in two hashtags. And if you aren't familiar with these hashtags and what I'm doing here, this is markdown formatting. So, we obviously want to format this prompt. They're using markdown formatting because a lot of the models created by OpenAI were trained on markdown formatting. So, it's kind of you can think of it as the the language of the AI. How the AI actually looks at this is going to be through markdown. And the better that we can present this prompt to it, the better it's going to understand it and read it and contextualize it. And so if we can put in markdown, it's going to help out quite a lot. So, we're going to put in the task. And then below that, we're going to put in our three sections. We're going to put in buying, probably put in selling as well. And then renting will be a third category here. Uh, so we got three different categories. And if we go ahead and take this prompt, head over to a platform called Markdown Live Preview. It's as it sort of says, it just previews markdown live. If I paste that in here, you're going to see what it looks like for the AI to be reading your prompts specifically. This is something that I really like to use when actually building out these prompts because we can sort of exactly see what we're doing and how this prompt would actually look to a real, you know, AI when it's looking at it. So, other stuff that I'll add later on when we start to to really refine the prompt and clean it up is just small things like bolding elements. So, we might want to bold the name of the agent here, Emily. That might help out. Inflate Real Estate Services. We could bold as well because these are key parts of it and things that it should really know. Just bolding it can add a little bit of emphasis to it and just make it more accurate overall.
So, the first thing that we're going to be doing is the buying pathway. So, "Great. Which of these properties were you interested in?" We're going to grab this question and chuck it into our buying category right here. So, this is the pathway to follow when someone is interested in buying one of our properties. So, we'll just put this here to really let our AI know exactly what to do. If somebody is interested in buying, of course, it needs to follow this pathway. There's a pathway to follow. And what we can do is just start to number these questions. So, number one, "Great. Which of our properties were you interested in?" Looks like we haven't added that. Interested in. "Great. Which of our properties were you interested in?" Number two, jump back to here. "Thank you. Could you please provide us with your full name?" We'll add that in there as well. Number three, we're going to add in, "Thank you. Do you have another property you need to sell first?" We'll add that in here as well. So, we got three questions there. We're going to have sort of a pathway change when we look at having either yes or no. So, we can put in here, "If the user responds yes to the previous question, please ask this." And then we could very simply put, "If the user responds no, please ask this question here." So, super, super simple. Obviously, this is right out the gate. This is not exactly going to be the the most best prompt ever. This is just what's coming off the top of my head. And we're going to be testing the system obviously to be able to see how good this prompt actually is. Numbering is is a good strategy. Obviously, this is kind of just good formatting technique to really structure it step by step, which is always the best thing to do, just step by step. You know, the more clearly that can be communicated, the AI is going to obviously have a much better idea as to how to handle these situations with our prompt.
Now, one thing that we need to do next is jump into a secondary flow after this flow. So, what I might do is actually make this a little bit better. So, if the user responds yes to the previous question, rather than saying "previous question," I'm going to say "to the third question" because we've numbered that. If the user responds yes to the third question, if they respond no to the third question, just so we don't lose context as to what question we're referring to. And then we're going to add in a little bit more conditioning here. So, we're going to add in this question. "If the user says no, please respond with this." Put that in there. What we can do is start to add in a little bit more markdown into this. So, obviously starting to get a little bit confusing even when I'm looking at it here. So, what we can do is just add in some apostrophes through some of these questions. And what it does to markdown, sort of just highlights it and puts it in its own box and that obviously makes it really clear to distinguish the difference between this and this as this is obviously going to be something that we want the AI to follow specifically. So, "If the user says no, please respond with this." "If the user says yes, please respond with this." And that is going to be in apostrophes as well. We're going to indent that as well. Grab that question and throw it in there. So, there we go. So, "If the user says yes, please respond with this." It looks like it has formatted itself quite poorly there. So, I'm not sure if we just add in a space and that will help out or not. We'll just do that and that's going to help out quite a bit. We can also space this out as well. Add in that as there. And then we can see. So, obviously if we didn't have this markdown formatter, that would have been pretty hard to actually understand why that wasn't formatted. So, great.
So, almost at the end here for this flow, which is the most complex one. So, we're just going to obviously finalize a couple questions here. So, "Great. Could you please provide us with your current address?" And then we're just going to say, "Thanks. Would you like to book a walkthrough for the new property?" So, we'll just add this here. "Next, please ask this question." "Do this." Apostrophes. Put that in there. Cool. So, we've got "if the user says no, want this." "If they say yes, do this." And then we're going to ask this as the next question. We only want to have that ask this as the next question. If they say yes, we probably want to add that under here. So, we want to make it super, super clear to how to structure this. Maybe we'll say, "Next, if the user had said yes, please ask this question." So, there we go. We should have that flow ready to go there. We got everything that we need here. And that is all concluded. So, if they say no, "No worries. Would you like to book a walkthrough?" So, we can obviously jump in and now add some conditioning for what that walkthrough would look like. That in this case would be an appointment booking. I'm not going to dive into building out an entire appointment booking flow. This video is directly looking at prompt engineering obviously and everything to do with prompt engineering, but we could obviously build out a booking flow and add that into this as well for booking a walkthrough.
So, the next flow that I'm going to jump into is obviously going to be renting and that is just going to be using the next obviously subheading down here. So, renting. We're just going to jump all the way down to the bottom. Can even swap these around. Doesn't really matter. I'll just jump into renting and just map out these questions here. So, should be super easy. This is the pathway, or I might just copy this all the way to the top here. This is the path I follow when somebody is interested in renting one of our properties. So, similar flow to the buying. So, what I can do is probably just copy this and add it right below the buying flow cuz they are similar. So, that will help out with overall contextual understanding of of what to do in that case. So, "Renting. Great. Which of these properties were you interested in?" is going to be the first question, exactly the same as here. So, we should just copy this as well. The more similar that we can make it between all the different flows, the more better it's going to understand how to read our prompt. So, if we can make it similar, definitely recommend to go ahead and do that. So, "Great. Which of our properties were you interested in?" We're just going to copy a lot of the same stuff here. So, "Great. Provide us with your full name." "What's your timeline for moving in?" That's a new question we'll add here. Three. Add the dot. "Thank you. What is your timeline for moving in?" Cool. We're then going to add in, "No worries. Would you like to book a walkthrough?" That'll be the last one there. So, super simple for renting. We're just asking four direct questions, but obviously with the booking a walkthrough, that's going to be something that we can obviously add on top with a with a real booking, once again.
All right. So, we're going to leave the renting pathway here for now, and we're just going to quickly speedrun through the selling pathway. So, that pathway is all good to go for renting. We're now going to quickly speedrun through building out the selling pathway as well. Very similar questions. We're just going to copy all of this. This is the pathway to follow when somebody is interested in selling their property. Obviously, it's not ours. So, we're going to put that there. First question probably not going to be "which property are you interested in?" So, it's going to be, "Please provide us with your address." Put that in there. Next question, "full name." So, we've already got that one. We're going to copy that over as well. "Reason for selling." We're going to add that question in here. And then number four, "Have you done any repairs recently?" Number four. And then lastly, "What is your timeline for getting this place sold?" Cool. So, we've got all these questions in there. Five questions that it should follow along with in the buying, renting, selling categories. And awesome.
So, now what we want to do is obviously see how well it's going to be responding to our current prompts. Now, next to this, we want to start to add in some sort of pronunciation stuff. So, when it comes to reading out an address, that's not something that's super easy to do. That's actually quite a complex thing to do when it comes to voice agents. Um, and you'll see that when we start to test it. A lot of it's to do with pronunciations and the capturing of information is something that we have to do a lot of work on. So, I'll showcase doing a little bit of that. I'll actually reference a lot of stuff that we have within our own internal SOP documents for prompt engineering to to help out with that. But, I'm also going to show you obviously testing these systems and and exactly showing you what the kind of issues that we get when we don't do that. And so, we'll get to that later as well.
So, I'm just going to go ahead and just copy this prompt. We'll hit copy at the top here. Headed back to Vappy. And we're just going to throw this entire prompt into Vappy right there. There is our prompt. We're going to hit publish. And what I'm going to do is just start to call this thing. We're going to start to call it. We're going to see if we can follow our instructions. How well does it work out of the gate? How perfect does it work? What are the changes that we're going to need to make? And how can we really start to push the system and stress test it? So, there are already some spelling mistakes that I can see here, but obviously we're going to publish this and let's give it a chat.
>> Hi, this is Emily. I'm an AI agent from Inflate Real Estate Services. How can we help you today?
>> Hey Emily, I was actually having a look at some of the properties on your website. I was interested in renting one of them. Um, would you be able to do that?
>> Great.
>> Absolutely. I'd be happy to help you with that.
>> Great. Which of our properties were you interested in?
>> Yeah, I saw 123 Main Street. That looked pretty awesome.
>> Perfect choice. Could you please provide your full name?
>> Yeah, it is John Smith.
>> Thank you, John. What is your timeline for moving in?
>> Probably next week. Uh, we want to move in pretty quick.
>> Got it. Next week sounds exciting. No worries. Would you like to book a walkthrough for 123 Main Street?
>> Yeah, that'll be awesome.
>> Great. I'll arrange a walkthrough for 123 Main Street. I'll ensure our team reaches out to you shortly to confirm the details. Is there anything else I can assist you with today?
So, that was pretty awesome. That actually followed through with our instruction set really well. That actually went through uh each step by step and followed through with each of the questions for that particular flow. That was the easiest one that I just picked out. So, we're going to jump into buying as well and just see if we can get it to trip up from our manual testing, but obviously from a very brief manual test. It seems like it works quite well, but I can guarantee you that when you start to put this system in production, obviously having just one shot of this prompt, that's when you're going to start to see the flaws over hundreds and hundreds of calls. And so, that is the point of this video, of course.
So, we are going to go ahead and try to see if we can get the bot to trip up on the buying flow. Now,
>> Emily, I'm an AI agent from Inflate Real Estate Services. How can we help you today?
>> Hey, I was just interested in buying one of the properties I saw.
>> Great. Which of our properties were you interested in?
>> Yeah, I saw 123 Main Street. That looked quite nice.
>> Thank you. Could you please provide us with your full name?
>> Yeah, my name is John John's.
>> Thank you, John. Do you have another property you need to sell first?
>> Yeah, we do, actually.
>> No worries. John, would you like our assistance in selling this property?
>> Um, how much would I get for it?
>> Great question. That's a great question. One of our local agents would be happy to help you estimate the value based on your area and recent sales. Would you like us to assist with selling your current property?
>> Yeah, that's fine.
>> Great. Could you please provide us with your current address?
>> Yeah, it is 57 um Wood Road.
>> Thanks, John. Would you like to book a walkthrough for the property interested in at 123 Main Street?
>> Yeah, that'd be perfect. Thank you.
>> You're welcome, John. I'll have one of our agents reach out to confirm the walkthrough.
There we go. Once again, awesome. That actually went through really well. Obviously, our prompting there has done a good job to to run through each of the individual questions and pathways and flows to be able to handle that and do everything we needed to. So, obviously that stuff is is quite simple and straightforward and black and white as as to what it needs to do. We're not doing a lot of complex steps in this particular bot for obviously, you know, just going from step by step. Each is a different step. When it starts to trip up could be when we start to actually validate a lot of the information that we're capturing. So, that could be when we're capturing an address, when we're capturing a name. A lot of that stuff is a lot more complicated. And so we can start to jump into that as well. We can start to add in some conditioning and start to see just any changes that we need to make to obviously make it a smoother experience.
So, coming back to our markdown live preview, what we'll typically do uh when we're building out one of these systems for a client is add in a confirmation step. So, if we're going to be capturing an address, we want to make sure that we capture the right address. And so, rather than just asking them for what what their address is, we want to confirm that back to them. So, for all of our different pathways, we are just saying, "Great. Could you please provide us with your address?" We can add in a secondary question. So, we're sort of skipping past this question here, and we're just going to confirm that with them. So, "Provide us with your address." Number two is going to be, "Thank you. Just to confirm your address is..." and we just add some brackets here. "Read out the address captured." And then I'm just going to renumber these questions to be corresponding. So, perfect. We're adding the second question there. "Thank you. Just to confirm your address is..." and then just read out the address captured. So, I'm going to add this to everywhere that we're capturing an address. Although I believe selling is the main area we're capturing an address. We're going to be obviously asking which properties they're interested in. We can do the same thing there as well for the property that they're interested in. So, we can repeat it back to them. Although I think the main part is going to be in the selling flow.
So, before we do any sort of conditioning for capturing it, I am just going to copy our prompt and add this back to Vappy and we're going to try and see how well it's able to capture our address and sort of accurately read that back out to us. So, we're just going to put that in there. We're going to hit publish and we're going to hit "Test the Assistant."
>> Hi, this is Emily. I'm an AI agent from Insight Real Estate Services. How can we help you today?
>> Hey Emily, I was looking to sell my property. We are currently located at 57B um Woodson Boulevard here in LA.
>> Great. Thank you for providing the address. Just to confirm, your address is 57B Woodson Boulevard in LA. Is that correct?
So, we can see there that also worked really well. So, I was able to take our address, capture it, output it. Did it perfectly with no issues there. So, that's cool for now. But one thing that we do want to add in obviously probably going to be later on as we start to test it and push it'll break it a lot more is just a little bit more prompt conditioning when it comes to reading out an address. And so this is something that once again need to test it quite a bit to be able to realize that you actually need this. And so if we add in just a note here at the bottom, I'll add in sort of an "address pronunciation" section. And this is really helpful for reading out an address, especially with a voice-based system. And so I'll add this right here. And this is a prompt that we've used previously when needing to read out sort of house addresses and addresses. As you can see here, even in the markdown format, we have a little bit quite a bit of prompt conditioning that we're using here to just split out, you know, an address that's being captured. As you saw before, when I provided the apartment number that I was that was referring to, so apartment 57B, it was output from the AI itself as as 5B or 57B. Now, it worked quite well when I listened to it through the Vappy system there. Although, that's not going to be the most consistent, the most reliable way to approach it. We're going to be splitting it up into actual words that is going to be far more reliable when it comes to being read out from a from a speech model. Having it put into words like this is just a far better way to make sure that it's consistent and reliable at doing that. So, really good to to have these conditions in here. Obviously, I've already done the work to build out this particular prompt here. This is definitely something to add in this particular instance. Once again, this is something that, you know, needs to be worked on over time to be built on from very extensive levels of testing.
So, the last thing that I'm going to do before we start to jump into automated testing and really hammering these prompts to try and break these systems and find a lot of the faults from this particular prompt here and just see where we can improve it to make it better and better is going to be with our full name. So, we're going to be capturing a name from the particular user in this case. Usually, in a in a real production-ready, a real system, we're not just going to ask for their name and then just, you know, not clarify or or confirm it. We obviously want to make sure that we've gotten the name quite correct from what we're capturing. So, I'm going to add in an additional question here. We're going to add this in as the third question. And we're going to change that to the fourth. And this is going to be confirming their name with them. So, number three, "Once their name has been captured, please repeat it back to them letter by letter." Let's remove that. "Once it's been captured, repeat it back to them letter by letter." So, to make this even a little bit more realistic, I'm going to change this first question from "Please provide us with your full name" to actually spelling out the full name. So, "Thank you. Could you please spell out your full name for us?" And then once it's captured, "Repeat it back to them letter by letter." And so this is just super, super helpful for obviously just making it more reliable over time. We obviously aren't able to rely on the transcribers as much as we'd like to to be able to capture a name that accurately, especially if you're going to be using this to plug into a CRM system. You don't want it to be putting in the wrong person. So, that's a pretty critical thing to do. And we're going to add that into each of these steps here. So, cool. There we go. Let's go ahead and add this prompt to Vappy and see what we can get out of it. And then we're going to jump into automated testing to really stress test this.
>> Hi, this is Emily. I'm an AI agent from Insulate Real Estate Services. How can we help you today?
>> Hey Emily, I was looking at selling my property.
>> Great. Could you please provide us with your address?
Yeah, we're located at 57B Avenue. Thank you. Just to confirm, your address is 57B Avenue. Could you please spell out your full name for us?
Yeah, my name is John Smith. J O H N S M I T H.
Thank you, John. Let me repeat that back to you. J O H N S M I T H. What is your reason for selling?
Yeah, we just looking to get the place sets a little bit bigger. We're obviously in a very small apartment at the moment.
Thank you for sharing that. John, have you done any repairs recently?
Uh, no, we haven't.
No worries. What is your timeline for getting this place sold?
Yeah, probably next week. We do want to get out of here pretty quick.
Got it, John. Thanks for the information. We'll make sure to assist you in getting it sold quickly. If there's anything else you need, feel free.
So, there we go. Awesome. Pretty simple. pretty straightforward. Just ran through each of those questions there uh and captured the information that we needed to. Obviously, in terms of improving the system immediately, obviously some of the questions need to be spaced out quite a bit. It went from one question to the next almost immediately without any sort of natural pauses. So, that is something that we can add in relatively easily. We can add pauses between these questions by prompting it to do so and just obviously adding some further conditioning to make it a bit smoother. But, I think promptwise did everything that we needed to from the sort of very small initial set of test calls that we've done. But obviously what I want to prove is that this prompt isn't still the best prompt. This prompt hasn't been tested enough to really be putting this any in any production environment. And we're going to be putting this through reliable to really stress test it. So jumping into reliable. This right here is the reliable dashboard. This is everything that we're going to be using to really stress test our voice agents. Really pin them to the ground, hammer them, and really figure out what we need to do to improve them. So this right here is the main dashboard. The way that the reliable works is that this plugs directly into at the moment retail and vapy. So this sports retail and vapy. So we can directly just import our Vappy agent. So what I can do is jump up into the agents right up here in my workspace. Hit create new agent. And you're going to see this pop up right here. This is going to be allowing us to add our agent in. And this is obviously, as you can see here, AI provider. As you can see here, AI provider, Vappy, or retail. So we're going to select Vapy, of course. Heading back to Vapy, we need our API key and assistant ID to connect this agent to it. So, what we're going to do is go ahead and jump into our API key section here. We're going to copy our private key and we're going to throw that into right here. We then need our assistant ID. That's going to come from the top here. We're going to then put that in there. And that is going to be able to connect directly into our agent. Before we do this, I do want to connect this to a phone number. So, what I'm going to do is click create new phone number. I'm going to get a free number from Vappy because why not? Free number. We're going to hit create with just a random area code that I've given it there. I'm just going to activate this number. I'm going to use this to do all of our test calls. So, I just purchased this phone number through Vappy. So, Appy does allow you to actually purchase some numbers from them for free. Got an account with some credits in it. So, cool. We've just got a number. I'm just going to name this the liable number. Doesn't really matter. Whatever you want, obviously. Hit save. We're going to jump into the inbound settings and we're going to select our assistant that we've just created. And that just allows it. So, this number will be connected to our agent obviously. So, we'll hit save. And then what we can do, jump back to reliable. We're going to hit next. And that is going to import our agent directly. And there we go. So, we can see we got our phone number that we just purchased and connected to our agent that has been pulled through as well. And we get our prompt that has come through as well. So awesome. That's our prompt on Vappy that's pulled through. Selected inbound. It's an inbound agent. We're going to hit create. And that is going to import our entire Vappy agent into reliable. And this is going to allow us to start to really test the system. So the first thing to do once we've imported our agent here, you can see the prompt is all uh added here. You can actually sync your prompt. So best way to go about this for adding stuff to the prompt. Heading back to Vapy, we can just build on our agent here and just do everything as we were. And when we've got a new prompt, we can just hit sync prompt and it will just update the new prompt directly into here. But what I'm going to jump into doing is I'm going to jump into test cases. So test cases is what we're going to be using to evaluate the performance of our agent. So the the way this works is that ultimately it's going to look at our prompt and it is going to determine what are all the little small bits of actions that need to occur in order for us to have our prompt successfully followed. And so we're going to sort of split our prompt up into all these different test cases. And we're going to evaluate every single conversation that happens with our agent against every single one of these test cases. And so we're going to hit generate test cases. Essentially, we're using AI to do this. AI to look at the AI prompts. So for this particular prompt, I am going to add 20 test cases. The way that we can sort of assess how many test cases we need. Looking at our conversational flow here, how many sort of different steps that we have? So we've obviously got three here. So 3 6 9 12. We've probably got about, you know, 20 different steps here that we want to follow through with. obviously including the first message and the different flows and and the yes and no pathways here. Obviously, buying, renting, and selling. So, we're going to add in 20 different test cases. We're going to hit generate test cases. This is going to obviously go ahead and start to look at our prompts and determine 20 different steps that we can evaluate against our agent. So, there we go. Those test cases have just generated and we can see here all of this has just generated for our particular prompts. And once again, like I said, this is looking at a prompt and it's sort of just splitting it up into multiple different sort of parts of the prompt. So obviously, as you can see here, we've got 20 total test cases. First one that's popped up is introduce as Emily. So if I click the edit button, we can see right here, the AI agent must introduce herself as Emily from Inflate Real Estate Services at the beginning of the interaction. This sets the context for the call and establishes the agent's persona as defined in the prompt. Consistently using this identity is crucial for maintaining brand consistency and a professional tone. So awesome. That's pretty accurate. Obviously, we prompted it to to have, you know, a part of this and it's expanded upon that and it's obviously put out the importance of it as well. So we want to make sure that we're introducing it accurately every single time. You'll see here test case priorities. So we can set different priority levels for different test cases. Obviously not everything about our agent is going to be the most important thing ever. Obviously in the case of introducing as Emily it's not absolutely critical. It's not business critical if this does fail. But obviously we do want to make sure that it's pretty consistent every time at doing that as it's the first thing that it does. We don't want that to to get it wrong. For the test case priority these are all assigned using AI once again. So obviously we can see here quite a few determined as high. Some are determined as critical. Some are also determined as medium. We've got one here that's determined as low. So obviously you can click into this and determine if it is either less or more important for that particular test case. But either way, this is a good starting point obviously if you want to get started pretty quickly. But I would always recommend going through every single one of these test cases really looking through and making sure that you're covering all the different aspects of your particular agent as this is something that you just have to set up once and then it's obviously going to be very very useful as we go on. So for some of the other test cases that we have here, we've obvious got buying, ask property of interest. We can jump into this. When a caller indicates they are interested in buying, the agent's first question must be to ask which specific property they're interested in. This is the designated starting point for the buying pathway and is essential for guarding the conversation correctly. The agent should not proceed to other questions in this flow until the information is solicited. So cool. Uh that's pretty accurate as to our prompt. Not going to go through every single one of these as there is quite a few here. But we can see here with low when reading a street address back to the caller if the street number is not a common whole number like 100 or 1100 the agent must produce each each digit individually. So this once again is just adding on top of our conditioning of the prompt readout and just making sure that it's done properly. So we've got plenty of test cases here to obviously assess against many different areas of the agent across all the different pathways of of buying. As you can see renting here, selling here and then obviously some of the notes we got at the bottom. So also we can add in some test cases manually if there's anything that's missing or you want to add anything on top. So once this is generated what we'll jump into is metrics. The next page here is metrics. So this is something that will get evaluated against all of the tests that you do. So these are a little bit of a much deeper dive than obviously if you're going to be manually testing. You're not going to be looking at or assessing any of these particular metrics. This just helps to give you a much more well-rounded understanding as to how well the agent is performing. Not so much on the prompt side of things, but a lot of it is to do with the quality of the voice. A lot of it is to do with the quality of uh the ability to do turn taking. So you're going to ask a question. It's going to provide a question back. It's going to look at the conversation just assess how smooth was it conversationally. And a lot of these metrics are going to provide you with some insights as to if it needs to be improved or not. So jumping in to the main part of reliable and the the main purpose of using this particular platform is to stress test and simulate real conversations with our Vappy agent. And so what we can do is jump into the simulation section right here. And what we have are personas. So personas is what we're going to be using to sort of mimic a particular audience or a particular customer that's going to be calling this particular AI agent. And so this is going to really help shape a lot of the unique aspects and and things that we can do with automated testing. So when I'm calling the system, I'm obviously going to be calling in with one Australian accent. I'm going to be calling in with one type of personality. It's very hard for me to simulate all of these different people that are going to be calling in. Uh and this is what's going to help out with that. So what we can do, as you can see here, I've got a bunch of personalities that I've already created within my particular workspace. I've got quite a few different Australian accents, quite a few different American accents. Got Indian as well. We can see here we've got a 73y old, a 23y old, 38-year-old, 29year-old. Many different types of individuals that we've created here. And we can use these to shape our callers that are going to be calling into our particular agent. So, we're going to go ahead and hit the generate personalities button. We're going to jump into our stereotype description. So, this is going to be once again, this is not specific to the scenario. So, this is not going to be what they're calling in about or or what their problem is. This is just going to be about them. And so what I'll type is create a persona around an elderly male who who is very frustrated with life and has no patience. So we've added in sort of a persona for them. It's somebody who's old. They're frustrated with life. They've got no patience. Obviously we want to use this to stress test our system and we really want to hit it hard by giving it someone that's probably going to be a little bit harder to deal with. So we're going to generate this once and we're going to hit generate personality. So there we go. Here's our generated persona. We've got Stanley Stan Miller. It's even created a a middle name for him. Stan Stanley's 79year-old retired mechanic and Korean War veteran. I can jump into the rest of that uh just in a minute as well. Gender is male, accent is American. We jump into the edit button. We can just scroll down a bit and we can see the description as well as the prompt that's actually going to be using in its response. We can set we can see here description. Stanley is a 79-year-old retired mechanic and Korean War veteran from a working-class neighborhood in Chicago. His life has been hard blah blah blah. Obviously, there's a lot of context in here to really just generate that persona. You see, he's stubborn, independent, fiercely independent, stubborn, self-reliant, communicates through size and brutal blunt observations. So, this is awesome. This is going to be able to really stress test our system quite a lot and really obviously push it and break it and try to see if we can get our agent to trip up. And this is going to be a pretty perfect example to use. The prompt is going to be speaking in short clip sentences. The tone should be perpetually exasperated and gruff. So, awesome. It's just going to use all of this to to really contribute to a difficult conversation that we obviously want to use. This is going to be probably a more tough conversation. You can obviously generate all different types of ones. We've got right here. Moreno, Donald, Latasha Williams, all different types of people that we can have calling in. You don't have to go this crazy, this hard for the test, but obviously we're going to go ahead and run this and just see what comes out of it. Jumping in to the run section next. What we're going to do is actually start to create the scenarios that going to be called in for. So, we've got our persona. Now, we've got Stanley and he's obviously going to be calling in about a particular scenario. And so, that could be he wants to buy a property, he wants to rent a property, he wants to sell a property. We're just going to let the AI generate the particular scenario that Stanley's going to call in about. So we can see here, hit generate tests. We're going to select Stanley from the list right here. And we're going to select our test cases. We can see we got all of our test cases. We want to evaluate it against all of them. So we're going to keep them all there. And let's just generate one for the sake of simplicity. And we're just going to hit generate. So here we go. That just generated. I haven't seen it yet. Let's hit edit. And let's have a look at what it's created. So you are Stanley Miller. Your kids are forcing you to sell your house you've lived in for 50 years because it's getting too hard for you to manage. calling in the agency reluctant and annoyed by the whole process. When the agent Emily answers, you state your purpose without any uh pleasantry. So, you can see here, obviously, this is awesome. This is following through with what we wanted it to. Stanley's going to be pretty mad calling in. He's obviously looking to sell his house. And it's just got a bunch of fluff here that we're going to try to trip up our agent with. And obviously, the whole purpose here is to trip up our agent. We want our agent to be able to handle something like this. This is, as crazy as it is, I'm sure it's not even that unrealistic to to have a situation like this. call in for one of these people and we want to be able to handle it. And so we're going to go ahead and select this run right here. We're going to hit run and we're just going to name this. We'll just call this our first simulated run. We're going to run this once, but we can run this, you know, between one or five times at a go. Uh we're just going to hit create run. So I've just gone ahead and actually generated four additional simulated scenarios that I'm going to be testing against. I think a really big part of this platform is that we can actually run such a large volume of test calls. Obviously, giving your agent a call and a talk just a few times is is not really testing it. it's not pushing the boundaries of the agent. You're not really looking for any issues and so we need to run it at scale, really start to find where its flaws are and all the areas that need to be improved. So, I've got a total of five different simulated scenarios here. We're going to select all of these and we're going to run it in one run. So, we're going to go ahead and just run this once. I'm just going to call this five scenario test. We got five different scenarios, five scenarios test. We'll hit run and we're going to go ahead and see what kind of results we get back from this. So, jumping in to the results page right here, just under the run tab. Once we've run our tests, we're going to be able to jump into the results page. And this is just going to show us all of the results from all of our tests, and it is going to score our tests, and it is going to provide us with all of the insights of what failed, what didn't work, give us some suggestions for how to improve our agent. And we can see right here, it is still running, although we can see at the moment it is scored at 61 12%. So, the way that this works is that our agent is going to get scored against all of our test cases. And depending on the priority of the test case, if it's critical or if it's low, they're all weighted differently. So if a critical test case fails, then that's obviously a very bad thing. And our score is going to be reflectant of that. So 60% sort of expected for a system that hasn't been tested at all. But what I'm going to do is jump into this and have a look through a lot of the test cases and a lot of the data to see where we can improve. So we can see here it's actually just dropped down to 51%. One of the tests might have just concluded. So we can see here this is our sort of main dashboard. We've got an average score 50%. You know, not not amazing. This is not great at all. Obviously, it's in the poor section right here. At a minimum, we're obviously looking to push past 70% for a system. Just at a minimum, we really want to be 70% above. Average latency is excellent. Obviously, that's operating quite well, which is nice. We're using a GPT40 model and we're using a very short prompt, so latency really isn't an issue here. Words per minute. We're looking at 119 words per minute. This is okay. As it says here, okay, we've classified that as okay. you'll listen that the voice that we've chosen, particularly that vapy female voice, is not the quickest voice. It's not terrible. It's definitely usable. Although, for this particular instance, maybe we want to make it a little bit quicker. If I just scroll down a bit right here, you're going to see all of our test cases and all of the test cases pretty much seen here. And we can see if it failed, if it didn't fail, we can see here if it worked every time, if it didn't work every time, and this is going to allow us to to jump in and see where the system needs to be improved. And we can see here quite a few different areas that it might need to be improved on. So, we can see here here are our tests. We've still got two running. I'll just hit refresh to see if they've now concluded. Otherwise, I'll just jump into some of these conversations to really showcase exactly what's going on. Okay, so all of the tests are now completed. You can see all of the conversations have occurred here. For the most part, most of them go for about 2 minutes, almost up to 3 minutes, if 3 and 1/2 minutes here. This one was only 40 seconds. Something's gone on in that call. Otherwise, I'm going to jump into this one right here where it's at 28%. So, quite a few things went wrong for this particular call here. And we can jump into the test cases and have a look at what exactly went wrong. And before I dive into every single one of these test cases to look at what needs to be improved, we can see here this is the conversation between the two AIs. Obviously, this is two AI speaking to each other. We're going to go ahead and listen to this conversation.
>> Hi, this is Emily. I'm an AI agent from Insflate Real Estate Services. How can we help you today?
>> I'm looking into that new place on Oak Street.
>> Great. Which of our properties on Oak Street were you interested in? for crying out loud. Just the new one. Don't you know your own listings?
So, there you go. We've obviously got two AI speaking to each other. Pretty awesome there. Both of these voices sound really good. Obviously, the test agent and the main agent are both quite realistic sounding and using quite high quality voices. And we can obviously skip through it and look through all the conversation here, as well as view the transcript of the conversation to really skip through and see what's going on and debug our agent to make it better. So the recording is obviously for looking at and assessing a lot of the quality stuff and and how it's speaking, what it's saying and just getting a lot of the tone inflections, everything about that uh through the recording. And then the transcript is obviously our prompt engineering side of it. So we can see here are our test cases and there's quite a few that have failed here. So so the way that we classify a particular test case is that it either failed, it was successful, or it was not relevant. And we can see here if I click onto this success, failed or not relevant. In this case, we've got quite a few that failed and a few that succeeded as well. What I'll jump into is this second one right here. the address being captured. So this is something that we've obviously prompted it to do. If we jump into our prompt, capturing the address and have a look into our flow here is obviously a part of our flow. We want to capture the address from the user depending on their particular inquiry. So jump into here. I'm looking into that new place on Oak Street. So we don't actually know if they're looking at uh renting or buying in this case cuz they haven't actually said that. But obviously that's something that we want to add into our conversation immediately. That's something that we haven't really added in. Great. Which of our properties on Oak Street were you interested in? We can see here it didn't provide us with the the address. who didn't actually capture the address. We can see here if a caller is on the buying pathway, accepts the offer of assistance for selling their current property, the agent must ask them for their current address. Capturing the address is the next logical step in initiating the selling process for them. Uh it's necessary to proceed with the selling assistance. So we can see here caller showed interest in selling assistance by asking about the cost. According to the prompt, the immediate next step should be been asking for that address. Instead, the agent first asks, would you like to book a walkthrough for the Oak Street property? And that was not what we prompted it to do. So we jump back into our workflow right here. We wanted to obviously ask them if they had a property they wanted to sell and and if so, we can say great. Could you please provide us with your current address and if that's the case, we want to obviously then book them through into the walkthrough. So, it's saying that this question right here was skipped. If we jump back into reliable right here, we can scroll down and get to that part of the conversation. Would you like assistance in selling this property? Jump back to here. That's this question right here. How much is that going to cost me? It looks like after just saying that, it just said we can discuss it in more detail. Would you like to book a walkthrough for the Oak Street property? So, I know this is such a small thing to look at, but this is obviously just one of the the parts that we need to be looking into very deeply to build out a production ready system. I know that it's just a very small thing, but these things add up. And obviously, this is not something that I was able to pick up on on my couple of small tests. And if we jumping back into the workflow, we obviously wanted to capture their current address. If we jump back into reliable, that didn't happen. This failed and told us about it. If we jump, you know, obviously lower, it said that it asked about the current address after this. So, obviously, that's not what we wanted. It failed at doing it. it picked it up and it told us about it. So, one thing that we can do to obviously go ahead and mitigate this and fix this is jump back into our prompt to fix it. So, what we could do is add in some prompting to say that we must follow each step step by step and that we must follow each question in the logical order. So, we haven't prompted it to do that at all. Clearly needs it cuz it failed at doing that in that particular case that we ran through. And this is the process of prompt engineering. Jumping into another test that we ran here, we can see we've got a a few other fails. Not as many fails as the other test case. You can see that there was quite a few that succeeded which was great and obviously some were not relevant as well. Like I said there's going to be many different cases especially in ours we've got three different pathways for buying, selling and renting. Obviously all of the selling and renting pathways not going to be relevant for this particular conversation. We can see here that it was obviously indicated that it wasn't relevant. So if I scroll up a little bit we can jump into the the failed test cases and have a look at the reasons as for why they failed. We can see here the confirm name spellings. We can jump into this and see what happened. We can see I'm looking for a place for my grandson. One of your rentals. I don't know which one. And we can have a look through here. Great. Let's start with gathering your full name. Could you please spell it out for us? They said, "Why do you need my name? It's for my grandson." Fine. It's John. Thanks, John. Last name as well. Thank you, John Smith. Use your timeline. So, we can see here in the conversation we're looking at. Hi, Emily. I'm looking for a place for my grandson. One of your rentals. I don't know which one. They said, "Let's start with gathering your first your full name. Could you please spell it out for us?" Then the agent said, "Why do you need my name? It's for my grandson." Fine. It's John. Thanks, John. Do you mind providing your last name as well? and they started complaining a little bit more to capture their name. But ultimately, as you can see here, it asked to capture their name, captured their name from them, although it didn't repeat it back to them. So, it says here, "Thank you, John Smith." Although it didn't repeat the name letter by letter. And we can see here after capturing the name John Smith, the agent said, "Thank you, John Smith." It did not repeat the name letter by letter as required by the prompt for the renting pathway. So, it failed. Ultimately, it didn't do what it should have. It didn't repeat it back to them. And that's obviously a fail right there. We need to jump back into the prompt and fix it. And once again, this is something that we didn't pick up on from our manual testing. This is not something that we could have picked up on just because there's so many different scenarios and situations that can trip up our agent to to do this. So clearly our test agent here was was capable of tripping it up by talking about their grandson. Our agent got confused and uh it didn't repeat the the name back to them. And so exactly what we were looking for, we wanted it to fail and it did. Here is another test that we ran through. This was the one that went for about 40 seconds. This didn't go for very long. And we can have a look at the some of the test cases here as well. So we got a failed test case buying offer selling assistance. So if the caller in the buying pathway confirms they have a property to sell first, the agent must then ask if they would like assistance from us to sell it for them. So in this case, I believe what happened is that the individual called through. They said they wanted to purchase a property, but they've got the current one to sell first. And our agent didn't go ahead and and jump to that. It just decided to say, "Great. Which one of our properties were you interested in?" It said, "Just get to the point. I haven't decided yet. No worries. decide to move forward, feel free to give us a call again with the details and have a great day. So, our agent pretty much shut down the customer right away. Our agent just said, you know, no worries, whatever, go away. Obviously, that's not fantastic. You know, our agent isn't prompted necessarily just to hang up like that. Agent is prompted just to go through the the script that we have and go through the process. And it seemed like it just gave up on that, which is a pretty critical flaw of our agent right here. That's that's absolutely terrible. We can see here in the failed test case that that's exactly what it picked up on. In this case, this should have been probably even more of a critical particular issue. In this case, that's something we need to fix. We need to make make sure that it follows through with each of our questions. You know, obviously from a prompting perspective, maybe we could add in some conditioning so that if somebody does have a question right away that we can answer that question for them rather than going through our pathway. But even if it didn't even go through our pathway in the first place and so that's obviously a pretty big issue and once again something that you just wouldn't have picked up on if you just called it manually yourself. You couldn't have. You need to run a very large amount of automated tests under all of these different scenarios. And this is just one scenario. I mean, there are so many other scenarios that we can hit this system with that's going to find so many more flaws, which is why it takes so long to actually build these systems. And this is just an incredible way that you can obviously pick up on these issues and and fix the agents. Another key part of reliable is live monitoring. So, at the moment, we're able to run simulated conversations with our agents and we're able to compare that against all of our test cases to assess its performance. But we can essentially do the exact same thing with live calls. So we can essentially look at all of the calls that are coming into the system once it's deployed and use all of the test cases that we've already created to assess them as well. The exact same way that we're assessing these simulated calls, we can assess real calls. And so the way that we can do this is really quite easy. We can go to the agent settings, head to our settings, and we can enable the call monitoring. What this will do is it will take a web hook, an end of call report web hook, and add that to Vappy from our end. So we can start to look at the calls that are coming through. That'll get sent to the reliable system right here. Hit refresh. That should remove this. And all of the real calls will start to come through. And then we can assess the real calls against their test cases. And this is absolutely critical for us particularly as an agency that builds these systems for real clients to be able to assess its performance after it's deployed is is absolutely fundamental to understanding how well it performs. Especially for a lot of the critical test cases that we're assessing. If we have a home services example that has an emergency pathway, that's absolutely critical. We want to make sure that it does that every single time that we can. And if we can set that as a critical test case to follow through with that particular set of instructions for an emergency. And if it doesn't and if it fails, we can get notified right away of that. And so that is what live monitoring is for. So if I go ahead and actually call this agent over the phone myself, we're going to see this populate right here.
>> How can we help you today?
>> Hey Emily, I was looking at getting a rental property property. Yeah, we're looking at the 123 Main Street property.
>> Could you please out your phone?
>> No, I'm not going to spell out my name. I don't want to give out that information at the moment.
>> Would you like to book a walk through property?
>> Um, a walk through? No, not really. support if it's feel free to run. Have a great day.
So, there we go. I just gave it a call. I was pretty harsh and tried to break it. And you can see there it kind of just gave up on me. So, that's obviously something we want to fix for and we want to obviously prompt to fix. We can see that the call populated on the screen right here. We can see the call is being evaluated now against all of our test cases. And hopefully that comes up as being a critical failure because it obviously didn't go through what we wanted it to. You can see that it kind of just gave up on the questions that we were were giving it. So, we'll see what this test case evaluation gives us. There we have it. Our call has just been evaluated and we got a score of 48%. Not amazing. Obviously, that is not great at all. We obviously want to be pushing once again 70 plus% at a minimum. All these agents just from my own perspective, but we can see here we've had three failures. The renting offer and the renting ask move uh in timeline. So, two parts of the renting flow failed. I would agree it's I believe it missed out on a few questions just from the conversation that I had. We can listen to the recording here as well. Look at the transcript that I just had there. works here. The agent did ask about booking a walkthrough, but it did so without first asking about the movein timeline as required by the script. The objective states this question must be asked after the timeline question, and the agent skipped that step. Absolutely, I agree. It did do that and that is something that we need to prompt to fix. The agent did not ask about the caller's move in timeline after the caller refused to provide their name and the renting pathway went directly to asking about a walkthrough. So, very similar parts here. I believe this actually mentioned both parts of the the script that it skipped on and yeah, exactly. It didn't do what it should have. picked up on that and we're able to obviously then use that data to fix our agent. I hope this video has been a helpful insight into really looking into how to build a production ready AI voice agent. Obviously, we can go ahead and build our prompt on Vappy and no matter what we do in terms of structuring it and trying to use markdown formatting to make it better and better. If we call it a couple of times, it's going to sound great. Although, when we start to run automated testing, we start to stress test it. We get Stanley in our case to really hammer our system. we realize it's got quite a lot of flaws and that we need to fix it quite extensively and we just wouldn't have known any of this if we didn't have a system to test it. So, it's pretty important. If you do want to check out Reliable, once again, I've got a link in the description if you do want to check it