Transcription
As the title probably suggests, I just built another app. If you're new here, welcome to the channel. My name is Chris, and I build productivity apps. I usually focus on these two productivity apps, but today I wanted to talk about a new one I'm working on called Lily.
I actually wasn't planning on making another app, but I did, and I wanted to tell you guys about it. So I've only been working on the app for a week, but there are two reasons I'm making this video. The first is I've actually learned a ton of things in the last week that I really wanted to share with you guys. The second is, even though this app is just getting started—we're in the early stages—I have a feeling that it's going to become a little bit more big, bigger, and serious in the future. And if that ends up being true, I really wanted to capture the early stages of building this app so you guys can see how messy and unplanned things really were at the beginning. So those are the two reasons. Okay, so what does this app do, and why did I build it?
During the day, I'm in a lot of video meetings, and sometimes during these meetings, I end up zoning out. And by the time I realize what's happening, I have no idea what's going on in the meeting. Whenever I realize I miss something or I'm unsure about it, I usually DM someone live during the meeting while it's going on, asking them, "Hey, what did I miss?" or "Hey, what did Sarah say her role was?" Then I had this thought: What if I could build a bot, maybe that lives on my app that I can take with me during the meetings, and it can be that person that I can DM and ask, "Hey, what did I just miss? What was Sarah's role?" Instead of having to DM a person that was on the meeting.
So I thought to myself, "How hard can this be? Maybe I can just code it up during the weekend." Something else I've been interested in is testing local AI models. So these are models that live on your device, so they're not being sent to a server; they're completely private and they're completely free—something I've never tried before, but I've been very interested in. So I thought, maybe I can pull off this transcription bot thing using local models. That's what I decided to build over the weekend. It was a real-time meeting helper bot, and it would use local models, so it would run privately and for free on my phone. And here's actually what the final result ended up looking like. So it's an app that you can take with you; you can put it right next to your computer, and it will listen to the meeting, and anytime you have questions, or you zoned out, or anything, you can ask it, "Hey, what did I miss? What did Sarah say her role was?" Right there. So that's the final result, but let me show you guys how I got there.
First, I wanted to mention that I think AI wrote like 90% of this code—that's how I was able to move so quickly. If you guys are interested, comment below; maybe I can make a tutorial in the future on how I use AI for coding and how I'm able to move so quickly with it. But for now, just know that's what I did. So the first thing I wanted to do is explore these local AI models. If you're not familiar with AI or how the OpenAI or the cloud models work, basically what people typically do is they send requests to a server like OpenAI or Claude, and then they get a response back. So, for example, I can feed in the meeting transcript and then a question of, "What did Sarah say her role was?" and the OpenAI LLM will respond back saying, "Sarah's role was this," based on the transcript. There's usually an API; it's a third-party server in the cloud, and you've got to pay for every single one of these requests. The local model is the exact same thing, but instead of hitting up a third-party API, it's actually on your device itself. So when you ask a question, it's going to hit the LLM up that lives on your device and return instantly. The big benefits here are: it's completely private because it lives on your device; it works totally offline; and the biggest benefit is it's completely free. So you download the model, it lives on your device, and you feed in questions just like you do for these third-party API-based models.
There are two components to the app: there's this transcription piece, so I need a local AI that will actually transcribe the audio into text; and then a second LLM, which I can feed in the transcript and ask questions. So this is the chatbot. So the first thing I did was tackle the transcription piece. So I used an open-source local model called Whisper. So you feed an audio; it's going to transcribe it, and it'll spit the text back out. After a few minutes—probably like 20 or 30 minutes—I was able to get this running, and it was really cool because it was completely free and it was running decently on my device. And when I say decent, I mean decent in terms of accuracy—like it was able to pick up a lot of the words that I was saying. So once I finished that, the next thing I did was tackle the chatbot. So after a few minutes, I got this running as well, using a local model called Llama, and was fully running on my device.
Something cool I also learned in the process was I got these models from this service called Hugging Face. I'd heard about Hugging Face in the past, but I didn't really understand what it was. So the way Hugging Face works is it's a huge database of these local models; pretty sure they have over like a million models that you can download and use on your device. So what apps like mine can do is have a list of models to choose from, and when a user clicks it, it'll download from Hugging Face to the user's device so they can use it, which is really great. So that way, if you have a higher-end device that has a lot more power, you can take advantage of these bigger models on Hugging Face that maybe other devices can't. So giving users this choice of model is really cool, and Hugging Face is something that enables that. You say you're not sponsored? Oh yeah, I'm not affiliated or sponsored by Hugging Face by the way; it was just a really cool service that I found.
So once we had both local models running—Whisper for the transcription and Llama for the chat—this is where the problem started coming up. First problem was that the chat was super slow. So if you've used ChatGPT before, you know that it's really quick; when you ask a question, you usually get a huge wall of text really fast. But with my chatbot, we were getting one word every 2 to 3 seconds. Imagine you're using ChatGPT and it took 2 to 3 seconds for each word to print out. That is horrible; that is not viable at all. So I quickly realized that the local models are really impacted by the device that it's running on. I also noticed that the same thing was happening with the Whisper transcript, too. Initially, it was running pretty decently; it was transcribing things. However, as the transcription ran longer—once you ran it for over 5 minutes—I noticed there was a bit of a delay; it started lagging a little bit, and it took a couple seconds for the words to come in after I said them. And when I dug more into it, it's because it uses the previous transcription to optimize future transcription, which means as the transcript's getting longer, there's more for it to continue to process, which is a problem because most of the meetings are at minimum 30 minutes. So I need this thing to be running for a very long time, and these performance issues were happening just when you ran them separately. Then when I combine the two—where there's a local transcription happening and I'm chatting with it at the same time—it was so bad. Instead of one word every 2 to 3 seconds, it was one word every 15 to 20 seconds. For Whisper, there was a massive delay with transcribing; so if I said something, it took about 30 seconds for the words to show up on the screen. Also, a side note: my phone was burning up while this was happening, so it was very clear that it was eating up all my phone resources as this app was running.
I am very confident there are ways for me to optimize both of these things so they're both running very performant on the device; I just didn't have time to do this. I was trying to finish this app in 48 hours. Maybe the local models are not feasible for something like this. So the next thing I did was move from a local model to these third-party API-based solutions. Instead of using Whisper for the transcription, I used a service called Deepgram, which is very expensive, but it promises better accuracy, and it even has this cool feature called speaker diarization, which is where it can identify who is talking in a transcript. For the chatbot, I switched from Llama to something called OpenRouter, and OpenRouter is just a service to connect to other LLM providers like OpenAI and Claude. And the reason I use OpenRouter instead of going directly to OpenAI or Claude is because it lets me switch out the LLM providers with one line of code. So there is a little bit of a fee, but in my opinion, during development, it's very worth it so I can test out a bunch of these different models without having to plug in their API. So that's a third-party API service I'm using for the chatbot.
So once both of these were implemented, I noticed a huge difference; the speed for the transcription and the chat was incredible, and they were both running so well on the device. I had the transcription running for an hour, and it was working flawlessly. So sadly, I decided for this app to work well, I probably had to use these third-party API-based models. I'm definitely going to revisit the local models in the future and see maybe I can optimize things so it works performant. For the sake of speed and trying to get this out there and usable in my hands, I decided that the third-party APIs were just the better solution for now.
So once I got the core functionality going—where the meeting is transcribed, and in real time you can ask questions to a chatbot about what am I missing, what is Sarah's role—once that was done, I decided to work on the designs. So I called in a favor, asked my girlfriend to help me, and we came up with a really cool solution where you can see the transcript and the chat at the same time, and it's really cool because you can actually drag in the center and change the height of each of these panels. I really like this design, and it feels really good on the device with haptics and the animations and everything. And in less than 48 hours, we had this prototype which I can put on my desk, and it can listen to meetings, and I can ask questions. Once Monday rolled around, I started using it during meetings, and it was actually really helpful. I did have to make some adjustments to improve the chat, but I was really impressed by what a basic prompt and a really cheap LLM could do.
Here's something cool I also wanted to share about building a good prompt that I learned while doing this. Typically, when you write a prompt for an AI, you just write out a set of instructions. A technique I learned about is you can actually format it in what's called XML. This is the format of what XML looks like; it has this title, description, type format. If you write it like this, I notice that the AI has a slightly higher chance of following your instructions correctly. And since I was using OpenRouter, which means I can switch out models with just one line of code, I was able to test a bunch of different models, and it was actually really cool to see how, as I used more expensive models, the performance really did improve. It was able to answer questions a lot better; it understood the context a lot better. And that's something I'm playing around with because, instead of just throwing the most expensive model at it, I want to see, "Okay, what's the cheapest model that I can get away with using that delivers really good results?" So that's something I'm still playing around with, but OpenRouter really helped with testing that out.
So I tweeted out a teaser that I'm building this app, and I was curious if anyone was interested in it, and I was actually surprised that quite a few people actually reached out saying that they thought this would be helpful during their meetings. I threw together a quick waitlist and landing page and actually got like 50 signups from that single tweet, and that signaled to me that maybe people, aside from myself, would actually find this useful, and I decided to take this a little more seriously. So over the next week, I fleshed the app out; I added some basic features like accounts; I even added the ability to have an organization because I plan on using this for work; I really wanted to be able to share meetings with the people that I work with, which I then ended up removing a few days later because I thought maybe this isn't actually that necessary for beta testing or for launch. And I even had this ability to generate notes about the meeting, so you can actually ask the chat, "Hey, generate notes for me," and it'll type a bunch of notes, and then you can actually view it in the app, which is really cool.
So all of this stuff happened on the feature side; a lot of things happened on the branding and the naming side, too. So if you've been following along, you know that I love to name my apps after my pets, so that's why we have Ellie and Luna, but I ran out of pets, so I had to get creative with the name. So the name changed a couple of times; it went from Juno to Stoa, back to Juno, and eventually the final iteration is Lily. I did a ton of branding and logo exploration, too, just in this short week. We tested some concepts from lily pads to frogs to ducks, and eventually we landed on Lily the Ghost. And the main reason I chose the ghost as a mascot is I wanted this app to be something that kind of lives in the background and is just helpful until you need it. Originally, I was like, maybe a fly, because flies are kind of on the wall and they're listening, but I don't think anyone wants a fly on their app home screen. So after a little more research, I realized maybe a ghost is a good idea because ghosts are kind of living in the background; they're kind of ephemeral, and that's kind of what I want this app to embody. And so that's how I came up with Lily the Ghost.
I think in my next video I'll actually probably do a branding and logo illustration video because I already have some really cool illustrations and empty states that I'm playing around with right now, and I've gotten a lot of comments asking how I've done this. So hold off on sharing more for now, but maybe that'll be my next video. So that's where the app's at. It's been a little over a week now that I've been working on this, and it's kind of crazy how quick some of these things move, but that's the main thing I wanted to show in this video: just how quickly things evolve from a simple exploration. I mean, we went from this to this in the span of a week, which is kind of crazy. But I wanted to show you guys what's possible and what happens when you just experiment and iterate and just try different things out. You never know where it's going to take you. And I have no idea where this app is going, but I do have a feeling that it's going to end up being something a little bit bigger than it is right now. And if that ends up being true, I really want to capture this early stage so you guys can see how unserious and unplanned and messy things actually were at the beginning. And if I can do something like this in a week, you guys can definitely do it, too.
If you enjoy this kind of content, check out my Instagram and TikTok; I post almost every other day about building productivity apps. And obviously, if you like this content, don't forget to subscribe. I'll leave a link to the waitlist for Lily down below; I'll be sending invites out there very soon. But thank you guys so much for watching, and I'll see you guys in the next video.