Transcription
All right, guys!
So, super excited about the video today. I'm going to show you how you can clone any personality that you look up to and turn them into your life coaches, ghostwriters, and business advisors using Crew AI.
Crew AI is a multi-agent framework. Whether you want to have Mark Andreessen as your co-founder or David Goggins as your life coach, all of this is going to be possible by scraping videos from YouTube or Instagram. I'm going to show you how you can actually have those people write articles or give you feedback on your marketing campaigns.
I've already built it for an agency that specializes in chatbots for content creators, and it's worked wonders. I'm very excited about this video! I'm going to walk you through all the fundamentals of Crew AI and how you can implement this in your business.
All right, so before we dive into the specifics of how we're going to build a crew, let's just stop for a minute on what Crew AI is.
It's an open-source platform that has been making a lot of noise recently because I think 50% of the Fortune 200s have been adopting and implementing Crew. They just announced a partnership with IBM. You can see all the logos of companies that have been implementing Crew.
The way it works is essentially that for every agent, you're going to give them different tasks that they're going to have to accomplish. Together, the sum of all these tasks is supposedly going to help you accomplish your goal.
In the example that I'm going to walk you through, if you want to have David Goggins turn into your gym buddy, you're going to have different agents. One is going to scrape videos about David Goggins, another one is going to push all that information into a database, and then you're going to have an army of other agents trying to retrieve information depending on your needs.
So, if you actually want to generate your own training plan or if you want to have motivational quotes, you're going to have different agents that are going to get called. This is the power of Crew AI; it's super versatile and performing really well. You can see here that everyone is going crazy about it, and I just cannot stop talking about it either.
There is a bit of a learning curve, but it's for sure a lot easier to implement and adopt than LGraph or LangChain. I'm going to walk you through the example, so let's go!
All right, so this is how the crew works. We have, as an input, the YouTube handle, and then we have an army of agents who are all specialized and will execute different tasks.
In this case, the first agent scrapes the video, the second agent adds it to the RAG database, then we have a third agent who's inferring values from the content in the database, a fourth one who's finding the business info, and finally, we have an agent who's converting all of that information into a report about the creator. This is the report that you can use for the different use cases that we discussed.
What's important to see here is that every agent has access to different tools. In this case, the agent who's scraping the video has a custom tool that allows him to scrape YouTube and find the most relevant videos. Then there's another agent who's capable of pushing the information from YouTube into the RAG database.
We also have a third tool called Query Database, which I will show you, that allows the different agents to query and ask a list of questions to the database. Finally, we have a last tool that allows the agent who's creating the report to tap into templates and examples that we already have to help him figure out how to create a report that is going to be helpful later on.
So, the final output is the creator report, and as I said, the input in this case will be the YouTube handle.
Now, let's dive into the code.
Okay, so now we're on Cursor. I'm going to show you all the code, and we're going to go through it, but I just want to quickly show you how the agents work.
I'm going to start with the input, and then I'm going to show you the kind of output that you might expect. Here, I'm actually going to just launch the main script that we're going to go over.
In this case, it's going to ask me for the YouTube channel handle that we just saw: Mind Motivation Coaching. Now you can see that the crew started.
Here, the scraper agent is getting the 10 most relevant videos for the channel. Now, another agent is adding the information to the database that we're going to be able to query, so we can ask questions about the content of those videos.
Based on all of the information that we have about David Goggins, it's going to turn this into a narrative. So, if you want to build a bot where you just want to be able to send WhatsApps to David Goggins to get his feedback, that's essentially how the tool works.
Those videos are focusing on extracting biographical events of David Goggins, but you could also prompt and request the agents to extract information about his business. If you want to learn about how to build a similar business, he probably has more videos where he talks about that.
Then, you can also replicate that across more channels. Maybe you want to extract information from the news or Instagram, so you could actually duplicate this crew and just extract information not from YouTube but from Instagram. You get the idea.
Now, we're going to go over to the code.
All right, so this is the script where I'm orchestrating all the different agents together. Essentially, what you have here are all the agents that I'm initializing.
We have the database manager agent, the scraper agent, and as I said, the scraper agent is actually going to scrape YouTube. I'm going to walk you through everything, but essentially, this agent has access to a tool called "Fetch Relevant Videos from YouTube Channel."
Then you have another agent who's adding the information to the vector database. You have a general research agent who's going to query and ask questions to the database, and so on and so forth.
You essentially have to initialize all the different tasks that I just mentioned. Every agent can have one or several tasks. Generally, the best practice is to create as many agents as you have tasks, as it allows you to have more customization and make sure that every agent is best suited to achieve these tasks.
That's essentially the configuration of the crew. Finally, here you have the crew function that we are creating, where you consolidate the agents, the tasks, and the process.
We're setting it up to process sequentially, which means that every task will be run after the other, but you can also run it asynchronously. You can also have a hierarchy, which essentially means that you have an agent who oversees other agents. This allows you to replicate the potential hierarchy structures that you have within your own company.
That's how the first script works. Now, let's dive into the agents.
As I said, there are different agents that we need for this crew. As you saw in the main script, we have a database manager agent, a scraper agent, and a vector DB agent. Here, you have to configure and specify a little bit about their goals and backstory.
When you have an agent, it's super important to define their thought process. This will serve as the foundation of how these agents will think. In this case, for the first few agents, it's quite straightforward because they have a very narrow scope of work.
But as we go on and look at the general research agent, who's specialized in querying the data, this is where it gets super important to be as specific as possible.
In this case, I'm creating an agent called the research agent. The goal of this agent is to consolidate information about the creator, and the backstory of this agent is that she is a doctor. So, you kind of try to give credentials to build up the credibility of this agent so that the agent can infer specific things from the data.
You help the agent evaluate how to define whether she accomplished her task.
Now, if we go over the set of tasks, this is a bit more text-heavy, but essentially, like I said, every agent has a specific task. In my case, I have the database manager agent, who's going to accomplish the task of resetting the vector database.
I'm doing this because I want the crew to reset so that it doesn't retrieve information from different content creators. This is helpful if you want to use the tools on different creators and want to consolidate all the information.
Then you have the task of scraping the YouTube channel. Here, as you can see, the task is to fetch the 10 most relevant videos. This is the scraper agent who's going to be responsible for that.
Processing the video is essentially adding all the information from YouTube into the vector database. Then we have a set of tasks where we explain how to find the achievements.
In the case of David Goggins, we have the general research agent that I just showed you, who's actually going to accomplish this task. Here, you explain and be super specific about how the agent needs to accomplish this task.
That's why agents are very powerful if you know what you want to automate. In this case, there's a lot of trial and error to come up with a final version, but the goal is to give a very clear-cut methodology on which questions to ask the database and how to analyze the information.
So, the analysis protocol, what kind of information does the agent need, what are the quality controls—you need to set up constraints. Finally, evaluation criteria.
In this case, because LLMs are very likely to hallucinate, the goal is to reduce the likelihood that they do. You want to give hallucination evaluation criteria to ensure that they come up with relevant and fact-based information.
For every agent, you essentially want to ask the database for information and then structure it, adding it to a file in a very structured manner. I'm going to use Pantic objects, and essentially what it does is use unstructured data from the database.
For instance, different anecdotes about the author. By defining in the task that we want to use specific achievements, let's say that David Goggins became a Navy SEAL. In the achievements, you're basically going to ask to turn this anecdote from joining the Navy SEALs into achievement object number one.
This is where it gets very powerful because you can do this across different formats. If you want to parse PDFs or extract, for instance, income or revenues, you would ask the agent to scrape all the PDFs.
Then, from that PDF, you would be able to say, "Okay, extract me the revenue." This is how you should find the revenue; it's supposed to be in this tab. As soon as you find it, you add it to this Pantic object.
Then, essentially, you have the task of querying the database with different lists of questions to infer different things.
In the first case, we had the "Find Achievement" task, which is a set of questions to find out more about achievements. Then we have a task about finding challenges: "What are the challenges that David Goggins has encountered?"
Those are the questions that the agent should ask the database: "What obstacles has the author overcome?" Then it's going to use all the replies to that query to the database and populate the Pantic objects.
Finally, we have the prompting RAG task, where we're actually going to use the consolidated information about the content creator—all of the achievements, all of the values, all of the challenges—and then we're going to ask the agents, "Okay, based on templates that you have and all the information that you know now about David Goggins, turn this into a report."
This is the final output of the agent. Here, I'm walking through, like, "Okay, you have values, so give me the name of the value, the context, and the description."
So, that's essentially how it works. Now we can go over the tools because, like I said, each of these agents is only as good as the tools that they have.
It's just like when you're in your job; you might be the smartest person out there, but if you are not set up for success and you're not given access to the right databases, you cannot make up information out of thin air.
This is a bit like the same logic here with tools.
Let's go over to the tools now. To fetch relevant videos from a YouTube channel, this is essentially the script that you need to generate to help the agent achieve his task.
This is what it takes as an input: the YouTube handle. The output is a list of video info, which includes the video ID, title, description, publish date, and so on. The script is essentially going to fetch all the information from the YouTube channel.
I think all the code will be available to you, but I think what's quite interesting and powerful in your case is that here, essentially, I'm using an LLM that is going to scrape all the descriptions of the videos and then rank them.
If you want to go over the last 200 videos and rank them based on your goals, let's say you want to find out and have David Goggins act as your life coach. You probably want to tap into the knowledge where he talks about his own life.
So, if you use the descriptions of the videos, then you're going to be able to say, "Okay, I actually want to scrape the 10 most relevant videos where he talks about his life." This is going to be the database that you're going to tap into for your use case.
But if you want to actually use David Goggins as a business coach, you might say, "Okay, find me the most relevant videos where he talks the most about his business." You would have to tweak that tool, but essentially, it works the same.
That's pretty much it. Then I'm using another tool that allows me to add all the contents, the video descriptions, the transcripts, and so on to the vector database.
This tool is called "Add Video to Vector DB," and essentially what it does is use MChain. I'm going to show you; it's essentially a free tool that allows you to add information in different formats—audio, text, video, whatever you like—into a database that you're then going to be able to query very easily.
In this case, I'm resetting the tool app so that it starts from scratch. Essentially, what it's doing is creating an instance of the vector database that I'm initializing every time, and then I'm adding information.
In this case, I'm adding the transcript text. I have another video where I'm showing you how you can create your own YouTube summarizer, so it works exactly the same.
Finally, what you have is the query. It's very easy; essentially, it's using the list of questions from the different tasks that I showed you. Then it's going to query the database with those lists of questions.
The way it works is it's going to call the instance function app and query the database with that list of questions. It's going to do that for every agent, asking the list of questions from every agent, and then slowly but surely, it's going to start populating this content creator info, which is essentially the list of all the information that we are collecting about David Goggins.
If we go over the main crew, this is the content creator info. Essentially, here, if we want to generate a report about David Goggins, we are going to get a list of his life events, business information about him, values, challenges, achievements, his first name, and last name.
We're essentially consolidating all of that, and this is going to serve our last agent, who's going to use this object to turn it into a report.
If we go over the final output here that I showed you earlier, this is the Pantic object containing all the information about David Goggins.
As you can see, we have content creator info: first name David, last name Goggins, life events—those are all the life events that the agents have managed to collect.
Then there's the other agent who managed to collect information about his business, the agent who specializes in inferring values, who finds those values like self-discipline, scientific curiosity, overcoming adversity, challenges, and so forth.
Finally, we have this prompting RAG agent who's going to use that object and turn it into the report, which is here: "Your name is David Goggins, you're an English-speaking male," and so on.
So, yeah, that's pretty much how it works. Incredibly powerful! You can replicate that across YouTube, the news, or any kind of content format that you have. You can even consolidate information from different YouTube channels that you may encounter.
Hopefully, you found this helpful and interesting, and hopefully, I'm going to have some more videos in the coming weeks about more AI agent use cases!