Transcription
Hey everyone, my name is Ren. Today, I'm going to show off how you could build a recommendation engine in 10 minutes. It's going to use Crew AI, We8, Docking, and the EXA Search tool in combination.
Our recommendation engine is going to take an input of a TV show or movie, and we're going to find the most similar one based on the given input. So if that sounds interesting to you, let's go ahead and get started.
So how is this going to work? We're going to need to pre-process data. Our data source is a spreadsheet of a bunch of movies. This is available on Kaggle, so you could easily download this and follow along. I'll link this down below as well.
Our movies have titles, what kind of movie or TV show it is, the director, a cast, the country, date added, description, and listed in. There are a couple of columns here that are super important. What we need to do is chunk for every single row in this spreadsheet. We're going to have a couple of things that we could store as metadata, and we're going to generate embeddings and store those into a vector database.
This is what we need to do first: have good data in, and hopefully, we could retrieve some similar movies when we use our Gensim RAG application. Then, we're going to create a vector query tool that our agents will use to find relevant movies and TV shows based on a given input.
This is kind of like the orchestration flow that I have defined. What we're going to input is a movie or a TV show, and the recommendation system will do a vector search on We8, finding, let's say, the top three to five recommendations based on a given TV show or movie. From that list, we're going to enrich that data, finding relevant details online using the Search tool, and then present all this in a structured recommendation output.
So let's get started with some code. Also, before you get started, we have some prerequisites for getting started officially. If you don't have Crew AI installed, you're going to need to install Crew AI using Crew AI tools. You need to sign up for the We8 vector database and set up your sandbox environment, and then sign up for the EXA Search tool as well.
Alright, with that, let's go to code. The first step that we need to do is generate the chunks and metadata. Our document converter is going to be imported from Docking, as well as our hybrid chunker. These are going to enable us to take a spreadsheet like this into something our vector database can use.
Then, the Docking document converter is going to convert that source into a Docking document that we can now use within our chunker. Our chunker is going to give us an iterable or a list of all of these sources. From there, we're going to take particular rows and columns and generate metadata using things like the director, the type of movie, who are the cast, and what was it listed in, like the genre.
What I'm going to do is show you this and kind of what exactly Docking does. I think the value of Docking is being able to transform this spreadsheet into something that we could easily see.
So we see the chunk over here is pretty much every single row in the data in our spreadsheet. We have chunk number 12, which is "The Starling." The director is Theodore Melfi, and it's about a woman adjusting her life, and so forth. These are pretty much every single row.
We're using the trimmed one for now just so it doesn't take as long, but you can see over here that chunk number 12 is "The Starling." It is a type of movie, and this is the metadata that we're extracting: the country is the United States, the release year is 2021, and it's listed as a comedy and drama.
Now, we're going to take this chunk and store that into our vector database. We're going to connect to our We8 cloud. Again, all you need is your We8 cluster URL and API key, as well as an OpenAI API key.
Why we need the OpenAI API key is because we're going to use OpenAI embedding models to generate the vector embeddings for every single one of our chunks. We're going to create a client called Netflix Data System. I already have one before for this example, but we're going to write it live and show you guys what it's going to do.
Again, the data that we're coming in is exactly the chunks that we've created before. We're going to iterate for every single one, and it's going to create the vector embeddings that store every single row of TV shows and movies into our We8 cluster vector database.
This might take a couple of minutes, but once it's done, what we're going to see is we're successfully added into our docs. I'll show you how it looks on the We8 vector database on the platform. Boom! This looks good. The client is ready.
Now, I think from there, we're going to batch process and put in all of these rows into our vector database. Boom! We've successfully added into our docs. To verify that, go over to collections, and now you have Netflix Data System 2. You can see all the details that we've stored, right from the description, release year, title, and so forth.
We're using the OpenAI vector embedding model. Boom! We've done pre-processing. Next, we need to create a vector search tool. Let me show you guys how to do that.
How do we build our vector search tool? When you're defining your tool schema, this is essentially what you want your agents to dynamically create. What makes it an agentic application or agenic use case? The query will dynamically be created, and we're optionally passing in some filter options, like filtering by genre or filtering by movie or show.
Outside of standard Python for instantiating and initiating a class, the most important things are just to connect to your clients and connect to the collection that you want to use, which is our Netflix Data System.
Then, this is the most important piece: take our Netflix Data System, and we're going to find the closest vectors based on a given query. We're going to limit it to five. I think our default limit is three, so that should be fine.
Then, what we're going to do is set filters if we do have any of them. Every agent tool output must be a string, so we might get JSON back. We need to do JSON to transform any objects or dictionaries into a string.
If we want to try this out, we could do something exactly like this: "Find me similar shows to How I Met Your Mother," and then we're going to print the result. This looks good.
Python copy-paste, and I think we just drop this. Boom! This is what we get. We get a result of three pretty close movies or shows to "How I Met Your Mother." The TV show we got is "Why Are You Like This?" It's an international TV show, TV comedy.
TV comedy is exactly like "How I Met Your Mother." There are a bunch of friends always hanging out with one another and intertwining in some romantic situations. I think these search results are pretty good.
Again, you could find and tweak them as you go, but what our agents are going to do is essentially create its own dynamic query when it's running this tool. The input that we will pass will probably be the title of a movie or TV show.
I want to confirm with you that these are not being made up from the LLM. If we go to "Can You Hear Me?" with the title of this and go to our spreadsheet, I'm going to search for "Can You Hear Me?" Let's see if all the details are right.
Melissa Bedard is the cast, but that's fine. It's based in Canada, the country. The release date is November 2, 2020. Perfect! It's an international TV show and TV comedy.
Again, I think one of their characters was from Canada in "How I Met Your Mother," so even closer when it comes to the recommendations that we're getting.
Now, what we're going to need to do is put all these things together in defining our agents and our tasks, and then enriching that with the EXA Search tool. I'm going to show you that next.
Now comes finding our agents and our tasks. Our first one is going to be our RAG agent, which is going to be called the recommendation agent. It is going to generate a list of recommendations based on the user query.
The query could be like the movie that we want to pass in, and then only use results found. You have access to the We8 database of Netflix data. You can use this tool to search for relevant information and so forth.
Then, we want to expand the details of the movies that we found using the EXA Search tool. You have access to the EXA Search tool to search for relevant information. You need to prioritize information and make sure the report is detailed and contains any and all relevant information.
Then, we're going to use our last agent to aggregate all of them together and provide a report with the director, the title of the movie or the show, the country, release year, and so forth, combining all those details.
Our agents are kind of mapped one-to-one with tasks. We have a RAG task that generates the list of recommendations based on the given query, and then reviews the context that you got, enriching that using the EXA Search tool to find more relevant information about the movies, the details, and the comparisons.
Then, our report task will put all of those things together. The EXA Search tool comes right out of the box within Crew AI tools. All you would have to do is do "pip install Crew AI tools," and you have the EXA Search tool.
All you would have to do is pass in your API key. However, for the We8 vector search tool that we've made, we've curated a particular one for our needs. We're going to pass in some of the headers, limit it from three to five, and our collection name is Netflix Data System.
We're going to change it to number two because we updated it on our embedding during our pre-process. The name that we want to put in here is whatever we have over here, and I think capitalizing it is fine.
When we're going to attach the tool to our agent, our first one is the RAG agent. We're going to give it the We8 vector search tool, and then we're going to give our second agent the EXA Search tool to expand the details that it's finding.
Then, the report agent is going to put everything together into a structured report for us to use for our end results. How we can trigger it is if you go to main.py. We're going to have a query like this, but instead of "How I Met Your Mother," maybe something different since we already got that.
Let's do a query of "Find me a show similar to The Office." All you have to do is CI run, and that will run our query.
We're running the Crew. Let's see our recommendation agents. We're getting a lot of recommendations. The five recommendations that we wanted can be seen. The query is "Find me a show similar to The Office," filtered by null.
Now, the recommendation agent is finding similarities from the movies that we're passing in, "The Office," through our EXA Search tool and generating more detail about that. Again, we're getting the scores for how close these queries are to our search query.
Let's see our end result. I think it's done. We got an error, but that's fine. It'll iterate and try to fix that. "Parks and Rec" is again very similar to "The Office," so that's a great recommendation.
Let's see if that was even recommended here as well. "Parks and Rec." Perfect! Nice! Now we're almost done. We're in the recommendation agent now, and here's our list of our five recommendations based on "The Office."
We got "Life on the Road," expanded details, "Receive the Movie with Mixed Feelings." Our second one was "Set It Up." Is that a TV show? A fresh take on the romantic comedy genre.
Then we got "Great News," and then we get "Parks and Rec." So let's try another one: "Find me a show like Squid Games." Try filtering for international genres. Boom! Let's try this out and see how it does.
Our recommendation agent is running, and we're starting to see our list. Boom! We have our query: "Shows like Squid Games," filtered by international.
Again, I didn't put "listed in." I didn't put how we want to filter. I just said "filter." Try filtering for international genres. "Find me shows like Squid Games," and we're starting to get exactly the results we wanted to find.
Right? International TV shows from Britain, South Korea, Singapore, and these are the metadata being generated. Just to show you guys, this data is here.
So we go to the sources and filter it from here. Want to command F? How do you command F? We can't find. Let's do it over here. It's the same. Boom! "Mind Game" is the TV show, and it's based in Singapore.
So great, great suggestions for what we are looking for. Again, this is kind of like the concept of an agentic RAG, where your AI agents are querying and filtering the way it needs to properly extract relevant context.
The EXA Search tool is finding relevant details within each show's funding and expanding the descriptions that we have to find more similarities as to why "Squid Games" is being recommended.
Let's see what our end result report agent is going to give us at the end of all these. Again, we have an expectation of a report with like 9 to 10 recommendations.
Let's see. Obviously, our first one is "Squid Games," and then we got "Loaded Busted," "Mind Game." It's in Singapore. Expanded details: the show intricately weaves psychological twists and suspense into narratives, challenging viewers to grapple with the questions about morality and justice, which is kind of what "Squid Games" is from that perspective.
Yeah, we get 10. We could turn this into structured data that you can actually use when you want to begin recommending TV shows or movies if this is a use case that you have.
Again, I think this is a great segue into being able to build out recommendation systems in just a little bit of time. Recommendation systems are things that usually take a lot of time and resources to create, but now with some data, you can easily start to scaffold an MVP of a recommendation system for any of your use cases.
If this was valuable to you, please leave a comment down below, give it a like, thumbs up, and don't forget to subscribe. Thanks again for watching and staying until the end. See you next time! Peace, guys! Bye-bye!
So yeah, I think this is a great start to a recommendation engine. In like 10 to 15 minutes, you're able to generate and pre-process embeddings to then leverage agents to query against a vector database.
This is a super simple guide to start building recommendation engines with AI agents. Again, it's super awesome to use tools like Crew AI, Docking, We8, and the EXA Search tool to put everything all together.
That's it for me. Comment down what you guys want to see next. This is AI agents, RAG, and recommendation systems. Thanks, guys! See you in the next one! If you found this video super valuable, be sure to leave a like and comment down below. Thanks, guys! Peace! Bye-bye!