Transcription
Good morning everyone, how's it going today? Welcome back to the channel.
Today I bring you a very fun tutorial right here. We're going to be building a complete deep research system with open models and with Firecrawl. It's going to be very fun.
Actually, what we're going to be building is right here. So it's going to be a complete system where you send your user query, then you have the system generate a very detailed research plan. And with that research plan, we're going to have another model split it into multiple sub-research tasks. And then we're going to sell the whole thing to a research coordinator. And it is this research coordinator who's going to have another tool or another ability to create sub-agents. And each sub-agent is going to research each sub-task that was generated before. And then once they are all done with the research, they're going to bring all the research back to the research coordinator, who is going to compile everything together and send it back to us. And that's basically the system that we're going to be building.
If you want to follow along in a written version, the article right here that you see right here is also linked in the description. There is also in the description the link to the repo with all the code right here in a modularized way. So you have, for example, the planner right here. You have the entry point, which gives you kind of a CLI tool to access the deep research. You can also access it in the description. But what we're going to be building in this video is this notebook that you see right here. So everything step by step. As you will see, I will explain and spell out every single step right here from the setup to how to generate a plan, how to call the open models, how to create your tools, how to split everything into sub-tasks, how to create your coordinator and your sub-agents, and everything very clear and very spelled out so that you don't get lost.
And just for the record, actually, this one right here is an implementation of this article that Anthropic shared a few months ago. In this article, they share how they build their own multi-agent research system for CLOD, and they share very interesting advice on how to build a multi-agent system. My version right here that I'm going to be showing you and that we're going to be building is a little bit more minimalist, of course, because we want to fit it into a single video, but it follows what they actually built. So without any further ado, let actually get right into building.
All right, so first, let's start off with our setup. We're going to start off like this. So we install small agents and we're going to install the MCP integration, LightLM, and OpenAI. What we're going to be doing is we're going to be using small agents, as I told you before. It's, in case you were not familiar with it, this library right here, a very minimalist and lightweight library that allows you to create agents with a few lines of code. It's very straightforward. I'm going to show you how it works in just a moment.
And we're also going to be using Firecrawl to actually scrape the websites that we're going to be using for our deep research, because of course, our deep research agents are going to require research tools such as search, as well. That's going to be super useful and crawling. Where, in this case, we're only going to be using the search and the scraping, but it's going to be very, very useful and it's very straightforward. The scraping is actually very smart. This one, let me show you. This one's actually a very simple example. So here we go. I am back in this Firecrawl side and let me just show you the scrape functionality. Basically, you send over to their service a URL of any site that you want to scrape, and it will return to you a markdown version of that same, of that same page, which is great for language models, of course. And they also have introduced a new search, which basically allows you to search the web, allows your agent to search the web, very, very straightforward. So, for example, I can go and look for top restaurants in San Francisco, and it will just give me the results right here that it can then use the scrape tool on top of to get the results. So, very, very useful. This is going to be extremely, extremely useful for our agent.
And the next thing that we're going to be using is OpenAI, but in this case, I literally just added it in case you want to try out some closed models as well, but it's not necessary. You can just stick to the HuggingFace tokens right here and with Firecrawl, and you should be good. I added my API keys right here. In case you don't remember how to get an API key from HuggingFace, it's very straightforward. You're going to go right here, you're going to go to access tokens, and you're going to create a new token. You're going to give it access to make calls to inference providers because that's what we're going to be using. Give it a name and just create it, and then you're going to copy your API key in prime. I'm just going to delete it because I'm not going to be using that one, but there you go.
And once that is done, let actually start with the first part of the whole process, which is generating the research plan.
All right. So before I forget, we're also going to have to add a few other things right here, which is the Hugging Face Hub dependency, which I forgot to add before. That one right there is going to allow you to connect to the inference providers that we're going to be using because we're going to be using open source models and we want to run them through Hugging Face Inference Providers API. Very, very straightforward.
Let me show you the first thing that we're going to do. So I was telling you, we're going to run this right here and we're going to create our research plan using Kimi K2. So the first thing to do is to go right here to Hugging Face. You're going to select the model that you want to use. In my case, I want to use Kimi K2, which is a really good model, especially for English and for planning. So it is, where is it? I just filtered by inference available and it is right here. Kimi K2 instruct. Nope, I want the Kimi thinking. Kimi K2 thinking. Let me look for it. So Kimi K2 thinking, here it is. I'm just going to copy it right here and I'm going to come right here and I'm going to say that the model ID that I'm going to be using is going to be this one right here, okay? And there we go. And let me actually copy and paste the whole thing. I'm going to use the provider. I'm going to set the provider to auto. I'm going to actually put the whole thing inside a function. So if you just copy and paste the entire function right here to show you what's going on. We're going to have to import from Hugging Face Hub. We're going to import inference client and my system instruction, which is going to be some sort of my system prompt, is going to be this one right here. Here it is. So we're going to tell my AI model: "You will be given a research task by your user. Your job is to produce a set of instructions for a researcher that will complete the task. Do not complete the task yourself, just provide instructions on how to complete it." And here are some guidelines, okay?
And the function looks very straightforward. It's like this. So we defined the model that I'm going to use. As I told you, I am going to be using Kimi K2 Thinking and my provider. There are a lot of providers in the Hugging Face Providers API, but feel free to choose the one that you like the most. In my case, I just set it to auto, and that way it automatically selects the best provider available. In my case, it's probably Novita that's going to run. So there we go. And just some logging right here. And what we're going to do is we initialize the inference client. We pass in our API key. We don't need this actually. And we select our provider, and that's all. That's all that we need. Okay. And then after that, we're going to initialize our completion. It's an API like SDK. So it goes chat completions create, just like in OpenAI, with the model that you want. And then right here, you select the, you send over the messages. The first message is going to be our system instructions, telling our model that we wanted to do this and this and that. And then we actually pass in the exact topic of research that we want. And that's basically all that we're going to want to do. And then from there, what's going to happen is that we're going to just return the whole thing, just like this. So the research plan is going to equal this completion choices. And then the first one, we're going to tap into the content of it. And then we're just going to print it right here and then return it. But just for the record, this, I mean, these weird characters right here just mean that what is printed is going to be printed in yellow, because why not, just to make it a little bit more nice to look at. So there you go. So my research plan, let's just say right here, and it's going to take, as we said before, our user query. And my user query is going to be "climate in northern France." And yeah, "research about the climate in northern France." And then we're just going to run it. And this is going to start executing and calling my Kimi K2 model. And there we go. This is the generated research plan. We can see: "I need you to conduct comprehensive research in climate in northern France. We have first section, geographic scope and definition, core climate variables, sunshine, blah, blah, blah." Well, it all looks very complete. So this looks like a complete research map. So that's the first part of our entire thing, of our entire pipeline right here.
The next part that we're going to want to do is we're going to take this research plan and split it into multiple sub-tasks so that we can have our research coordinator generate multiple sub-agents and assign one task to each one of them, okay? So let's get right into that.
All right. So in order to do this, what we're going to have to do is we're going to have to first import a few things that we need to import. Just zoom back in. So we're going to have to import JSON, Pydantic, and Typing. I'm going to show you why you're going to need this in a moment, but we're going to have to get a structured output right here, right? Because remember that what we want from this step right here is multiple different subtasks. And each subtask should actually come inside of a JSON-like object, right? So instead of a dictionary, we don't want it to be just in plain text, so that we can actually reuse them to create multiple sub-agents. And that's very straightforward to do, precisely because there is a possibility of doing that right here. Let me show you. If you go to the documentation right here, to inference providers, and we're going to go right here into structured outputs with LLMs, you will see right here that you can create structured outputs. You can have your language model return structured outputs to you without you having to manually parse the whole thing, which is great. But not all providers support this, and that's a very important thing to keep in mind.
So I'm going to go back right here. I'm going to sort of go to search my Kimi thinking right here. And I'm going to go to compare models right here. And I'm going to try to look at a model that supports structured output right here. So here you can see that Novita actually does not support structured output. So I'm going to have to switch to another one. I'm going to have to go to either, for example, Nebius or Together to make sure that this is actually supported. In the previous query, we actually just selected Auto because we didn't really care which provider was selected because we just wanted just plain text research plan, but we now actually want structured output. So we want to check for one that is supported right here. So that's what we're going to do.
The first thing is define our system instructions. Just like before, we're going to tell it and we're going to define a very quick prompt to do this. So it says: "You will be given a set of research instructions, a research plan. Your job is to break this down into a set of coherent, non-overlapping subtasks that can be researched independently by separate agents. That is where the multi-agent aspect of the whole thing actually shines. We're going to have one agent research thoroughly one specific subtopic about the entire research plan, and in the end, we're going to put everything together." Now, just a quick set of requirements and just a quick example of what the output looks like. This is not absolutely necessary because we're going to be passing over the actual schema in the function, I mean, in the request. I'm going to show you how to do that in just a moment. And there we go.
So let's actually initialize the whole thing by initializing our function that's going to work in this pipeline. So first thing, splitting to subtasks, it's going to take the research plan that we got from the previous function before, and we're going to select our model ID. Remember that in this case, we wanted Kimi K2 thinking. So I'm going to copy this one right here. I'm just going to make sure that we're on the same one. And for the provider, we wanted to select one that supports structured output. So I'm going to go to communicate to thinking right here, structured output. I can see that Together actually supports it. So I'm just going to, I could either just copy it right here. And actually, you can either paste the whole thing like this, or you can just write Together right here. That also works. All right.
Once that is done, we're going to have to actually make our API call. And that's very straightforward. All we're going to have to do, we don't need this right here. So the client, inference client, that we imported above, and we just initialize it again with our HF token as before, and the provider is the provider that we selected right here. And then now from here, what we're going to do, this is actually in the documentation if you want to take a look at it. So we're going to send over the model ID, the messages, which includes our system message right here, and the research plan, which is the one that the user is going to be sending. And then the response format, this is where we're going to specify what the expected result should look like because we want JSON format, right? Since we're going to be specifying this in the model request, the model is going to return a structured JSON object. And it's very straightforward to create it.
The first thing that you're going to want to do is you're going to want to create a subclass, sorry, a base model inherited class from Pydantic. It's very straightforward. In case you're not familiar with Pydantic, it's just a data validation library that allows you to create these data models that you can then reuse in multiple applications. In this case, we're going to use it as a model for our schema right here. It's very straightforward. Basically, you just create a class, you give it a name, you inherit from base model, and then you just list the properties that you want your JSON to have. So in this case, I want my JSON to have one property called subtasks, and inside of it, we're going to have a list of subtasks. And what is a subtask? Same thing. Let's define it. So my subtask is also another type inherited from base model. It has an ID, a title, and a description. And each description describes what each task is. So, for example, history, whatever. And then the title, short descriptive title of the task. And then a clear and detailed instructions of the sub-agent that will research this, for the sub-agent that will research this subtask. And there we go. And then this we're going to put it inside this object called task splitter schema. I mean, you can name it whatever you want, but this is what this function is going to take. And we're just going to name it subtask list. We're going to pass in the schema, which will just dump this model right here, and let's just set it strict to true because, you know, this is needed for this to work. And there we go.
Now we have our structured query completed. And once we have that, we can essentially just tap into it. So let's take a look at it. So we're going to just extract the message from the completion, and the SOC tasks are just going to be this SOC tasks property inside the JSON element that is in the content. And then we're just going to print it right here. And actually, for you to actually see it in real time, we're going to print each one of the tasks. I'm actually going to, I was actually going to need pretty print. I removed it before. From pretty print, import pretty print. And we're going to pretty print the description each time. And then we're just going to print an empty line. And then we're just going to return the subtasks like that. How about that? And then we can try it out. Let's just say that the subtasks are going to be split into subtasks from my research plan. I hope it is clear. It's very straightforward. We basically just re-initialized our inference client and we generated a completion, but with this new fancy structured property right here, which is a basic parameter right here, which tells my model that it does not want just plain text, but actually a JSON object. And which kind of JSON object, which is this one right here, and it is defined right here. And this one is defined by the subtask list, which is, and the subtask is this one right here. So very straightforward. And then we just tap into the message part of this. The message is actually a JSON object, so we have to load it using json.loads, and then we tap into the subtasks property, which is the one that we created right here. And once we have tapped into it, we just loop through it, through every single subtask, and just print it. So just to make sure I am using Together as I told you before, so let's just run this again and we should get my list of subtasks now in a JSON, and then we're going to loop through them right here. Let's take a look at them once this one is finished.
And there we go. I have to go back a little bit because I was having trouble with my provider, but I had to switch it back to Nebius right here because I had an error with Together for some reason, but just go to Nebius right here. In my case, I just selected Nebius and it worked correctly. So there you go. I have my structured formatted subtasks right here. So my generated following subtasks: geographic scope definition, and here's my, here the instructions for the sub, and here the instructions for the second sub, and et cetera.
And then what we're going to do is we're going to spin up one agent per subtask right here. So it's going to be very fun. Let actually go into doing that.
Okay, so now that we have completely generated a list of subtasks based on our research plan, which is in itself based on our query, we're going to be able to go to the next step, which is to create our research coordinator, which will itself create multiple sub-agents, and each sub-agent is going to research one of these individual subtasks or subtopics, right? Let me just show you what this looks like because this is just pretty printed, but let me just show you what each subtask actually looks like. As you can see, subtasks is basically just an array of multiple dictionaries right here, and each dictionary contains a bunch of like one subtask. So, for example, you have the title, which is, for example, in this case, "geographic scope definition." So, "define the boundaries of North France for this research," blah, blah, blah. Then "temperature data collection." "Collect comprehensive data temperature for all reference cities and subregions across time periods," blah, blah, blah. There you go. So each sub-agent is going to take one of these subtasks and is going to do very tough and thorough research on it. And then by the end, we're going to just put everything together because they are going to return the result back to the research coordinator, which will just put everything together and give us our final report for the deep research that we did. And it's very straightforward.
Now, how we're going to do it is very straightforward. First of all, we're going to have to initialize an agent. Now, if you're not familiar with agents, I have another video right here explaining what an agent actually is behind the scenes. In this case, we're going to be using small agents, this library right here, which is very minimalist. It basically just allows you to create an agent, plug in the tools that you're going to give it, and just leave it run, and it will return to you the result. That's just what it does. It's very straightforward.
So let's start off with this. The first thing that we're going to need is we're going to have to get the API key for Firecrawl because that's the one that we're going to be using to give tools to our agents. I'm going to show you more about that in just a moment, but what we're going to be doing is we're going to have to define the models that we're going to be using for this. And in my case, I'm going to be sticking to open source right here, but feel free to use any closed source model that you want right here. GPT 5.1, CLOD, OPUS for Sonnet. They're both, they're all great and they're going to do a great job right here. But I want to show you that you can also do this with open source and it's way, it's much more affordable, by the way.
So one thing to keep in mind when selecting your model is you have to select the model that has enough context window because agents have a tendency to generate a lot of messages because they do a lot of thinking and then they do a lot of function calling and all of that is going to remain within their context. So you're going to have to decide, you're going to have to choose an LLM that has this huge context window, which is usually not a problem for closed source models, but for open source that's a little bit more tricky to find. But let me show you how to find it again. We're going to go right here to Hugging Face. We're going to go to models right here, and you can either select, um, I mean, if you already know which one you're looking for, you can just go straight right here. But, um, in my case, I let's suppose that I don't know. So I'm going to go to compare all the models right here, and I don't want only this one, I want all the models. And right here, I'm going to, I'm going to sort them by context window. And as you can see, Llama 4 has a pretty good context window, so we could use that one. But in my case, I'm going to want to, I'm going to want to stick to Minimax right here. So I'm going to click right here on Minimax, and this is the one that I'm going to be using. I just want to make sure that it is capable of calling tools. And right here it says that it is capable of calling tools. So I'm going to call it and use this one. So I'm going to go right here. It's going to paste this and I'm going to say that the coordinator model provider is going to be Novita. I forgot to add these quotes right here, and there you go. So now I know that this model is going to be able to call tools and it also has a pretty huge context window. So there we go.
And we're also going to want to do the same thing but for the sub-agent. So sub-agent model ID is going to be this one, and it is also going to be the same provider because we're going to be using the same one. So let's also say that the sub-agent model provider is going to be Novita. So there we go.
And then once we have this, we're going to have to initialize our entire pipeline. And it's very straightforward. We're going to call it run deep research. And we're going to first generate our research plan, just like we did before. And then we're going to generate the subtasks based on this research plan right here. So by here, we're going to have this already. And once we have this, it's going to be time to actually initialize our coordinator. So I'm just going to log right here. Our coordinator logs right here, saying "Initializing coordinator with this coordinator model and with this sub model."
And in order to initialize an agent on small agents, it's very straightforward. We're going to be using tool calling agents. There is another kind of agent called code agent, but in this case, we're going to stick to tool calling agents. We go right here and we're going to go to tool calling agent right here. So the tool calling agent is this one right here, and you can just initialize it like this. It takes a list of tools, it takes a model. And in order to initialize the model, you're going to have to initialize it using inference client model. Let me just show you what it looks like. So we're going to go right here and we're going to initialize our coordinator model using this thing right here called inference client model, which we're going to have to import from, from small agents, actually. So from small agents, we're going to import inference client model, right? Just like that. And there we go.
Now, once we have that, we're going to have to also initialize our sub-agent LLM. And in order to do that, it's also very straightforward. Just do the same thing right here. And we're going to initialize our sub-agent client model and the provider. We don't want to hard code it. So we're going to just add this right here. Then for the sub-agent model, we're going to pat that in right here like this. And there we go.
Now that we have our LLM models defined, we're going to be able to actually pass them into our agents. But before creating our agent, we're going to have to create the tools so that we can actually pass them to the agents, right? Because in order to initialize an agent, let me just show you how to do that. Here it is. So in order to initialize an agent, you're going to have to import your tool calling agent and then pass in the tools. But since we're going to have to define our tools ourselves, we have to define the tools first so that we know what to pass in right here, right? And that's very straightforward. And that's literally all that you need to initialize an agent, by the way, using tool calling agent. And it's very straightforward.
The thing, though, is that we could really code the entire set of tools that we're going to give to our model, but Firecrawl kind of already did the job for us. Because if we go right here to the documentation and you go to their MCP server, you will see that they already have an MCP server working. And an MCP server is basically just a toolkit of a bunch of tools that you can already start using and that you can give your agent to use. And it's very straightforward, of course, if you already have an MCP client, such as for example, Cloud Code, Codex, GitHub Go Palette Klein, any of those, or Cloud Desktop, all of these work with MCP servers and you can connect this MCP server and turn your existing MCP client into an agent that can research the web. But it's not necessarily a deep research agent, right?
So what we're going to be doing is we're going to be taking the tools that they have already prepared right here. So as you can see, they have already prepared a bunch of tools. They have Firecrawl Scrape, which scrapes a website, Batch Scrape. They also have Map. They have search. Where is the search endpoint? They have search right here. So I mean, there's a lot of stuff going on and a lot of pretty cool functions that you can already use. If you want to write your own tools, I would encourage you to do it. Actually, this is just for a quick test and to show you the possibilities of what you can do. But if you want to have complete control of the tools that are going to be given to your agent, I would encourage you to write the tools yourself. But since in this case, we want to go pretty quickly and we want to just use the super cool tools that are already available in the MCP server, we're just going to initialize the MCP server ourselves within our deep research agent.
So let's actually reuse this MCP server. And there is actually a way of running an MCP server inside an agent. So instead of coding your own tool, you just import the tools from an MCP server and give them to your agent in small agents. So we're going to do that. And just for the record, most agent frameworks have this possibility. So if you're using LangChain or LamaIndex or any other, there is the possibility of initializing agents using MCP tools. So just load the MCP tools and give them to your agent.
Let me show you how to do that. So what we're going to do is, first of all, we're just going to say with MCP client, and we're going to have to import MCP client from small agents like this. Let me just put this right here. Import from small agents, import this and this. And now with MCP client, we give it the URL, which is going to be this URL right here. And we're passing the transport as MCP tools. Since this URL is already authenticated, no need to do anything else because right here within it is my API key. And then once we have done that, we can already start using the MCP tools. And this guy right here contains all the tools available in the MCP server. So that is great. And that's what we're going to be doing.
So with it, we're going to define a few tools in itself. But remember that we want our coordinator agent to be able to create multiple sub-agents. And each sub-agent, in order to create a sub-agent, we're going to give our research coordinator a tool titled "create sub-agent" or something like that. So that whenever our research coordinator calls this tool, it spawns a sub-agent with its given sub-task. So that's what we're going to be doing.
Now we're going to say, let me just show you, we're going to call it like this. So we're going to have to decorate it with @tool. So in order to create a tool in small agents, all you have to do is decorate it with this decorator right here that you also import from small agents like this. Oops, small agents tool. And now this tool right here is going to be usable by our small agent. And once we have done that, this tool actually, what it's going to do is it's going to initialize our sub-agent. So let's initialize our guy right here. So we're going to go right here and we're going to initialize sub-agent = tool calling agent. The tools that it's going to have are going to be our MCP tools. The model is going to be our sub-agent model, which we initialized above right here. Remember that this is not the ID, the string ID. This is the actual inference client model that we initialized before. We do not want to add base tools because if you set this to true, it's going to automatically have a few tools that all small agents have available. We want it to be very focused to only have our Firecrawl MCP toolkit. And there we go. So we're going to have to import our tool calling agent right here also from small agents. Now it's going to work.
And we're going to have to give it a prompt, right? Because even though the sub-agent is created, so that is great, we're going to need to give it a system prompt too, so that it knows what it has to do. Because right now, it's just a regular sub-agent. And it's very straightforward. I'm actually going to give to give you a pre-prepared sub-agent prompt template right here. I'm just going to copy it right here. I'm going to probably paste it above right here. I'll just paste it here. So this is my template. So it says: "You are a specialized research sub-agent. Here's the global user query, and here is the overall research plan. Now, your specific subtask, which is the," I'm just going to pass in the subtask ID and the title right here, say, and then pass in the subtask description right here, which is exactly this description that we have right here. Okay. "Run your results as markdown report with this structure," and then we just give it the title of the subtask, the summary. I mean, this is what it should give out, and then the bullet points, and then the sources as well, which is very important. And there we go.
Now, basically, this is the prompt that we're going to give to our research agent. And in order to actually, for this actually to be useful, we're going to have to fill it with the actual variables that we're going to be using. So in this case, the user query is going to be the user query. The research plan is going to be our research plan, the subtask, the subtask title, etc. Just, I mean, just to be extremely clear, remember that right here in the template, we added these variables right here. This is what is going to be replaced when we use this .format thing right here. Okay? So we take that sub-agent prompt template that we created, we do that format, and we format it so that by the end, the prompt looks like an actual prompt dedicated to this sub-agent. And then what we're going to do is we're just going to return our sub-agent run, and we're going to run it with this prompt that we just created right here. Okay? So looks great. And this tool right here is the one that our coordinator is going to be able to call in order to spawn or to create a multiple sub-agent. So it will be able to create 10, 15, or we should probably steer this in the right direction to not have it create 100 sub-agents. But there you go. I mean, basically, our main agent is going to be able to create as many sub-agents as needed. But in this case, it will only create one per subtask that we created, which in this case, it's 1, 2, 3, 4, 5, 6, 7 sub-agents. Okay.
And once we have that tool, something that we're going to have to do as well is we're going to have to create a coordinator. And the coordinator, just like before, it's going to be a tool calling agent. The tools, it's not going to be the MCP tools anymore. It is going to be this initialized sub-agent, because we don't want the coordinator to do any research. We just want it to be able to initialize the sub-agents and then by the end, just put everything together into a single report. Then the, then we're going to see, say that the model is going to be the coordinator model. The base tools are, we're going to set it to false, and the name is going to be coordinator agent, just to make it fancy right here. And then just close it. And there you go.
And then what we're going to do, it's very straightforward. We're going to get the subtasks in JSON right here. So we're going to dump them. So that's basically just going to structure it in in this very nice indented format. We're going to dump this subtask thing, which is basically this thing right here. And what's going to happen is we're going to come, we're going to fill out our coordinator prompt, just like we did with the sub-agent prompt. So right here, we're going to do the coordinator prompt template and we're going to format it. But before we forget, we actually have to initialize that coordinator template. So where is it? Let me look for it. Here it is. So let me just initialize it up here, just below this one right here.
And the coordinator prompt template is going to say something like: "You are a lead research coordinator agent. The user has asked the user query. A detailed research plan has already been created. This plan has been split to the following subtasks in JSON." And then we're just passing the subtasks in JSON right here. And then each element has the shape of this one right here, and then you have access to a tool called initialize sub-agent. And then we have these parameters right here. And we say, "Your job is that for each subtask in the JSON array, call initialize sub-agent exactly once with all of its parameters, of course, and wait for all sub-agent reports to come back. Each tool call returns a markdown report for that subtask. After you have results for all the subtasks, synthesize them into a single coherent, deeply researched report addressing the original user query, which is this one right here." And then the final report is going to integrate the whole thing, the highlight, blah, blah, blah. So basically, we are telling our research coordinator that it has access to this sub-agent tool, which will spawn a sub-agent with its own research plan. Because remember, we're also passing the full research plan as a context. And then all of them are going to return their final result back to the research coordinator, who is going to synthesize the whole thing, as you see right here. Okay.
So that's what we're going to be doing. We're just going to execute it and then we format it to complete, to replace the values of each variable. So the user query is going to be the user query, research plan and subtasks in JSON, which is the ones that we defined above right here. And then once that is actually finished, we are going to be able to just call it. So let's say that the final report is going to be our coordinator.run with the coordinator prompt and the coordinator, remember that has access to this initialize sub-agent function. And then we just return our final report. And that's all.
Now we should be able to just call this and it's going to take quite a bit of time because remember, just like deep research in Cloud, in OpenAI, etc., can take a lot of time, the same thing's going to happen right here. So I'm just going to execute it and show you how it looks when I start and. And oh, I did not execute it. I just created it. Let just run deep research right here and say result is going to be "research the climate in Northern France." And we're going to execute it. And as you can see, running deep research, sharing the research plan with Kimi K2 with Together. And I mean, I'm probably going to pause this a little bit because it's going to take a long time to finish. But there you go. We have the plan. And then it's going to split the research task, research plan into subtasks. Now it's using Kimi K2 thinking with Nibius, so that it's able to use the structured report. Let's just give it a little bit of time for it to generate that, and then I'm going to show you real quickly how it starts to call the sub-agents, and then I'm going to pause this so that we don't spend like 20 minutes right here looking at the agent do its job. There we go. And there you go. You can see that it generated a bunch of different subtasks. And now we have initialized the coordinator. So there you go. So let me just go back right here. New run with the coordinator agent. "You are the Lead Research Coordinator Agent," jara, jara, jara. And then you can see right here that it has all of the tasks that are replaced right here in the prompt, which is pretty cool. And then "each element has the shape," blah, blah, blah. "Your job is to do this," blah, blah, blah. So there you go. It's calling tool initialize sub-agent. So there you go. Now that one spawned a new sub-agent called "sub-agent geographic framework," blah, blah, blah. And then it's going to do the same thing for all of the subtasks.
All right. So right here, that's one new run. You can see right here that it's also, I mean, this, this sub-agent is already calling a bunch of functions from Firecrawl. So it's searching in Firecrawl "northern France administrative regions." Then it got an error, so it, so it fixed that error and then called it again. And then there you go. So down. And now it got some results using Firecrawl, the Firecrawl MCP tools. And there's, and there you go. Actually, I think there you go. It has it finished its report. That was pretty quick. Great. Okay. Okay. Final answer from the sub-agent. That was pretty quick. Okay. So it only took a couple of, a couple of searches. And now let's take a look at the rest part right here. So we have another sub-agent that was started, "sub-agent with contemporary climate," and it's also going to run. So it's going to continue like this for a while. As you know, deep research takes multiple minutes to complete. So I'm just going to pause the video right here and I'm going to come back once it finishes.
So it finally finished. So as you can see right here, the deep research run is super long. It took multiple minutes to complete. I didn't really count them, but it was definitely more than 10, probably 15 minutes to complete the whole thing. And it called all the agents, all the sub-agents it had to call. And then by the end, we have this result right here, which looks like a very complete executive summary. So "Climate of Northern France: A Comprehensive Analysis." "Northern France exhibits complex maritime to continental climate transition," blah, blah, blah. It's a pretty long thing. Actually, it goes far right away. That way, it's not just a single line right there. And there you go. There's a lot of material here to actually complete very complete research. And then it has a very detailed bibliography. So primary data sources with their links right here. Peer-reviewed literature, climate assessment reports, methodology references. Looks great to me.
So there you go. We have effectively completed a deep research pipeline that uses Firecrawl as our main tools for researching the web. It's super cool. I really recommend it. It uses small agents as our agentic framework. Very lightweight, as you saw. All we did is just initialize the agent, give it the tools, and give it a go and just say run. And it went with it. So that was great. And we used this very straightforward architecture where a research plan was generated first. Then it was split into multiple subtasks. Then we had a research coordinator create multiple sub-agents, one for each subtask. And then we had the research coordinator summarize the whole thing that all of these sub-agents gave back after each one performed very thorough research on their own subtask. And then we got the final result.
In the end, I think this is a great example. I do think that there are many areas that you can improve. So, for example, you can have right here another human in the loop part right here where the human actually validates the research plan or asks it to include multiple other things. Or you can also have the LLM ask the user some clarifying information before generating the research plan, kind of like ChatGPT's deep research does. That could also help a lot. There's also the possibility of using closed models if you want to use GPT 5.1 or Sonnet 4.5 and run evals to see which one gives you the best result. But in my opinion, for open source models, I mean, open source models are working great for deep research. And we have seen that even Kimi has their own deep research tool, and pretty much every deep research platform has their own deep research now, even DeepSeq and all of them. So, very, very happy with these results. Let me know what you think.
On top of this, I have actually released this, um, um, this repo right here that you can take a look at. It's very straightforward. It's the same thing as I showed you in the Colab, but a little bit more modularized so that you can actually run it directly. So it has an entry point with the main.py right here that requests the user for their deep research query. So "Enter your research query," and then runs deep research on top of that query. So its UI is inside the CLI. Then it has the coordinator right here, which has the run deep research function, which we initialized before. It has all the prompts centralized right here in our prompts.py file. It has the task splitter function right here, which also calls the LLM with this, with the research plan. It has the planner, which is the one that generates the research plan, pretty useful as well. And you have this README right here to follow if you want to to figure out how to go with it. That has this .env example. My recommendation for you right now, if you have finished implementing the whole thing right here in the Colab notebook, would be to try to put the whole thing into an actual repo like this one. So instead of cloning it, try to build the repo yourself and try to get it to run on a single input like this right here. And once you have that, you will be ready to actually put it into a graphical user interface using, I don't know, maybe FastAPI or maybe some sort of a Streamlit or Gradio frontend and actually display a real deep research application. Maybe we can do that in another video if you're interested. But thanks a lot for watching and I will see you in the next one. Thank you.