Transcription
Everybody knows about vibe coding. It's become extremely popular, and it's essentially that you just describe what you want, and an AI builds it. You don't really make that many decisions. You don't know what it's going to do. You just want it to build something.
Now, what if you could do the same thing with research? Yes, I'm talking about scientific research papers, experiments, machine learning models, anything that involves actually experimenting when you don't know what the right answer is going to be. Imagine an AI system that explores your topic, reviews the relevant literature, runs different experiments, even writes code or trains models, and then writes you a full professional research paper with its findings.
Now, that's exactly what I'm going to show you in this video using a really cool platform called Lema. Now, they've branded themselves as the Vibe Research Platform, where you give a prompt, you pick a particular method, which we'll talk about in a second, and it goes and does the research for you, similar to how you would vibe code a website.
Anyways, I want to show you how it works. Stay tuned. There's some pretty cool stuff that you can do with this. So, just to give you a taste of what we're going to look at later, you can see that I generated a proposal, had a detailed plan created, did an experiment that actually took multiple days, and then there was an in-depth research paper that was generated for me with graphs, charts, statistics, and talks about finally the conclusion based on the prompt that I passed it, which was to investigate whether an LLM can get less bias or less hallucinations. But we'll look at that later.
Now, this is all coming from a platform here called LMA. Now, this is built by a company called Analemma AI, and they've kindly sponsored this video and loaded me up with a ton of credits. However, it's free to try out on some small running tasks, and you can check it out from the link in the description.
Now, I want to explain what this actually is because it's really just not another chat GPT wrapper or generic AI search tool. Now, the way that most AI tools work right now is that you ask them a question, you get a response. Now, maybe it searches the web or it calls a few tools, but generally speaking, it's kind of a single turn. You prompt it, it responds, and that's it.
Now, Lema here is doing something fundamentally different. It's a multi-agent research system. So, when you give it a research question, it's not just generating a response or guessing. It's actually spinning up what's essentially a team of AI agents that plan, search, read papers, synthesize the information, run and execute code, and then validate the results, and iterate on top of each other.
Now, it's orchestrating pretty much an entire research team of AI agents autonomously. And the key thing that makes this worth paying attention to is that this is an AI for AI company. So they're actually using AI to research and improve AI itself. So the system they're actually going to look at here in a second was partially built with its own output. Now, it's a fundamentally different approach than most companies in this space.
And what I want to do now is talk about how this platform works and go into the four modes that it has that I'm going to show you some demos that I ran that took multiple days to execute. So the first mode we have is explore. This will create a simple report for you in 1 to 3 minutes. It's pretty basic and it works like a lot of the other AI tools out there.
Next, we have survey. This can take a lot longer. You can see the number of credits that it's estimated to use, and what it's going to do is generate long-form academic surveys with broad citation coverage.
Now, for me, the most interesting is this code one and FARS. This is fully automated research system. Now, I actually ran a few tasks before this video because they do take a long time to run. So, I'm going to go through them and show you exactly what they look like. And again, if you want to mess with this, you can do from the link in the description. You will have to purchase some credits if you want to do the long running task. This is the price of the credits as watching this video. And full disclosure, again, they loaded me up with a bunch of credits. I didn't pay to run these tasks.
So let's have a look at the code tab first, and we'll look at FARS, which is a much more complex task which can take a long time to run.
Okay, so the first thing to have a look at here is a relatively simple task that I built using the code feature. Now, I believe this took 2 or 3 hours to run, and what I asked this to do was build an image classifier that can distinguish between AI generated images and real photos. I told it to train this on a small data set and visualize what features it's picking up on.
Now, what I want to emphasize here is that this doesn't just immediately go and start writing code. Now, it did make code, which you can see here. We can see all of the generated files and the images, which we'll have a look at in a second. But it actually starts by doing research, reading the different methodology story and understanding what it should do to accomplish this task. So, you kind of would use this feature if you have an experiment or a research idea that you actually want to put into practice rather than just doing it theoretically. And that's kind of what I did here.
So, if we scroll through, you can see there's a good amount of code that was generated here. We can look at the data set. We can look at generating the data set, training, visualizing, etc. And then what it did here is actually created a bunch of different images. Let me move this over. And this is running all inside of Lema. So you'll notice there's actually a virtual execution environment that Lema connects to. This is not my computer. This is like what it built and what it's demonstrating to me. And if we check inside of here, we can see we kind of have the VS Code editor. And it's given us a bunch of different images here that it then goes and classifies.
So if I just pull up another grid here, because this one has the correct labels, you can see that if we scroll through this, it shows what the true label was, AI generated. And then it shows what the prediction was. AI generated, showing us the image below. So you can see this is AI generated. It predicted that next real. This one's real. AI generated. AI generated. AI generated. AI generated. Real, real, real. Right? So it's working properly and it's given us kind of this visualization like I asked for.
Now it also generated the training curves here. So we can see the loss and the accuracy over time. Now, given I didn't train a massive model here and I didn't want this to take a super long time to run, so it just used a small set of images. So, it's not going to be the best in the world, but it's kind of just proving the case that this can handle that.
Now, we can go and we can look at all of the bash that was executing here. We can see all of the different terminals that were open where it was running and going through and actually doing all of the training. And we can see the visualization here where it just kind of gives us this graph where it's saying, hey, it got everything perfect again because the training set was relatively small.
Now, if we have a look, we can see all of the to-dos that it executed. And we can scroll through and it kind of gives us a result here where it says, hey, it used Mobile net v3 small.5, fine-tuned it on 300 images, 150 real, 150 AI generated images, and gives us all of the results and how long it took to train on the CPU, again, because it's a very small model.
So now I can just download all of this code, start using it, improve it, or prompt it again and ask it to change this model, maybe train it on more data, do some more complex larger images. You guys get the idea. I think this is pretty cool, and again, it runs in this kind of virtual workspace which is really interesting. You don't need to have this open on your computer. It will run in the background and you can wait for it to finish.
Okay, now let's have a look at by far the more interesting task, which is FARS, again, fully automated research system. Now, if you want to trigger this task, it is going to take more credits, as I mentioned. And if you go to this tab from home, what you can do is choose if you want a research proposal, an experimental study, or a full research paper.
Now, in my case, I went with a research paper because that's what I was looking for. And if you just want a proposal, what you can do is generate multiple different proposals, see what it says, and then you can trigger either an experimental study or a research paper afterwards. So, you'll always start with some kind of proposal. Once you have the proposal, you can pick the one that makes the most sense to you, and then you can advance to the kind of more complex stage where it's going to take a little bit longer, run all the experiments, do training, write the code, whatever it needs to do in this kind of vibe research way without you really needing to direct it step by step.
So, let me show you an example of one that I ran here. We started with the proposal. So, I asked it, hey, I want to investigate whether asking an LLM to explicitly state uncertainty, so saying something like, I'm not sure, reduces hallucinations. Design a simple experiment with 20 questions and compare a normal prompt versus an uncertainty-aware prompt. Analyze the expected results and provide the practical recommendations.
Okay, so it gave me this research report here. So I was able to read through this and make sure that this kind of made sense before I advanced to the more complex step, which is going to take a lot longer. So I read through all of it. Looks pretty good. And then we went ahead and moved on to the experiment phase.
So from the experiment phase, this is the list of to-dos that it came up with. You can see it's 10 steps. Each step actually has substeps, as you can kind of see down here. It's just not letting me click into them right now. And you can see that we installed the dependencies and then we ran multiple different prompts on different models. So, it automatically picked Gwen 2.572 billion as well as GPT 4.0. It changed a bunch of different stuff here. And again, you can read through kind of what it came up with. And this is fully on the model. This isn't me telling it what to do. Like, I just gave it one prompt and then it just ran with everything. And you can see the full kind of to-do list here in this experiment phase.
Now, this experiment took a pretty long time to run. I believe it was 1 or 2 days. It used up 63,000 credits. And that's because a lot of the steps here just take a long time to execute because it actually needs to spin up the remote execution environment, run through, download the download the model, sorry, pass the prompts, train it, and do everything that it's doing in this kind of vibe research way that I'm not really involved in.
And then finally, it gave us the research paper here. I'm going to open it full screen and just show you a few of the findings because it's actually pretty interesting what this came up with.
Okay, so this is the paper here. I'm not going to read through everything, but I want to show you just some of the key results here. Now, it follows kind of the format that you're supposed to follow for a professional research paper. So, we have the abstract, right? We have the introduction, we have related work. So, again, it goes and finds what other people have actually done in the space so that it can do something unique and go based off of existing research rather than trying to reinvent the wheel on everything.
If we scroll down, it shows us the method. So, it says, hey, it's going to do different prompting conditions. We have a standard baseline, loose uncertainty, and then strict binary abstention. I believe that's how you say that. And if we go down here, it kind of gives us a graph of what this looks like. So we have some input Q and then we have a baseline, loose uncertainty, strict binary or structured confidence. And then it shows us kind of how it's going to be classifying the output here to see what these models actually give us.
Okay, if we scroll down, we can go to the evaluation metrics, coverage, false answer rate, all of this kind of stuff. And it gives us a chart on how these models perform. If we scroll through here, you can see all of the results that we get based on the data set that we're using, as well as the uh what do you call it? coverage, false answer rate, HCE, all of this kind of stuff which is broken down in more detail in the paper.
If we go down here, it gives us the exact values on how many were wrong. Use two different prompting data sets and then two different models. So, we have a little bit of variation here. And then it gives us the HCE. This is the hallucination coverage efficiency, as you can see here, which is the main thing that we are looking at.
If we keep going, it talks about the experiments that it ran. Gives us the graphs. We can see where everything lines up here in terms of the different models in the different data sets. And then we can see kind of the overall results here on base correct items and on baseline wrong items as they appear based on the prompting methodology. And if we go down here to the conclusion, I'll just read this because this is kind of the whole thing that this did. It said, we introduced binary abstention prompting, a zero-cost intervention that eliminates hedged hallucinations by forcing LLMs to either answer or explicitly refuse. This method reduces the false answer rate by 18 to 51% across two models and two benchmarks, achieving near-perfect abstention targeting on structured QA or question answering. However, binary abstention over abstains on knowledge-intensive longtail questions, revealing a regime boundary.
So essentially what this is saying is, hey, when we tell the LLM to tell us if it doesn't know or give us a strict answer rather than I think or maybe this is the case, we get a significantly lower hallucination rate and false answers. However, sometimes when it should know the answer, it doesn't tell us the answer based on us introducing that in the prompt. And that's overall what we came to. Again, this doesn't use a massive huge data set, so you can't necessarily trust the results 100%. But it's pretty interesting and reveals the fact that, hey, if you want to, you know, reduce hallucinations, this is actually a valid path you can go with, given this small drawback that it mentions.
That it goes through all of the references here of where it got all this information, all of the models it used, XYZ, all in this nine-page report that was generated while I was sleeping. Now, I don't know about you guys, but this is a better research report than I would have written when I was in university or high school. Pretty detailed, goes through a lot of stuff, and I feel like I've genuinely learned something from the conclusion because it actually went through and provided the evidence for why this is the case.
Now, of course, there's a lot of other stuff that you can do here with Lema in terms of the FARS. And I'm going to keep kind of vibing away and doing some more research here specifically related to AI because that's kind of what this platform is designed for. And then like I mentioned, there's a few other things you can do: explore, survey, code, FARS. So if you're interested in scientific research at all, I would definitely have a look at this platform. It's currently in beta. However, when this video goes out, it might be fully released. And I think this is giving us a glimpse of what the future is going to look like for in-depth research. We actually have AI doing research for us. So that informs the next level of AI. And it's kind of this self-improving iterating loop, which is pretty interesting and also a little bit scary at the same time.
So, all in all, what this is ultimately doing is democratizing the access to in-depth professional research that normally you would pay hundreds of thousands of dollars or need to be an expert to do. Let me know what you guys think in the comments down below, and I will see you in another video.