Transcription
There are three AI tools that I've been really impressed with with literature reviews recently, but here's the thing. You need to know which ones are hallucinating and how often they just make up references. You see, when you're doing a literature review, you want exactly that, literature. So, here are the best three AI tools that I've tested and how much they actually lie to you.
So I took three tools, Manis, Gen Spark, and Gemini AI from Google. And I wanted to know, are they just lying to you? So the prompt that I put in for every single research tool was this. If we head over to Manis to look at it, it says here, generate a structured literature review on the topic and then the topic. that I had. Please include, and this is the important bit, the key research themes, how they've evolved, important studies, conflicting viewpoints, gaps in the current literature, suggestions for future research. And I says, please use peer-reviewed sources. Present the output in a format suitable for inclusion into the introduction of a thesis. Add intex citations in IEEE format even if they are placeholders and make the review as long as it needs to be to cover the literature effectively.
So I did this for the three things and then if you can blur this editor these are the results. Oh let's get into it. So the first thing was uh Manis was uh the fastest to kick it out. It did it in about 3 minutes. So that's really good. But did it actually produce enough references? And were those references completely made up? So let's have a look here. These were the things that it kicked out. I love Manis because it gives you all of the kind of files that it used to generate the literature review. But this is what we're really interested in. Literature review PDF. We can open it up here. Let's go full screen on that. And you can see that it is a 14page document. And we'll go all the way down to the bottom. and it produced 38 references and I wanted to know are those references first of all actually sort of like saying what they want to say but also are they just made up.
So overall this literature review is good. It contains all of the important things that I want to know about critical analysis. Um and then it's got you know different kind of themes that pop up in this kind of research field. So it's a really good first start. But let's head over to my magical Excel document so we can see what has actually happened. So I went through every single reference and I actually looked on Google Scholar to see if it exists. And here are the results for Manis IM. So here it was 38 references down here. And you can see that it did an good enough job and it was actually um making up references and hallucinating for 16% of the time. So, I went through and you can see here where I put across this reference. It was just sort of like the wrong journal and the wrong year, but the title was right. And then I went down here and it actually repeated two references that were repeated up here in four and five. And then these ones, it just made up completely. Here it doesn't exist. Doesn't exist. And here the journal year and title was wrong, but it was kind of like a plausible one.
So, let's have a look at 18. And what I did for each one is if I couldn't find it like just by using Google Scholar, I went to the actual um journal where it was published. So, uh where are we? This one here. Look, morphology control blah blah. I was like, hm, I wonder if that exists. So, I went to journal of fizz chem and I tried to find it. And uh yeah, journal material chemistry. I used to put in the page number and the year and you can see that records not found. So ultimately this is what I did. I went in after using Google Scholar if I couldn't find it there. I went to the actual journal and uh yeah, it just didn't exist.
So overall, Manis did a pretty good job. And then you can see after that, I was expecting it to get worse as it kind of like went along as it was using more and more of its memory to hold on to different references. But in fact, you know, early on we got this one, which was wrong in a certain way. These ones repeated, then these three were wrong, and I was like, uh-oh, this doesn't bode well. But then all the rest of them were absolutely fine. So overall this gives Manis a 16% hallucination rate or just failure rate in some sense.
Now let's have a look to see how Gen Spark did. So Gen Spark is another tool that I've been absolutely amazed by for all sorts of different academic tasks. Let's see if it's lying to us. So I put in the exact same prompt as before and then I just waited. This one was the second to finish and it did it in about sort of like 5 to 7 minutes. So overall it was still relatively quick and you can see it understood the fact I wanted IEEE referencing and you can see that, you know, it did a relatively good job at getting all of this. One thing I don't like about GenSpark is that it's hard to extract it into like a a file. At least Manis gave us a PDF. Genspark. Um, yeah, it's not really a useful format. You can ask it to put it out in maybe like a latte or latte, whatever, however you want to say it. It's completely up to you. Um, but um, yeah, it it is a little bit annoying. But if I scroll down, you can see it gave us 19 references, which isn't quite as good as uh, Manis, which did it quicker, but Gen Spark uh, did uh, lie to us a little bit more. Let's check it out.
So, here, this is what we ended up. GenSpark actually ended up lying to us 26% of the time. And if I go through each of the references, you can see yes, yes, it was fine. It was fine. It existed. Then it just didn't exist. And then it just didn't exist. And then it just didn't exist. So this one lies more because over on Manis, we can see that we only had sort of two that didn't exist. Then we had ones that was repeated. And then here and here, um, you know, it uh, it just kind of like got it a little bit wrong. there was a very similar article but uh not exactly the same. So uh yeah, we just need to I think be aware of how different tools are actually hallucinating. And unfortunately, Gen Spark was the worst because it literally just didn't exist. There wasn't sort of like anything similar apart from the last title where it was like similar. It was close to another peer-reviewed article, but really it just yeah, didn't really exist. So overall, this gave us a 26% um sort of like hallucination or failure rate of Gen Spark.
So at the moment for me, Manis is winning. But there's one more tool that I've tested in the past that uh is just really great. Check out this one. So up till now, we need to make sure that every single reference that these tools give us, we need to double check. Now, I have produced a literature review, a really deep, detailed literature review with Gemini. Here's Gemini. I've paid for the advanced one and I used deep research. I used exactly the same prompt in this one and this is what it gave me.
Now, this took the longest. This probably took about 20 minutes and it is detailed. It is deep. It has um lots of tables um you know presenting the information in a really sort of like useful way. This is something that the other AI tools I've tested in this video didn't do. They just gave us text. Here we've got sort of like tables and not just one table, two tables. Then we got it split up into really useful kind of like um uh sections. It really is detailed. And if we scroll all the way down here, we can see that the references are actually there we go are actually links. So, do these links go somewhere and do they actually sort of like lead to real research? That's what I wanted to know. So, I actually exported this to docs because I did want to see how long it was. This is a massive document. It's 61 pages and 105 references. H, that was a long time going through each one, but I did it for you because I love you. Um, and so you can see that uh, yeah, it's a really detailed document. You've got all of the different um, things referenced here. It's not IEEE though. it is just kind of like numbers. So, it didn't really understand that brief. And also, you'll see in a minute that it didn't stick to peer reviewed in a sort of like very strict sense. It did include other references, but let's talk about that.
So, over here, um, yeah, you can see that it's just a really detailed thing. And if, look, let's just cut to the chase. If you were wanting to do a literature review at the moment and you are only interested in sort of like the uh finding the literature, getting themes, um coming up with a generated document, there's no doubt at the moment that Gemini AI is just doing so so well in that space. But that's not what we're here for today. Is it lying to us? I went through all of them and let's uh let's reveal this column. Boom. Here we are. Gemini was a 1% hallucination, but it wasn't really hallucination. And you'll see in a minute. Stay around because uh yeah, it went I went through each one and you can see that it didn't sort of like strictly stick to peer-reviewed papers that were published in journals. Um it did include thesis. It did include a website. I did like this that it did include it was the only one where I was like, "Oh, wow. They've used 2025 research this year." the other ones um they got close but I don't think I saw any like really up-to-date references. I really like that for Gemini. And then the one it failed down here was just like a book that wasn't accessible and that was the sort of like issue is that it did take me to a like a live web page but I couldn't access what it was actually citing. But I would take that over a completely madeup reference any day. So here you can see we got a thesis thesis and then down here we have another thesis but ultimately it gave us 105 references and all of them existed in some like way but this one did not exist.
Now uh there are some drawbacks like I said you know you can't extract this into something you work with in terms of like mendlay or endnote. You'd have to sort of like go and find each source and put it in to your reference manager um manually. But look ultimately it is a really great introduction and a great literature review. Look I love this. I love the tables. I love it that you know you can see here it's actually referenced. It says IEEE. You're just lying to me a little bit there, matey. That's a bit cheeky, isn't it? It says IEEE there in reference, but it's not even in IEEE. Oh well, it's okay. It's okay. I'll let you get away with that because you're doing well in other ways. But ultimately, yeah, this is just such a really great literature review. It's got all of the things. It's detailed. It's deep. it is the definition and I think the benchmark at the moment for academic um kind of uh searching the literature and producing a literature review. So really really impressive. Give them a go and let me know what you think.
So overall this is where we're at. Gemini there is no hallucination. It sort of like got a little bit wrong. Um Manis I really like. If you're going to use Manis just go through and double check. It's got about a 16% hallucination or like failure rate. I guess I can't strictly call it hallucination because it was only two titles that really didn't exist. The other one kind of got a little bit wrong, but there was kind of a reference like it. Um, and then it did repeat. So overall, this is probably less of a, you know, hallucination rate. Um, and then we've got Jen Spark, which yeah, it really did just make up stuff and it didn't give us many references. So in order, Gemini wins the crown. put the crown on. Manis in the second place. Oh, thanks very much. And then Jen Spark in third place. Oh, I'm sorry I did so badly, but I I tried my best. All right, then. That's where we're at with those. Uh yeah, give them a go for yourself and let me know what you find. If you like this video, go check out this one where I talk about writing a thesis with AI. You'll love it.