Transcription
Every researcher dreams of a magic button. One that can instantly synthesize the literature, write a full review, critique the latest papers, and do some of the heavy lifting for you. So, I tested whether or not Chat GPT's Pro version, the $200 version built for researchers, engineers, and anyone who wants research intelligence could actually deliver. And let me tell you, the results either blow your mind or make you laugh. So let's check it out.
So it's this that piques my interest is when they introduced chat GPT pro. I was like, okay, this sounds good, but is this bit a way for researchers, engineers, and other individuals who use research intelligence? Research intelligence. Isn't that what this channel's all about? So I wanted to know, could it actually do it? And look, it's got some good graphs. Look at that. PhD level science questions. Oh, well done, Chat GPT. What do you want? A medal for answering that? And then you've also got this. Well done. Yeah. Okay. Some great graphs. You're winning. I get it. But does it actually stand up in the real academic world?
So, this is what it looks like. So, I paid $200 for you so that we can test it together. And here you can see that I've got Pro, the research grade intelligence. Oh, that's what I love. I want that. I want that so much. And I put it through a number of tasks. The first one was uh pretty good. Check this out. All right. So, I asked it, create a literature review on nano composite self-healing devices. Focus on recent trends, research gaps, and current state of the literature. And it reasoned for 7 minutes and 37 seconds. And you can see all of the reasoning or thinking, I know some people don't like thinking. All of the reasoning it did down here. So, this model thinks very, very deeply about everything you chuck at it, which is a problem for later things. And you'll see what I mean in a minute. But ultimately, this is what it kicked out. So down here, it's got current state of the literature. And I went through it and I really like the details that it went into. And at first, I was a little bit kind of like disappointed. I was thinking, well, what did it actually look for online in those seven minutes? But if you click sources, there were loads and loads of sources. And look, it goes on and on and on. And I think this is exactly what a literature review should be about, which is just all of these things. It's done the heavy lifting for you in terms of going out and finding the papers. And the one thing I really like here is like selected recent and high signal references if you want to get familiar with the research field quickly. So it did everything I asked of it. What it didn't do is create like a literature review in the sense that it was just not pages and pages of synthesized literature. Other tools like Gemini actually do that really, really well. But I was very impressed with the amount of information it gave me and how it organized it. It did organize it in the way that I asked. So, if you're looking for a really expensive way to go find all the literature, it does actually work quite well. Is it worth the extra $200? Uh, not yet. But maybe this next thing will make it worth that amount of money.
One thing that's really annoying in literature is getting loads of papers and trying to synthesize and work out what they mean. So you can put it now into Chat GPT and does it provide extra deep reasoning? I wanted to know. So I put in all of these papers and I said, using the PDFs I uploaded, do a cross-paper synthesis. Create a papers times claims matrix marking support, contradict, and not address. This is the sort of level of thinking I need if it sort of is going to uh be the value that I want from $200 worth of a tool. That's per month, by the way, which is a lot of money. And so, here we are. It reasoned for 15 minutes and 11 seconds. That's actually quite a long time to wait, 'cause I was just sat here twiddling my little thumbs. Oh, there they go. Look. Oh, look. I can do that real fast. Anyway, I'm not here to show off with how fast I can twiddle my thumbs. Um, although I'm an expert at it. Um, you can see down here that it gave me a cross-paper synthesis and it only looked at my PDFs, which was great. And it gave me the claim versus all of the support, contradicting, and not addressed. So most of the time, you know, there's not much overlap, but in terms of giving me a detailed understanding of what's going on in each individual paper and where that matches in this kind of matrix of ideas, it really does help. So I can see that all of these claims here are actually kind of like generated in these papers. But this one is a bit of an outlier. So that doesn't really sort of do anything to the other papers. There's no crossover. And then here we've only got these three here that provide this claim. So overall, I think uh, yeah, it did a really great job. And then obviously we've got a lot of other information which is a bit superfluous. A lot of the times I find this tries so hard that it's almost a little bit annoying, a little bit tryhardy because look, I've got all of this and it's not formulated very nicely, even though it's giving me kind of like where it's finding it. Um, I'm not sure it's really, yeah, doing its best for me. It's just showing off with all of the things that it can provide. Nonetheless, I think if you are interested in deep diving in a range of things, this could be a useful tool for you. Although something like um Notebook LM also does this pretty well. Um, and you can upload a lot of uh papers and it can synthesize the ideas in them. Um, so yeah, you know, is it worth it? You may, maybe for this, I was impressed. It did sort of like spill out information that I was just too much and I didn't really want. But nonetheless, this next thing actually is pretty good and is maybe worth it because uh it's probably one of the best peer reviews I've ever got from an AI. Check it out.
All right, then. Peer review is so very important. And if you had a way of getting a genuine grumpy peer review on your paper before submitting it, it surely would help your career, wouldn't it? Well, this grumpy research grade AI may be what you're looking for. So, here we've got uh one of my papers that I uploaded. And then I said here, act as a journal reviewer to review the attached paper. Produce a structured review and about this with major issues, minor issues, statistical or methodological checks, and for each major issue include um those and a concrete fix for additional analysis. So it reasoned for six minutes, which is quicker than a normal peer reviewer. They're only just starting sipping on their tea in six minutes. But here I've got a summary and it's just like what someone would say about it. Now here is the major issues. Now, this is an accepted peer-reviewed paper. So, let's just see if this shouldn't have been accepted. Let's check it out. So, the first thing I was actually very shocked at is that equation 2 appears to be incorrectly rearranged. No, but I did actually go out and I found the original reference that I referenced, and it is right according to that. So, is that reference wrong? I'm not very good at rearranging equations to be honest. So I looked at it and I was like, well, no, I think this is okay, actually. Like I, I believe that this is uh correct, you know, like it's it's doing it's picked up something that is right and it is wrong. That's what I'm trying to say. Okay. But the rest of it I think is actually pretty good. Like it really did a deep dive into the paper and used proper reasoning. There's things here like annealing conditions, mismatch, severity high. We've also got like flexibility testing where I only did it once on one substrate. And yeah, that is probably a bit of an issue. Um, why wasn't that picked up? And also down here we've got figure of merit bias where I actually report on different um like figure of merits in a probably a way that benefits my paper more than the science and like makes me sort of like big note myself a little bit. Um, and also here in the title it says high throughput, but we didn't actually demonstrate high throughput. We just said it was a pathway, a potential pathway to high throughput devices. So anyway, I really think that this has captured the grumpiness of a peer reviewer perfectly. Um, it really goes into a lot of detail and I actually think this is one of the best peer reviews that I've got from AI. And uh, that's maybe because it's a little bit rude. And it clearly um, yeah, like it it goes into deep. And then we've got minor issues here. Okay. Terminology. We got typos apparently. Um, flexible OP devices. Okay. That should be oh no, itto row shows 0.6.2 and it should be 0.62. Oh no. So it's all of these things that you could just double check. These minor issues are those annoying things that give a peer reviewer kind of a sense of how much care has gone into a paper. And if you can pick up on all of these little things and clearly, you know, they've got less to complain about. Um, we've also got statistical and methodological checks. And here it says ev um warrant major revision before acceptance. Well, it's already peer-reviewed and published. So, take that, Chat GPT. So, yeah, I think that is almost worth the $200 because it really captures the essence of a grumpy peer review. Let me know if you've tried it for your paper. And also, this next thing is surprisingly rubbish. You'll see what I mean.
All right, then. I've paid good money for this. So, can it do images? Now, I wanted to know if it could create a graphical abstract. And I said here, create a professional graphical abstract based on the following text. And I put in one of my abstracts. And I wanted to know whether or not it could produce a uh graphical abstract. And it thought for six minutes and come up with this rubbish. Look at this. This is terrible. It is so terrible. It's And I think this is the problem. When you use this pro mode, it tries its very hardest to a fault because it's like bringing a tank to a knife fight. Like it really just overthought it. It did it in a weird way. And just to put it into perspective, Chat GPT, like the normal one, the like the not the pro version, just like the paid version, the $20 a month one actually does this really well. This is what I was able to do with uh just the non uh pro version, but it's still paid. Uh, this is just Chat GPT5. And I put in exactly the same prompt. And there we go. Yeah, that's close to what I want. It's got good text. It's got silver nanowires. Well, they're carbon nanotubes, but that's fine. We can deal with that. Um, these would probably should be swapped. Those should be silver. But, you know, these details can be changed in Canva. Go check out my other video where I talk about, you know, using Canva for creating graphical abstracts from AI images. Um, entangled opto-electronic applications. That's a weird thing that I put that like that little thing here maybe. But ultimately, this is much, much better than this rubbish, this mash of weird things that make no sense. So yeah, it really, really fails when essentially you need a quick answer. Like if you need a dumber um AI to provide a better answer, then this just overthinks it, overcommits to an idea, and it provides you with rubbish. So uh, yeah, sometimes you just need the quick, simple uh, non-fancy AI to give you something you can actually use, like this. So, um, at the moment, I think for research, unless you're in the peer review process and you really need that grumpy deep research kind of, um, uh, deep dive into your work, it's probably not worth the upgrade at the moment. I think Chat GPT like five and other AI tools do a lot of this a lot better. Um, but I hope that in the future these reasoning models can help you just with the uh process of science, coming up with research ideas, coming over getting over problems, um, you know, generating first drafts of of research articles. Um, but at the moment, I think uh, yeah, it is lagging behind, unfortunately. Is it research grade? Um, I mean, it can answer questions like a PhD student, but can it do the work? Let me know what you think. If you like this video, go check out this one where I talk about using Chat GPT agent, and I think it's a little bit better than this. Go check it out.