Transcription
[Music] You don't care if you're right. It's the way academics check each other's research papers. It ensures that only the good ones are published and prevents the bad ones from getting through, right? Wrong. Peer review does precisely the opposite of what you think it does. It prevents the good papers from being published and ensures that only the bad ones get through. Peer review is suffocating science. If we want to reverse the stagnation of science over the last 50 years, we've got to get rid of peer review.
In his splendid article, "The Rise and Fall of Peer Review," Adam Mastroianni describes peer review as a scientific experiment. We tend to think that peer review is just the way science works, but science has been working rather well for centuries before peer review became a thing. When Newton published his laws of gravitation three centuries ago, his work wasn't peer-reviewed. When Einstein published his laws of gravitation one century ago, his work wasn't peer-reviewed. At least, Newton and Einstein weren't peer-reviewed the way we do peer review today. Their peers did review their work. Their fellow scientists heard about their ideas, read their papers, and made up their own minds as to whether these new theories were revolutionary or rubbish. The way we do peer review today caught on around 1973, when the journal Nature began to require it. Yep, 1973. You'll no doubt recognize the year from my episode, "Why Has There Been No Progress in Physics Since 1973?" That's a coincidence. Correlation is not causation. Pure coincidence, right? Anyway, if peer review is an experiment, shouldn't we be assessing after 50 years whether or not it has been successful? And if peer review is intended to ensure that only the good papers are published and the bad ones are prevented from getting through, shouldn't these be the criteria for assessing whether it has been successful? Does peer review prevent the bad papers from being published? Do the good papers get through?
Okay, so here's how peer review works. You do some research. You're pretty excited by what you find. You'd love to write it up in simple, clear language, but you're not allowed to because the peer review process requires that you use opaque language that's full of technical terms. You'd love to write it up in an engaging way that communicates your excitement, but you're not allowed to because the peer review process requires that you do it in the driest possible way. Okay, so you submit your dry, opaque paper to a journal. You wait an interminable amount of time because the peer review process assumes that your research is inconsequential, so there's really no hurry to get it out there. A couple of academics, the type who would rather spend their time criticizing other people's papers than doing research of their own, get back to you with their objections. Generally, these objections follow the form: "Hmm, well, yes, that, that research you did, that's not how I would have done it. I would have done it like this. Do it exactly the way I would have done it, and maybe it'll pass muster." Now, you have a choice. You can redo your research exactly the way these academics would have done it, assuming, of course, that the academics would have done it in the same way as each other. Or you can submit your paper to another journal and hope you get different academics to get back to you with different objections. At that point, you have another choice. You can redo your research in exactly the way these other academics would have done it, assuming, of course, that they would have done it in the same way as each other. Or you can submit your paper to yet another journal and hope you get, yeah, different academics to get back to you with yet different objections, and so on. You can see how productive this process is for the advancement of science. But this process, however ponderous, prevents bad papers from being published, right? Wrong. With millions of academic articles published every year, there's no shortage of bad science. Indeed, there seems to be no shortage of fraudulent science. Some scientists simply make stuff up. Studies don't just make it into unknown journals; they make it into respectable journals like Science, Nature, and The Lancet. In other words, they make it past peer review. The thing is, peer review isn't designed to detect fraud. The academics who peer-review papers don't check the data in those papers to make sure it hasn't been just made up. I mean, how could they? To check the data, they'd have to reproduce the research in its entirety. People, if you can't detect fraud, you can't detect error. You can't even detect sloppiness. Peer review can't prevent bad papers from being published.
Thank you. If peer review does no good, then at least it does no harm, right? Wrong. The physicist Max Planck proposed a principle that's often paraphrased as "science progresses one funeral at a time." He knew that scientists are not very good at changing their minds. They learn theories when they're young. They invest decades of their lives in those theories. When they're old, they're so invested in those old theories that they're not so interested in any new theories that come along. They don't have the decades left in them to learn those new theories, so they dismiss the new theories. The new theories slowly gain ground, not through the old scientists changing their minds, but through the old scientists dying off. Science progresses one funeral at a time. What could be worse than having to wait an entire generation for any new theories to gain ground? Well, what would be worse would be giving the old scientists the power to prevent any new theories from being published. And that, of course, is precisely what peer review does. Peer review requires young scientists to get permission from old scientists to publish anything new. It's difficult to overstate what a disaster this is for science. If you have to get permission from those old scientists, the ones who, according to Planck, never change their minds, the ones who, according to Planck, must die before science can progress, well, then you can forget about publishing anything new. The only research you'll be able to publish is pedantic elaborations of the entrenched theories. And since, in academia, your career depends on publishing papers, this is the only research you're incentivized to do. The chilling effect of peer review is to encourage incremental improvements on tired old theories while discouraging bold new ideas. Peer review doesn't just prevent good papers from being published; it prevents good research from being done.
Okay, let's take a step back. Which is more important: preventing bad papers from being published or letting good papers through? Well, this is an easy one. Do you remember all those bad papers that were published in 1905, long before peer review was put in place to save us all from bad science? Well, no, of course you don't. I bet you can't name a single bad paper that was published in 1905. All the mediocre monograms, all the misconceived experiments, all the mistaken theories, all are long forgotten. Okay, so can you think of any good papers that were published in 1905? I can. Four papers come to mind: one establishing the existence of photons, one establishing the existence of atoms, one introducing the theory of relativity, one establishing the equivalence of mass and energy, you know, E=mc². Each of these four papers, published by Albert Einstein in 1905, was truly revolutionary. None of them was peer-reviewed. If they had been, Einstein's so-called peers would no doubt have prevented their publication, complaining correctly that his theories didn't accord with the well-established laws of motion, matter, and light. So, which would have been more important: preventing all the long-forgotten bad papers of 1905 from being published, or letting Einstein's papers through?
Okay, so peer review doesn't work. It doesn't prevent bad papers from being published. It does prevent good papers from getting through. Worse, it prevents good research from being done. But what's the alternative? We can't let just anyone publish just anything, surely? We need some kind of authority to put a stamp of approval along good science, surely? And we need some kind of authority to keep the bad science from corrupting how impressionable minds, surely? Well, I'm not so sure. In practice, this censorship might be preventing today's Einsteins from publishing their own bold new ideas. Indeed, discouraging them from ever embarking on what might prove revolutionary research. In practice, that stamp of approval isn't just giving us confidence in good research; it's giving us undue confidence in bad research, too. In trivial research, in sloppy research, in erroneous research, in downright fraudulent research. Which leaves me wondering, why exactly can't we just let anyone publish anything?
Since I launched the last theory to explore Wolfram physics, I've heard the same criticism of Stephen Wolfram over and over again. I hear furious complaints that none of the research that's come out of the Wolfram physics project has been peer-reviewed. I could respond by saying that that's not actually true. Jonathan Gorod, for example, has published papers on the Wolfram model in peer-reviewed journals. But I'm not going to take the easy way out here. The fact is, Stephen Wolfram has very deliberately circumvented the peer review process, publishing many thousands of pages of articles and books and many hundreds of hours of podcasts and videos without asking anyone's permission. And that's a good thing. The fact is, none of these articles, books, podcasts, and videos would have passed peer review. First up, peer review is designed for scientific papers like, you know, on a sheet of paper. Articles on this new thing called the web, not on paper. So, not allowed. And books, podcasts, videos, definitely not permitted as a way to promulgate science. Also, peer review is designed for a very specific form of research, written up in a very specific way. It's very, very important that scientific papers be written in language that's impenetrable to anyone other than a few dozen academics steeped in the minutiae. If it's written in language that's so simple and accessible that anyone can understand it, then it's just not science. And it's very, very important that scientific papers be crammed with the equations and derivations of the established mathematical paradigm. If it's crammed instead with the algorithms and simulations of a new computational paradigm, then again, it's just not science. Stephen Wolfram was right to put out his ideas on the web. His articles, books, podcasts, and videos have inspired others to pursue his ideas. Even if they're not scientists, they've inspired me to pursue those ideas, and I'm definitely not a scientist. I love this open way of doing science. Well, of course I do. I'm the founder of the Open Web. Of course I love all things open. Wolfram physics might prove to be a true model of our universe, and it might not. Either way, Stephen Wolfram's putting it out there, unsanctioned, uncensored, unobfuscated, unfiltered. In his thousands of pages of text and hundreds of hours of talk, he has given us all the opportunity to explore that possibility. Peer review would have never let these ideas see the light of day.
[Music] Peer review is suffocating science. If we want to reverse the stagnation of science over the last 50 years, it's got to go. Thanks for listening to The Last Theory. Join me for fresh insights into Wolfram physics every other week. Subscribe to the free newsletter, podcast, or YouTube channel at lasttheory.com. After all, this might be the most fundamental scientific breakthrough of our time. [Music]