Understanding Statistics and Research: Even if You Really, Really Hate Math

So let's go ahead and get started. Let me start by introducing myself for those who haven't seen me speak or don't have any idea who I am. I'm Steve Dykstra. I'm from just outside Milwaukee, Wisconsin. I grew up in Wisconsin my whole life. As I say before all these presentations, it's less relevant to this one, I'm a little bit unique at these reading conferences. I've never taught anybody to read, I've never done any reading research. I'm known mostly for punching a certain aspect of the world in the nose occasionally, in a somewhat entertaining fashion, for advocating, I think very passionately, for our children and for our cause. And I hope that at times I put things in a different light and have had some success explaining things that other people have figured out. And I help to explain those things and carry those things to other people. So I'm really happy to be here with you today.

The title of our presentation, as you see here, is very audaciously, "Understanding Statistics and Research." The subtitle is, "Even If you Really, Really Hate Math!" That's important because we're not really gonna completely understand statistics and research. You're not going to come out of here today, I don't think, with a comprehensive understanding of everything that goes into statistics and research, how the two interact together, how research design dictates what kind of statistics you use and the kind of statistics that are possible should influence your research design. And the two of them together define the limits of what you can do with the results. You know, one of the things that's most frustrating to me with research, this happened very recently. Somebody was criticizing a study, a research study in reading, and they said, "But it didn't show that synthetic phonics is better than analytic phonics." And I said, "Yeah, it wasn't about that. That's not what the study was designed to show." But they thought they had me over a barrel when they pointed out than it didn't. And so I pointed out to them that it also didn't prove that the Earth goes around the Sun, 'cause it wasn't designed to prove that, but I'm still reasonably confident that the Earth goes around the Sun.

So understanding statistics and research, even if you really, really, really hate math, I can't put enough reallys in there. I mean, I'm not being critical of anybody. And this is a little unusual for me, 'cause I always really liked math. I'm one of the few people in the world with a psychology degree who has two semesters of calculus. I took statistics and advanced statistics. I liked math. I came from a scientific, I came from a mathy household. My father is a chemist, and there was lots of math and chemistry and science laying around. So I have kind of a different take on this, but it's undeniably true that in education, I think particularly in elementary education, I don't think a lot of people say, "God, I love math. I love advanced math and statistics so much. Let's go be a first grade teacher." You know, those are somewhat different populations of people.

But as we talk about these things, we find ourselves more and more in society being asked to consume science. And today is not so much about doing science, doing research or doing statistics, but about being good consumers of science and being good consumers of statistics. I hope I don't need that, 'cause he just... (audience laughing) I've used it before. That's true. I don't need it, so that's good. So being very good consumers of science, because we're being inundated with science and sciencey things, and things dressed up as science that pose as science. You know, it's Halloween tomorrow, you know, and it's in the world of science, it's Halloween every day, because somebody's putting on a costume pretending to be science and knocking at your door and saying, "Buy this." And you really need to send them away. And hopefully after today you'll be a little bit better being a consumer of science.

So before we begin, there won't be any math. I can promise you that in a million different ways. There really won't be any math. You won't need the calculator on your phone. You won't need even to count anything. Things may get a little mathy once in a while, so no actual math, but maybe some mathiness. And I apologize for that in advance. There'll be some graphs and things which imply numbers and imply math, but I've stripped all the math away from them. If the mathiness gets to be a little bit too much for you, I encourage you to breathe. You know, hold the hand of somebody next to you. And this is supposed to be both informative and fun. I'm glad to be going near the end of the second day. This should be fun. This shouldn't be heavy stuff. We don't want anybody to be intimidated by it, so we're supposed to have fun. If it's neither informative or fun, that's your fault, not mine. (audience laughing) And so do your part by asking questions, trying to keep a good attitude, and literally, and truly, truly ask questions. I think it'd be interesting to ask questions. I think there are people in the audience who probably know more about this stuff than I do, but I have the microphone, so you're gonna have to live with my answer.

So our learning objectives, and I don't normally do learning objectives, but for some insane reason when I wrote this, I was inspired to sort of formally put in learning objectives. So gain a basic understanding of what statistics are trying to tell us. You have to understand what statistics are trying to tell you. You know, there's this famous quote from Benjamin Disraeli, you know, when he was Prime Minister of England, that there are "lies, damned lies, and statistics." You have to remember that was told by somebody who had been caught lying and cheating, so this was his excuse. He was insisting that people were misusing statistics to catch him in his lies and his cheating. They were not. The statistics were telling the truth.

Get a sense of how to consider experimental design when weighing the meaning of statistics. You can't understand statistics unless you understand the design of the study that created the data upon which the statistics have been done, because that's where you figure out the limitations of the statistics. I told the story, you know, I mentioned earlier that, you know, you can't use a study on reading to say something about how the Earth orbits the Sun. It's not designed for that. But you can't say, you can't use a study on one aspect of reading to say something about another aspect of reading. And the devil is often in the details. And there's a lot of research, a lot of "research" out there that's being billed and promoted and pushed as telling you something that it was never designed to be able to tell you.

Get better at spotting sloppy science, and I'll say the word outright bullshit, because there's an enormous amount of sloppy science out there, and even more outright lies. I mean, people just putting things out there that are not what they appear to be. I suspect for this audience, it's gonna be a lot of fun. I've been billed now to give the same presentation in Columbus, Ohio, you know, the mothership of Reading Recovery, to a favorable audience, but in an unfavorable town. And there's some things I'm going to say later about some science about Reading Recovery that might get me killed, so this might be the last chance, this might be one of the last chances to see this.

I'm gonna encourage you, this is a name you've probably never heard before in your mind, but I'm gonna encourage you to build a shrine at least in your head to a guy who's been dead for a long time named Thomas Bayes. Thomas Bayes had some really important insights on how we interpret data and statistics and science and truth and fiction, and how we interpret that in an ongoing way. If there's anything that drives, besides the things I've described, one of the other things that really drives me crazy is people asking me, "So, tell me what the study is that says this," as though there's one study, as though there's one study that says this. As Professor Moats pointed out earlier, these are things we discover from an enormous body of science, of converged science, that if I look at the problem this way, it seems to converge over here, and if I look at the problem this way, it seems to converge over here. And look, as I look at the problem from lots of different angles using lots of designs and methodologies, they all seem to be converging around the same answer. Converged science, converged knowledge is what science is really about. It's not about, and I'm not going to, you know, name them, What Works Clearinghouse. It's not about counting studies. It's not about counting studies in isolation from a larger body of research. That's a prescription for being duped. It's a prescription for being tricked and sold a bill of goods.

And get vaccinated against the most common statistical misunderstandings. There are misunderstandings in science and statistics and how they work, and you're gonna get vaccinated against them today, which is a good thing, 'cause it's the flu season and vaccines are good. Always get your vaccines. And the first one is to P or not to P. And that doesn't, you know, if you don't know what that means, you're gonna know what that means. So to understand what the letter P means in statistics, and why you need to understand what it is and what it isn't. And then what the hell is an effect size anyway?

So we're gonna get on with that. Statistics, the design of a research study, should dictate the statistics that you do. Statistics should not be limited. The statistics should not be decided by the kind of statistics a researcher likes or by some effort to trick you or deceive you. Research determines the kind of statistics. Certain kinds of research studies, correlational studies naturally lead to certain kinds of statistics. Multivariate studies force us into certain kinds of statistics. And the other thing that determines the kind of statistics we use is whether the variables we use are what are called continuous or discontinuous. So reading achievement is a more or less continuous variable. It sort of moves along a variable continuously. And we should use statistics that acknowledge that people's performance varies continuously. If we are doing a medical study, and the outcome of the study was going to be measured by whether you were alive or not, so there were only two possible outcomes, there's alive or dead. Alive and dead is a discontinuous variable; you're either alive or you're dead, and that would force you to use different kinds of statistics. And those are just sort of simple examples of how statistics and research naturally feed off of each other.

And there are two kinds of statistics, and therefore two kinds of research. And this is something, this is very basic stuff. I don't know if everybody's heard this before. If you did, many of you forgot. I'm assuming you forgot, so that's why you're here today. But there are descriptive statistics and inferential statistics, and they're exactly what they appear to be. Descriptive statistics simply describe something. You can't make inferences from descriptive statistics. You can just describe something. You have to do something else with them later. You have to plug those descriptive statistics into other statistical formulas to drive inferential statistics. Common descriptive statistics in our life are things like the percentage of kids who pass a given test, the batting average of your favorite baseball player, the free throw percentage of Stephen Curry. These are just descriptive statistics. You can't infer anything on that. Stephen Curry makes 92% of his free throw. Somebody else makes 88% of his free throws. Is Stephen Curry, can I infer that he's a better free throw shooter than the other person? You need to know more than just that to make a judgment about who's the better free throw shooter. And that gets into inferential statistics.

So descriptive statistics can tell you how many or how few of something they are. Simply counting things in the room. I could count all of you. I'm not going to. I could ask you to count off. I'm not going to. And that would give us a descriptive statistic. How many people were in the room when Dykstra got up and gave his talk? The average, the middle of those things. We could have you all stand up and we could measure your height, and we could describe the average height of people in the room at the time that I gave this talk. That would be a descriptive statistic. How spread out those things are around that middle. That's a very important thing to keep track of. So the average of the middle of something, there's different ways to measure that. The most common one is an average or a mean. We describe that as central tendency. Where does the middle of the group tend to be, and how spread out are of those things around the middle? Some measure of variance. 'Cause we could have a room in which the average person in this room was 5'4". We could have a room where the average person out of 50 people was 5'4" because everybody in the room was 5'4". We could have another room where the average height was 5'4", and they could range in height from like two feet to like 8'3", and the average height could still be 5'4". So we need both of those statistics. But that's about it. That's about all that descriptive statistics can tell you, is those really basic simple numbers. But from those numbers, everything else comes.

Now this is, I want you to recognize that everything that's covered with a beige block was actual math that I covered up for you so that nobody would get upset today. And I had to learn certain PowerPoint and word processing skills in order to do that. And this was hard for me, 'cause there was math back there. There was little lowercase Greek letters and all kinds of stuff that makes my heart sing. And I thought, "It's sad. I'm covering up the Greek letters. I like the Greek letters." But as you see here, we have four distributions, four graphs. It's a little mathy, but it's not math. We have four distributions, and what you see here is these three in the middle all have the same central tendency, but all have very different levels of variance. We have a green one over here that has a different central tendency, and has its own degree of variance around that central mean. Interestingly enough, the amount of area underneath each of these graphs, so that's kind of mathy, but the amount of area underneath all of these are exactly the same. So if this was describing a population, those would be populations of the same size, but they would have different characteristics. These are just descriptive statistics. That's all this is. But from descriptive statistics, everything else eventually grows.

Inferential statistics, on the other hand, these are the really good statistics, unless you don't like the results of your study, in which case then they're really bad statistics. If somebody brings you inferential statistics that tells you something you don't wanna know, then you say something like was said to me in a meeting about reading. I brought somebody a study. I say, "Here's some empirical data." And she passed it to somebody next to her, and the teacher said, "What's empirical data?" And she whispered, she said, "It's the bad kind."

So inferential statistics ultimately tell you, I'm making this as simple as possible, but it's all very true. They ultimately tell you only one of two things. They tell you either how different things are when we predicted, note the fact that I've made predicted a different word, a different color, 'cause it's such an important word here, either how different things are when we predicted they would be the same. The other one is easy to predict, understand what's coming next, or how similar things are when we predicted they would be different. These are the two things that happen in research. Either I predict that things are going to be different and I check to see that they were actually the same, and I'm surprised, and go, "Woo, they were the same." Or I predict that they're going to be the same, and it turns out they're actually different.

Now here's a twist on research. How many of you in the course of your studies ever heard the phrase "the null hypothesis?" The null hypothesis is this dirty little secret of science that most people, the average person doesn't understand. That is for the sake of research, if I set up a study and I say that if my child goes, if this group of children who are struggling to read, go to tutoring 30 minutes a day every day, five days a week for 10 weeks, they're gonna read better than a group of kids who are at the same level who didn't get that. That's my prediction, but my null hypothesis is it's not gonna happen. My null hypothesis is that they're going to end up the same. So I'm trying to disprove that, and I have to use inferential statistics to see, did they actually turn out to be different, or did they end up still being the same?

On the other hand, when we look at correlational studies, a lot of times we're looking to see how similar things are when we predicted that they were actually different. Let me ask you a question just off the top of my head. And we're not going to do it. I'm not like that ridiculous. But if we went and got some descriptive statistics on all of you, say height and weight, or height and shoe size, or height and inseam, or something like that, how many of you think height and weight are very different statistics? How many of you think height and weight are nearly identical statistics? How many of you think height and weight are really, really similar statistics? We could do correlations on height and weight. We could correlate everybody's height with every weight, and we would find, I guarantee you, a positive correlation. Taller people tend to weigh more, and shorter people tend to weigh less. There's a lot of variability around that, but you would get something of a correlation. They're more the same than people might sometimes realize.

What if we went into your classroom and we measured statistically how well all your kids could read and how well all your kids could do math, and then we correlated them with each other? Would we find a positive correlation? That is, would they be more similar to each other than we thought? Would it turn out that the kids who read the best also tend to do the best in math? That tells us like a Venn diagram, and you're gonna see one of those in a minute, another mathy thing. Well, we're gonna have a Venn diagram that tells us that while we may predict that math and reading and math skill and reading skill, we might start out by predicting they're completely different from each other. It turns out there's an enormous amount of overlap, while the outcome of the test looks very, very different. Some of the skills and abilities and the neurology that underpin those skills share a lot more in common sometimes than people realize. I said to somebody one time, if you wanted me to pick the best math students in third grade, but you wouldn't let me give a math test, I would just give a reading test. And they said, "Well, that would be ridiculous." And I said, "Well, I'd like to bet you a lot of money that it wouldn't be ridiculous." And they communicate.

So inferential statistics communicate this information by telling us how likely it is that things would look this way by random chance alone. How likely is it that things would look this way by random chance alone. If you just did this, if this just happened randomly, for instance, if I had a big bin of ping pong balls and there were half the balls were black and half the balls were white, and I started pulling, I just started looking in, not cheating, just pulling balls out randomly from inside and you started sorting them out, you wouldn't expect them to come out exactly 50/50. If I had 10,000 balls and I pulled out 100, you wouldn't necessarily predict 50 black and 50 white balls. But how different would they have to be before you should reasonably suspect that I cheated? Statistics tell us how likely it is that whatever is happening with me pulling the balls out could just happen by random chance alone, and how different does it have to be before you should think that something is up?

The Securities and Exchange Commission and other major political bodies in the world use some very advanced number theory about the distribution of numbers in the world. So certain digits come up more than others, and there's a very advanced reason for that. So when people file reports that should describe random activity and certain digits don't appear, certain actual numerals don't appear as often as they should, that's a sign to international governing bodies that you cheated. Now, the good news about this is the math that lets them predict that is so freaking complicated and advanced that people who cheat are too lazy to learn the math that would allow them to trick the system. So they've caught people falsifying medical information and securities information and stock trading information just by being able to start with how likely it is that this could happen just by random chance alone. But remember, it's just likely. It doesn't mean it didn't happen by random chance alone.

What's the likelihood that one of us is gonna be hit by lightning on the way home? Extraordinarily low. Now, God forbid one of us gets hit by lightning on the way home, doesn't mean something nefarious and non-random has gone on. Really unusual random things do happen, which is the reason that you can't be overly dependent on one study. 'Cause one study could just be one weird random study. A guy did a study a number of years ago. It made "The Today Show." He gathered some very legitimate data. He didn't fudge his data, but he gathered some data that showed, appeared to show that a particular rare species of West African frog could predict earthquakes. West African frogs cannot predict earthquakes. But in one study, the random vagaries of science and chance sort of waved in the breeze a certain way and appeared to show data that these frogs living in Aquaria, someplace, I think, in South Carolina or wherever the hell it was, were able to predict earthquakes. So other people then, important word in the world of science, tried to replicate that study and couldn't do it.

You'll all be interested to know that all of whole language as we understand it, you know, as it descended from Ken Goodman, was based on one improperly done study that was poorly designed, and by the time anybody could come out and replicate the study and show that it actually showed the exact opposite of what dear Ken Goodman thought he found in his original study, whole language had taken off like a wildfire, and by then they said, "Ah, we don't care anymore." Nobody cared that they couldn't replicate it. Nobody cared that the actual attempts at replicating it, that corrected the design flaw, showed the exact opposite. By then it was a phenomenon. It was funny that a movement which ostensibly started because of the results of a single empirical study, when shown to be empirically false, then turn around and say, "This isn't the kind of thing that can be proved empirically." It's just, it's a heuristic. It's just an emotional thing that we just know is true.

No matter how fancy they are, all inferential statistics are built from the same basic descriptive statistics. Everything you read, every study you read that says it's got some version of a Danish, you know, a Danish modification to some finished version of a multi-variate statistic based on with this process, and it's all written out in language, and your head starts to spin, and you think, "I don't think I wanna read this anymore." Ultimately it's all based on those same basic, simple, descriptive statistics which are built up through a very fancy set of mathematical equations. They use the same basic measures of samples. Samples. If you wanna study all the first graders in America, you can't study all the first graders in America. There's too damn many first graders. By the time you round them all up, they're second graders. I mean, you can't even study them as first graders. You have to study a sample. You have to study a sample. How many of you are familiar with the National Assessment of Educational Progress? One reason that test can be so good, 'cause they make no illusion about testing every fourth grader in America. They test a carefully selected sample of fourth graders. Here's another thing. The test is so long and so good that one person doesn't even take the whole thing. Those students who are samples of other fourth graders each take a sample of the test. You gotta do some really good math and be really on your game to pull it off. And then the results have to align with all the other times they've done sampling. That's why really high quality testing is really hard to do. But they all use the same basic measures of samples, and they do fancy math on them to tell us how similar or different two samples are and how likely it would be to get that result from chance alone. Regardless of any statistic you're ever looking at in any study that's ever been done, that's all they're doing. They're all doing some version of that. They're doing a different version of that, because they had a different kind of study using a different kind of variable done with a different kind of sample. Whether it's an ANOVA or some other multivariate statistic, whether it's a regression analysis, you see a million different kinds of regression analysis.

By the way, most of the most advanced statistics we have now are simply made possible by computing power. When I first started in graduate school, I could do ANOVAs by hand. You can't do multi-variate, linear regression path analysis with a Danish twist and a Finnish topper. You can't do any of that. You can't do any of that by hand. Nobody can do that by hand. It's impossible. I learned to do ANOVAs using a thing, using a matrix called a cornfield tookie, which was invented and developed by statisticians in Iowa. And there was a little song, a little mnemonic that went along with it so you would understand, you had to figure out what the denominator was for a fraction that you would calculate, and a cornfield tookie helped you choose the right denominator. And there was a little song you sang in your head so you would know like there was this matrix and you pick the right one. Nobody does a cornfield tookie anymore. This is all done for us by Bill Gates, you know? That's all there is. So as you see, I've scribbled out the math here. (audience laughing)

All this shows, so we see two distributions over here, one represented in blue, and one represented in some shade of red that mostly overlap but not quite. They're the same height, they're the same width, they're exactly the same shape. How far apart would they have to be before we would say, to a reasonable level of confidence, they're different? Inferential statistics help us make that statement that they are different. These two over here. How much do they have to overlap before we would say they are the same enough that I think they share something in common, and how much do they share in common? By the way, looking at this one over on the right, what's the accepted standard for how confident we should be before we say that they're different? The likelihood that it happened randomly should be below what number? Anybody know?

"5%."

"5%." 5%. You know what the standard is in physics? They should have to be like one millionth of 1%. The reason for that is in physics they can control all kinds of other crap. You know, when you fire a bunch of photons down a racetrack, a super cool helium racetrack under Switzerland, and you speed them around at the speed of light and you slam 'em into each other, you control everything. You control where the photons came from, you control the magnets that speed them up. You control everything. When you're studying human beings, you don't control squat, man. I mean you have like no control. So we give ourselves the generosity of saying when there's only a 5% chance that this is just random, we'll say they were really different. There's still a 5% chance we weren't, that this just happened randomly. But it's hard to get to better numbers than that. It's also why we have a higher need to replicate those studies, because we have more uncertainty. And the other problem with that, and it should be obvious, is that even when you think you're controlling for these other variables, a lot of times you forget something. You're not controlling for variables. And we'll talk about some studies where they did a horrible job of controlling for variables.

So reporting similarity and difference. Some kinds of designs and measures report this with these other letters like T and F and a few other letters of the alphabet. Those reflect certain kinds of research. ANOVAs report as an F value. T-tests, remarkably, report it as a T value. Other kinds of designs and measurements report them as lowercase r. That's a symbol that's typically used to refer to the correlation between things; r squared refers to the variance, and a few others. Here's what you need to know about that. This is the one thing you have to remember about that. You don't care. You really don't care. Unless you really wanna be a statistician, when you get to that part of the study, God bless you for picking up the study and trying to read it. When you get to that part of the study, here's what I want you to say in your head as you read it: "Blah blah blah, blah blah blah blah, blah blah blah, blah blah blah blah." This is the one time I'm gonna pay an homage to Yetta Goodman, who said, "When you came to a word you didn't understand, just say 'Moses' and move on." That's a really bad idea when you're reading for meaning, until you're doing this, and then that's exactly what you should do. You just go, "Moses, Moses, Moses, Moses, Moses, Moses." Because you're not going to understand it. I kind of understand it. I've forgotten a lot of this stuff. I kind of understand it. There may be some people in the room who actually understand it, but we don't need them. You can't call them all up, right? You can't do that. "Steve, what does this mean?" I'm gonna remind you, you don't care. And there's a good chance what I tell you if I try to explain it to you, I'm gonna be wrong anyway. So who cares? It doesn't matter. Don't be distracted. It's just a distraction. That's one of the challenges of reading research.

One of the things I've become... How many of you are familiar with me on SPELLTalk? So one of the things I'm most famous for, I think, on- Oh yeah. Oh yeah. One of the things I think I'm most famous for on SPELLTalk is a study comes out, and like seven minutes later I've explained what's good or bad about it. The reason is, I don't read all the crap that you don't have to read. I'm not reading all of that. I'm looking at the design, I'm looking at the methodology, I'm looking at certain basic aspects of the results, and I'm starting from the... And if I find a fatal flaw, I don't read the rest of the study. There's a reason we call it a fatal flaw. That sucker just died. You don't have to, you know, it's sort of like, if you, you know, if doctors find a body and it has no heartbeat, they don't take their temperature. You know, you don't need to do that. There's fatal flaws in research. You don't need to go any farther when you encounter a fatal flaw. And we're gonna talk about some research in a little while, which was rife with fatal flaws.

Reporting the likelihood of the same results. So you'll see things like this, that P was less than 0.05. I suppose this violates my promise to not have any math. You know, for those of you who had a hard time your whole life remembering greater than and less than and all this kinda stuff. So p, meaning the probability that this happened by chance alone is less than 0.05. Here in this case, this is probability reported as a decimal, so 5 would be a 5% chance. The smaller that number is, the better. You'll see 0.05, 0.01, 0.001. I got a result in my doctoral dissertation, there was like seven zeros and then a one. I was really, really proud of that. It was meaningless data, but it didn't matter, 'cause I got a really good result. You know, when you're doing this kind of stuff as a graduate student, like you do not wanna go in and defend a dissertation that got no statistically significant results. It may actually make an important contribution to the research. Like if you go out and do a study, this is one of the things that happens, too. There's this thing in publishing, that there's a strong bias for positive results. If you go out and do a really good study, a really good study of Reading Recovery and its effect on kids and its comparison to other ways of reading instruction, for one thing, you'll be the first one. (audience laughing) So besides that, if you show no positive results for Reading Recovery, you may find it very difficult to get that study published, simply on the grounds that people don't like to publish studies that have no positive results. So one of the things that happens is somebody does a study, a graduate student or a faculty member does a study and gets no result and they throw it in the garbage, and somebody else someplace who didn't know about that does the same study, gets no result, throws it in the garbage. If 50 or 100 other people do the same study, get no result, throw it in the garbage, the 101st person, not knowing that the first 100 people did it, do that same study again, gets a positive result and publishes it, and everybody goes, "Wow, this is really amazing. Frogs can predict earthquakes." Because nobody saw the first 100 studies that said frogs can't predict earthquakes. And you don't have a chance to realize very easily that it's just in this percentage down here that was just random chance. This was just, if you did the study 100 times, 5 of them should come up randomly, should come up with positive results. Now, interestingly enough, 2 1/2 of those will come, I don't know what half a study looks like, but 2 1/2 of them will come up with a positive result one way, and 2 1/2 will come up with a positive result the other way, which refers to a feature of statistics as to whether they are one-tailed or two-tailed. If you really want to know what that means, come up and talk to me afterwards. But you don't have to worry about it. In a study that makes a prediction in a direction, that actual likelihood that this has happened is actually half what you see there, so.

p represents a problem. You could get these results from chance alone, assuming the study was not biased, assuming nobody had their thumb on the scale, either deliberately or accidentally. And most bias in studies is accidental. Some of it is just sloppiness, occasionally it's deliberate. p alone, and this is one of the things you absolutely have to remember. The average person who knows a little bit about research reads the p value as saying that there is only a 5% probability. That means that there's the other part of it, 95%, a 95% probability that the treatment was effective. No. It's a 95% probability that something happened. You still have to figure out what that was, and figuring out what that was is based on not only the design of the study, but all the other things you know about the world, and everything you know about how the world works, including other studies. You can't read any study and understand any study in isolation, not only from other science, but in isolation from what you know about the world. If I do a study tomorrow that says I did a great job teaching kids to read by writing words on Tic Tacs and having them swallow them with their eyes closed and I get great results, you need to interpret that in the light of everything you know. It doesn't matter if the results seem to be valid. It doesn't matter if I didn't cheat. You need to decide, what are the odds that actually happened? What are the odds that actually happened, and what are the odds that it's just a weird thing that happened?

So a zillion people bought lottery tickets a while ago to try to win $1.6 billion, which only turned out to be $1.5 billion. I hope that person wasn't disappointed. The reason it was only 1.5 was they were trying to make a statistical prediction about where it would be when they got there at the same time, but nobody had ever been monitoring the growth of a lotto that was getting that big at the same time that another lotto was getting that big, and so money got split between the two of 'em. They didn't get to $1.6 billion. Interesting test of human behavior. But just, you know, recognizing that these things, you know, if everybody who bought a ticket had a rabbit's foot, and then somebody wins, should you make the judgment that their rabbit's foot works and everybody else's rabbit's foot doesn't? That's one possible interpretation of what happened. "My god, you've got the best rabbit's foot ever." Or you sit there and you say, "Wait a minute. Let's test this again next week. Buy another lotto ticket with your rabbit's foot and let's see what happens."

Now people read this and go, "Disappearing elephants." He's gone round the bend. What the hell is he talking about?" No, I'm gonna try to describe to you that you are naturally wired. You understand the importance of interpreting data in the light of what you already know around the world. Imagine that I was David Copperfield or some other similarly showy magician, and I took you out into a field and I had a great big elephant. He actually did this trick one time. I had a great big elephant, and I put a big curtain around it. We pulled the curtain away, the elephant was gone. All the data you have tells you that elephants disappear, that elephants can be caused to go out of existence, that a several ton elephant made of matter, of stuff, of substance, of guts, bone, skin, and hair, can (snaps fingers) disappear like that. That's what the data is telling you. But none of you would believe that, because you would interpret that in the light of everything else you know about the world, which includes the obvious observation that shit doesn't just disappear. So you would tend not to believe that. So if somebody comes to you with the Tic Tac study, you shouldn't do it. And see, here's the problem. If they've got four Tic Tac studies, they're gonna get a high rating in What Works Clearinghouse, because they've gotten to the magic number of four. Nobody knows if there's 200,000 Tic Tac studies that failed that are in the garbage someplace, and nobody's gonna interpret this in the context of a larger body of data. But you will when you see the elephant. You'll say, "I've known since I was a child." You know, when you were an infant, you didn't know that. Somebody showed you something, they covered it up, you thought it was gone. "Oh it's back. Now it's gone. Ah, now it's back." Then you developed object permanency. You didn't believe that anymore. You know, you were interpreting data in the light of other data.

The importance of understanding all of the data we see and observe and gather from research in the light of all the other data we have, remember what I talked about with convergence, is attributed most closely to a fellow named Thomas Bayes. Now as you can see, Thomas died in 1761 before the invention of the personal phone and digital photography, so we have no good photos of Thomas Bayes. This is the nearest we have. This is a primitive photo, taken in 1761. Thomas Bayes came up with this notion of what's called Bayesian inferencing. Thomas Bayes would hate p values, 'cause what Thomas said was, instead of using the criterion in an individual study or an individual data point, you need to consider all data in the light of all other data. And according to Bayes, that reasonable people... So if one person starts out with a belief way over here, and another person starts out with a belief way over here, and the actual true belief is something like right about here, that an honest mind and an inquiring mind, an appropriately skeptical mind who appreciates the data and understands the data, over time, they will both come to this spot regardless of where they began. If they engage in an honest pursuit of the data and an honest understanding of the world around them, both of them will come to the same understanding.

Unfortunately, in the kind of tribal science that we see in reading and climate change, and turns out in like whether the world is round or flat, there's some sort of tribal science going on. You know, when you find out that there are whole websites devoted to the flat earth and proving that there's flat earth, you realize that the shit going on in reading is not that bad. (audience laughing) The difference is, the difference is the people who believe in a flat earth aren't hurting anybody. They're not in charge of like mapping out routes for planes or shipping or anything like that, so they're not like damaging anybody. But the people who believe the flat earth version of reading, they've got the keys to the schools. I mean, this is not a good thing. So, you know, we've gotta go get the keys. I'm not sure how we're gonna do that, but we're working on it, and being here is a good part of that.

So Bayes advises us to do several things. Consider the entire body of knowledge, not just the study in front of you, but similar studies and very diverse studies. What we know about how the brain works from studies that have nothing to do with reading. What we know about how learning works that has nothing to do with reading. What we know about human beings. If somebody comes to you with a program that says, "I have this television program, this video program that you can put in front of kids, and they can learn to read by the time they're eight months old." What we know about children... People sell that crap. What we know about children says that's just not true. You have a really steep mountain to climb to prove that to me. It's not that that is necessarily impossible, but the standard of proof for that is much higher than if you come to me and say, "Kids learn best when we teach them to read sometime around the beginning of school age," as opposed to somebody saying, "No, you should teach 'em to read, you know, when they're six months old." One of those claims has a steeper curve, and the standard of proof for that should be different than the standard of proof for something which agrees with the rest of the data. Somebody could come tomorrow and make measurements that say that the Earth weighs half what we think it does and the Moon weighs twice what we think it does. They could, but the standard of proof for that is extraordinarily high. It's not that they're necessarily wrong, but the standard of proof for that is extraordinarily high. These things go on in physics more often than they go on in reading or human science, where people propose models of physics that will completely change everything. It's not that they're wrong, it's that the standard of proof for that is extraordinarily high. So you should consider the entire body of knowledge. Consider and weigh all possibilities, including that the people who did the study designed it wrong. That just random chance wielded its ephemeral hand over the results. That there was something going on in the design of the study that was not even reported. Occasionally people cheat. And be skeptical and cautious. If Bayes has any advice for us, it's that we must be skeptical and cautious. And we must be skeptical about our own research, too. You don't get to be skeptical about their research, but look at your research and say, "Yes, I just believe everything that comes to me, as long as it's something I like. Then I suspend skepticism and I'm not cautious and it's just wonderful." We need to be skeptical and cautious for everybody. For one thing, it helps us to get really good at being skeptical and cautious, because we get lots of practice at it, and because it's fair and it's the right way to approach things. And one of the dangers is that as your side of the argument accumulates things which have not been appropriately weeded out by skepticism in the hard edge of science, you accumulate your own fat, your own gristle, your own garbage, and the other side uses that against you. So we need to be merciless about our own science so that when we go in and do the scientific version of battle, we are keenly armed. This reminds us, again, of the interplay between statistics and research. So research designed for skeptics in a single slide. So I've managed to take like a lifetime of

research design across a number of studies. I've whittled it down to this slide. This is it. Drop the mic. This is my contribution to the scientific world.

So one of the keys is the basics of research are, first of all, control and comparison. You need to look for evidence of control and comparison. If we're trying to do a study, if we're trying to do a study of the impact of phonemic awareness instruction on the outcome of children's reading, then you wanna vary the phonemic awareness instruction you give, and as much as is humanly possible, control everything else. Keep everything else the same.

You can manipulate multiple variables at a time. Like you can manipulate two variables in two conditions. You can have a high and a low condition for two different variables, which now creates four conditions. Or you could have two variables in three conditions each, which now creates nine conditions. Or you could have seven variables in seven conditions each, which some people do in a study, which produces an ungodly number of variables. In each of those overlapping spots, you're supposed to have a minimum number of subjects in each one of those spots. I just reviewed a study for somebody that had more overlapping variables, more conditions than they had subjects in the study. So there was less than one subject in each one of the conditions. Each one of those conditions is like its own little study. You can't do a study on less than one person. Well, the study came out of Brazil, so maybe in Brazil you can. I don't know.

So control and comparison. You have to try to grasp the amount of control they exercise. Did they have appropriate control? And what exactly were they comparing, and did they design that well?

Bias and blindness, which go heavily towards control. Here's what I have to say about bias. Everybody is biased. I'm a psychologist. Trust me, everybody is biased. We're designed to be biased. You can't say, "But they're good people, so they're not biased." "But they were teachers and they're such nice people, so they don't have to worry about their bias." Everybody is biased, and you need to manage bias. And one of the chief ways you manage bias is through what's called in design blindness. Whenever possible, I want people to not know who's in the study, who's in the control group, who's in the comparison group. Ideally, I don't want them to know. Ideally, I'd like 'em to not know a study is going on. I'd certainly would like them to not know what I'm predicting in the study. That's almost never the case in the research that we do. And one of the things that happens leads to a lot of bias in blindness.

And then we want our measurement to be of high quality. Somebody was talking to me recently about running records in the observational survey, and I said, "That's interesting data." I said, "What's the validity and reliability of observational surveys?" And they said, "It's not that kind of an evaluation." I said, "That's my point, not yours." You know? (audience laughing) You know, it's not that we can't use that data, but we have to understand the limits of that data going forward, that this is stuff that may lack validity, and certainly lacks reliability.

Reliability, by the way, if I have a ruler, if I have a measuring tape, reliability refers to the idea that if I walk up with this measuring tape and I measure the width of the stand and I step away, and I come up and I do it again, and I hand it to you and you come up and you do it again, we should get more or less the same result every time. Validity refers to whether that result is correct. So if I get the same result every time, I have a reliable measure. If the result says that this is three and a half miles wide, it's not valid, but at least it's reliable. There's a lot of measures being used out there by some people, which are neither reliable nor valid. And it's very difficult to do statistics. All statistics assume that the measure you use is both reliable and valid. There's assumptions in every statistic. They assume that they're reliable and valid. All statistics are built on a list of assumptions. All studies violate numerous of those assumptions. Most of those assumptions are not essential, so those are not fatal violations. Violating the expectation of validity and reliability is a fatal violation. You have to have at least minimal reliability and validity.

So now we're gonna look at... This is fun. This is fun in this audience. If you do this in a different audience, it's not as much fun. If I did this at certain presentations, like people wouldn't show up. But these are the kinds of things, you know... I want you to know, I feel the love. I really know that there's a lot of people in this room who hold me in high regard because of their impression of me, and I really appreciate that and I enjoy that deeply. (chuckling) There are also a number of people in the world who just really seriously hate my guts. And what you're about to see has something to do with that.

So the people who did Leveled Literacy Intervention have a study out there that they list on their site which they describe as the "Independent, gold standard study." And it kind of is the gold standard, because there was random assignment of people. of subjects in a quasi-experimental design, so that appears to be the gold standard. Undermining the notion of a gold standard is that the researchers who did this study were paid by Fountas and Pinnell to do the study. That doesn't mean the study is wrong. We're gonna go back to be Bayes. We're gonna be Thomas Bayes. That doesn't mean the study is wrong. We don't throw it out for that reason. But that's something which, you know, if I tell you that they paid for the whole thing, and they controlled when and where it was published, that should raise your level of skepticism, not reduce it. You just need to be aware of that as a level of skepticism.

Control and comparison. There were matched controls who got no treatment. There were matched controls. So one group got Leveled Literacy Intervention, and a matched group got nothing. So they paired them up in pairs. Matched controls is, if you do it right, is a really good design. So there's some significant control and comparison.

One of the other things you have to keep in mind, and this is always something that goes on in these studies. This was being done entirely at balanced literacy schools. So while there was random assignment of the students, there was not random selection of the schools. Let you in on a little secret. There's never random selection of the schools. The government does not come around and tap a school and a teacher on the shoulder and say, "Congratulations. You're gonna be part of this study. You're now gonna teach like this." These are schools that had long since chosen to teach according to balanced literacy. So what we're looking at here is a study of in a balanced literacy school, do the kids who fail at balanced literacy in the core instruction show some improvement when we give them souped up performance, enhanced balanced literacy later on. It's not a comparison of balanced literacy to systematic instruction, it's just a comparison within this universe of balanced literacy, right? So if I've previously been feeding you nothing but Twinkies and not letting you exercise, and you're getting sick, would you get less sick if I fed you all the same Twinkies but let you go for a walk every afternoon? And if the answer is yes, that is not an endorsement of feeding you all Twinkies, but people will interpret it that way. And these are the moments in the presentation that are gonna get me killed when I'm in Columbus, Ohio.

Bias and blindness. Now, this is important. The teachers who are all deeply committed to balanced literacy and all deeply trained in LLI, and many of them were also Reading Recovery teachers, they knew who was in the study. And when they gave them the observational survey, which is one of the measures that was used, which is that the observational survey a subjective measure of reading or an objective measure of reading? It's subjective. That doesn't mean we dismiss it, but we keep the fact that it's subjective in our head. Their own teachers gave them the observational study and made these subjective judgements. "Oh, Johnny, I've had you in LLI for 34 weeks. Let's sit down and let me decide now, based on my subjective judgment, whether my instruction to you has been effective or not. Oh, it has been. Miracle of miracles." "Jimmy, you haven't been in class with me. You were matched to Johnny. I know that. I shouldn't know it, but I do know it, 'cause the study didn't make me blind. Let's see how you've done. No, you haven't done as well." I mean the opportunity for bias is overwhelming.

And measurement. They used two different measures, a subjective measure and an objective measure. I don't have to tell you about the subjective measure. According to the subjective measure, which was designed specifically for LLI and is incorporated and embedded within LLI, the kids did dramatically better on LLI. Let's see how they did on the objective measure. Now this, first of all, is the quote. They said, "Thus kindergartners who participated in LLI showed more significant gains on subtests of the DIBELS as compared to those who did not have LLI." Wow, that's remarkable. Let's see how that worked out. One of the things when you look at a table of numbers like this, try not to see the numbers. Here's what I want you to look for. I want you to look for asterisks, okay? 'Cause traditionally when these things are reported, you put an asterisk next to things that have a certain level of statistical significance. So one asterisk is 0.05, two asterisks is traditionally 0.01, three asterisks, woo, the big effect size, is 0.001. We only see two asterisks. So this is the treatment condition over here. We only see two asterisks. And when you realize that the top one is the aggregate, it included these kids. So these ELL kids were the only subgroup that showed any benefit on a DIBELS nonsense word fluency. They're the only ones. Nobody else had a statistically significant benefit. They're included in this group up here. If you take them out, and you just aggregate everybody else, there was no significant benefit to LLI. And they had predicted. their prediction was that LLI would benefit everybody. Except it didn't benefit everybody. And they didn't just give DIBELS nonsense word fluency. Spot the asterisk on this page. Can anybody find it? Those ELL kids, again, phoneme segmentation fluency. It's the only one. In the aggregate, again, they not only didn't do better in the aggregate, they didn't come close to doing better in the aggregate. And remember, the kids that are being compared to got nothing. They didn't get some other instruction. They got nothing. And Fountas and Pinnell says this is their gold standard study. This is the best evidence they've got for LLI. Now, if you read the executive summary of this, it touts their tremendous performance on all these DIBELS subtests, because they're counting on you not wading down to page 97 through 112 or whatever it is to find the stuff and sort this out. They're counting on you like having a mild seizure along the way, reading all their crap and just giving up and saying, "You know, I'll just trust you." Never trust anybody. This is the first rule of statistics. Never trust anybody.

So that's kindergarten. So for first grade, "overall similar significant differences..." Yes, very similar. "...between treatment and control groups were seen with first grade DIBELS." Once again, "Oh look, it's bigger," but this time it was only the Hispanic and Latino kids who benefited. If you take them out, you still get some benefit. Look at the ELL kids who did better. There's only three of them in the treatment group with only 10 over here. Why did they divide them 10 and 3? These are questions that I asked. Everybody else got divided pretty evenly. Why did they do this? But again, most of these groups simply did not benefit. I can't find it right now. There's one of these groups, actually, let's see if we can find it, that actually did worse when they got LLI. And again, yeah, it's better than nothing. And again, here we have letter naming fluency, only the ELL kids, DIBELS oral reading fluency, only the Hispanic, white non-Hispanic kids. The other thing that you see is as you start seeing these other results, that an asterisk here, an asterisk there, you see no pattern. You see no pattern from one test to another, from one grade to another. If there was an actual thing going on here, we might see, well, consistently the ELL kids benefited, or consistently the white kids benefited, or consistently the special ed kids benefited. But it's not consistently anything. The only thing that's consistent is almost nobody benefits. (audience laughing lightly)

The good news is by second grade nobody benefits. Things get very clear. By second grade, literally nobody. And the other thing is... Let me show you this. Let's count them together. 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, so almost 20. Statistically by random chance at 0.05, out of 18, how many would we expect to get a significant result just by random chance? Out of 20, just 1. How many did we get a statistically significant result? 1. Random chance is a bitch, (chuckling) let me tell you. This is what it looks like for second grade. Random chance has abandoned them completely. Random chance has gone home to take a nap. It's an extraordinary thing in terms of what's happened here, and yet they bill this as the gold standard of evidence for the effectiveness of LLI. And even if you wanna believe their subjective observational survey, their own prebuilt test that they teach to for the entire length of treatment, remember, this is compared to kids who got nothing. If this is proof that LLI is better than anything, even if you don't wanna be skeptical, it's only proof because of the design of this test that it's better than nothing. Almost everything's better than nothing. I mean, can you imagine trying to sell this in the marketplace? Go to schools, "We have a product here for you. It's very expensive. You're gonna invest your children's lives in it. We have good news. It's better than nothing." (audience laughing) You know, nobody's gonna say that.

What would Thomas Bayes say? Thomas Bayes would say, "Be skeptical." Consider these results in the context of everything else we know about reading. About what we know about how brains work, what we know from the first grade studies, what we know from Jean Shaw, what we know from the work of Dr. Moats, what we know from the work of Marilyn Adams, what we know from the review of the National Academy of Science, from the National Reading Panel, what we know from the brain imaging studies the last 15 to 20 years. You have to consider all this in the context of all that. It doesn't stand by itself. You have to consider it in the context of all that, which is a lot of work, but I encourage people to try to do it, to be skeptical. What would Bayes say? Bayes would laugh at these people. Well, he wouldn't laugh, 'cause he's been dead for 300 years. But, you know, he would have a problem with this.

So now we're gonna go up to the big kahuna, the i3 study of Reading Recovery. You know, there are people who I think sleep with this thing under their pillow. They're convinced that it solved all their problems. "This is it. We have the i3 study of Reading Recovery. They can never argue with us again. Pretty soon Reading Recovery will be everywhere, 'cause that's what the i3 study said it was better than anything, right? Isn't that what the i3 study said?" Let's check. So they gave a portion of the Iowa Test of Basic Skills, and the results were significant. No question about it. Kids in Reading Recovery did better than the comparison group on the Iowa Test of Basic Skills. By the way, what did the kids in the comparison group get? Nothing. So we're back to that better than nothing thing. Now, interestingly enough, May and the other researchers went, they like contorted themselves in this incredible position to make the case that kids in the control group, that many of them did get intervention. You know what? So 31 or 39% got nothing, but the rest of 'em all got some intervention. You know what some intervention means? If a teacher walked by you one time, if a teacher said in a question that she walked by the student one time in the control group and gave them 30 seconds of help with something, that counted as intervention. So they would count you as having gotten intervention. You didn't get systematic intervention. You didn't get scientific intervention. You got the watered down crappy intervention from the same teacher who was doing Reading Recovery, who, by the way, control and comparison, they say there was some treatment for controls, but most got nothing. And again, these were all balanced literacy classrooms. These were schools that already had Reading Recovery. These were schools that made a massive investment in Reading Recovery and balanced literacy. What's another word for balanced literacy? (chuckling) Whole language. Yeah. Louisa feels so good right now. You know, we all know that. So that's a good thing. I had somebody argue with me in a state panel in Wisconsin when we were talking about balanced literacy as though it was whole language. She said, "It's not whole language. They're different words." And I said, "Yes, we know that." You know, it's different words. You know, white nationalists aren't Nazis, you know? I didn't say that, 'cause I thought it might be inflammatory, you know? It would be. So these were all balanced literacy schools and classrooms.

Bias and blindness. Here's the amazing thing. The researchers went out of their way to make sure that when they administered an objective multiple choice Iowa Test of Basic Skills, they went out of their way to say, "Oh, and we had blindness. We brought in outside teachers to administer the Iowa Test of Basic Skills, and they had no idea whether the subjects were in the experimental or control group." La-di-freaking-da. It's an objective study. How much bias can you have on an objective test? But during treatment, and during the observational survey they gave, they made no attempt to create blindness or bias. So from the beginning, the teachers knew who was in each study. They were free, and we should be skeptical and believe this may have happened. A teacher who is deeply inve- Have you ever met Reading Recovery teachers? They give like pounds of flesh to this thing. I mean, God bless them. They're incredibly dedicated. They've invested their careers in this. We have to assume that people who are that invested, knowing that Johnny over here is working on Reading Recovery and he's part of this study and it's my job to prove that Reading Recovery is great. Yay. Reading recovery. And his yoked comparison student is sitting over here. He's not supposed to get any intervention, but I have a chance to influence his instruction. The inherent human temptation to mess with that kid's instruction, to tell him F makes the P sound, for instance. (vocalizes P sound) You know, I mean, tell him whatever. You can do all that stuff because nobody's monitoring. You know who's... They literally told them who was in the control group. You know why they told them who was in the control group? 'Cause the teachers refused to participate unless they knew who was in the control group. After getting the grant to do the study, they had to renegotiate the terms of the study with the teachers in schools, all these teachers who were Reading Recovery specialists, because they refused to participate in the study the way it was originally designed. There's official complaint. One of the originally expected measures was a measure of the Slosson, an objective measure of oral reading, the Slosson Oral Reading Measure. They refused to give the Slosson. And I believe the reason they refused to give the Slosson is 'cause Reading Recovery would not agree to have results of the Slosson laid alongside the results of the observational survey, which is also a measure of oral reading, because it would've given us a chance to compare them and contrast them and see that they don't line up. They're measuring very, very different things. They should be measuring the same thing, but they'd be measuring very, very different things. - [Audience Member] There are no pictures on the Slosson. - There's also no pictures on the Slosson, right. I mean, even I have pictures. I had Thomas Bayes, you know?

Control subjects were, they admitted control subjects were under the direction and control, their entire instruction during the study was under the direction and control, direct and indirect, of Reading Recovery teachers who were participating in the study. These were all schools that were deeply committed to Reading Recovery. We didn't just pluck them up. And again, they weren't being compared to systematic... So even at its best, even at our least skeptical, all we can say this is, is that, and I've said this, I've said this in writing, if you're absolutely determined to teach according to guided reading and whole language, you should probably make the investment in Reading Recovery, 'cause you're gonna break so many kids. You could do a lot better and you could spend your money a lot more wisely and you could torment a lot fewer children and families. But yes, if you teach really, really badly, according to guided reading, you will help a few of those kids read a little bit better if you also give them Reading Recovery. Now, the problem is if you just taught them the right way, all those kids could have ended up over here. But this study very carefully avoided the kind of data that would've allowed them to show that. So what would Bayes say? Bayes would laugh at this. Plus we have to consider these results in the context of everything else we know.

We have to consider it in the context of, like one of, so, I contacted the lead researcher, a guy named May. I contacted him, and I said, "What were you doing? What were you thinking?" And he said, "Well, you know, we didn't have that much money. It was hard to do the study." They had $6 million just for the study. He said $6 million wasn't enough to get it right. You know what kind of research I could do with $6 million? I could do really good research and buy a $5 million house. (audience laughing) I mean, it's like... But they insisted that they didn't have the resources necessary to do the study that they would've liked to have done. But that doesn't come out in any of their publications. And while some of this was published in peer reviewed journals, much of it was directly published and paid for. The other thing they did was they took federal money, and then they added all kinds of other private money to it, and that private money had something to say about where these things were published, the nature of the study, or we'll pull our private money out.

Now I'm gonna talk a little bit about effect sizes. This gets a little dicey, because everybody loves effect sizes when effect sizes tell us what they want. But I'm gonna tell you what effect sizes are. You have to be very, very careful with effect sizes. Effect sizes reflect the size of the statistical effect, not the clinical effect. They are a measure of our confidence that something happened. They are not the best measure we have of the magnitude of the thing that happened. The two oftentimes, there's a lot of overlap between our statistical confidence and the clinical magnitude of the effect, but they are not one in the same. It is possible, and I would have to draw graphs for you and this would get very mathy, so you're just gonna have to take my, believe in it. It is possible to have two different, we'll use the example of reading, but we could use anything. It's possible to have two different treatments for reading, for reading intervention, one of which raises all the students one unit, right? The average growth for this group is 1. The average growth for this group is 2. It's possible that this group could have a larger effect size than that group. It has to do with the distribution of the samples, the shape of it. It's never gonna happen that dramatically. But effect sizes are influenced because effect sizes are all done by dividing the average of something by the standard deviation of something. So distributions which are broad and varied and spread out will have their effect size will be smaller, even if the overall average magnitude of change was larger. If you understood all that beyond, that's fine. If you didn't, just trust me. Narrower sample distributions will always yield larger effect sizes. In fact, if I can narrow up the sample so that they're all very homogeneous, I can get really big effect sizes. One minute? Okay.

Their use, effect sizes are most useful for aggregating similar things like they did on the NRP. Take a lot of people who did research on the impact of phonics and phonemic awareness instruction. Those are similar studies. They use different statistics and different designs, so use effects sizes to aggregate all that together, and say something about the pool of that data. They're much less useful when you're using them as they do in What Works Clearinghouse to compare one thing to another and say which one is better. They're not really meant for that. To know which one is better, you have to go back to Thomas Bayes, and really understand huge bodies of research, or learn to trust people who are digesting and compiling huge bodies of research for you, like I hope the Reading League is gonna be able to do for people. They're much less useful for comparing dissimilar things. They're inherently non-Bayesian. You know, nobody had invented any of these when Thomas Bayes was around, but they are inherently non-Bayesian.

I wanna thank you all for being here. I hope you've learned a little bit. And by being here, I don't mean just being here right now, but being here for this whole thing, being here in the world in everything you do. I'm gonna repeat something I said last night, and I don't want put anybody on the spot, but I wanna give a special thank you to Professor Moats. When I first got involved in this field, she was waiting there for me. Work she had done, stuff she had made understandable, other people's work that she had interpreted for me, for you, for other people, for the mothers and fathers of dyslexic children, for dyslexic children who grew up to be dyslexic adults. I said something last night. There's a theological concept of prevenient grace. That is the grace that was there for you before you knew you needed it, that was created for you and waiting for you when you arrived at a certain place in your life, that it was already there waiting for us. Professor Moats and some other people were waiting for us when we came to this fight because she was there a long time ago. And I would be remiss if I did not thank her for that. (audience applauding)