Transcription
Hi class. So today, we're gonna pick up where we left off. Previously, we had been discussing experimental research, and now we're going to be moving on to non-experimental research design. And this is really the seat of what we see in most social science research is that it follows non-experimental research design. As we talk about this today, we're gonna see why that is.
Now, as you can probably tell from the term, non-experimental research is defined by exclusion, right? So it's non-experimental. It's some it this requires us to define an experiment before we go on. So the distinction between experimental research and non-experimental research is based on the degree of control that the researcher has over the subjects and the conditions of the research. Now, the keywords here are manipulation and assignment versus observation. An experiment is a kind of investigation in which some variable is manipulated. So the researcher has enough control over the situation to decide which participants receive which conditions at which times. So suppose you were interested in the difference in learning between students who only studied a book and students who read the book and also attend lectures. A researcher who's able to perform an experiment would set up the conditions, manipulate a variable, and assign subjects to the assigned subjects to them. As we're going to see, this assignment of subjects to conditions makes it possible to determine with more certainty whether the differences between the conditions actually caused any difference in behavior.
Now, if the researcher cannot assign subjects to groups but must only observe how students in already-existing classes at a college learn according to the same two methods, we do not have a true experimental study. In this case, we would find that many things could have caused any differences in learning between the classes. For example, the students may have signed up for one class or the other based on their preference for an instruction, the instructor, the teaching method, or even just the time of day that the class is offered. Any of these factors could cause differences in learning.
A second characteristic of non-experimental research is that the data collection procedure often must forfeit some degree of control in return for obtaining the data. For example, we might decide to study public records that may be almost, but not exactly, in the form we desire, or we might have to keep a questionnaire short to help gain the cooperation of students or subjects. So non-experimental research is often called correlational research because it seeks causes of behavior by looking for correlations among variables. Now, the term is somewhat misleading because all research is correlational to the extent that it seeks functional relationships between variables, but calculating correlations among variables does not make the research correlational in the strictest sense. We often compute correlations among variables in the in the truest of experiments. What makes research correlational in the common usage is the ability to manipulate some variable independently. In correlational research, relationships are studied among variables, none of which may be the actual cause of the other. So, you know, this is where we get to the correlation is not causation reminder. And one can only speculate the causes of the relationships. We demonstrate causation when we can decide which variable caused the other, and this is best done in an experiment.
In a true experiment, non-experimental research is often a first step in starting to answer theoretical questions by empirical methods. Experimental research frequently is done as a follow-up to previous non-experimental observations. For example, experimental research on physiological and behavioral factors in alcoholism has found has followed from the non-experimental observation that alcoholism tends to run in families.
Now, it's convenient to distinguish several varieties of non-experimental research. The first may be called observational research. That is, research in which the researcher simply observes the ongoing behavior. Examples are field observation of ducks from from a blind or television monitoring people in a store. The second category is archival research, and in this method, existing records are examined to test hypotheses about the causes of behavior. For example, a researcher might study crime statistics in different countries to see if there's a relation between capital punishment and the murder rate. In the third category, which isn't on this slide, but we're going to talk about it in a little bit, yeah, we call the case study. This category is different from others in that the researcher investigates a particular situation that has come to to their attention, and the situation may be a practical problem that must be solved as soon as possible, or it may be an event, a person, or animal that intrigues a researcher. An investigator might study the victims of a natural disaster to determine its effects on their psychological health or the impact of severe social isolation on an individual's ability to learn language, as was demonstrated in Jones' 1995 article where they looked to just that. Case studies are typified by the varied nature of the methods used to study the problems and intensive descriptions of a single individual or a single group of individuals. So Freud's early works were prime examples of case studies. They really are a wonderful starting point. They're observational in nature. They focus on few, if not just one, subject. But can we say that they're generalizable? Well, no, we can't say that they're necessarily generalizable. However, they can give us a really excellent starting point and may describe circumstances which we see in the broader population if we can confirm that using more experimental methods.
Now, we also have a fourth category of non-experimental research, which again is not on this slide, what we're going to talk about, which is the survey. And, you know, we talked about the design of various metrics to measure behavior. Well, a survey is one of the most common that we use in the social sciences, and a survey is probably something you're pretty familiar with. You've probably taken a number of them online. And in this method, participants are requested to cooperate by responding to questions. Now, nearly everyone has taken part in a survey, and we're gonna talk about all these different, all these different methods. But when you think about the about designing a metric that collects self-report data, we're really talking about a survey.
Okay, so moving forward. So we talked about that here. Surveys assess public opinion or individual characteristics by the use of questionnaire and sampling methods. So what we mean is, is that we are sampling data from the larger population, right? So the population is what we're studying or what we're hoping to to get an estimate of what is actually happening in the general population. And we are taking a sample, a subset of that larger population through a survey.
Now, before we really go into more detail on the various non-experimental methods, we need to take a short detour to discuss what it is that scientists using these non-experimental techniques are trying to accomplish in the first place. Now, you might be inclined to wonder why this is necessary to talk about this, because all psychologists are trying to understand behavior, right? Well, the problem is that is that it's more difficult to determine that in correlational research, and it's often difficult to decide which variable caused the other, or whether both variables are caused by a third variable. You know, we talked about this quite a bit already. So this problem has led some social scientists to consider their task to be the discovery of meaning of a behavior rather than the cause of the behavior. So these scientists consider their task to be the attempts to understand and interpret behavior rather than search for its causes. So essentially, the question being asked is not, "Why did the behavior happen?" but rather, "What did the behavior mean?"
Social scientists have adopted the term hermeneutics to describe methodology that looks more at interpretation than causation. So this term is borrowed from the field of biblical interpretation, where the task of the scholar is to find out what the original meaning of a text was to the people originally meant to read it. So psychologists who use hermeneutics, a hermeneutic approach, may try to interpret the meaning of two people who are holding hands, for example. So if we attempted to explain the behavior in terms of cause and effect, we would have to try to determine what caused pull a hand holding on the particular occasion for the couple. This attempt to explain the interaction in terms of cause and effect would be much clumsier than to interpret its meaning to the individuals as a public indication of close relationship, as Goffman's 1971 work described. So as we proceed through the lecture, we'll find that in some places, the traditional cause-effect approach seems more more appropriate, and in others, uh, the hermeneutic approach is more natural.
So what we're essentially talking about, yeah, when we talk about non-experimental methods, as we're talking about observational research, and there's various subtypes of this, but observational research involves recording ongoing behavior without attempting to influence it. So this method takes two general forms: naturalistic observation and participant observer research. So naturalistic observation is research conducted in such a way that the subject's behavior is disturbed as little as possible by the observation process, and the observation is made in the environment where the behavior naturally occurs. And you've probably seen films showing naturalists or observing birds and/or other animals from within a blind, which is a device for screening the observers from the view of the animals being studied. You might have seen something that looks like this. In the social sciences, naturalistic observation is also referred to as unobtrusive research. All right, so the idea here is that the term unobtrusive really just refers to the effort that researchers make not to influence or obtrude on the behavior being studied. Still another term for naturalistic observation is nonreactive research, and the term nonreactive emphasizes that the subjects are unaware that they are being studied and therefore do not react to the presence of the observer.
So why would all of these observational methods take all of this effort into account to not let the subjects be aware that they are being observed? Well, for one simple reason that might sound familiar from previous lectures: it's that we do not want to confound the report of the data, or the data that we're getting from people knowing that they're being observed. So think about this in terms of a reality TV show. All right, let's say, for the sake of argument, that there's no screenwriters or no individuals who are kind of trying to weave together a specific kind of storyline to influence the participants in this reality TV show. Now, that you might say, "Okay, then we're actually seeing reality." Well, you could say that that might be true, except for the fact that all of the participants in the reality TV show, all the actors, as it were, have cameras in their faces. So if they have cameras in their faces, think about it from your perspective. If you had a camera on your face, would you feel more natural? Would you feel more able to express yourself as you would if you didn't know that there was a camera there, or would you be acting in a certain way to either look your best or respond to what you think that the person who is taking the film or the video of you wants? The idea here is, is that if you introduce the knowledge that the person is being observed, then it is very likely to influence their actions and their behaviors, right? So the idea behind observational research, naturalistic observation, is various types of naturalistic observation in in particular, is that we don't want the participants to know that they're being observed, or we don't want them to be interfered with in their natural behavior because that represents a confound, and that's what we've referred to numerous times in the course before, this idea of a confounding variable. You, as the researcher, don't want to be the confounding variable yourself.
So methods of observational research are really as varied as the subject matter being studied. So researchers have been very ingenious in devising unobtrusive measures of behavior. One broad category of unobtrusive measures is known as physical trace measures. These measures make use of physical evidence of some behavior. So, for example, researchers have studied graffiti in school restrooms to discover attitudes towards racial integration of schools, smudges on pages of library books to see which pages are most read, and grease prints on display cases in museums to see which displays are most interesting to children. So these examples illustrate the behavior of a group of individuals can sometimes be studied quite effectively by naturalistic observation, even though it may be unethical to observe an individual's behavior in certain circumstances. So, for example, it's illegal to obtain a person's borrowing record from the library, right? So, you know, you have to do this within ethical boundaries, but there are ways to use physical traces of behaviors to get a sense of what people are doing and surmise why they are doing it.
Now, although much has been made of the cleverness of social scientists in using physical traces of behavior, naturalists have for many years used scratches on trees to study territoriality in bears, droppings to study the eating habits of owls, and similar, similar physical traces of animal behavior. And as the field of psychological research was very influenced by behaviorists, physical trace research played a very large role in in in psychological research on humans as well. However, it clearly is not the only way that we can look at psychological behavior.
Another example of naturalistic observation is provided by Don Zimmerman and Candace West in their 1975 study. They were interested in whether patterns of speech would reflect power through dominance of conversation. In particular, they were interested in looking for sex differences in interruptions and other turn-taking behavior in conversations. They tape-recorded conversations between sets of two persons in public places around university, such as a coffee shop. They chose conversations that could be over heard easily in the normal course of being in a public place. They stayed 10 female-female pairs, 10 male-male pairs, and 11 cross-sex pairs. Whenever possible, they obtained the permission of the conversationalists to use their recordings. Sometimes, however, people felt people left before the researchers could approach them. All personal identifications were edited out of the transcripts of the recordings, and there was also informed consent, privacy, and debriefing that took place whenever possible. And what Zimmerman and West found was that males made 96% of interruptions in the cross-sex pairs. These investigators decided to learn whether this finding was limited to the type of setting or to people who might be already acquainted. So they brought pairs of people into a laboratory where they were asked to chat and get acquainted before the experiments, before the experiment began. And in five conversations between males and females, all of the males did more interrupting, with men doing 73% of the interrupting on average. The amount of interrupting was unrelated to which person talked first. So the Zimmerman and West research illustrates a typical admiration in observational research, from idea to naturalistic observation to laboratory. This move to laboratory allows researchers to rule out sources of confounding variables that would occur in the naturalistic situation. It also illustrates that observational research is not limited to field settings but often occurs in the laboratory, where it's called laboratory observation.
So another kind of observational research that has yielded important results is participant observer research, in which investigators participate in groups and record their observations. And one of the most famous of these studies was that of Leon Festinger's, H.W. Riecken, and Stanley Schachter in 1956, who joined a group that believed the world would come to an end at a certain time. The group members believed that they would be rescued by a flying saucer. The psychologists carefully observed interactions among group members and the effect the disconfirmation of their prediction had on their behavior. Surprisingly, when the world did not end, the group members began to be more open and less analytical about their beliefs. This research was instrumental in the development of Festinger's theory of cognitive dissonance, which predicts how people deal with conflicting beliefs. We should note that careful records and diaries are crucial in evaluating participant observer studies because of the increased possibility of subjectivity in these situations.
So when is participant observer observation appropriate? Well, the usefulness of participant observer research is limited to certain types of situations. Jorgenson's 1989 study showed really talked about this, and it's it's most useful in studying a small group that is separated from the population as a whole, when little is known about the group, or when the group's activities are generally not available to public view. So an example would be what the study that we just mentioned, where you might be looking at groups that are cults or criminal enterprises or groups that engage in certain behaviors, right? So in my own research, when doing my own literature reviews on heavy metal music, there was a lot of participant observer research on the sociological and social psychological sides among metalheads and people that were within the heavy metal community because it made up a subcultural group. So that would be a very appropriate context to engage in participant observer research. Another important level of this layer of this to keep in mind is to take the insider in-group perspective, which is really taking the point of view of those that you're studying. So participant observation is characterized by the effort to view some behavioral activity from the viewpoint of an insider to a situation, and the methodology needs to be open-ended, it needs to be flexible, and it needs to be opportunistic. The approach to theory is often hermeneutic, emphasizing interpretation and understanding. Some participant observers will take the role of a central participant because it would be impossible to learn much without doing so, and as in studying the people who hang out in a bar in a really rough section of town, that would be an example. Other times, a researcher can stay more on the periphery, as in studying an organization such as a religious group by definition, participating in a group leads to problems of objectivity. The researcher must strike a balance between taking the viewpoint of the group members and maintaining scientific objectivity. A researcher who's studying police officers tries to take the point of view of the officers, but he, she, or they may find that their attitudes towards criminals and towards society as a whole start to match those of police, who may happen to be more prejudiced or more authoritarian than the population as a whole, based on the nature of the culture, their own experiences, or they may have different views. But the idea here is, is that it's, it's not unheard of for the researcher to really truly become part of the group. It's also not unheard of for researchers to be converted by a religious group that they were studying. Some researchers maintain their objectivity by regular contact with other researchers who debrief them about their experiences, so that it's really important to keep in mind and engaging in this kind of almost anthropological research, that there are certain risks associated with it, and the loss of objectivity certainly is one.
Now, sometimes it's possible to be open about the fact that you are there to study a group, as when a person openly enters a fundamentalist church for the purpose of research and is welcomed as a potential convert. This is known as undisguised participant observation, right? And this is when we talk about issues of gaining access. By contrast, certain groups are hostile to the larger society and suspicious of anyone who shows interest in them. Admitting that you are there to study them might result in your getting kicked out or worse. In that situation, the researcher would use disguised participant observation to hide their true purpose from the people being observed. In addition, it's often unrealistic to expect a researcher to inform each and every member of a group the first time they meet that he/she or they are there for the purpose of research. It would be highly artificial to begin every interaction with a new person by saying that you're a researcher, and there's no hard and fast rule about whether to inform the group. We should consider the ethics of that also. When we talk about informed consent, it's best to be truthful where possible and to adopt a disguised strategy only when necessary. So, kind of harking back to our discussion of the ethics code, using deception only when necessary. But still, let's think about this in terms of the ethical, ethical nature or lack thereof.
So obviously, there's certain problems with observational research. Two important problems are present in participant observer research. First, by entering the group, the observer is the observed by definition, changes it to some extent. Therefore, the act of observing the behavior changes the behavior to be observed, which is the big problem. A large group may not be influenced much by the observer's presence, whereas a smaller group may be influenced pretty considerably. And in general, participant observer research is done in unusual groups that can absorb an observer whose presence would have little effect. The second problem is the ethical question of invasion of privacy. So we talked about the ethical question of deception, but this invasion of privacy piece is huge. Participant observers cannot always obtain informed consent from the people they study. Some researchers hold that participant observer research is therefore always unethical, and that's something that we should think about. Is it always unethical? Others point out that professionals, such as journalists, are permitted to engage in this type of practice. They argue that if psychologists do not perform participant observer research, they're withholding the application of psychological techniques and insights to social problems. So these are of key importance and of key concern.
So what about archival data? We mentioned that a little bit earlier in the lecture. Well, the term archival research refers to research conducted using data that the researcher had no part in collecting. And in fact, a lot of research done in the social sciences uses archival data. It's often a lot easier if you have a good dataset that was collected for researching something else or answering other research questions to use that data to assess your own research questions. In fact, it's one way of preventing a very long and drawn-out IRB application process because usually, as long as you have a right to that archival data, that kind of research is either exempt or subject to an expedited review. So essentially, the researcher here simply examines or selects the data for analysis. And archival research is appropriate in many instances. So data that bear on the hypothesis may already exist. In collecting new data may be wasteful, or ethics or logistics may make it infeasible to conduct an experiment relating to variables of interest. So think about research that studies suicidal behavior or completed suicides, that looks at sex crimes. Both topics would be inappropriate for experimental research, but if there's pre-existing data on that, it would be really perfect for archival research because you don't need to do anything unethical in order to answer a lot of questions that you might have for a new study.
Now, obviously, archival research has its limitations. However, first, most archival data are collected for non-scientific reasons. Governments and private agencies collect the data for their own purposes, and such data often do not suit the purposes of a scientist. For archival data to be scientifically useful, the agency collecting the data must ask questions similar to the scientist's, or must inadvertently collect data that are valuable to the scientist. Second, because archival research is by nature carried out after the fact, ruling out alternative hypotheses for particular observed correlations may be difficult. So a researcher who relies on archival data is at the mercy of any biases that may have occurred in collecting the data. So police records are notoriously subject to bias. Many categories of crimes are seldom reported to the police, and it's estimated, for instance, that only one in four attempted rapes is reported. So if you have data on those that are reported, it may be skewed if you're trying to do forensic psychology research, and that's according to the Bureau of Justice Statistics in their 1990 publication. So this isn't just coming from me and my views on it. So there are certain considerations to keep in mind when using archival data, primarily, what is the source of the archival data and what biases would you need to keep in mind? And also talk about if you choose to move forward with a study using archival data from those sources, because you would need to include that in your papers that you would be doing or your research projects that you would be doing.
A successful use of archival research can be seen in David Phillips' analysis of motor vehicle fatalities in his 1977 article. Authorities had long suspected that many automobile fatalities were suicides that were completed rather than accidents. And Phillips hypothesized that if suicides are triggered by reports of other suicides, then motor vehicle fatalities should increase just after the publicized suicides. He studied all motor vehicle fatalities in California during the week that followed the reporting of suicides on the front page of the state's two largest newspapers. Then he compared the fatalities with those in a control period in another year. He found a 9% increase in the number of fatalities in the week following the suicides, with a maximum increase of 30% on the third day after the stories appeared in those newspapers. By correlating the increase in fatalities following each story with the total circulation of all newspapers that covered the story, he showed that the publicity was likely responsible for the increase. Further analyses bolstered his conclusion that the increase was the result of the publicized suicides. Details were similar between the suicide stories and the types of accidents that increased following the stories. So this is really interesting stuff. This is a really good way of using archival data to answer questions about human behavior that had nothing to do with initial psychological variables. It was just essentially accident and crime statistics, and piecing those together to talk about human behavior became something that was very valuable as well.
So case studies. I promised you that we talk more in detail about this, and case studies constitute a category of research that is difficult to characterize with a simple definition because case studies often include the use of observation and archival methodologies. The distinction among them are not always clear. Nevertheless, it's possible to say that case studies tend to involve an ongoing situation that presents itself for investigation. The principal characteristic of case studies is that they examine individual instances or cases of some phenomena. Sometimes this is an individual person, as in a psychiatric case. Other times, it's an individual institution, as in the case of mass hysteria at a school, for example, or it can be an individual method or treatment, such as the effectiveness of a new surgical intervention on patient's health. The individuality of the situation or case characterizes a case study.
So Robert Yin's 1994 article defines a case study as "an empirical inquiry that one investigates a contemporary phenomena within its real-life context when the boundaries between phenomena and context are not clearly evident and in which three multiple sources of evidence are used." So according to Yin's definition, the multiple approaches distinguish a case study from other non-experimental methods. Many case studies result from problems that present themselves to researchers as opportunities that must be grasped quickly or are lost. Little time may be available for planning, and the study often must be conducted under difficult conditions. An example of case study research is the study of unilateral visual spatial neglect reported in a 7-year-old boy by Rebecca Billingsley and her colleagues in 2004. In this neuropsychological disorder, patients ignore stimuli in one half of space. So patients will draw a flower with petals only on one side, for example, or eat the food on one half of their plates. So frequently, this disorder is observed after a stroke in elderly persons, but in this case, it was observed in a young child, so it was really interesting and disturbing at the same time. When the little boy first visited the medical team, he had experienced difficulty in moving his right hand. They had no trouble in paying attention to anything in his environment. An MRI of his brain revealed the presence of a cancerous tumor near the ventral fold of the brain, and after surgery to remove the tumor, the child began to exhibit symptoms of unilateral neglect. So just leaving off the beginning, a few letters when attempting to read long words, and drawing a clock with numbers only on the right side. And actually, one little side note is that a neuropsychiatric screening tool for individuals that may have visual spatial neglect is just to tell them to draw a clock, and so if they only put the numbers on one side, that suggests that that suggests that this issue may be present. So getting back to the story at hand, fortunately, six months after his surgery, he was able to overcome these difficulties and again attend to space somewhat normally. Now, this case assists in demonstrating specific brain regions that are tied to unilateral neglect and the development of visual spatial attention. And this visual, this study of visual spatial neglect is typical of case studies. The efforts of the physicians and psychologists were directed towards a practical problem. The case appeared unpredictably and required prompt attention, and a multidisciplinary team approach was used to elucidate the problem. Members of each discipline used several techniques to rule out explanations and narrow the possible causes. Other case studies are quite varied. In fact, one of the few generalizations possible about case studies is that it's difficult to generalize about them. It's difficult to generalize anything about the idea here, really, is is that it often focuses on a single case or a single group, and it really is difficult to generalize. But it serves as an excellent starting point and can reveal very valuable data. But one of the the difficulties here is that you can't generalize because there's too many potentially confounding variables, and you only have usually one or a few data points. There's not a lot of standardization that takes here. And one thing you may remember from our discussion about experimental methods is that standardization is key. You want the conditions to be as identical as is humanly possible between subjects so you can truly say something about the nature of the behavior that you're trying to measure and the conditions you're trying to measure that behavior in, absent any potentially confounding variables. So interesting stuff, important stuff as well.
So when we talk about research using observation and case study methodologies, we often start with some general questions in mind, but without specific categories of behavior defined. In fact, this characterizes a lot of observational research. So what you want to do is you want to have field notes, which are really a notebook of all of your observations and interpretations. So the next couple of minutes, I'm going to be talking to you about how to put together good field notes for these kinds of observations because in any good research methods class, it would be a bit of a how-to guide in documenting your data, especially in observational research, especially in non-experimental designs where you want to get as much of your data down and as much of it accurate as possible to eliminate potential confounds.
So the first point would be to be systematic. So you should keep a notebook of all your observations and their interpretations and make certain that you think about the categories of behavior that you're concerned with and clearly define your behaviors of interest. You should also decide the length of your observation and observational intervals. So checklists and other recording devices that you could use can help considerably in keeping clear records. Now, this documentation may sound kind of daunting, and unfortunately, it often is. You may be so busy observing that you don't have time to take notes, or you may be in a social situation where note-taking is impossible, either because it'll inhibit the behaviors you're observing or because you're taking part in the situation. In such cases, you should consider inconspicuous recording devices with obvious due consideration to ethical issues, or at least write up your observations as soon as you leave the situation. You might be tempted to skimp on writing up your notes later because you're tired or you think you'll remember what's important. This is often the case, and taking notes after class, you may experience this, but if you think you won't forget, you're probably wrong. So just try to remember everything you did a week ago. Well, what did you wear? What did you say? What did you do? Probably not going to come up with a lot of information. So your notes should include as much detail as possible, including the time and the setting. Later on, the significance of these details may make sense in a way that they don't when you're writing down those notes. And you may want to write your notes in stages. You can just jot down some brief notes as you work and elaborate them, elaborate on them later. However, remember that memory is reconstructive, so the more you can get down as soon as possible, the more accurate you're going to be.
The other piece is be selective. The problem of having too much to observe can often be solved by taking samples of behavior. Instead of watching television all day long for a week to observe the content of commercials, you might watch only the first batch of commercials after the top of the hour. Instead of watching all channels, you might focus on only one. Instead of watching everything that all children in a daycare center do for a whole day, you might watch four instances of one kind of behavior, such as fighting, or only those behaviors that you're prepared, that you prepared on your checklist. So, you know, keep in mind there are ways to kind of hone in on what you're looking for and record your raw data. Use audio or visual data if at all possible.
So that brings us to content analysis. And when dealing with textual or photographic materials, you don't have to take field notes because the material is already in a permanent form. The problem remains, however, that of deciding how to deal with what may be a large body of material. Suppose you're investigating the sexually explicit content of magazines, for example. How do you decide what to record? There are two basic approaches to content analysis. The first one is based on counting the frequency of some objective measure, such as frequency of certain words that may be sexually charged. Now, this would be known as coding the manifest content of the text because it's relatively simple to count the number of times the word "kiss" or "love" or whatever appears in a body of text. The coding of manifest content is very reliable. There may be a problem, however, if words are used in different ways. That could be a big issue. The word "love" appears many times in a certain chapter of the Bible where it has nothing to do with sex. You would draw the wrong conclusion about the point of the chapter if you just counted the word "love."
The alternate method of analyzing the content of text or photograph is coding its latent content. With this method, the researcher reads a passage of text or looks at a photograph and interprets the presence of a particular theme. That same passage, the Bible, would easily be seen to be about what might be called brotherly love rather than romantic love, right? And because latent content analysis is inherently subjective, though, it runs the risk of being less reliable than a manifest content analysis. So it's a good idea to do both manifest and latent content analyses when you're doing work like this and compare the results. If the results turn out the same by both methods, pretty strong evidence of the validity of your results. If they differ, you can look for the reasons why they might differ.
So what's the reliability of content analysis? Well, whichever method of content analysis that you choose, there is a problem of reliability. If you use latent content analysis, another researcher might interpret the same passage differently. Even with manifest content analysis, there is the possibility of coders making mistakes when going through large amounts of data. So to establish the reliability of your coding scheme, it's important to use at least two coders. So if you're part of a research team, this is helpful. If you're helping somebody out with their research, when coding raw data, you can help them, and they can help you with your research. But having multiple coders helps. Now, there are various techniques based on statistical correlation that are available to quantify the reliability of coders and coding schemes, but the the primary method that I think is is used with this from my experience and from the experience of the researchers that I've worked with is a reliability analysis. So it's a correlational statistical method of determining a degree of reliability between two coders. And the the idea here is that of inter-rater reliability, the idea that you want those two coders to be interpreting the same things in the same ways as much as possible. If it diverges too much, then it calls into question the reliability of your coding scheme.
So an example of content analysis is that of Regehr, Bledsoe, and Rasmussen in their 1991 study. And they used content analysis to study personal ads in newspapers. So this is kind of old-school. So think before online dating apps. So think, "Single white female desires to meet professional male who enjoys dancing, long walks on the beach, etc., etc." So the researchers were interested in discovering which ads were most successful in eliciting responses from readers. And categories they coded were statements about looks, such as attractive, petite, tall; son status, such as affluent or employed or professional; on intelligence; on sincerity; and age. And these categories were based on previous investigations. So the category "looks" was used to code information that other investigators had labeled attractiveness, appearance, sexuality, or physical status, because it was an inclusive category that encompassed all others. Similarly, they chose "status" instead of "financial security," which in a previous, which a previous study had used, because it would incorporate social prestige, so a career person or a gentleman or a professional, as well as material possessions. So two of the authors coded all the ads independently. Then they got together and discussed all the ads until they agreed on the coding of each one. After the coding was complete, they sent a questionnaire to the advertisers through the newspaper asking about the results of their ads. They found that ads placed by younger women were more successful than those by older women. And in contrast, ads by older men were more successful than those by younger men. So we could kind of see some of some cultural biases coming through in the results of that research. The authors were surprised to find that advertisers who claimed to have good looks and high status did not receive many more responses than those who did not. And the researchers interpreted this finding as an indication that most of the ads contained such claims, so little information was to be gained by considering looks or stats. The content analysis was an essential component of this study and then defined the independent variables against which the outcomes were correlated. And inter-rater reliability was huge because everybody had to really be on the same page in how they were coding the information.
So when we talk about when we talk about your procedure in terms of recording methods for experimental research, procedure is a bit different from method. A method is a broader term that encompasses all aspects of the study, including the logic of the design and the steps for carrying it out. But procedure refers only to that latter point: what the researcher does in translating the design into action. I'm picking this apart because method, when you think about procedure, you might think, "Oh, this is the method section of all these papers that I'm reading, and the methods section that I'm going to be writing in my term paper." Well, it's a component of it, but it's not exclusively it. So the design, for example, of your experiment may be an experiment with two conditions that are both experienced by each subject. So procedural concerns include whether each condition is tested on the same day or different days. Procedure also involves instructions and how they're given, the debriefing, and so forth. The actual procedure of your research, many details must be worked out in the course of translating a design into practice, and the step of going from design to procedure can be a difficult one for a lot of people, especially students who are just learning this, because it involves going from a logical plan in one's mind to a practical plan of action in, let's say, a laboratory study. After the design is completed, you should develop a clear idea of the exact procedure or protocol you will follow.
So what's a protocol? Well, this was mentioned a little bit earlier in the course, but a protocol is a list of the exact steps needed to test a subject from start to finish. A written protocol is helpful for beginning researchers, especially when the one researcher will help run a given study. And if this seems familiar to you from our ethics lecture, it should. This is required. A protocol is required to submit to your IRB before your study begins. Write a detailed list of how this is going to take shape. Now, once a protocol has been developed, you probably should do a pilot study to find the bugs in your procedure. There are almost always problems to be smoothed out. One of the researchers, let's say your advisor or a friend, should be run through all the steps of the study exactly as it will be carried out. And although the temptation to skip the pilot phase may be great, you should resist it. Nearly every experiment that I've performed or others that I know have performed without testing some pilot subjects has been either disappointing or had a lot of bugs in it because this is where a little effort can greatly increase the precision of a study. The idea of a pilot study is you're going to list all the steps that a subject goes through in a study. You're going to execute that in a smaller scale, and then you're going to work out the bugs, right? So it's, it's the idea here is, is that you're working out the bugs before you do the full procedure.
And the the last piece for you to consider for the purposes of this lecture is guarding the integrity of the data. And all the care expended on designing and conducting a study is really a waste if this, if the data is compromised by care, carelessness in recording or handling. And in fact, the IRB is going to want to know that you have a plan for guarding the integrity of the data. So before the study begins, the researcher must have a plan for recording and handling the data. A good procedure is to keep all the data and other research details, such as, let's say, you're doing a study on psychotropic medication, so the dosages, or if it's a behavioral study, the stimulus settings, and like, in a notebook, right? So you want to keep this together in a notebook. All pages should be dated and identified in such a way that they can be replaced if any sheets became lost. Back up your data. I tell students about this with their term papers, but it applies especially for your for your research projects as well. Often, it's inconvenient to reproduce a blank sets of data sheets with spaces for all necessary information so that nothing will be overlooked. So data should be placed in a file and kept in a secure place. They should never be carried around in a briefcase or a backpack, so so it could be potentially lost. I've heard horror stories of people leaving leaving raw data on buses or subways or or in their car, and their car's been broken into. So it's really important to guard the integrity of the data and the safety of the data because it is considered an ethical violation if something happens to data with identifying information. So all important things to keep in mind.
All right, so that concludes our lecture for this week. You will have your discussion questions posted, and the dates are the same as they always are for each week. So if you have any questions or concerns, please do not hesitate to message me on Canvas. I look forward to reading your discussion posts and I look forward to our lecture next week as well. All right, take care.