Written by [NOTE: this is the fourth in a series of posts about a report recently issued based on a study done by Marzano Research Laboratory. Part I is here, Part II is here, and Part III is here.]
PART IV: INTERNAL VALIDITY
“Internal Validity is the approximate truth about inferences regarding cause-effect or causal relationships” (Trochim, 2006).
I consider research questions as one of three types: descriptive, relational and causal. From there, certain research designs lend themselves to best answering the research questions. For example, naturalistic inquiry methods such as those used in ethnographies or case studies are suited only to answer questions of description.
Marzano and his team set out mainly to answer a question of causality: does use of the IWBs cause improvement in student achievement? How they chose to do that is less than ideal, but we know that was their intention based on the use of quasi-experimental designs in each of the 85 classroom-based studies. Experiments and quasi-experiments are designs intended to address questions of causality.
When such designs are employed, the primary consideration of trustworthiness is internal validity. “All that internal validity means is that you have evidence that what you did in the study (i.e., the program) caused what you observed (i.e., the outcome) to happen” (Trochim, 2006). In any effort to prove causation, one key is to be able to rule out alternate explanations. In other words, alternate causes are threats to internal validity.
There are any number of ways to think about possible threats to internal validity, and there are many ways to describe them. There are at least a couple of general threats to internal validity for each of the 85 classroom-based studies that Marzano and his team used for his meta-analysis. The threats result mainly from the fact that, at least in the secondary schools, the same teacher taught the treatment and the control groups. At one level, that might seem like a benefit in that it eliminates teacher-level confounds. However, consider:
Social threats to internal validity – also known as or related to intervention or exposure bias. The students in the control group are presumably in the same classroom (at a different time) than the students in the treatment group. They see the IWBs. They also probably hear about the teachers using the IWBs from their friends in the treatment class. Might there be some compensatory rivalry and/or resentful demoralization? Trochim (2006) defines the latter:
Here, students in the comparison group know what the program group is getting. But here, instead of developing a rivalry, they get discouraged or angry and they give up (sometimes referred to as the “screw you” effect!). Unlike the previous two threats, this one is likely to exaggerate posttest differences between groups, making your program look even more effective than it actually is.
Experimenter’s bias – it is impossible to tell from the report all that the teachers knew and/or were told about the study. We know that in the instructions to the teachers, it says “Thank you for agreeing to participate in an action research [sic.] study regarding the effectiveness and utility of the Promethean technology in your classroom.” So, they knew what Marzano and his team were looking at/for. This knowledge could easily cause the teacher to pay more attention to her/his teaching in the treatment class. Additionally, just the fact that the teacher had to take an existing unit and figure out how to integrate the IWB technology means that the teacher was biased toward that group (i.e. she/he was more planful about that teaching).
Earlier, I wrote that I’d never seen meta-analysis included purposefully as an a priori part of a separate research design. Frankly, I’ve never seen or heard of a quasi-experiment in education where the (non-random) selection is at the classroom level and yet the treatment and control classes still have the same teacher. I could be wrong here, but in summary, this approach raises a number of general threats to internal validity for each of the 85 individual studies upon which the meta-analysis was based.
COMING NEXT:
FRIDAY – Part V: Summary and recommendations
I found your review both fair and balanced on Marzano’s research. There are too many polemic rants that fly around the blog/twittersphere about this research and I believe that Marzano’s research is a very basic starting point for figuring out exactly what IWB’s may be doing for us.
Your insights regarding the design of his research are excellent and thought provoking. I wonder if the problem with Marzano’s research isn’t more with all of his research, not just with this particular example of his research?
However, I think both of following two criticisms of his research while valid, are really criticisms of a lot of research into educational technology.
“Suffice it to say that none of the measures of student achievement used by any of the teachers who participated in the study are either valid or reliable.”
A valid measure of achievement may be a test score, but do test scores really represent a valid measure of all the learning that is going on (I believe that they don’t) and if they don’t, then what would a valid measure be? That isn’t to say, that the teachers did use a valid or reliable measurement of student achievement.
“just the fact that the teacher had to take an existing unit and figure out how to integrate the IWB technology means that the teacher was biased toward that group (i.e. she/he was more planful about that teaching).”
Don’t most teachers who integrate technology do so willingly? Isn’t the biggest problem with integrating technology convincing people to do it? If you included people who didn’t want to use the technology or were ambivalent, wouldn’t that bias the study the other way.
Also, are teachers who are more planful, more likely to integrate technology, since they are planning (ie. thinking what they could do vs. doing what they always have done) to teach content?
In conclusion, I think your review was excellent, however I tend to see Marzano’s research as more a very basic starting off point than anything at all conclusive.
“Marzano and his team set out mainly to answer a question of causality: does use of the IWBs cause improvement in student achievement?”
From what little I know about research in instructional technology and instructional design, there’s already a huge body of good research that has repeatedly demonstrated that a specific media has very little to do with increases in learning. This is not a new question. It is asked every time a new technology is introduced into learning environments. The repeated and, what I think empirically valid result from media comparison studies over 50 years shows that if instructional methods stay the same, learning stays the same. (Clark, 1994; Dillon and Gabbard, 1998)
This is not an insignificant body of evidence. If the same instructional methods were used in non-IWB classrooms and in IWB classrooms, and everything else was controlled(not even close for Marzano), the result that one would expect from this history of research would be one of no causality. A causal effect from using a different media would be like finding a human fossil in the cretaceous period.
From this study, we have no clue about instructional delivery. That’s key to learning and the development of instructional strategies that take full advantage of the capabilities of interactive technologies should be the focus of any school that purchases these devices. That’s the really, really hard part.
“there’s already a huge body of good research that has repeatedly demonstrated that a specific media has very little to do with increases in learning.”
I’m not sure I am as comfortable with your rather sweeping generalization. Look for example at the research that has come out of Maine in their laptop program.
Your right about the lack of information about instructional delivery, IWB’s mediate instruction (ie. the teacher instructs the students through the technology) unlike a lot of other instructional technology. Therefore it is imperative that any research that looks into IWB’s, explores exactly what the teachers are doing with them and why (what training and support did they receive) b/c the technology is simply a mediator for the instruction between the teacher and student, unlike one-to-one laptops in which the relationship of technology, teacher, and student is different.
Hi, I need some better study skills and things to help me remember… if u hv ADD – pl see a doctor. dont take medication but learn new techniques to concentrate eg mind-mapping.