Unless you live under a rock (or if you don’t track education policy matters through social media – same thing), you know about the “big” study about teacher effects that was conducted by Chetty, Friedman and Rockoff, disseminated through NBER and reported in multiple outlets, most notably the New York Times.

This is an important study for at least a couple of reasons. First, methodologically, the study is massive and novel in some important ways. Second, from a policy perspective, even if the authors overreach in their interpretation, the study adds to the growing body of literature on teacher effectiveness and value-added measures. The more empirical evidence we have, the better; that’s the nature of scientific research.

I haven’t read the paper yet, so I won’t comment on it. What stood out to me over the last few days, though, is how scholarly communication unfolded. The authors published the paper through the National Bureau of Economics Research (NBER), which is a reputable research organization, but one that publishes working papers that don’t undergo traditional academic peer-review. The paper is dated December 2011, but it was essentially “released” when the New York Times reported about it on Friday (January 6). Within the last 48 hours (over a weekend!), we already have a number of scholars/academics who have issued a range of reviews of the study. Consider just the following:

• Bruce Baker, professor in the Graduate School of Education at Rutgers, The State University of New Jersey, offers a fairly comprehensive review.
• Sherman Dorn, associate professor of education at the University of South Florida, has already written two posts about the study: one is a brief review of the study and the other is about the reporting of the study.
• Cedar Riener, assistant professor of Psychology at Randolph-Macon College, writes about the study and about the reporting.
• Matthew Di Carlo, a senior research fellow at the Albert Shanker Institute, offers a comprehensive critique of the study.

Through traditional forms of scholarly communication, this would have taken months to a year+ to get published and hashed out. Instead, we get the paper and 4 thoughtful reviews, all within 48 hours! Our traditional forms of scholarly communication are broken and woefully outmoded. And, education scholars wonder why research is ignored and ill-respected.

One other note – consider the disciplinary backgrounds of the four reviewers: Baker (economics), Dorn (history), Riener (psychology), DiCarlo (sociology). Publishing to the open web also helps us break down disciplinary silos.

Well, certainly the best one I’ve ever read (and I’ve read dozens).

This is the dissertation recently defended by one of my new colleagues, Dr. Hilary Hughes-Decatur. It is an absolutely brilliant piece of work; it’s rich with issues to be explored, ranging from how we deal with “bodily-not-enoughness” in schools to how we elicit the voices of students as researchers and practitioners. Her understanding and use of phenomenological inquiry is outstanding, and the substantive issues addressed in the dissertation are vitally important. What jumps out to me, though, is the format. It’s not so much the magazine-style and the multigenre writing that are noteworthy as it is how that all interacts with the particular topic of the study. It’s all very purposeful and wonderfully effective, IMHO.

Dr. Hughes-Decatur writes on page 15, “I don’t write like this to turn the academy on itself – I write like this, because that is what just comes out.” Well, I’m so glad it came out like that… and I hope it helps turn the academy on itself.

[UPDATE 10/30/11: Dr. Hughes-Decatur realized that the dissertation as I had posted it is under review by a publisher to be published as a book. Therefore, the right thing to do is to not post it here. Hopefully, the publisher will have the good sense to publish it exactly as is. Until then, if you'd like a glimpse into the work, be in touch with me or Dr. Hughes-Decatur.]

What the best and wisest parent wants for his own child, that must the community want for all of its children. Any other ideal for our schools is narrow and unlovely; acted upon it destroys our democracy. – John Dewey (1900, p. 3)

I am by no means the best or wisest parent. However, lately, I find myself thinking about what we have been able to provide for my son before he even enrolls in kindergarten. He lives in a house full of books and other print reading material (the floor in his room is often invisible under a pool of books). He has his own laptop. We talk to him, lots. He talks to us, incessantly (I mean that in the best way; I think). Though he was eligible to begin kindergarten this fall (he turned 5 in June and the cutoff date in VA is October 1), we chose to enroll him in an amazing preschool for another year.

In other words, our son (and daughter, but she’s only one) is awash in social capital and cultural capital. We have provided him with a literacy-rich home environment and we have engaged in LOTS of what Hart & Risley (1995) call “extra talk.”

As a result, when he begins kindergarten in Fall 2011, he will be more than “ready.” In fact, I am confident that he will be much more “ready” than most other kids starting kindergarten at the same time as him. This phenomenon ( i.e. the contribution of home literacy environments or literacy-rich homes to differences in school readiness) is well documented, and I suspect well within the conscious mind of many educators (especially elementary educators). Kids show up to school with hugely different levels of “readiness,” and those differences relate strongly to subsequent educational outcomes.

***

A lesser known phenomenon is the seasonality of student achievement. I live in the world of educational research and question a lot of what passes these days as “evidence” in education. However, there are a few studies that I point to regularly as of high quality and meaningful. One of those studies is written about in an article called Schools, Achievement, and Inequality: A Seasonal Perspective by Alexander, Entwisle and Olson (2001).The authors used data from schoolchildren in Baltimore, where kids took achievement tests not just in the spring, but also in the fall. The figure below is a rough approximation of what they found with respect to reading achievement.

It is noteworthy that during the academic year (between fall and spring), all kids gained about the same. That is, the slopes of the lines between fall and spring are nearly identical across all three SES groups. It is during the summer months where the gains (or losses) are disparate across SES groups (i.e. the spring-to-fall slopes are significantly different). “Lower SES youth start out behind (i.e. the baseline differences are significant) and during the school year they keep up, but during the summer periods their gains fall short of those registered by upper SES youth” (p. 182). As a result of starting behind and the “summer learning loss,” achievement gaps are exacerbated over time.

This is not to say that families and communities are the only holders of the keys to student learning. In fact, the Alexander, Entwisle & Olson study points to the extremely important compensatory role schools play.

Schools do matter, and they matter the most when support for academic learning outside school is weak. School-based public resources do not completely offset the many and varied advantages that accrue to children of privilege by virtue of private family resources outside of school (e.g. Coleman, 1990)…The powerful role of schools in fostering achievement of all children is one lesson informed by a seasonal perspective on learning.

One might even go so far as to suggest, based on this study, that, on the whole, schools do a reasonably good job of serving all children, not just those of wealthy families. That is, schools are not “failing.” Berliner and Biddle (1996) refer to this as The Manufactured Crisis.

***

I don’t know if we qualify as mid- or high-SES, but I know we’re not low-SES. So, what does our son do over the summer? He goes to camps: nature camp, pottery camp, general day camp, etc. We travel over the summer, too. In other words, my son’s learning does not stop because school is not in session.

Thus, at the risk of sounding terribly elitist, I want for every child in our nation what my son has. For that to happen, we cannot and must not talk about school reform without talking about equality of opportunities for kids outside of K-12 classrooms. If we are serious about closing the achievement gap, we need to:

1. Commit, first and foremost, to high-quality preschool for all kids.
2. Consider policy efforts to improve the literacy richness of homes of children in low-income families.
3. Think seriously about shifting to year-round schooling, especially in low-income communities (NOTE: I’m not necessarily talking about extending the school year beyond 180 days; rather, I’m talking about shifting away from school calendars based on the agrarian calendar).
4. Make schools community centers that are open beyond the school day, where kids can regularly access media centers and computer labs.

So, while policy makers are arguing over labor matters and the intricacies of school governance, and while we’re all waiting for superman, my hope is that local communities commit themselves to providing meaningful learning opportunities for all children beyond what is provided during the traditional school day.

[NOTE  #1: this was originally posted on February 7, 2008.  I am re-posting it here as part of the #edublogBT idea/meme about which I wrote yesterday.]

[NOTE #2: at the time, I had designs on posting a weekly blog post about educational research.  I think I got as far 4 or 5 entries before that idea faded. I'd like to, at some point, revisit the "scientifically-based blog post" idea. I think it's a niche I can help fill.]

***

Might as well start right at the top of my own pecking order. To me, one of the very best, if not the best, articles I have read is called, Factors Affecting Technology Uses In Schools: An Ecological Perspective, by Yong Zhao and Kenneth A. Frank from Michigan State University. This well-designed, mixed methods study examined teacher and student use of computers from an ecological perspective. The authors start by referencing the introduction of zebra mussels into Lake St. Clair. The mussels were first introduced in 1988, and by 1990, they could be found in all of the Great Lakes. In fact, in a very short period of time, the introduction of the zebra mussel has caused tremendous ecological change in the Great Lakes. While recognizing that computers and zebra mussels are very different, the authors imply that their goal is to understand why “computer uses” as an invading species have not caused disruption to the schooling organization.

Their first major finding confirms some of my own empirical conclusions; the vast majority of the variance in computer use exists WITHIN schools, not between schools. I think that’s a really important but rarely understood finding. One of the many implications is that we are much more likely to be able to identify high-end computer using teachers than high-end schools. Schools are not the right unit of analysis in examining differences in technology use. Other findings:

• Teachers niche in the ecosystem – English teachers were much more likely to use computers than their peers.
• Teacher / Ecosystem Interaction – teachers who reported feeling pressure and support from colleagues were more likely to use computers more. Also, where there were too many competing invading species (other “programs” or “innovations”), computer use suffered.
• Teacher-Computer Predisposition for Compatibility – teachers who found computers to be more compatible with pedagogical beliefs and practices used computers more.
• Opportunities for Mutual Adaptation – teachers that had more time to “play around” with computers used them more for teaching/learning.

Overall, the authors found great support for the ecological framework. I think the study is framed beautifully, carried out well and reported eloquently. Most importantly, the findings resonate with my own empirical understandings of technology integration. I’d love to hear your thoughts…

[NOTE: this is the final post in a series of posts about a report recently issued based on a study done by Marzano Research Laboratory.  Part I is here, Part II is here, Part III is here, and Part IV is here.]

PART V: SUMMARY AND RECOMMENDATIONS

[NOTE #2:  I know, I know...I'm a couple of days late on this one. Sorry.]

Before I sum up and conclude, I should point out one other major flaw in this study.  Marzano and his team use percentile ranks incorrectly.  On page 18 of the report, they write: “Of particular interest is the column entitled ‘% Gain.’ Again, this column contains the percentile gain (or loss) in achievement associated with the treatment (i.e., use of Promethean technology).”  Two problems here.  First, percentiles are not the same as percentages (or % as it is written in the report). Second, they then go on to write: “This value [the percentile gain] was determined by consulting a normal curve table for the area for each reported effect size.”  This would be fine if the scores on the dependent variables are normally distributed, which they most definitely are not.  For Marzano to go around saying that incorporating Promethean IWBs into instruction will improve student achievement by 17 percentiles is wrong on lots of levels.

It should be clear by now that if I were reviewing the Marzano IWB study report as a manuscript submitted for publication in a peer-reviewed journal, I would reject it.  I would not even mark it as “revise and resubmit.”  The problems with the work are too critical and, in most cases, impossible to fix.

In summary, those problems are:

• Misuse and misapplication of meta analysis.
• Incorrect usage of key terms.
• Serious problems with measurement validity and reliability.
• Major threats to internal validity.

Those last two points are with respect to each of the 85 classroom-based studies that serve as the basis for the meta-analysis.  The ultimate problem, then, is that the hallmark of good meta-analysis is the use of strong criteria as decision points for including individual studies.

As a point of comparison, I’m linking to two reviews of research.  Each is described as having used “best-evidence synthesis” which very closely resembles meta-analysis.  The methods used in the studies reported in the articles below are also consistent with those used by the What Works Clearinghouse.

Effective Programs in Elementary Mathematics: A Best-Evidence Synthesis

Effective Reading Programs for Middle and High Schools: A Best-Evidence Synthesis

In the first article, you’ll notice on the seventh page of the document (p. 432 of the article) a list of criteria for inclusion.  The authors of those articles also provide a list of studies that were considered for inclusion but that were ultimately excluded along with the reasons for exclusion.  This combined approach is critical; it gives the consumer of the research confidence that the data used in the meta-analysis come from many solid studies.

The impact of sample size for any given study included in a meta-analysis is another important point raised in the articles above.  According to the authors of the second article, “[p]revious research (e.g., Rothstein et al., 2005; Slavin, 2008; Sterne, Gavaghan, & Egger, 2000; Taylor & Tweedie, 1998) has shown that studies with small sample sizes report larger effect sizes than studies with large samples.”  As a result, in their meta analysis, the authors weight the individual findings by sample size.  In each of the separate sites/studies used by Marzano and his team in their meta analysis, sample sizes were tiny.  Consider for example site #34, teacher #57 where there were 9 students in the control group and 5 in the treatment group.  There is no way that study gets included in any decent meta analysis.

There is a bit of irony in my choice of articles to post as exemplars.  The lead author in each of those studies is Dr. Robert Slavin, the developer/founder of Success for All.  Slavin has been frequently critiqued for being the lead researcher/analyst/author on many evaluation studies of Success for All, the program that he created.  In other words, he has been accused of producing biased research.  I don’t know enough to say if his research is biased or not; it’s certainly legitimate though to raise the question of bias where he is involved in the research.  What I do know, though, is that each of the articles appears in one of the most well-respected, highly selective peer-reviewed journals.  The math study appears in the Review of Educational Research which is dedicated to only publishing exquisite and top-notch reviews, syntheses, and meta-analyses in education.  Thus, there is good reason to believe that those two articles present exemplars of how meta-analysis type research should be done.

I wrote earlier that doing good, comprehensive program evaluation in education is difficult and resource-intensive.  That said, I believe it would actually be reasonably easy to evaluate the impact of IWBs on student achievement.  In this era of standards and accountability, in any given state, we have year-to-year state test scores (at least in math and reading/language arts) from grades 3 to 8. So, Marzano’s team could have focused on one or two grade levels in one or two subject areas in one state.

Let’s say they focused on 8th grade student achievement.  All they needed to do was to find about 20 middle schools that were willing to participate.  In those 20 schools, there would be one subject-area teacher teaching in a classroom with the IWB and one teacher teaching a comparable class (NOTE: comparable here refers to students who are demographically similar and who are no different with respect to student achievement at baseline) without the IWB.  Surely there are at least 20 middle schools in any state where there are two 8th grade teachers teaching comparable classes.

A common way to get schools and teachers to participate in such a study would be to offer an incentive.  For Promethean, the promise of a free IWB to the teacher/classroom in the control condition the year after the study would be a wonderful incentive.  Given this sampling framework, Marzano’s research team could work with the schools, districts or state departments to get student achievement data on the students in those 40 classrooms (20 treatment + 20 control).  This could easily be done without violating any privacy laws.  The students’ scores on the 7th grade state exams could serve as the pretest or the covariate.  Their scores at the end of the 8th grade year would be the dependent variables.  Over 40 classrooms, we’d be talking about a sample size of well over 800, with well over 400 students in each condition.  Such a study would have lots of power.  Analytic decisions would have to be made with respect to the unit of analysis.  Marzano and his team could use the classroom as the level of analysis and conduct matched-pairs statistical test.  Or, they could use the student as the unit of analysis and account for the nesting or lack of independence by using multilevel modeling techniques.  Either way, this design would be much more appropriate and powerful for estimating the effects of IWBs on student achievement.

In the last couple of days, I spoke about this series of posts to two professors who I respect greatly.  Interestingly, each one was very surprised to hear my opinion that Marzano was affiliated with sloppy work.  One said, “he’s always been so careful.”  That may very well be.  I don’t intend for this series to be an indictment of Marzano (or even of IWBs).  My hope is that I’ve provided a sensible critique of research that is being widely disseminated.

I often lament that decisions in education are too often made in the absence of empirical evidence.  I wish policymakers in education would consult research more often.  However, if educational decision makers decide to make an investment in interactive white boards, I would strongly urge them to do so for reasons other than the evidence offered by the Marzano Research Labs.

[NOTE: this is the fourth in a series of posts about a report recently issued based on a study done by Marzano Research Laboratory.  Part I is here, Part II is here, and Part III is here.]

PART IV:  INTERNAL VALIDITY

“Internal Validity is the approximate truth about inferences regarding cause-effect or causal relationships” (Trochim, 2006).

I consider research questions as one of three types: descriptive, relational and causal.  From there, certain research designs lend themselves to best answering the research questions.  For example, naturalistic inquiry methods such as those used in ethnographies or case studies are suited only to answer questions of description.

Marzano and his team set out mainly to answer a question of causality: does use of the IWBs cause improvement in student achievement?  How they chose to do that is less than ideal, but we know that was their intention based on the use of quasi-experimental designs in each of the 85 classroom-based studies.  Experiments and quasi-experiments are designs intended to address questions of causality.

When such designs are employed, the primary consideration of trustworthiness is internal validity.  “All that internal validity means is that you have evidence that what you did in the study (i.e., the program) caused what you observed (i.e., the outcome) to happen” (Trochim, 2006).  In any effort to prove causation, one key is to be able to rule out alternate explanations.  In other words, alternate causes are threats to internal validity.

There are any number of ways to think about possible threats to internal validity, and there are many ways to describe them.  There are at least a couple of general threats to internal validity for each of the 85 classroom-based studies that Marzano and his team used for his meta-analysis.  The threats result mainly from the fact that, at least in the secondary schools, the same teacher taught the treatment and the control groups.  At one level, that might seem like a benefit in that it eliminates teacher-level confounds.  However, consider:

Social threats to internal validity – also known as or related to intervention or exposure bias.  The students in the control group are presumably in the same classroom (at a different time) than the students in the treatment group.  They see the IWBs.  They also probably hear about the teachers using the IWBs from their friends in the treatment class.  Might there be some compensatory rivalry and/or resentful demoralization?  Trochim (2006) defines the latter:

Here, students in the comparison group know what the program group is getting. But here, instead of developing a rivalry, they get discouraged or angry and they give up (sometimes referred to as the “screw you” effect!). Unlike the previous two threats, this one is likely to exaggerate posttest differences between groups, making your program look even more effective than it actually is.

Experimenter’s bias – it is impossible to tell from the report all that the teachers knew and/or were told about the study.  We know that in the instructions to the teachers, it says “Thank you for agreeing to participate in an action research [sic.] study regarding the effectiveness and utility of the Promethean technology in your classroom.”  So, they knew what Marzano and his team were looking at/for.  This knowledge could easily cause the teacher to pay more attention to her/his teaching in the treatment class.  Additionally, just the fact that the teacher had to take an existing unit and figure out how to integrate the IWB technology means that the teacher was biased toward that group (i.e. she/he was more planful about that teaching).

Earlier, I wrote that I’d never seen meta-analysis included purposefully as an a priori part of a separate research design.  Frankly, I’ve never seen or heard of a quasi-experiment in education where the (non-random) selection is at the classroom level and yet the treatment and control classes still have the same teacher.  I could be wrong here, but in summary, this approach raises a number of general threats to internal validity for each of the 85 individual studies upon which the meta-analysis was based.

COMING NEXT:

FRIDAY – Part V: Summary and recommendations

[NOTE: this is the third in a series of posts about a report recently issued based on a study done by Marzano Research Laboratory.  Part I is here and Part II is here.]

PART III:  MEASUREMENT VALIDITY AND RELIABILITY

As I wrote yesterday, Marzano and his research team had a “dependent variable problem.”  That is, there was no single, comparable measure of “student achievement” (his stated outcome of interest) that they could use as a dependent variable across all participants.  I should note that they were forced into this problem by choosing a lazy research design.  A tighter, more focused design could have alleviated this problem (more on that in Part V).

Let’s visit their research methods briefly.  Marzano’s research team asked 79 teachers to teach a unit with the IWB and one without it.  In secondary schools, that meant two separate classes of students.  In elementary schools, that meant teaching two “similar” units, one using the IWB and one without using it.   Students were to be given a pretest before the unit and a posttest after the unit. For the elementary teachers, that actually meant giving four tests; a prettest and a posttest for each of the two “similar” units [NOTE: I'm putting quotation marks around "similar" because this raises a question of research ethics for me.  But for this study, would the elementary teachers really teach two units on the same topic?  One golden rule of educational research for me is that research should never drive pedagogical decisions.]

In the instructions to participating teachers, Marzano and his team wrote:

To be involved in a study you must be willing to do a few things. First you should select a specific unit of instruction, or set of related lessons on a single topic (hereinafter referred to as unit) and design a pretest and posttest for that unit. It is best if the unit is relatively short in nature. For example, if you teach mathematics, you might select a two week unit on linear equations. At the beginning of the unit, you would administer a pretest on linear equations. Then at the end of the unit you would administer a posttest. This test could be identical to the pretest, or it could be different. The important point is that you have a pretest and a posttest score for each student on the topic of linear equations. Ideally the pretest and posttest are comprehensive in nature.

Then, later, they write:

Finally both pretest and posttest scores should be translated to a percentage format. For example, if your pretest involves 20 points and a particular student receives a score of 15, then translate the 15 into a percentage of 75% (i.e., 15/20 = .75 x 100 = 75%) and record that as the pretest score for the student. If your posttest involves 80 points and that same student receives a score of 75, then translate the 75 into a percentage of 94% (75/80 = .94 x 100 = 94%) and record that as the student’s posttest score. The same procedure would be employed if you used a rubric. For example, if a student received a 2 on a 4 point rubric on the pretest, this score would be translated to a percentage of 50% (2/4 = .50 x 100) and this would be recorded as the student’s pretest score. The same translation would be done on the student’s rubric score for the posttest.

Those posttest scores became the dependent variable in each of the 85 separate studies (the pretest scores were used as covariates).  In other words, the measure of student achievement in each of the 85 studies is “% correct or % proficient on a teacher created test of a single unit.” We can quibble about how to best define and operationalize student achievement, but that measure is unlikely to satisfy any legitimate educator’s conception of student achievement. Furthermore, what matters most here is the trustworthiness of the actual measure(s) used.

In the field of measurement, trustworthiness is operationalized in terms of validity and reliability. In the most general terms, validity is about the accuracy of a measurement (are you measuring what you think you’re measuring?) and reliability is about consistency (would you get the same score on multiple occasions?).  I won’t write a whole treatise here about measurement validity and reliability; that would be a waste of our time (especially in an era where information is not scarce and you can find credible and accessible descriptions in places like this).  Suffice it to say that none of the measures of student achievement used by any of the teachers who participated in the study are either valid or reliable.

To repeat what I wrote yesterday about meta-analysis, it is a powerful technique for combining the results of  lots of studies, each of which is fully reported and each of which was selected for the meta-analysis because the full report of the study allowed the analyst to determine its trustworthiness. Because we do not have full descriptions of each of the 85 studies, we do not know what the dependent variable was in any of the studies; we just know that the teacher made up a test and converted the score to a percentage.  In other words, as best we can tell, each of the 85 studies included in the meta-analysis suffers from a serious lack of measurement validity and reliability.  That renders the whole meta-analysis invalid.

NOTE BENE: I know this is more of an issue of research design (the topic of Part II) than a measurement issue (the topic of Part III), but I needed to add this bit of information.  At the end of the letter to the participant teachers, Marzano and his team wrote “Thank you again for considering involvement in an action research project.”  Additionally, in a blog post about this study, Sonny Magana wrote, “Over the past academic year, Dr. Robert Marzano conducted a much-anticipated meta-analysis of numerous action research studies on the direct effect of Promethean’s transformational technologies on academic achievement.”  Again, I’ll spare you the detailed description of action research, and I’m by no means an expert on action research.  But, I know enough to state with total confidence that what the teachers did FOR Marzano and his team was NOT action research.  Action research is NOT simply teachers collecting data in their own classrooms; it is a much more complicated and sophisticated process.  For me, the misuse of the phrase “action research” calls into question the credibility of this study as a whole.]

COMING NEXT:

THURSDAY – Part IV: Internal validity issues

FRIDAY – Part V: Summary and recommendations

[NOTE: this is the second in a series of posts about a report recently issued based on a study done by Marzano Research Laboratory.  Part I is here.]

PART II: RESEARCH DESIGN ISSUES

From a research design perspective, this study (or collection of studies?)  is best described as unusual.  In fact, what Marzano’s research team tells us is that they conducted 85 separate small studies and then “synthesized” the results of those studies through a meta-analysis (a set of complex statistical analyses used to examine “effects” across multiple studies).  Meta-analyses are not unusual, and are quite helpful as a way of combining the results across lots of studies.  However, it IS very unusual to use meta-analysis as an a priori technique built in to an evaluation.  Frankly, I’ve never seen it before.  Meta-analysis is more typically used when there is a mature body of research within a topic area comprised of studies by multiple researchers across a number of years.  Furthermore, they are typically done using lots of studies, each of which is fully reported and each of which was selected for the meta-analysis because the full report of the study allowed the analyst to determine its trustworthiness.  That, to me, is one of the biggest problems with this study/analysis; we don’t know enough about each of the 85 individual studies.

Is Marzano’s meta-analytic approach “wrong?”  Not necessarily, but to me, it’s indicative of a certain laziness.  High-quality evaluation research in education is complicated and costly.  It requires a ton of coordination and planning, especially to make sure that key data are high-quality and comparable.

Ultimately, Marzano got 79 teachers to agree to “participate” and got data on over 2,700 students.  That’s commendable.  But those teachers varied by grade level taught and subject taught (other than the elementary teachers; more on them later).  They also taught in different states.  So, he had a huge “dependent variable” problem.  In other words, there was no single, comparable measure of “student achievement” (his stated outcome of interest).  He needed a way to account for that and chose to deal with it by way of analytic techniques (i.e. meta-analysis), rather than by focusing the study (perhaps within a single state within one or two grade levels). He also chose a lazy way to get data on student achievement (more on that in Part III).

COMING NEXT:

WEDNESDAY – Part III: Construct validity and reliability issues

THURSDAY – Part IV: Internal validity issues

FRIDAY – Part V: Summary and recommendations

Two of the more well-known brand names in education recently combined forces.  Robert J. Marzano (wildly popular consultant/author/speaker) produced a report of a study he conducted of Promethean ActivClassroom (wildly popular interactive white board (IWB) technology).

The report has received lots of publicity; I have seen multiple references to it on Twitter and elsewhere. You can get a copy from Promethean, but only by first providing them with lots of contact information here. I wasn’t willing to make that exchange, but the day after discussing the report on Twitter, Sonny Magana, the Director of Education Strategy at Promethean, Inc. was kind enough to e-mail me a copy of the report.

Marzano’s work has not yet been formally reviewed by any “peers” (at least as far as I can tell).  While I am very critical of the way peer-review is typically conceived and carried out in academia, there is real value in the process.   Therefore, I’m using this space to do just that.  This first post is a bit of an introduction.  In subsequent posts, I’ll address methodological and analytical issues.

In this first post, I’ll try to do two things simultaneously: address a key criticism and establish some semblance of credibility as a reviewer.  The report states that it was prepared by the Marzano Research Laboratory for Promethean, Ltd.  That undoubtedly means that Promethean funded a study of their own product(s).  Such an arrangement, which certainly gives the appearance of a lack of objectivity, is not unpredecented and not even unusual.  I should know; I’ve done plenty of evaluation research as a “third-party, independent” evaluator funded by vendors.  For the better part of ten years, I was part of a research team that conducted evaluation research funded by private vendors such as Lightspan [since purchased by Plato Learning], Scholastic, eChalk, Jostens [sic.], etc to study their own products/programs.  Based on my experiences, I can state confidently that those sorts of arrangements should be viewed with skepticism and examined critically.  I stand by much of the work I did and would defend the work against any critique.  However, there were certainly instances where the vendor/funding source “influenced” the contents of the final report. More often, the final report was written in a way that would be most palatable to the client.

[NOTE: of the privately funded evaluation research I was a part of, I've only been able to find one report that is publicly available.  This report of a large-scale evaluation of Scholastic's READ 180 (funded by Scholastic) happens to be one by which I swear.  There are a number of reasons why this study is credible, but the most important factor is that the main stakeholder was the Council of Great City Schools and not Scholastic].

Ultimately, without being present at the initial negotations between the parties and without being privy to conversations between the researcher(s) and the client, it is hard to know how “objective” or “honest” a research report is when the study is of a product/program and the study is funded by the vendor of said product/program.  The best we can do is to (peer-)review these sorts of reports against the standards of educational research.  Onward then…

PART II: Research Design

You couldn’t expect a guy with a doctorate in the politics of education to let the education portion of Wednesday night’s presidential debate go unchecked, could you?

Apparently, both candidates support charter schools.  No surprise from McCain; the Republican party tends to support most forms of school choice (that’s a gross generalization, but I’ll live with that for now). That Obama supports charter schools signals an interesting policy shift, assuming that Obama’s stance is representative of the Democraticy party as a whole (which it may very well not be).  I’m personally pretty mixed on charter schools.  On one hand, I think any policy or set of policies based in free market principles where perfect information on the part of consumers is assumed is highly problematic in the field of education.  On the other hand, these days I’m for anything that attempts to disrupt the status quo in public education.

And, speaking of “perfect information,” both candidates at least implied that charter schools “work” or that they are “effective” in some way (as opposed to voucher programs, where there seemed to be some disagreement).  Well, that’s less than perfect information.  Consider this study conducted as part of NAEP’s pilot study of charter school performance in 2003.  According to the executive summary for the report, “After adjusting for student characteristics, charter school mean scores in reading and mathematics were lower, on average, than those for public noncharter schools.”

I also bring your attention to more recent research conducted by colleagues and “social associates” (I’m not sure I can call them “friends,” but I have been out socially with them on multiple occasions) Sarah and Chris Lubienski.  Based on their analyses of NAEP data, Sarah and Chris concluded that “charter schools, privately operated and publicly financed, did significantly worse than public schools in the fourth grade, once student populations were taken into account.”

(BTW, teachable moment…the next sentence in the NYT article is as follows: “In the eighth grade, it found, students in charters did slightly better than those in public schools, though the sample size was small and the difference was not statistically significant.”  That’s a nonsensical statement.  If the differences were not statistically significant, then nobody did better than anybody else; not even slightly better.  So, the first part of the sentence cannot logically precede the second part.  This bugs me!)

The body of research on charter schools and school choice policies more generally has become overly politicized.  There are too many researchers with agendas dabbling in that field, and too many policy advocates who cherrypick a single study to support their argument.  However, in my reasonably well-informed opinion, the two studies above are as “independent” as they come.

So, Senators Obama and McCain, I’m in favor of exploring any and all educational policy options, including choice-based alternatives.  But, let’s please not mislead the American public.