Information for:

Vanderbilt University Peabody College of Education and Human Development

HOME > News & Events > Gold mettle: How to make informed decisions about education

Ideas in Action, Feature

Gold mettle: How to make informed decisions about education

topics: Assessment, Education Reform, Psychology and Human Development

Fall 2007, Kurt Brobeck, Ideas In Action

Educational researchers have long been criticized for emphasizing qualitative over quantitative research. More importantly, observers lament the faddishness of education reforms and view the research coming out of education think tanks as justifying pre-established political agendas. In turn, policy-makers pick and choose among research studies and run with the results that most closely match their own biases.

In recent years, however, the landscape has begun to change. Among current topics in education research, few garner so strong a reaction as the subject of research methodology itself. In the U.S. Department of Education, when it comes to sponsored research, a new day has dawned.

These changes took form with the Education Sciences Reform Act passed by Congress in 2002. The act created within the Department of Education the Institute of Education Sciences, with a mission “to bring rigorous and relevant research, evaluation and statistics to our nation’s education system.” IES has been headed since its founding by Grover J. (Russ) Whitehurst. Whitehurst has worked assiduously to move federal involvement in education toward evidence-based practice, and in doing so he has prioritized answering questions about “what works in education.” IES views this as a causal question that is best answered with the same “gold standard” applied to medicine and related fields.

So what exactly is the gold standard? David Cordray, professor of public policy and psychology and director of Peabody College’s Experimental Education Research Training (ExpERT) program, says, “The gold standard is the randomized control trial (RCT). This is the only class of research design capable of producing adequate evidence for making decisions about the effects of educational practices, about policies, about programs.”

The most prominent characteristic of research meeting the gold standard is the randomization of individuals, teachers or schools to conditions. As in the sciences or medicine, experiments involve two or more conditions, one of which is a control group that represents the status quo. According to Dale Farran, professor of education and psychology, “An intervention condition is anything imposed in an education setting that is designed to change the behaviors of the recipients. A new curriculum could be an intervention, or professional development for teachers. Any time education scientists propose doing something different from business as usual, that is an intervention.”

Through random assignment to conditions, two similar groups are assembled. One receives the intervention to be tested, while the other does not. Comparison of the outcomes from these two conditions enables investigators to more accurately determine the effectiveness of a given intervention while controlling the influence of numerous sources of personal and technical biases. “Because of its ability to control biases, the RCT is a gold standard when we want to estimate the causal effect of interventions,” says Cordray. “And you’re taking out all of the biases that emerge when you let people just choose the condition they want.”

As Cordray acknowledges, conducting RCTs in real-world settings can be difficult: “You are dealing with children, and children are in classes, and classes are in schools. It’s not always possible to randomize the kids or the conditions because of basic practical problems like contamination. Teachers talk, materials get shared, so it isn’t a good idea to randomize at that level.” Instead, Cordray says, researchers need to conduct what are called cluster RCTs. These call for working with large numbers of schools, and randomizing teachers or schools.

According to Cordray, “It’s become clear that an RCT is the best tool for knowing what works. It is an important first step. Ultimately, we want to answer questions not only about what works, but under what circumstances does it work, for whom does it work, how does it work, and why does it work. These questions require additional statistical and research methods within an RCT. Although these methods are no longer considered experimental, the answers they produce are important because those are the ones that practitioners and policy makers are also interested in.”

ExpERTs Wanted

While investigators nationwide have been scrambling to make their grant applications and research designs more competitive, education schools have also begun to address the question of how to prepare the next generation of education scholars. At Vanderbilt, Cordray and 27 Peabody faculty members have developed a program intended to equip future education scholars with the requisite skills. In 2004, Peabody was among the first of 10 institutions nationwide to receive grants from IES under its predoctoral research training program. Peabody’s Experimental Education Research Training (ExpERT) is the result.

Doctoral applicants who have been accepted for admission, as well as doctoral students still in their first year of study, are eligible to apply for a predoctoral fellowship in the ExpERT program. Fellows undertake an integrated sequence of graduate courses in statistics,measurement and design, along with courses in educational practices, contexts, pedagogy and learning. The courses may go beyond the requirements of their doctoral curricula. Research, and how to conduct it, is strongly emphasized, and students take part in extensive field research working collaboratively with one or more of the college’s faculty members. They also attend monthly interdisciplinary lectures and colloquia, teach classes and attend professional conferences.Most importantly, they acquire direct expertise in planning, executing and analyzing RCTs firmly grounded in theoretical frameworks and supported by empirical evidence.

The program offers full-tuition support for up to five years of study, a generous 12-month living stipend, and funds for research and travel to professional conferences. Forty students spread evenly across four of the college’s departments are participating. “Dean Benbow has been especially supportive of ExpERT, including providing tuition and stipends for two of the five years,” says Cordray. He says that most of the fellows are working with multiple faculty members on several projects, or they may work with one faculty member on an extended series of papers. Over half of the fellows have contributed at least one paper or poster.

In 2005, Vanderbilt won a second grant from IES for postdoctoral research training. The program provides funding for up to four fellows for two years apiece. Three postdocs are currently participating, and a fourth just completed his fellowship. ExpERT will run through 2010. With up to 40 Ph.D. graduates having received the ExpERT training, Cordray believes that Vanderbilt will assert considerable influence on education research and practice: “IES wanted to know what works, for whom, and under what circumstances.We’ve added the how and the why questions to that. And we’re creating our program around the methodologies that are necessary to answer those questions. Our graduates will be right at the forefront of being able to tell superintendents the circumstances under which a given kind of program is effective, for what kinds of schools, what kinds of teachers, what kinds of kids, and they’ll be able to do that in a way that is state of the art.”

The Fidelity Hurdle

One thorny problem with RCTs has to do with the question of intervention fidelity. Put simply, this means the degree to which the intervention tested is faithfully applied in the real-world setting during field trials. According to Cordray, few current projects measure fidelity in their evaluation of a trial’s outcomes. “People will just do a simple assessment of whether or not the intervention itself is put in place, but they don’t really check to see if what was put in place actually differs from what was in practice already.We need to be much clearer about what to measure in terms of what was actually received by kids in all conditions. IES is basically making that a requirement of their research grants,” he says.

A third grant from IES, at just under $1.5 million dollars, will enable Cordray; Mark Lipsey, of the Vanderbilt Institute for Public Policy Studies; and Dale Farran to develop a reliable set of measures by which fidelity can be measured. The grant involves several components. First is a synthesis of the existing literature on prior attempts to asses the integrity or fidelity of interventions. The second component will examine established databases that have incorporated fidelity assessments of some sort into their data collection. The team will apply new techniques of either measuring or scaling the fidelity index and looking at its correspondence with outcomes.

The last piece involves collection of new data in Nashville for the Building Blocks program, a pre-K math program being installed in about 60 classes or Head Start centers. Cordray, Farran and Lipsey will collect and review data to determine whether Building Blocks is delivered as intended and, if not, the consequences of deviation from that. The Building Blocks curriculum was developed by Douglas Clements and Julie Sarama of the State University of New York at Buffalo and is currently undergoing scale-up. Says Cordray, “We’re trying to develop indices that are as closely aligned with the Building Blocks theory as possible. But we’re also testing whether or not it matters that you become more and more meticulous and focused on the elements of Building Blocks or whether you can use indices that are less costly and easier to measure. Do you get the same set of relationships when you use a simple measure versus a complicated measure?”

Establishing simple indicators of fidelity is especially important when interventions reach the scale-up stage, Cordray believes. This is the stage in which investigators ask whether an intervention will work in a wide array of schools. The teachers implementing an intervention may no longer be volunteers, so they may be more or less interested in participating. As a result, scale-up becomes the setting in which infidelity is more likely to creep in.

“You can’t spend all of your money getting at the nitty gritty details of what transpires in the class; you just don’t have the luxury or resources for that. So you have to have indicators that are well connected but not expensive,” Cordray says. In addition to the Building Blocks project, he is conducting a similar evaluation for the Midwest Regional Education Lab that involves 18 schools and about 75 teachers. The teachers are being randomly assigned to receive (or not) the Measuring Academic Progress program, an intervention that provides information on student progress to enable teachers to reconfigure instructional strategies to optimize students’ performance on high stakes tests. “We’re trying to test whether or not the teachers actually differentiate instruction—that’s one of the critical fidelity components in our assessment. If student outcomes improve, it is important to demonstrate that differentiated instruction also improved, otherwise we are still in the dark about the what of our what works claim.”

Opposition

Not everyone agrees with the seeming primacy afforded randomized control trials by IES. In April, a committee of the American Educational Research Association released a report encouraging greater use of large data sets maintained by the Department of Education to analyze cause and effect questions. The authors argued that technological advances and research techniques used in other fields can move educational research beyond correlational studies. AERA’s grants board, which authored the report, also questioned whether the use of RCTs—where some students are randomly assigned to a control group and may not receive important educational benefits—is always ethical.

More recently, a report from an interagency panel chaired by Education Secretary Margaret Spellings—the Academic Competitiveness Council—met with criticism for overemphasizing RCTs. In Science magazine (June 2007), Jere Confrey, of Washington University, said, “They missed the idea of multiple evaluations, using multiple methods, to come up with a theory of change.”

Dale Farran thinks there is still plenty of room in the education community for traditional research. “If you look at the proportion of RCTs published in education journals versus others, it’s still a tiny percentage. There are a lot of other things that people might want to do in research, things people want to know, including descriptive research. There need to be other mechanisms for that, whether through foundations or the NSF. RCTs frighten people because they’re just so hard to do. But they are certainly not the dominant paradigm.”

For his part, Cordray acknowledges the resistance but sees an underlying issue. Many educators are just not trained in the methods necessary to do experimental work. “If you want to answer causal questions, you need different kinds of methods. For answering other questions, the people who work in education are doing a fine job. The question is: Is the result of the work they’re doing answering practical questions about how to make the educational experience better for kids? That’s where the battles get enjoined—the nature of the evidence.”

Cordray is also quick to point out that the current emphasis on evidence-based practice is not new. “Most of us around Peabody have been talking the same line for years,” he says.He points for example to the research of Douglas and Lynn Fuchs in the Department of Special Education.

Nor is the Department of Education unique among federal agencies in adopting more rigorous research standards. As Cordray says, “Across the board the idea of doing well controlled studies of the effectiveness of products, and processes and policies is something that’s here to stay.

Team Players

Dale Farran, professor of education and psychology, was one of the first Peabody faculty members to become involved with the current IES push to fund more randomized control trials. With Mark Lipsey, director of the Center for Evaluation Research and Methodology and a senior research associate at the Vanderbilt Institute for Public Policy Studies, Farran conducted her first RCT in 2002, while IES was still getting underway.

"Mark and I began working together in our Preschool Curriculum Evaluation Research (PCER) grant. Seven Middle Tennessee school systems agreed to work with us in a randomized control trial comparing the effects of two alternative curricula and a business-as-usual control group," Farran said. "Our project was one of seven projects funded nationally in the first round of IES' PCER awards. We're following these students through third grade, and we're starting to get their latest scores in now."

At the time the grant was awarded, Farran says, Russ Whitehurst wanted to show Congress that RCTs could be done in education, that educational research could be done in a rigorous way that would justify the government's continued investment. IES chose early childhood curricula as its first initiative. Competition was tough, Farran recalls, "You had to have good designs. You had to have good ideas but you had to have good designs. You had to know something about power, something about effect sizes and something about how many subjects you would need to see an effect. Mark brought this to the table."

Farran sees collaboration between methodologists and researchers who are content experts as essential to successful research to establish evidence-based practices. "We have extraordinarily strong content people here. And you want the methodological approach not to be external. You want them to be melded in with these people. We have really terrific methodologists. We have Dave [Cordray]; we have Mark Lipsey; we have Dale Ballou. Lynn Okagaki, the commissioner for education research, has begun to call this 'the Vanderbilt model.'"

Like David Cordray, Farran sees Peabody's ExpERT program as an essential part of the mix, as well. "One of the things the ExpERT students have helped to do is bring some of that methodological knowledge and consciousness into the department to be a part of teams that include these strong content people. ExpERT has brought people together. It's too complex a world for any one person to know it all, so you need to have teams," she says.

Training Regimen

In June, Peabody College served as host for an Institute of Education Sciences Research Training Institute on Cluster Randomized Trials. Out of approximately 160 applicants from across the nation, 30 participants were selected for the 12-day intensive training. The institute included lectures, workshops, and group experiences.

Faculty for the institute included Howard Bloom, chief social scientist, MDRC; Margaret Burchinal, senior scientist and director of the Data Management and Analysis Center at the FPG Child Development Institute and research professor in psychology, University of North Carolina; David S. Cordray, professor of public policy and psychology; Fred Doolittle, director of the Policy Research and Evaluation Department, MDRC; John W. Graham, professor, Department of Biobehavioral Health at Pennsylvania State University; Larry V. Hedges, Board of Trustees Professor of Statistics and faculty fellow, Institute for Policy Research, Northwestern University; Mark W. Lipsey, director of the Center for Evaluation Research and Methodology and senior research associate at the Vanderbilt Institute for Public Policy Studies; and Ina Wallace, senior research psychologist at RTI International.

David Cordray was pleased by the hands-on nature of the experience and its end results. “The goal of the group experience was for people to take what they learned each day and develop a research proposal they would present at the end of the two-week period. Every day at the end of the lecture, people would get together and Mark, Larry and I would help them work through their proposals and apply what they learned. At the end of the Institute each group gave a one hour presentation and they were great.”

Unlike the doctoral students in Peabody’s ExpERT program, participants in the summer institute were experienced in the field. “They were already either in research groups or companies, or the educational regional labs, or from universities,” said Cordray.

Curriculum for the institute focused on the principles underlying RCTs, understanding the structure and context of educational interventions, selecting measures for assessing outcomes, describing implementation fidelity, and acquiring the knowledge to design and conduct a cluster randomized trial.