Preschool Curriculum Evaluation Program
Implementing pre-k programs is one of the major initiatives states have undertaken in recent years to improve educational outcomes for economically disadvantaged students (Barnett et al., 2007). Participation in formal pre-kindergarten does appear to improve some aspects of school readiness at kindergarten entry (Barnett et al., 2007b; Gormley et al., 2005), but evidence for longer term effects is mixed and a matter of some debate (Barnett, 1998; Magnuson, Ruhm, & Waldfogel, 2007). Other research suggests that, while preschool programs may improve basic pre-reading skills, their influence on complex language skills, mathematics, self-regulation, and social skills is less clear (Gormley et al., 2005; Jackson, et al., 2007; U.S. Department of Health and Human Services, 2005).
This situation makes the question of curriculum effectiveness important. An effective curriculum is one that tells teachers how to configure prekindergarten instruction to reliably promote school readiness and long-term school success. The most rigorous and comprehensive evaluation of preschool curricula to date is the IES funded PCER project, which launched 14 randomized trials around the country (Preschool Curriculum Evaluation Research Consortium, 2008). Most of the curricula tested had a literacy or general developmental focus (with one focused on math). Overall, 10 of these curricula showed no statistically significant differences from business as usual instruction in the control classrooms on any of the student-level outcomes, and only two showed significant differences on even one outcome measured in kindergarten. In light of these disappointing results, IES terminated the national evaluation of these curricula after the collection of the kindergarten follow-up data.
The Research Question
The project in Tennessee investigated the effects of two contrasting pre-k curricula, relative to practice as usual, on subsequent academic achievement. One curriculum had a strong literacy focus, the other was a less didactic "developmentally appropriate" curriculum that allowed children to have more influence on classroom activities. The purpose of the research was to determine if either of these curricula provided advantages for improving the pre-k, Kindergarten, and/or first grade academic performance of economically disadvantaged children in rural Tennessee.
The Research Design and Results
The study was initiated at the beginning of the 2002-03 school year in seven school districts in six rural middle Tennessee counties. The preschool programs in these schools were funded by the state and enrolled students meeting criteria for economic disadvantage or other risks for school failure. The sample of children (N=549) consisted of the students enrolled in the 36 pre-k classrooms who were age-eligible to attend kindergarten the next year (mean age = 4.4 years) and received parental consent to participate in the study. These students were 70% white, 19% African American, and 4% Hispanic. Slightly more than half were male (53%) and 10% had identified disabilities (IEPs).
Two different pre-k curricula were studied in comparison to practice as usual. The Bright Beginnings (BB) curriculum has a strong literacy focus with a didactic approach to instruction. Creative Curriculum (CC) has a developmental orientation and takes a constructivist approach to learning. Practice as usual in these school districts did not require any specific curriculum but, rather, left that decision to the discretion of the teachers. Teachers assigned to each of the experimental curricula received training the summer before implementation with a follow-up session during the school year. The curricula were implemented during the 2002-03 nine-month school year in full-day pre-k classrooms.
The study design was a randomized field experiment. The participating schools were blocked by district and, within each district, randomly assigned to the BB curriculum, CC curriculum, or practice as usual. All the pre-k classrooms in each school were assigned to the same condition though, in most instances, there was only one classroom per school. Data was collected on classrooms through extensive observations three times during the year, and on students through direct assessments and teacher reports at the beginning and end of pre-k, end of Kindergarten, and end of 1st grade.
Multi-level analysis was used for the outcome data to account for the nesting of students within classrooms and classrooms within schools. School district was also included as a blocking factor along with covariates that included pretest measures for the respective posttests and selected student characteristics (age, sex, minority status, and disability status). Because of the small number of classrooms involved in the curriculum comparisons, the alpha level for statistical significance was set at .10. Only a scattered few of the comparisons between pre-k outcomes for conditions reached statistical significance. It is especially notable that, despite the much greater emphasis on literacy, the children in the BB classrooms did not score significantly higher than the children in the control classrooms on any of the literacy and language related outcomes.
Analyses similar to those for pre-k outcomes were conducted on the analogous outcomes at the end of kindergarten and end of first grade. Again, there were only a few significant differences among all these comparisons, and none showed better outcomes on language or literacy measures for the BB curriculum.
As a follow-up to these early educational analyses, for children reaching the third grade in school year 2006-7, the Tennessee State Department of Education provided scores from the statewide mandated achievement tests, known as TCAP (Tennessee Comprehensive Assessment Program). Four scores were available from this test: Reading/language arts, mathematics, science, and social studies. Of the 549 children in the initial sample, at least one of the posttest or follow-up achievement outcome measures (WJIII, PPVT, or TCAP) was obtained for 531 of them (97%).
Multilevel analysis of the TCAP scores was conducted with students nested within pre-k classes and schools and with school district as a blocking factor. A composite pretest covariate was constructed from the principal components factor score for all the WJIII and PPVT pretest measures and included in the analysis with age, gender, minority status, and disability status as additional covariates. Across all the TCAP subject areas, the children in the BB and CC curriculum conditions performed at least slightly better than the children in the control condition, as shown by the positive effect sizes. On the TCAP Reading/Language Arts achievement test, the BB children scored significantly higher than the control children (effect size=.27). On the TCAP Mathematics test, both the BB and CC children scored significantly higher than the control children (effect sizes = .34 and .41 respectively).
Because these positive effects on TCAP reading and math scores seem anomalous in light of the essentially null effects found on the direct assessment measures taken at the end of pre-k, kindergarten, and first grade, further analysis was done to probe their plausibility as pre-k effects. Note first that the TCAP scores changed the most in the subject areas given the most emphasis in pre-k-early reading and math-and not in the areas given little attention-science and social studies. The strong math findings, however, are unexpected because math is given much less attention than literacy and language in most pre-k classrooms. Further analysis showed that the gains made during pre-k on the WJIII and PPVT literacy/language and math measures were predictive of the TCAP reading and math scores. That is, children who made the most gains during pre-k were those who showed comparatively better TCAP scores with baseline levels, age, gender, minority status, and disability status statistically controlled. The modest curriculum differences observed on the available measures during the early years thus do appear to reflect at least some gains related to later performance on the achievement tests. Further analysis is underway to more fully explore the nature of the relationship between pre-k experiences and outcomes and TCAP performance at third grade and beyond.
Dale Farran, Principal Investigator
Kerry Hofer, Data Manager/Analyst