This study developed from a discussion with Dr. Nancy Holmes, Associate Superintendent for Curriculum, Harnett County, in the beginning of the academic year 2001 – 2002. Dr. Holmes identified several reading comprehension skills that have had a negative impact on student performance in Harnett County.

TCR/NewsBank, at the direction of Ann McCormick, Director of the Media and Technology Department, designed a study addressing goals for reading in the district. The plan called for using TCR Connections for Kids and Connections to assess three specific reading comprehension skills.

During the design phase of the study, "No Child Left Behind" (NCLB) guidelines for research-based programs and adequate yearly progress for student performance and teacher quality were released by the Federal Department of Education. Ann McCormick and TCR recognized that this study would accomplish three major objectives: (1) meet Federal guidelines under NCLB for a research-based program; (2) promote adequate yearly progress for students; and, (3) provide administrative support for improving teacher quality.

The project began in earnest in November 2002 with a planning meeting in Harnett County. A decision was made to use the January through March marking period to gather data on student performance. The design and methodology for the research employs a pre- post- individual student performance model in Lillington and Overhills. The target groups differed at the two schools: Lillington included 4^th and 5^th grade students. Overhills identified Level I & II At-Risk students in their after-school program.

Both the treatment and control groups at Lillington used the feature article and self-check components of Connections for Kids for assessment. Lead teachers Kristina Magyar and Jennifer Blaylock received introductory training from TCR/NewsBank. Under the guidance of Principal Linda Stewart and district Technology Director, Tom Frye, Kristina and Jennifer introduced Connections for Kids to all 4^th and 5^th grade children. The treatment group in the study completed the entire activity; the control group read the feature article in Connection for Kids, but did not complete the assessment or other elements of the activity.

At Overhills the treatment and control groups are all grade 7 students. Each student used the feature article and self-check components of Connections for Kids. However, the students at Overhills urged their teachers, Ms. Clark, Ms. Weathers and Ms. Daughtry to move them into the more challenging Connections reading. Therefore, the post-test reading was drawn from BOTH Connections for Kids and Connections at Overhills. The student population consisted of Level I & II students who were invited to participate in the after-school tutorial program. The Treatment group is comprised of students who attended the program; the Control group includes any student who elected NOT to participate in the after-school program. Therefore, the control group for the Overhills study is self-selected. All 7^th grade students were given the pre- and post-tests to mask the participants’ treatment or control identity.

A "treatment" group is defined as the group identified for measurement of the program on reading comprehension and reading level. Therefore, a "control" group has the same/similar group characteristics, but does not receive a "treatment" --- in this case, the use of the program.

Both Lillington and Overhills teachers met the study criteria of a minimum 180 minutes of classroom use of Connections for Kids reading and activity components during the marking period. In some cases teachers and students may have used the program more often. This amounts to a minimum of 20 minutes of program use each week during the marking period. In addition, reading from the program was assigned outside of class at Lillington. Overhills students asked their teachers to move them from the Connections for Kids to the Connections reading. The children felt that they had mastered the reading in Connections for Kids mid-way through the marking period.

Each student in the study was assigned a random number to ensure anonymity. Then, again randomly, they were distributed to either the treatment or control groups. Grading, analysis of information related to student performance and analysis of student –related data was anonymous. The recording of all information when not in the hands of either Ann McCormick or Tom Frye maintained anonymity.

Ann McCormick’s support for this study was essential. Members of the Media and Technology department, especially Kay Wheeler and Tom Frye, provided extensive support.

Linda Stewart, Principal at Lillington Elementary School arranged for training, adjusted schedules and provided general support for implementation plans at Lillington. Linda gathered the relevant curriculum content information used to introduce the program to Kristina and Jennifer.

Jennifer Daughtry, Alfreda Weathers and Denise Clarke provided an invaluable team approach to identifying curricular areas to measure the impact of the after-school program on reading comprehension and reading level.

Dr. Holmes’ insightful analysis of district needs and suggestions for the reading comprehension variables enabled the research team to develop a design to measure the appropriate reading skills for district students.

An independent researcher, Dr. Wayne Foster, analyzed all data. Dr. Foster is a North Carolina educator currently responsible for the program for Exceptional Children in Alamance-Burlington School District. Dr. Foster has been with Alamance for the past nine years.

Scale-scored data without student names was provided to Dr. Foster who analyzed the data and tested for significance.

A total of 153 fourth and fifth grade students participated in this project. Students were assigned to a treatment or control group at each grade level. Table 1 shows the distribution of students across grade and group. A chi square analysis revealed no significant differences between the control and treatment groups with respect to race or gender in either the fourth or fifth grade. (In this project race was classified as either white or non-white.)

Table 1

Distribution of participants by grade and group (treatment and control).

The chi square analysis for age revealed significant differences in the fourth grade (χ² = 5.85 ₍₁₎, p < .05). The fourth grade control group had significantly more 11-year-old students (as compared to 10-year-olds). In the control group 59% of the students were 11-years-old while in the treatment group 32% of the sample were 11-years-old. In the fifth grade the age distribution was equivalent across the two groups. The chi square statistic was not significant.

Testing was performed prior to participation in the project and at the conclusion. The scores described briefly below were utilized in this data analysis. Means and standard deviations are shown in table 2.

Star reading level (RL). The Star Reading Level is a measure of each student’s reading grade level. In addition to reading level, the Star testing produced grade equivalent scores, percentile ranks, and normal curve equivalents for each student. Reading level was used in this study but any of these above scores could have been used. The bivariate correlations between all the scores were all in excess of .92. Basically, all these scores were generated from one raw score. The outcomes reported in this analysis were essentially the same regardless of which of these scores were used.

Recognizing supporting details (RSD). A single pre- and post-test question was used to determine students’ ability to recognize supporting details in the materials read. This item was scored as ‘0’ for incorrect and ‘1’ for correct.

Making inferences (MI). As with RSD, a single question assessed students’ ability to make inferences from the materials they read. The item was scored as ‘0’ for incorrect and ‘1’ for correct. It was measured at pre- and post-project participation.

Recognizing author’s purpose (RAP). Students were assessed with a single question on their ability to recognize the author’s purpose in materials presented to them to read. A pre- and post-project score was obtained and, as with RSD and MI, it was scored as ‘0’ for incorrect and ‘1’ for correct.

Composite score (CS). The composite score is the sum of RSD, MI, and RAP. It could range from 0 to 3.

Table 2

Means and standard deviations (in parentheses) for the measures used in this analysis.

Note. RSD = recognizing supporting details. MI = making inferences. RAP = recognizing author’s purpose. CS = composite score.

Fourth grade. Paired sample t-tests were used to determine if the post-project scores differed significantly from the pre-project scores. All analyses were performed with a type I error rate set at α = .05. Both the treatment and control group showed significant gains in the measured RL. (Control group: t = 3.61₍₃₁₎, p < .05, Treatment group: t = 3.45₍₄₀₎, p < .05.) It should be recalled, however, that the 4^th grade control group had significantly older students than the treatment group. When age was controlled in the analysis the control group’s RL t-test was no longer significant. Figure 1 shows the RL pre- and post-project scores. In sum, when age is controlled in the analysis for 4^th grade, only the treatment group exhibits a significant difference between the pre- and post-project scores. RL was the only measure showing a significant difference.

Figure 1 Pre- and post-project reading level scores for the 4^th grade.

Note. In this analysis age was controlled.

Fifth grade.

Paired sample t-tests were again used to determine if the post-project scores differed significantly from the pre-project scores for the control and treatment groups. Neither group exhibited a significant gain in RL. Both groups showed significant improvements in RAP. (Control group: t = 2.69₍₃₃₎, p < .05, Treatment group: t = 1.97₍₃₉₎, p < .05.) Figure 2 shows the pre- and post-project RAP scores for the 5^th grade participants.

Figure 2

Pre- and post-project RAP scores for the 5^th grade participants.

The 4^th grade control group had significantly more 11-year-old students than the treatment group. Without controlling for this age difference, both groups showed significant gains in reading level (RL). However, when age was controlled, the control group no longer showed significant gains. The treatment group in the 4^th grade made meaningful gains in reading level between the pre- and post-project period.

No significant gains in reading level were observed for the 5^th grade samples (treatment and control). Both groups exhibited significant gains in the ability to recognize the author’s purpose from read material (RAP).

This analysis is based on 35 students who had both pre- and post-test scores. Data does exist for 77 students but for over half either pre- or post-test scores were unavailable.

Pre- and Post-tests. The students participating in this project were given pre- and post-testing. Individual items in these tests related to (MIDC) making inferences/drawing conclusions, (RSD) recognizing supporting details, (IMI) identifying main idea, (RAP) recognizing author’s purpose, and general comprehension (GC). Two levels of difficulty are represented in the testing (Levels I and II). RSD and MIDC (Level I) were assessed at both the pre- and post-testing period using single questions. GC was assessed during the pre-test with three questions and for the post-test GC with one question. RSD, MIDC, and RAP (Level II) were also assessed during the post-test with a single question for each. A post-test average of the Level II questions was also utilized in the analyses. No Level II pre-test questions were presented.

Table 1 shows the assessments used in this project. Table 2 shows the means and standard deviations for these scores. Of the 77 students who participated in this project only 35 had adequate numbers of scores to be included in this analysis. The means and standard deviations shown in table 2 are for these 35 students.

Table 1. Pre- and post-testing scores.

Table 2. Means and Standard Deviations for the Pre- and Post-Test Variables.

Note. IMI = Identifying main idea. MIDC = Making inferences/drawing conclusions. RSD = Recognizing supporting detail. RAP = Recognizing author’s purpose. GC = General comprehension.

Group level analysis. Figure 1 shows the pre- and post-test comparison of mean scores for RSD, MIDC and GC (Level I) for the 35 students who had both sets of scores. One sample t-tests were performed to determine if the pre-test scores for RSD, MIDC and GC significantly differed from the post-test values. (All analyses were performed with the Type I error set at α = .05.) RSD and MIDC were significantly different (RSD: t₍₃₄₎ = 8.11, p < .05, MIDC: t₍₃₄₎ = .409, p < .05.) The GC comparison was not significantly different.

Figure 1. Pre- and Post-Test Mean Scores for Recognizing Supporting Detail (RSD), Making Inferences/Drawing Conclusion (MIDC) and General Comprehension (GC).

Note. Pre- and post-test scores represent a similar level of difficulty (Level I). Pre- and post-RSD and MIDC were significantly different (α = .05) but GC was not.

Individual level analysis. The performance of individual students was assessed using K-means cluster analysis. K-means cluster analysis is a statistical method for determining whether scores tended to differentiate into separate groups. Using this method the 35 students could be assigned to a performance group based on how they scored on the three measures used in this analysis.

Three scores were used to differentiate between clusters of students. The RSD and MIDC (Level I) gains were used because the pre- and post-test scores for the entire set of students were found to differ significantly. Since the GC measure did not differ from pre- to post-test, this score was not used. The RSD gain score for an individual student could be –1, 0, or +1. MIDC gain was the second score utilized. Like the RSD gain, this score could be a –1, 0, or +1. Finally, the higher difficulty (Level II) post-test score was averaged and used as the third variable for the analysis. The (Level II) post-test RSD, MIDC, and RAP scores for each student were averaged. This averaged score ranged from 0 to 1.

Two clusters of students were identified. Cluster 1 is the lower performing cluster of students. This group made no or negative gains on the RSD and MIDC variables. This group also averaged 0 on the Level II post-testing score. Cluster 2 was the higher performing group of students. This group had positive gains on two or more of the measures. Table 2 identifies the cluster for each of the 35 students included in the analysis.

Twenty students fell into cluster 1 and fifteen students were identified as being in cluster 2. Due to some missing scores, several students were placed in a cluster based on their performance on two of the three variables.

Table 3. Cluster analysis results for the 35 students included in the analysis.

*Note: # 11111 above was assigned to a student for whom no ID # was provided in the data recorded.

TCR/NewsBank’s instructional strategy is to present an engaging "feature" article and, through additional reading, related websites, graphic organizers, and rubrics, to enable students and teachers to explore and extend the students’ understanding of the material. The interactive elements and Self-Checks in Connections for Kids and KidsPage allow students to evaluate their interpretation of the reading. The architecture of Connections for Kids and KidsPage supports differentiated instruction to promote reading comprehension. These programs encourage teachers and students to explore sources within a highly selective database of quality children’s publications like Cobblestone, Super Science, DynaMath and Footsteps.

In the course of a single nine-week marking period, teachers, trained by TCR/NewsBank curriculum specialists, but with limited prior experience with TCR Connections for Kids and KidsPage, were able to raise 4^th grade student reading scores significantly. Interesting to note, Lillington teachers, Kristina Magyar and Jennifer Blaylock observed that some of the 4^th graders in the "control group" were surreptitiously using the parts of the program reserved for the "treatment group." While the fact that some of the control group subjects were "self-treating" undoubtedly mitigated the degree of improvement measured between the "treatment" and "control" groups in the study, the teachers were delighted with the "self-treaters" initiative and obvious enjoyment of reading their own selections rather than the prescribed material.

At-risk 7^th graders at Overhills Middle School asked to read material from the upper grade program, Connections. They told their teachers that the material in the upper grade program was "more interesting" and provided them with more of a challenge --- the lower grade material was "too easy".

TCR also determined that remote access as well as in-school program use increased dramatically in Harnett County during the timeframe of this study. There was a 52% increase in usage in January 2003 compared to January 2002; March 2003 usage increased by 49% over March 2002.

These unexpected ancillary outcomes in the Harnett County study demonstrate the potential "halo effect" of the classroom research in compliance with NCLB guidelines. The enthusiasm and special attention of teachers and administrators, as well as the "treatment" vs. "control" group structure stimulated the natural curiosity, spirit of adventure and competition in students who previously had not applied these motivations to reading.