Home Page Table of Contents
PreviousResearch Paper ToC


3.3. Results and Discussion

Table 1 shows the percentage correct, standard deviation (in parentheses), and N (second line) for the groups on the four tests. Note that because of the high absenteeism that is typical of city schools particularly near the end of the school year, there were a substantial number of students who missed one or both days of the assessment. For each test, a between-subjects ANOVA was run with three levels, Comparison, PUMP+PAT, and Scholars-Comparison. The addition of the Scholars-Control group provides a upper edge comparison for the intervention. The results of these tests are shown in the 5th column of Table 1.

The 6th and final column shows effect sizes in terms of standard deviation units (sigma) of the PUMP+PAT group above the Comparison group. Effect sizes provide a metric for assessing the impact of instructional interventions. The Bloom (1984) result that individual human tutors can bring students 2 sigma above normal classroom instruction sets a standard of comparison for the impact of intelligent tutors. Previous studies have shown cognitive tutors to yield as much as a 1 sigma effect over control conditions (Anderson, Corbett, Koedinger, and Pelletier, 1995; Koedinger & Anderson, 1993).

Table 1. Results of Final Testing of PUMP+PAT, Comparison, and Scholars-Comparison Classes
Comparison PUMP
+PAT 		Scholars-
Comparison 		F value and
significance 		sigma
Iowa Algebra
Aptitude 		.46 (.17)
80 		.52 (.19)
287 		.68 (.17)
34 		F(2,398) = 17.0
p < .0001 		0.3
Math SAT Subset .27 (.14)
44 		.32 (.16)
149 		.42 (.15)
15 		F(2, 205) = 5.1
p < .01 		0.3
Problem Situation
Test 		.22 (.22)
42 		.39 (.33)
127 		.38 (.26)
20 		F(2, 186) = 5.3
p < .01 		0.7
Representations
Test 		.15 (.18)
44 		.37 (.32)
124 		.12 (.16)
18 		F(2, 183) = 13.4
p < .0001		 1.2
        

On the Iowa Algebra Aptitude test, PUMP+PAT scores are significantly higher than the Comparison (p < .05), a 0.3 sigma effect. They are significantly lower than the Scholars-Comparison (p < .01). On the SAT subset, the PUMP+PAT scores are higher than the Comparison scores (0.3 sigma), but there is a lot of variance in this smaller sample and PUMP+PAT is not significantly higher than Comparison (p > .05) nor significantly lower than Scholars-Comparison (p > .05). The largest effects come on the new NCTM-oriented tests. On the Problem Situation test, PUMP+PAT scores are significantly and substantially higher than Comparison (p < .01), a 0.7 sigma effect. They match up with the Scholars-Comparison (p > .05). On the Representations test, PUMP+PAT scores are significantly and substantially higher than Comparison (p < .01), a 1.2 sigma effect, and than the Scholars-Comparison (p < .01).

To summarize, the PUMP+PAT classes scored about 1 sigma better on the NCTM-oriented tests that were the target of the curriculum. Their scores were about 100% better or double those of the Comparison classes. These learning gains appear to occur at no expense to basic skills objectives of standardized tests. In fact, PUMP+PAT classes scored about 15% better on these tests.


Next Page