Executive Director of Secondary Education David A. Volrath gave an update on high school reform at a recent Board of Education meeting and it was a whopper in more ways than one. The sixty-page, chart-laden report was handed to the Board just prior to the meeting rather than the Friday before as is standard practice. Unusual, especially given the size, the import and the level of public interest in the reforms also known as CSSRP.
Hopefully the Board will put this report back on a future agenda, because it was hard to dissect all the data on the fly and a closer look has revealed some misleading first impressions. Kind of like watching a magic show; you want to believe the illusion, so it’s a let down when you notice the trick.
One example is the analysis of Advanced Placement (AP) Exams. High school students who choose AP classes can also take an end-of-course AP Exam, a national test administered by the College Board. Looking at the report’s soaring bar charts, you might conclude AP results in Harford County are improving rather dramatically. It’s quite a show, but there’s a sleight of hand.
Look closely and you’ll see that all of the charts tell you the same thing. That is, more AP exams are being administered. That’s especially good if it means more students are participating in AP work. But high-performing students often take more than one AP class, so the increase may simply be double- and triple- counting a small group who are beefing up their resumes in light of stiffer competition for college slots. More than likely the increase is due to some combination of the two, but it’s a distinction that should have been made.
Even more than the increase in the number of exams, what matters is student performance. A score of 3 out of 5 on the AP Exam is the usual cutoff to earn college credit. But the only charts devoted to test scores measure the number of “exams at 3 or higher” within a rising number of exams given, giving the distinct impression of improving achievement.
But the measurement is misdirection. To demonstrate student performance, you’d need to show how many exams were scored 3 or better as a percentage of the total number of exams given. With all the bar charts flying around, you’d think at least one of them could have been devoted to that display.
The information is certainly chart-worthy. Students who take the AP Exam plunk down $86 for the privilege and hope to get college credit or at least impress college admissions officers. A poor AP score has the opposite effect, so students who expect a score below 3 tend to skip the exam and save the cash.
So, is performance on AP exams rising or falling in Harford County Public Schools? The chart below would have answered that question if anyone had bothered to put it together: The blue bars are the years prior to CSSRP; yellow is after:
Turns out the percent of exams with a score of 3 or better is basically flat. It hovered between 63% and 65% for the past 5 years and it has gone down slightly in the two years since CSSRP was implemented.
How could the Board know any of that from the presentation they were given? They’d have better luck getting Houdini to share his secrets.
The full CSSRP Update: