A friend recently sent me an article written on the FiveThirtyEight blog (remember, Nate Silver's organization?) supporting the use of value-added modeling (VAM) in order to evaluate teachers across the country. I find it truly interesting that so many researchers continue to use this data despite such vilifying evidence against it.
In this article, the author explains a new method being used that allegedly makes the use of VAM more reliable and valid than ever before.
As a brief background summary: VAM is a method whereby statistical analysis of standardized tests is used in combination with demographic backgrounds of students and teachers in order to calculate how much a teacher should improve a student's ability vs. how much they a actually do. I have found this method very tenuous ever since reading Gary Rubenstein's critique of it back in 2012.
In this article, the author explains a new method being used that allegedly makes the use of VAM more reliable and valid than ever before.
As a brief background summary: VAM is a method whereby statistical analysis of standardized tests is used in combination with demographic backgrounds of students and teachers in order to calculate how much a teacher should improve a student's ability vs. how much they a actually do. I have found this method very tenuous ever since reading Gary Rubenstein's critique of it back in 2012.
One of the best pieces of evidence is the graph to the left. It shows the correlation between teacher ratings in 2008 (x-axis) and 2009 (y-axis) in New York City. If teachers were consistent year-to-year, one would expect the ratings to be fairly linear. As can be seen here, that is patently false. In fact, the correlation is rated as 0.35, something incredibly low for us to think about. If you read further into his report, you will find out even more scary pieces of data and graphs for those who might want to use this method for evaluation.
What the FiveThirtyEight author argues in this article, however, is that there are new method to get around these issues of consistency because there are more randomized trials that can be performed. Even though students are not randomly selected and sent to classrooms across the USA, teachers move somewhat randomly (or, at least, in a way that can be calculated as random) so that the difference in one class and the next can be quantified in connection with the teacher. Unfortunately, because of the myriad studies demonstrating the lack of consistency of tests and students, it is impossible to designate the teacher as the sole reason for student achievement.
The top comment on this article comes from a gentleman named BIll Honig, who accurately states (and cites):
The top comment on this article comes from a gentleman named BIll Honig, who accurately states (and cites):
Studies have shown that under current tests, a teacher who tests rank her at the 50th percentile could be anywhere from the 85th percentile to the 15th A significant number of teachers bounce from top to bottom or vice versa from year to year. (Reseach cited below) A recent report from the Federal Department of Education (of all places) found very high misidentifications even with three years of data per teacher. One fourth of teachers identified as needing special treatment were actually at the mid-range of performance and one-fourth of teachers who were deemed as average were actually in need of special treatment. http://ies.ed.gov/ncee/pubs/20104004/pdf/20104004.pdf
I really hope that those advocating value-added modeling read these studies carefully and spend some time in a classroom to see the effects of these policies.
RSS Feed
