After class on March 2, high school students in Albuquerque protested new assessment tests. Credit Roberto E. Rosales/The Albuquerque Journal, via Associated Press
In 2004, the Chinese government decided there were too many accidental deaths. China’s safety record, it decreed, should be brought in line with those of other middle-income countries. The State Council set a target: a decline in accidental deaths of 2.5 percent per year.
Provincial authorities kicked into gear. Eventually, 20 out of a total of 31 provinces adopted “no safety, no promotion†policies, hitching bureaucrats’ fate to whether they met the death ceiling. The results rolled in: by 2012 recorded accidental deaths had almost halved.
It wasn’t, however, all about increased safety. For instance, officials could reduce traffic deaths by keeping victims of severe accidents alive for eight days. They counted as accidental deaths only if the victims died within seven.
Related Coverage
As Common Core Testing Is Ushered In, Parents and Students Opt OutMARCH 1, 2015
In a study of China’s declining deadly accidents, Raymond Fisman of Columbia University and Yongxiang Wang of the University of Southern California concluded that “manipulation played a dominant role.†Bureaucrats — no surprise — cheated.
Photo
Third graders at Public School 169 in Queens, preparing to take a standardized test. Credit Uli Seit for The New York Times
This is hardly unusual. It is certainly not exclusive to China. These days, in fact, it has acquired particular importance in the debate over how to improve American education.
The question is, what will happen when teachers are systematically rewarded, or punished, based to some extent on standardized tests? If we really want our children to learn more, the design of any system must be carefully thought through, to avoid sending incentives astray.
“When you put a lot of weight on one measure, people will try to do well on that measure,†Jonah Rockoff of Columbia said. “Some things they do will be good, in line with the objectives. Others will amount to cheating or gaming the system.â€
The phenomenon is best known as Goodhart’s Law, after the British economist Charles Goodhart. Luis Garicano at the London School of Economics calls it the Heisenberg Principle of incentive design, after the defining uncertainty of quantum physics: A performance metric is only useful as a performance metric as long as it isn’t used as a performance metric.
It shows up all over the place. Some hospitals in the United States, for example, will often do whatever it takes to keep patients alive at least 31 days after an operation, to beat Medicare’s 30-day survival yardstick. Last year, Chicago magazine uncovered how the Chicago Police Department achieved declining crime rates, simply by reclassifying incidents as noncriminal.
“We don’t know how big a deal this is,†said Jesse Rothstein, a professor at the University of California, Berkeley, who has criticized evaluation metrics based on test scores. “It is one of the main concerns.â€
American education has embarked upon a nationwide experiment in incentive design. Prodded by the Education Department, most states have set up evaluation systems for teachers built on the gains of their students on standardized tests, alongside more traditional criteria like evaluations from principals.
Fourteen more states are expected to have fully developed systems this academic year, according to the National Council on Teacher Quality — an advocacy group that supports rigorous assessments. All but six states are expected to have one by the 2016-17 school year.
The assessments are backed by sophisticated research. An important study by Professor Rockoff and two Harvard professors, Raj Chetty and John Friedman, found that teachers who improved students’ scores, termed high value-added teachers, raised the students’ chances of going to college as well as their salaries later in life.
But teachers — and parents — are up in arms. In New York, the teachers’ union strongly opposes Gov. Andrew Cuomo’s proposal to increase the weight of test scores to 50 percent of a teacher’s evaluation. The governor is being hammered over the issue in opinion polls.
“People who claim to be market-based reformers want to sell the theory that there is a direct correlation between test scores, the effort of teachers and the success of children,†said Randi Weingarten, who heads the American Federation of Teachers. “It just ignores everything else that goes into learning.â€
Critics have questioned the Harvard scholars’ findings. Teachers argue there is no way they could isolate the impact of teaching itself from other factors affecting children’s learning, particularly such things as the family background of the students, the impact of poverty, racial segregation, even class size.
Photo
Children join a rally opposed to a proposal in New York to increase the importance of student test results on teacher evaluations. Credit Jake Naughton for The New York Times
Professor Rothstein at Berkeley suggested that sorting plays a big role in their results: better-ranked teachers got better students. Other studies found teachers’ scores jump around a lot from year to year, putting their value into question. Professors Rockoff, Chetty and Friedman have defended their results.
In this heated debate, however, it is important not to lose sight of Goodhart’s Law. Most of these studies measured the impact of test scores when tests carried little weight for teachers’ future careers. But what happens when tests determine whether a teacher gets a bonus or keeps his or her job?
From Atlanta to El Paso, school officials have been accused of cheating to improve their standing on test scores.
Fraud is not the only concern. In one study, schools forced to improve grades by the No Child Left Behind law were found to have focused on helping children who were at the cusp of proficiency. They had no incentive to address those comfortably above the cut or those with little hope of gaining enough in the short term.
A survey of teachers at a school district in the Southwest that awarded bonuses based on test scores found that many tried to avoid both An earlier version of this article misstated the number of states expected to have fully developed teacher evaluation systems based on student gains on standardized tests, according to the National Council on Teacher Quality. It is 14 more this academic year, not 14 in all.
Loading…
Grading Teachers by the Test - New York Times
No comments:
Post a Comment