Even Bias Measures For Standardized Tests Are Biased

If you want a certain job or admission into a top-notch college, the ability to excel on standardized tests is a must. But not only are such tests often biased — one new Indiana University study finds that the measures used to check tests for bias are also flawed.

How did the study's authors arrive at this finding? Researchers used super computing technology to examine tens of billions of scores from tests used for civil service employees and university applicants. According to Herman Aguinis, director of the Institute for Global Organizational Effectiveness at IU’s Kelley School, the study demonstrates “that bias can be present but not be detected by even the top experts in the field.” In other words, though exams may be biased, even experts told to look for such bias have trouble detecting it.

The testing industry is a multibillion-dollar business, one that corporations who crank standardized tests out won't easily walk away from. Nor will the companies and institutions that rely on them. But we all know that no test — even an unbiased one — can ensure that someone will be a perfect fit for or succeed with an organization. People are too complex.

I’m not advocating that companies or colleges do away with standardized tests altogether. Studies like Indiana University's, though, show that it’s high time institutions stopped emphasizing test scores so much. Too often, America's wealthy are able to pay for test-taking training. Meanwhile, bias embedded in standardized exams continue to leave people of color behind.

As Aguinis puts it, “While the academic community has demonstrated repeatedly that different racial or ethnic groups’ cultural frames of reference and identity may play a role in affecting test scores, we have not used that knowledge to sufficiently advance testing processes."

Aguinis should know. In 2009, he co-authored an amicus brief in the Supreme Court case Ricci v. DeStefano. In that case, firefighter Frank Ricci sued the city of New Haven, Conn., for failing to promote him to lieutenant despite the fact that he’d come in sixth out of 77 firefighters on the qualifying exam. The fire department had thrown out all test results because only one firefighter of color passed the exam, causing the city of New Haven to fear that minority firemen would sue for discrimination if only whites got promotions. While it’s easy to sympathize with Ricci — who did, in fact, win the case — look at all the resources he used to ace the exam. A dyslexic, he paid someone more than $1,000 to read textbooks onto audiotapes, took practice tests, worked with a study group and engaged in mock interviews, according to the New York Times. Unfortunately, many minority job and university applicants don’t have the luxury of investing more than a $1,000 to ensure that they pass a test. When we're talking about so-called standardized tests, the ability of upper-class whites to invest in their scores may tip the scales in their favor as much as test bias does.

Now that the results from the IU study are in, test bias researchers are predicting that organizations will review their existing exams and create new ones. I hope that's the case, but either way, we should go one step further — and stop taking test scores so seriously in the hiring and admissions process in the first place.

Photo Credit: Dave Bleasdale