Evaluating Research Papers, Studies, and Reports

"Autism research" spans the full range of academic subjects. It includes psychology and cognition, biology and health, educational theory, language and communication, the social sciences from service delivery to attitudes and belifs, even robotics and the engineering of assistive technology, all forms of multidisciplinary studies, and many, many other fields. The results of research is then published in scientific journals, which have varying degrees of quality.

"Science reporting," the popular translation of scientific research by the press to the public, can frequently misrepresent the actual content of scientific studies or academic papers. See my favorite example of "the daydreaming study" which was not about daydreaming and never mentioned the term. Problems with science reporting are so endemic that dealing with the press shows up (along with tips for scientists to cope with it) in places like Winning the Games Scientists Play by Carl J. Sindermann,

Always keep in mind that the principal job of a reporter is to develop a story that will sell newspapers or attract and retain television viewers or radio listeners. To this end, the reporter will not usually deliberately misrepresent what you have said, but may emphasize what to you are minor points, may weave your comments in with those of others, may use part of your statement out of context, or may eliminate many or all of the qualifications that you have placed on your conclusions. Then the headline writer may create an erroneous impression about the already distorted content, leading you to complete frustration and a pledge never again to give even the correct time to a reporter.

While few (any?) may be able to read and understand all of the primary sources that science reporting on autism comes from, including being able to access the articles in the first place as many journals are subscription only, I do think there is value in skimming primary sources when possible. Even when the highly technical subject matter is incomprehensible, it's usually at least possible to make a comparison between what the press says versus what the study says to know how much "salt" to take the popular account with.

One of my most favorite "cheat sheets" to quickly evaluating a scientific research study without reading it in depth is Efficient Reading of Papers in Science and Technology (PDF) (Michael J. Hason / Dylan J. McNamee). The pamphlet was written with more engineering-type-science in mind, but the tips and advice it gives, including how to assess the relative validity and quality of the study, applies well to evaluating the quality of a research article in any field.