This year I’ve been single-mindedly focusing on finishing my book about consciousness, but now I’m re-connecting with other projects, including this blog.
Theists & Atheists: Communication & Common Ground is dedicated to discovering both practical and conceptual common ground among theists, atheists, and agnostics. Today’s posting may seem unrelated to this subject, but I’ll explain the connection shortly.
These days epistemology is a hot topic in several fields of study, including medicine and nutrition. Skeptical scholars have pointed out serious weaknesses in the data that guides our dietary and medical choices. John Ioannidis, for example, looked at 49 highly influential research studies, each of which has been cited over 1000 times in the medical literature. Out of these prominent studies, 16% were contradicted by subsequent studies and 16% showed effects that were quite a bit stronger than those of later analyses. In 24% of the cases there was little or no attempt to replicate findings. Bottom line: Just 44% were successfully replicated (Journal of the American Medical Association, July 13, 2005, pp. 218-228).
Other disturbing articles include “It Ain’t Necessarily So: Why Much of the Medical Literature Is Wrong,” by Christopher Labos (http://www.medscape.com/viewarticle/829866) and John Ioannidis’ “Why Most Published Research Findings Are False” (http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fjournal.pmed.0020124). That’s an unsettling title!
I’m not an expert in research design, but several points made in these and other articles seem very troubling. For example, the article on why most findings are false points out that “As research efforts are globalized, it is practically the rule that several research teams, often dozens of them, may probe the same or similar questions. Unfortunately, in some areas, the prevailing mentality until now has been to focus on isolated discoveries by single teams.” Of course, we should be looking at the overall pattern of findings, not isolated reports that may be outliers. He also notes that many studies are motivated by the desire for tenure or a promotion. These objectives create a built-in bias toward finding something positive to write about. Who wants to conduct a study and admit that it produced no substantive findings?
As Labos notes, “There is a way to guard against such spurious findings: replication. Unfortunately, the current structure of academic medicine does not favor the replication of published results …”
The same is true in many other fields. I have read many studies about theological attitudes, and I find that many scholars cite single reports as if these findings are conclusive and replication is unnecessary.
Atheism and agnosticism are especially under-investigated. I recently read that among self-identified atheist and agnostics, 16% were women in 1993, and 43% are today (Cathy Lynn Grossman, Religion News Service, Oct. 24, 2014, http://ncronline.org/news/people/secularism-grows-more-us-christians-turn-churchless). That is a huge change in just two decades. Is it true? False? Basically true but exaggerated? Without replication, it’s hard to know.
Those who do believe in God have been more extensively studied, but many research projects are weakened by the use of an extremely vague definition of “God,” or no definition at all.
So the next time you see a headline with an amazing new factoid about religious attitudes, hum a few bars of that old Gershwin classic: It Ain’t Necessarily So.
Roger Christan Schriner
To subscribe to Theists & Atheists: Communication & Common Ground, click the “Follow” link on the upper left.