Ed Yong, ‘Psychology’s Replication Crisis Is Running Out of Excuses’, The Atlantic, 19th Nov 2018: https://www.theatlantic.com/science/archive/2018/11/psychologys-replication-crisis-real/576223/

In my last post I forecast a new direction, but November isn’t really a time at work when new directions get going easily, so here is something more reactive.

This is an interesting report on a big new attempt, including hundreds of people, to reproduce some famous psychological experiments. It isn’t scornful in doing so, but it clearly thinks that the failure of 50% of these replications poses a severe challenge to the scientific respectability of psychology, and especially the behavioural sort of social psychology that’s most often at stake. One reason this matters is that the studies being put in doubt are often the kinds of thing that cross over and get mainstream attention. But generally the assumption is that if you can’t get the same results when re-running the experiment under sufficiently similar circumstances, then we didn’t have the knowledge we thought we had, at all.
      I have written about replication problems a few times, such as here and here. I’ve generally found myself feeling sympathy for those who, in good faith, have persuaded the psychological community of their findings, and have then found themselves exposed to various kinds of negativity, implicit or explicit. I’m wondering now whether this is more than just a matter of human sympathy. I think my assessment of the value of findings in cognitive science is not decisively affected by replication problems — I’m less bothered than I could be, or should be. So I thought I’d try to write about why.
      Perhaps exact replication is something that is basically at odds with a focus on human beings as your source of data? Is it at all tenable to see each psychological experiment as an attempt to tell us something about the mind, which uses certain methods to get there, but stands as an argument (like one in the humanities even more than in the social sciences). People can disagree with it, try their own methods, construct new views: there will be periods of convergence and settling, but there will also be times when the debate shifts rapidly and contentiously.
      Perhaps the question is not why certain effects cannot be repeated in another population of individuals, but whether ‘generalizability’ of this sort is a necessary or clinching criterion for the value of a finding? For example, I used a phrase above, ‘sufficiently similar’. This refers to the careful efforts made in psychological experiments to create the right conditions for an experiment, and in replications to recreate those conditions. But there will, of course, be many differences between those populations and their situations, and the priority given to isolating something undeniable and identifiable that they share may not lead to the liveliest or most truthful conversations about how our minds work alone, let alone socially.
      So I am wondering how actively and explicitly to think that the replication crisis reveals to me what I get from psychological research, and may even reveal things about how to understand psychological research. When human beings are the subject, when we’re looking at ourselves, maybe we have to take on non-reproducibility more positively? I would happily say about Hamlet that my arguments will not be convincing to some people according to their principles (many of which I would share), and they will be set aside by future generations, and (this is the point) that’s something that makes them worth having, not something that makes them pointless. Could that possibly also be true of studies focusing on a selected population of people turning up one day to answer questions in a lab?
      I realise, having said these things, that I’m rehearsing entry-level stuff in the philosophy of science. And that it may have nothing whatsoever — least of all reassurance — to offer anyone trying to solve the practical problems of non-reproducibility or to work out where some bits of Psychology go next. Nevertheless, it’s quite important for my interdisciplinary thinking to have gone through some of these steps, to have worked out what I make of scientific conclusions, what the deal-breakers are, and now I need to spend more time thinking about whether I am doing it right.

Update, some hours later: As I have been thinking about this, I have asked myself the question, does it make a difference to me if a result has been successfully replicated, or not? And the answer seems to be yes, which isn’t surprising, but cuts back a little against the grain of the post.