Ed Yong, ‘Psychology’s Replication Crisis Is Running Out of Excuses’, The Atlantic, 19th Nov 2018: https://www.theatlantic.com/science/archive/2018/11/psychologys-replication-crisis-real/576223/

In my last post I forecast a new direction, but November isn’t really a time at work when new directions get going easily, so here is something more reactive.

This is an interesting report on a big new attempt, including hundreds of people, to reproduce some famous psychological experiments. It isn’t scornful in doing so, but it clearly thinks that the failure of 50% of these replications poses a severe challenge to the scientific respectability of psychology, and especially the behavioural sort of social psychology that’s most often at stake. One reason this matters is that the studies being put in doubt are often the kinds of thing that cross over and get mainstream attention. But generally the assumption is that if you can’t get the same results when re-running the experiment under sufficiently similar circumstances, then we didn’t have the knowledge we thought we had, at all.
      I have written about replication problems a few times, such as here and here. I’ve generally found myself feeling sympathy for those who, in good faith, have persuaded the psychological community of their findings, and have then found themselves exposed to various kinds of negativity, implicit or explicit. I’m wondering now whether this is more than just a matter of human sympathy. I think my assessment of the value of findings in cognitive science is not decisively affected by replication problems — I’m less bothered than I could be, or should be. So I thought I’d try to write about why.
      Perhaps exact replication is something that is basically at odds with a focus on human beings as your source of data? Is it at all tenable to see each psychological experiment as an attempt to tell us something about the mind, which uses certain methods to get there, but stands as an argument (like one in the humanities even more than in the social sciences). People can disagree with it, try their own methods, construct new views: there will be periods of convergence and settling, but there will also be times when the debate shifts rapidly and contentiously.
      Perhaps the question is not why certain effects cannot be repeated in another population of individuals, but whether ‘generalizability’ of this sort is a necessary or clinching criterion for the value of a finding? For example, I used a phrase above, ‘sufficiently similar’. This refers to the careful efforts made in psychological experiments to create the right conditions for an experiment, and in replications to recreate those conditions. But there will, of course, be many differences between those populations and their situations, and the priority given to isolating something undeniable and identifiable that they share may not lead to the liveliest or most truthful conversations about how our minds work alone, let alone socially.
      So I am wondering how actively and explicitly to think that the replication crisis reveals to me what I get from psychological research, and may even reveal things about how to understand psychological research. When human beings are the subject, when we’re looking at ourselves, maybe we have to take on non-reproducibility more positively? I would happily say about Hamlet that my arguments will not be convincing to some people according to their principles (many of which I would share), and they will be set aside by future generations, and (this is the point) that’s something that makes them worth having, not something that makes them pointless. Could that possibly also be true of studies focusing on a selected population of people turning up one day to answer questions in a lab?
      I realise, having said these things, that I’m rehearsing entry-level stuff in the philosophy of science. And that it may have nothing whatsoever — least of all reassurance — to offer anyone trying to solve the practical problems of non-reproducibility or to work out where some bits of Psychology go next. Nevertheless, it’s quite important for my interdisciplinary thinking to have gone through some of these steps, to have worked out what I make of scientific conclusions, what the deal-breakers are, and now I need to spend more time thinking about whether I am doing it right.

Update, some hours later: As I have been thinking about this, I have asked myself the question, does it make a difference to me if a result has been successfully replicated, or not? And the answer seems to be yes, which isn’t surprising, but cuts back a little against the grain of the post.

It’s anniversary time again. Five years of the blog!

I wrote a review of the year for 2016-17 here, for 2015-16 here, for 2014-15 here and for 2013-14 here, and here’s another. This one announces a few changes ahead, though.

* I wrote four posts about the early 16th-century poet John Skelton, a maverick talent who seems to me to be revealed interestingly by a cognitively-tuned approach. The trail goes from here to here to here to here.
* I wrote a couple in which I reflected on my experience in trying to design experiments that answered my questions about literature at the same time as answering those of my psychology collaborators: this one and this one.
* The year was bookended by posts about the ways that generalised approaches, and individualised approaches, offered problems and opportunities to psychological research: the former, then the latter.
* I offered a few thoughts about predictive processing and the free energy principle, which offer interesting links with literary questions. Once, twice, thrice. Vera Tobin’s book on surprise, noted here, connects to that general theme.
* And finally, sneaking in under the radar at the time perhaps, was a post about memory, prompted by Jon Simons’s lecture at the Royal Institution. The post is here, and the lecture can be viewed on Youtube: see below. Jon gives a little namecheck to me and to Charles Fernyhough (website here), because we are plotting interdisciplinary work in which literary as well as psychological perspectives on the subjective experience of remembering will be brought together.

And it’s this last topic which is going to become the main focus of the blog for a while. I would like to do something focused that develops and grows over time. It may take a little while for this to get going, but it should be a nice change, for me at least, to have the blog aligning more clearly with my day-to-day priorities.