* Romy Lorenz, Adam Hampshire, and Robert Leech, ‘Neuroadaptive Bayesian Optimization and Hypothesis Testing’, Trends in Cognitive Sciences, 20 (2017), 155-67.
* Siobhán Harty, Francesco Sella, and Roi Cohen Kadosh, ‘Mind the Brain: The Mediating and Moderating Role of Neurophysiology’, Trends in Cognitive Sciences, 20 (2017), 2-4.

This is a second post in a row about issues in experimental design. Even though I rarely design experiments myself, it’s important to understand what sort of compromises are accepted, and what sort of changes are underway. These are two essays about how the methodology of cognitive science could improve, not least to address the ‘reproducibility crisis’. Various solutions have been suggested to deal with the current situation, in which a failure to replicate a number of well-known experimental findings has cast general doubt on methods and conclusions. There is a worry that various biases in the system — for example towards the publication of successful rather than unsuccessful experiments — that obscure the true shape of the discipline. One suggestion is ‘preregistration’, a move towards open-ness in the publication of data and methods so that everything is open to scrutiny.
      Lorenz et al. are proposing a different solution, which is to take advantage of two technological advances. The first is the development of real-time analysis of brain imaging, which means that experimenters can broaden their hypotheses and designs, and can observe neural functions in a more flexible way. The other advance is in the use of ‘active sampling approaches’, where the selection of samples is made by the computer, which progressively ‘learns’ according to an algorithm defining how to refine and direct the search. More ground can be covered, and the criteria for the choices involved (including the algorithm itself) can be made public.
      They give an example which, I think, helps get this across. Some cognitive tasks ‘recruit a combination of spatially overlapping yet distinct frontoparietal networks’, and ‘understanding their exact functional role remains a challenge’. It seems possible that the method described, involving more flexible testing of brain activity and a search mechanism based on emerging facts rather than on the scientist’s expectations, could help uncover what links to what.
      It’s not the easiest read — there’s a description of ‘experiment space’ in 2D and 3D that (I’ll be honest) is presumably metaphorical but how, and to what extent, I don’t know. And I suppose this tends to change the point at which the scientist’s choices impinge, rather than eliminating them (as if that were possible). However, people are worrying about cognitive science at a pretty basic level so maybe there are ways in which machine learning can help.

*

Harty et al. look at experiments on behaviour and argue that they aren’t sufficiently designed to account for ‘the critical antecedent of behavior, the brain’. It’s time, they say, to take into account how neurophysiology affects ‘mediating or moderating’ roles. Some variables are taken into account regularly: gender, age, socioeconomic status, level of education, and others. But brains are very complex, and there may be pertinent physical reasons why this individual, and this one, but not that one or that one, respond in a certain way to a behavioural stimulus: ‘We should endeavor to design our studies and analyze our data in ways that can address questions about why, how, and for whom the experimental manipulation is effective’.
      They hope that this will improve ‘prospects for reproducibility’: the more accurately the experiment is designed, the more parameters that are covered, the better. Also there is the possibility that this could lead to more ‘personalized cognitive interventions’ — not just experiments, then, but better applications in general. I have sometimes find myself worrying that distinctions taken by experimenters as stable (e.g. the ‘gender, age, socioeconomic status, level of education’ mentioned above) are a lot more subtle than they are taken to be, but there’s got to be an interplay between categories (can’t do without them) and refinements.