To make replication studies more useful, researchers must make more of them, funders must encourage them and journals must publish them.No scientist wants to be the first to try to replicate another’s promising study: much better to know what happened when others tried it. Long before replication or reproducibility became major talking points, scientists had strategies to get the word out. Gossip was one. Researchers would compare notes at conferences, and a patchy network would be warned about whether a study was worth building on. Or a vague comment might be buried in a related publication. Tell-tale sentences would start "In our hands", "It is unclear why our results differed …" or "Interestingly, our results did not …".
Next month Las Vegas will host the Final Event of the DARPA Cyber grand Challenge as an all-computer cyber-defence Capture the Flag tournament. From an initial field of over 100 applicant seven teams will compete for the $3.5 million prize pool. Reproducibility is a key aspect of a sound scientific design. While perfect system state replay is impossible without a full system event recorder, DECREE has been designed to allow high determinism and reproducibility given a record of software and inputs. This reproducibility property has been built into DECREE from kernel modifications up through the entire platform stack.
When functional magnetic resonance imaging (fMRI) was introduced in the late 1990s, it drew raves for its ability to show brain activity—and concerns that it might be the modern equivalent of phrenology. Now, that debate could spring to life again with revelations that the popular imaging technology could have been flawed for years. As Kate Lunau writes for Motherboard, new research suggests that software used to analyze fMRI results could invalidate up to 40,000 brain activity studies.
A whole pile of "this is how your brain looks like" MRI-based science has been invalidated because someone finally got around to checking the data. The problem is simple: to get from a high-resolution magnetic resonance imaging scan of the brain to a scientific conclusion, the brain is divided into tiny "voxels." Software, rather than humans, then scans the voxels looking for clusters. In this paper at PNAS, they write: "the most common software packages for fMRI analysis (SPM, FSL, AFNI) can result in false-positive rates of up to 70%. These results question the validity of some 40,000 fMRI studies and may have a large impact on the interpretation of neuroimaging results."
Nearly one-third of junior scientists spend no time validating antibodies, even though accurate results depend on these reagents working as expected, according to the results of a survey reported today in BioTechniques. "This is quite alarming," says Matthias Uhlén, a protein researcher at the Royal Institute of Technology in Stockholm who heads an international working group on antibody validation, but who was not directly involved in the survey.
This week in science, academia and publishing for reproducibility.