Many scientists worry over the reproducibility of wet-lab experiments, but data scientist Victoria Stodden's focus is on how to validate computational research: analyses that can involve thousands of lines of code and complex data sets. Beginning this month, Stodden — who works at the University of Illinois at Urbana-Champaign — becomes one of three ‘reproducibility editors’ appointed to look over code and data sets submitted by authors to the Applications and Case Studies (ACS) section of the Journal of the American Statistical Association (JASA). Other journals including Nature have established guidelines for accommodating data requests after publication, but they rarely consider the availability of code and data during the review of a manuscript. JASA ACS will now insist that — with a few exceptions for privacy — authors submit this information as a condition of publication.
To make replication studies more useful, researchers must make more of them, funders must encourage them and journals must publish them.No scientist wants to be the first to try to replicate another’s promising study: much better to know what happened when others tried it. Long before replication or reproducibility became major talking points, scientists had strategies to get the word out. Gossip was one. Researchers would compare notes at conferences, and a patchy network would be warned about whether a study was worth building on. Or a vague comment might be buried in a related publication. Tell-tale sentences would start "In our hands", "It is unclear why our results differed …" or "Interestingly, our results did not …".
Next month Las Vegas will host the Final Event of the DARPA Cyber grand Challenge as an all-computer cyber-defence Capture the Flag tournament. From an initial field of over 100 applicant seven teams will compete for the $3.5 million prize pool. Reproducibility is a key aspect of a sound scientific design. While perfect system state replay is impossible without a full system event recorder, DECREE has been designed to allow high determinism and reproducibility given a record of software and inputs. This reproducibility property has been built into DECREE from kernel modifications up through the entire platform stack.
When functional magnetic resonance imaging (fMRI) was introduced in the late 1990s, it drew raves for its ability to show brain activity—and concerns that it might be the modern equivalent of phrenology. Now, that debate could spring to life again with revelations that the popular imaging technology could have been flawed for years. As Kate Lunau writes for Motherboard, new research suggests that software used to analyze fMRI results could invalidate up to 40,000 brain activity studies.
A whole pile of "this is how your brain looks like" MRI-based science has been invalidated because someone finally got around to checking the data. The problem is simple: to get from a high-resolution magnetic resonance imaging scan of the brain to a scientific conclusion, the brain is divided into tiny "voxels." Software, rather than humans, then scans the voxels looking for clusters. In this paper at PNAS, they write: "the most common software packages for fMRI analysis (SPM, FSL, AFNI) can result in false-positive rates of up to 70%. These results question the validity of some 40,000 fMRI studies and may have a large impact on the interpretation of neuroimaging results."
Nearly one-third of junior scientists spend no time validating antibodies, even though accurate results depend on these reagents working as expected, according to the results of a survey reported today in BioTechniques. "This is quite alarming," says Matthias Uhlén, a protein researcher at the Royal Institute of Technology in Stockholm who heads an international working group on antibody validation, but who was not directly involved in the survey.