Next month Las Vegas will host the Final Event of the DARPA Cyber grand Challenge as an all-computer cyber-defence Capture the Flag tournament. From an initial field of over 100 applicant seven teams will compete for the $3.5 million prize pool. Reproducibility is a key aspect of a sound scientific design. While perfect system state replay is impossible without a full system event recorder, DECREE has been designed to allow high determinism and reproducibility given a record of software and inputs. This reproducibility property has been built into DECREE from kernel modifications up through the entire platform stack.
When functional magnetic resonance imaging (fMRI) was introduced in the late 1990s, it drew raves for its ability to show brain activity—and concerns that it might be the modern equivalent of phrenology. Now, that debate could spring to life again with revelations that the popular imaging technology could have been flawed for years. As Kate Lunau writes for Motherboard, new research suggests that software used to analyze fMRI results could invalidate up to 40,000 brain activity studies.
A whole pile of "this is how your brain looks like" MRI-based science has been invalidated because someone finally got around to checking the data. The problem is simple: to get from a high-resolution magnetic resonance imaging scan of the brain to a scientific conclusion, the brain is divided into tiny "voxels." Software, rather than humans, then scans the voxels looking for clusters. In this paper at PNAS, they write: "the most common software packages for fMRI analysis (SPM, FSL, AFNI) can result in false-positive rates of up to 70%. These results question the validity of some 40,000 fMRI studies and may have a large impact on the interpretation of neuroimaging results."
Nearly one-third of junior scientists spend no time validating antibodies, even though accurate results depend on these reagents working as expected, according to the results of a survey reported today in BioTechniques. "This is quite alarming," says Matthias Uhlén, a protein researcher at the Royal Institute of Technology in Stockholm who heads an international working group on antibody validation, but who was not directly involved in the survey.
This week in science, academia and publishing for reproducibility.
Jeremy Berg, the incoming editor-in-chief of Science magazine, will be grappling with a number of issues plaguing science and science publishing when he takes over that role, Retraction Watch's Shannon Palus writes. Berg has previously supported efforts to bolster reproducibility and transparency, Palus notes. He tells her that there are a number of efforts aimed at improving reproducibility underway at Science, but as he hasn't started the position yet — he's to take the helm in July — he needs to catch up on what's already been done. He says various issues could be behind the irreproducibility problem and, to be effective, any response has to be tailored to that issue.