SIGMOD Repeatability Effort

As part of this project, in collaboration with Philippe Bonnet, we are using (and extending) our infrastructure to support the SIGMOD Repeatability effort. Below are some case studies that illustrate how authors can create provenance-rich and reproducible papers, and how reviewers can both reproduce the experiments and perform workability tests: packaging an experiment on a distributed database system (link in title).

How Bright Promise in Cancer Testing Fell Apart

Research at Duke University in genomics that involved fighting cancer by looking for gene patterns that would determine which drugs would best attack a particular cancer (no more trial-and-error treatment, considered a breakthrough). This research turned out to be wrong, due to flaws in the research (found by statisticians); if the research was reproducible, errors could have been found earlier and the patients could have continued their treatment.

It’s Science, but Not Necessarily Right

NY article discussing the issues with scientific reproducibility: "Why? One simple answer is that it takes a lot of time to look back over other scientists’ work and replicate their experiments. Scientists are busy people, scrambling to get grants and tenure. As a result, papers that attract harsh criticism may nonetheless escape the careful scrutiny required if they are to be refuted."

The Legal Framework for Reproducible Scientific Research: Licensing and Copyright

The code, data structures, experimental design and parameters, documentation, and figures are all important for scholarship communication and result replication. The author proposes the reproducible research standard for scientific researchers to use for all components of their scholarship that should encourage reproducible scientific investigation through attribution, facilitate greater collaboration, and promote engagement of the larger community in scientific learning and discovery.

  • open access
  • Sweave

    Sweave is a tool that allows to embed the R code for complete data analyses in latex documents, and is automatically packaged in R installations. The purpose is to create dynamic reports, which can be updated automatically if data or analysis change. Instead of inserting a prefabricated graph or table into the report, the master document contains the R code necessary to obtain it. When run through R, all data analysis output (tables, graphs, etc.) is created on the fly and inserted into a final latex document. The report can be automatically updated if data or analysis change, which allows for truly reproducible research. It does not, however, track provenance.