The Recomputation Manifesto
The abstract of the manifesto is this:
Replication of scientific experiments is critical to the advance of science. Unfortunately, the discipline of Computer Science has never treated replication seriously, even though computers are very good at doing the same thing over and over again. Not only are experiments rarely replicated, they are rarely even replicable in a meaningful way. Scientists are being encouraged to make their source code available, but this is only a small step. Even in the happy event that source code can be built and run successfully, running code is a long way away from being able to replicate the experiment that code was used for. I propose that the discipline of Computer Science must embrace replication of experiments as standard practice. I propose that the only credible technique to make experiments truly replicable is to provide copies of virtual machines in which the experiments are validated to run. I propose that tools and repositories should be made available to make this happen. I propose to be one of those who makes it happen.
The manifesto itself contains six theses, as follows
- Computational experiments should be recomputable for all time
- Recomputation of recomputable experiments should be very easy
- It should be easier to make experiments recomputable than not to
- Tools and repositories can help recomputation become standard
- The only way to ensure recomputability is to provide virtual machines
- Runtime performance is a secondary issue
I discuss these positions and argue for them. I also summarise briefly the recomputation.org mission.
The paper will soon appear on arxiv and I will update this post with details when it does appear.
This is Version 1, or to be more precise 1.9479. But the four digit number just refers to the revision in the subversion repository it comes from, so there have not been that many version of it: there are many other papers in the repository I’m using.