Welcome to CP 2013 Authors
Lars Kotthoff and I would like to give a warm welcome to
recomputation.org to any successful authors with papers at the CP 2013 conference.
We will be giving a tutorial on recomputation at the conference, and we want to report on recomputation of as many papers from the conference as we can.
We believe that experiments in Computer Science should be “recomputable”. By this we mean that experiments should be repeatable, as exactly as possible (within reason) as used to make the published results, and that these recomputable experiments should be made available openly for as long as possible. There are many reasons for doing this, and some are outlined in “The Recomputation Manifesto”. There are two more important reasons not emphasised there. First, by making your current experiments recomputatble, you will find it much easier to do your next experiment. And second - though I state this without evidence - if you make your experiment recomputable, I expect you might get more citations in future: the added information you will give the community will be of great use.
We do have ideas as to how to make experiments recomputable. To test these ideas out, we would like a range of real life scientific experiments, and so you can help us by providing yours. Since we come from Constraints, we thought that working with authors of accepted papers at CP 2013 would be an ideal starting point.
A couple of quick responses to points which are often raised after people read The Recomputation Manifesto.
First, we do wish to provide future researchers with copies of virtual machines in which they can run your experiments. This does have overhead and bandwidth and storage implications. However, this does not mean that you necessarily have to give us a virtual machine. In most cases people run their experiments on hard silicon rather than virtual ones. We would hope in many cases to be able to import them into VMs without you needing to construct and send one to us, although we will discuss this on a case by case basis.
Second, a very common point is concern about the manifesto’s claim that runtime is a secondary issue. We do think runtime is very important, and it is often the crux of a given scientific paper. Thsi is true. But our goal is to make the experiment recomputable, even if runtimes are not similar to what you achieved on bare silicon. This does not detract from the value of the experiments you report in your paper. But, by enmabling recomputations of experiments, the community will get access to your results on a wider variety of execution environments, and that may be interesting in itself.
Ideally, we would like to make all experiments available from this site by serving them out as downloads and/or to be executed from this site. One issue that might limit this would be if your experiments are just too HUGE in disk space or run time. That is not a bad thing in itself! But we would still love to see what we can do. The second obvious issue is licencing. We would very strongly urge you to make your contributions open source, as many people have recently been advocating. However, there may be issues which prevent this for your whole experiment, e.g. use of proprietary library. While we might not be able to host your experiment in these circumstances, we would still love to work with you: and even the we still urge you to make your contributions open. It’s not us just saying this. Look at the
Scientific Code Manifesto
or not just
two pieces in Nature on this topic.
A few quick reassurances. First, participation in this event for CP2013 has no effect on the publication of your paper: it is an optional extra. Second, there is no deadline for participation. But we do hope you will participate and contact us as soon as possible to do so. The sooner you do the more time we have to make your experiment recomputable. Also the sooner after you did your experiments, the fresher they will be in your mind, and the easier to help us make them recomputable. Finally, of course expressing interest in no way commits you to proceeding further if you do not like what we are doing.
This is all a very long way of saying … please join us in our efforts!
To any CP 2013 authors or anyone else who wish to contact us, please email us at our new mail address email@example.com. Of course you should also feel free to contact us through emails to our normal addresses to.
And finally, many thanks to Christian Schulte (programme chair of CP) and Laurent Michel (tutorial chair) for their help with our efforts.