Terrible Acronym - Great Summer School

Last week, in St Andrews, we held the first
Summer School on Experimental Methodology in Computational Science Research.

So the acronym is EMCSR. Not only is this terrible since it doesn’t flow off the tongue or make a word, but there was a more popular event with the name EMCSR 2014.

Nevertheless we held the summer school, and it went very well. By “we”, I should say, we mean not just those involved with recomputation.org,
but also all the other organisers: in full that was Ian Gent, Tristan Henderson, Alexander Konovalov, John McDermott, Angela Miguel, and Lakshitha de Silva.

Two things in particular went much better than I expected.

First, I was really pleased to learn a lot from the various talks we held. I mean, it’s not that I thought I knew everything about reproducibility in computational sciences, but I was really pleased to see so many interesting points being made, and issues brought up.

Second, we always planned to write a paper by the end of the week. This seemed ludicrously ambitious, since participants would also have lots of work to do attending lectures (and eating meals and things like that.) I had in mind something like that people would split into groups and perhaps each one would make an experiment recomputable, reporting on any obstacles they overcome.

I was right about splitting into groups, but each group was far more impressive than I’d expected. One made three experiments recomputable, and had an interesting spin as these were not computer science experiments but from other disciplines. A second group did experiments on how performance of parallel experiments varies under different environments (virtual and real). And a third group looked at ethics forms from universities around the world. I’ll admit freely that this didn’t sound too promising to me, but it turned out remarkably interesting. Almost nothing is shared between these forms, meaning that you might be able to reproduce an experiment from another lab, but have no idea if it will be ethical in your university just because it was in theirs.

It’s worth saying that the groups and topics were self-selected, and I’m sure that helped groups get really into it.

And indeed, we got a paper written by the deadline of Friday night. It’s now available on arxiv, although even the author list is not correct in this draft, so it is not in any sense finished. We hope to revise it in time for a journal deadline at the end of the month.

The paper is also intended to be open and reproducible, and to that end you can see every commit made on github if you wish.

Many thanks to all the participants, speakers, and our sponsors.

Who knows? There might be another summer school next year, and if there is maybe we can see if we can reproduce this year’s paper!