paper.tex

\documentclass[11pt]{article}
\usepackage{todonotes}

\title{Approaches to Reproducibility of Computational Experiments: A Survey}
\author{Ian P. Gent}
\begin{document}
\maketitle
\begin{quote}
\textit{It's not really for the benefit of other people. Experience shows the principal beneficiary of reproducible research is you the author yourself. \hfill ---~Jon~Claerbout}
\end{quote} 
	
\section{Introduction}
Reproducibility is fundamental to the modern theory of science.\todo{Cite some boilerplate}
Computational experiments have become fundamental to the modern practice of science. How are the two combined, in theory and in practice? 

Reproducibility in CS has the problem that it is an area dissipated across the discipline. Meaning that similar baby steps are taken in many areas and it doesn't come together. On the other hand, we cannot let it become ghettoised as its own research topic to which nobody else pays attention.

\subsection{Some notes on this survey}

I think it is useful to make some notes on attitudes I am taking in this survey. 

The literature on this topic is not all contained in scientific papers. It would be quite improper not to cite important work and observations made in other forums, such as blog posts and even tweets. I will attempt to give these due credit. 

Most importantly, in some sense this survey is very early. There is relatively little work, and very little of it is definitive. I am not apologetic for writing this survey too early. The topic is too important to leave until all the dust is settled.  

\subsection{A reproducible survey}

VM, github or something.

\subsection{Types of Reproducibility}

Carole Goble thing

\begin{verbatim}
+ things like R, ipython, including data in paper... 
\end{verbatim}


\subsection{Jon Claerbout}

http://sepwww.stanford.edu/data/media/public/sep//jon/reproducible.html
Admirable attempt going back a long way
Not VMs but codebases
Reproducibility tested 
"It's not really for the benefit of other people. Experience shows the principal beneficiary of reproducible research is you the author yourself.”

\subsection{Ten Simple Rules}
http://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1003285

\subsection{New Gold standard}
https://www.cam.ac.uk/research/news/new-gold-standard-established-for-open-and-reproducible-research

\begin{verbatim}
+ famous excel spreadsheet story
+ virtual machines - considered harmful 
+ digital vellum
+ reprozip, CDE
+ containers
+ build 
+ debian reproducible builds
+ bit reproducibility
++ Drummond
++ Considered harmful

+ reproducibility chart thing and then reproduction of that
+ http://reproducibility.cs.arizona.edu/
+ http://cs.brown.edu/~sk/Memos/Examining-Reproducibility/


+ efforts by conferences and journals 
++ IPOL
++ failed journal - Open Research Computation
++ ACM JEA
++ recomputation
++ reproducibility papers anywhere

+ http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3878063/
+ RRE / RRS 

+ Kotthoff cloud results

+ Case studies of failures...
++ Prosser, Prosser/Beck

+ Stodden 
https://web.stanford.edu/~vcs/papers/CiSE2012-LMS.pdf

+ Workshop paper output with lots of authors Stodden 

+ Arxiv 17 author papers thing 

+ Davison from Luke's thesis

+ Counterpoint views
++ http://www.nature.com/news/reproducibility-the-risks-of-the-replication-drive-1.14184
+ 

+ Pontifications
++ Recomputation manifesto
++ Peng 2011 http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3383002/

+ Related topics
++ Provenance 
+++ http://www.sci.utah.edu/publications/freire08/cise2008a.pdf

\section{Challenges}

\subsection{Research Challenges}

\subsection{Cultural Challenges}

\subsection{Legal Challenges}

\subsection{Ethical Challenges}

\subsection{Regulatory Challenges}

Ref



\end{verbatim}
\end{document}

\section*{Acknowledgements}

I am very grateful to
Christopher Jefferson
.... add yourself here with a pull request if you fix something.