Based on a thread on the DP-JEDI discussion list:
Submissions to "my" journal ... are almost entirely quantitative. I want very much to adopt a requirement that authors must make their code/syntax available: 1) post it to a repository (e.g. OSF), and 2) include an anonymized link to it in the manuscript text.
We have so far only "strongly encouraged" it, and as things stand very few authors do it.
Before pulling the trigger, I'd like to ask: are there drawbacks I should consider?
My response, which might be the kernel for a page:
For what it's worth, a few contributions to answers (not sure I have all the answers, but I do have a few revisions of replication packages under my belt...)
(the we/us below is the group of us, primarily in economics, but some finance, some management, collected as the Social Science Data Editors group. Nothing formal, and I do not speak for the group - nobody does...)
- choice of where replication packages should go (the fact that OSF is great for sharing, not always great for permanent archives was already mentioned). We wrote this up a few years ago, it may still be mostly relevant: https://zenodo.org/records/10983029 (perspectives from 4 journals, having chosen 3 different archives)
- quality of the package: those of us (not my journals) who have decided to require (not just suggest) the use of our Template README have seen great success in completeness, and something that is easy to check (you either used it or not, which is easy to check, and you may have gotten some sections wrong, which is also easy to check). None of us want to go back. https://social-science-data-editors.github.io/template_README/ (at my journals, we only suggest that you use it, but if you don't have all the elements we require in the README you DID submit, we kindly suggest that you do look at the template, which usually solves all the problems).
- some of us have developed checklists that authors are required to submit (after conditional acceptance, usually), which also helps a lot:
-
Economic Journal: https://ejdataeditor.github.io/checklist.pdf
- AEA Papers and Proceedings: https://aeadataeditor.github.io/aea-de-guidance/preparing-for-pandp.html (we don't do that for the regular journals, except in a pilot project)
- some journals (not us... ) have required that code be shown to be reproducible on specific platforms. This may or may not be constraining. Codeocean comes up, a new kid on the block is Nuvolos, and I have a pilot project (aforementioned) that provides such evidence. It is not always a good idea - point-and-click people will struggle, and it depends on discipline-specific coding practices (which is why for my journals, it is only a pilot project).
Finally, as to why you want to do this, I like to quote Guido Imbens, here indirectly via Timothy Salmon:
https://hdsr.mitpress.mit.edu/pub/z9fpx2js/release/1?from=11495&to=12158
"Guido Imbens [...] argued that allowing empirical researchers to hide their methods like this is similar to allowing theorists to publish theorems while keeping the proofs hidden. [...] We do not, however, allow theorists to avoid providing proofs."
And then I would like to quote Tim Salmon directly:
https://hdsr.mitpress.mit.edu/pub/z9fpx2js/release/1?from=12898&to=13901
One reason to be concerned about that burden from a journal’s perspective is that these requirements may decrease submissions [...] At EI, [...] submissions dropped off by about 15%. [...] Further, the last journals in a peer group to adopt these policies should expect to receive submissions by any authors who do not wish to make their research methods transparent. As an editor, I do not want to receive those submissions and so I am willing to accept this trade-off.
Hope all of that is useful!
Based on a thread on the DP-JEDI discussion list:
My response, which might be the kernel for a page:
For what it's worth, a few contributions to answers (not sure I have all the answers, but I do have a few revisions of replication packages under my belt...)
(the we/us below is the group of us, primarily in economics, but some finance, some management, collected as the Social Science Data Editors group. Nothing formal, and I do not speak for the group - nobody does...)
Finally, as to why you want to do this, I like to quote Guido Imbens, here indirectly via Timothy Salmon:
https://hdsr.mitpress.mit.edu/pub/z9fpx2js/release/1?from=11495&to=12158
"Guido Imbens [...] argued that allowing empirical researchers to hide their methods like this is similar to allowing theorists to publish theorems while keeping the proofs hidden. [...] We do not, however, allow theorists to avoid providing proofs."
And then I would like to quote Tim Salmon directly:
https://hdsr.mitpress.mit.edu/pub/z9fpx2js/release/1?from=12898&to=13901
One reason to be concerned about that burden from a journal’s perspective is that these requirements may decrease submissions [...] At EI, [...] submissions dropped off by about 15%. [...] Further, the last journals in a peer group to adopt these policies should expect to receive submissions by any authors who do not wish to make their research methods transparent. As an editor, I do not want to receive those submissions and so I am willing to accept this trade-off.
Hope all of that is useful!