#14 An empirical analysis of journal policy effectiveness for computational reproducibility · injunctive
“Reproducible after tweaking (e.g., missing parameters required fiddling to find, missing modified code lines, missing arguments required for differing architecture; missing minor method step)” — 5% of sample; “Minor difficulty in reproducing” — 18% (p.2587)
#20 Successes and struggles with computational reproducibility: lessons from the fragile families challenge · tacit
“All 14 manuscripts received a decision of ‘revise and resubmit,’ and authors were instructed to improve both their manuscripts and code on the basis of the feedback that was provided” (p.7)
#23 Creating and sharing reproducible research code the workflowr way · injunctive
“An analysis file is only published and committed if it runs successfully; on failure, wflow_publish() aborts, and neither code nor results are committed to the Git repository (R code that does not work can still be committed to a workflowr project via other methods, e.g., directly using Git, but it will not be associated with a committed results file).” (p. 6).
#44 Active maintenance: a proposal for the long-term computational reproducibility of scientific results · tacit
“Errors were classified based on whether a moderately trained replicator would be able to overcome them without substantially updating the script for the statistical analysis or after receiving help from the study authors” (p. 463)
#52 CODECHECK: an open science initiative for the independent execution of computations underlying research articles during peer review to improve reproducibility · injunctive
“Codecheckers record but don’t investigate or fix… the job of the codechecker is not to fix these problems but simply to report them to the author and await a fix.” (Core Principle 1, p. 7)
#54 WORCS: a workflow for open reproducible code in science · injunctive
“Any alterations to the data – including processing steps and even error corrections – should be documented in the code, instead of applied to the raw data” (p. 40)
#66 Reproducible research publication workflow: a canonical workflow framework and FAIR digital object approach to quality research output · tacit
“Code review and reproducibility verification… could lead to the identification of errors which would then lead to new calculations and a new paper submission” (p.308–309)
#74 Reproducibility as a service · tacit
“users can easily fix 23.1% of the errors that appear when executing datasets with RaaS if they are library and working directory errors. We refer to these fixes as easy as they do not require significant code restructuring” (p. 1563–1564)
#80 Replication of the morning after: report from the nottingham replication games · tacit
“We could not detect any coding errors or other issues and the reproduction was seamless.” (p. 3)
#86 What makes computational communication science (ir)reproducible? · injunctive
“Major code rewrite: When the execution failure was related to code quality issues, i.e. bugs, we attempted to correct for the bugs and rerun.” (p. 9)
#93 Unintended code errors and computational reproducibility · tacit
“This is an iterative process in order to resolve any computational reproducibility issues” (§1.1); “errors… were resolved prior to publishing… often by updating files or publishing ‘curator notes’” (Executive Summary, p.1)
#95 Computational reproducibility in finance: evidence from 1,000 tests · tacit
“the verifier tried to fix bugs and errors by adapting the code and/or the computing environment… stayed within a one-week time budget for each team” (p. 3567–3568)
#98 Mass reproducibility and replicability: a new hope · tacit
“We instead focus on coding errors which could have had an impact on claims and conclusions of articles.” (p. 24)
#99 Introducing reproducible research standards at the world bank · tacit
“the review process is iterative, with as many resubmissions as required to get to a fully functional package” (p.5); reviewer “suggests solutions to authors for all issues other than version control problems” (p.5)
#100 Computational and robustness reproducibility of “UN peacekeeping and democratization in conflict-affected countries” · tacit
“we could not detect any coding errors in the data manipulation part” (p.3); “could not detect any coding errors or other issues” (p.4)
#103 The Challenges of Replicating Volatile Platform-Data Studies: Replicating Schatto-Eckrodt et al. (2020) · injunctive
“We allowed for minor code modifications to accommodate changes in software and dependencies over time, ensuring that the research remains adaptable and relevant.” (p. 4); applied at p. 5 re: the unrecalculated t-statistic (“this non‑execution did not alter the test outcome’s results substantially… thereby not affecting the core conclusions”). Also: “it is recommended that a seed be determined for illustration methods, incorporating a randomness component to ensure consistent and reproducible visualizations.” (p. 5)
#104 Reproduce to validate: a comprehensive study on the reproducibility of economics research · injunctive
“we … limited the time to find a solution … to about a week. If unsuccessful … the reproduction attempt was marked as ‘not reproducible’” (p.970)
#109 A protocol for structured robustness reproductions and replicability assessments · injunctive
“For all other errors due to missing data or incorrect code, document these problems if you were able to identify them, but do not yet attempt to solve them.” (Appendix, p. 8); “replicators carefully inspect the code to identify objective (i.e. unambiguous) coding errors… In case errors are found, the replicator corrects them.” (p. 4–5); explicit list of objective errors includes: “variable misspecification, coding switcheroo, outlier management, missing data handling, sensitive commands such as merging, appending, reshaping, and dropping data.” (Appendix, p. 10)
Note: The paper defines a two-stage procedure: (1) first run without corrections and document errors; (2) then in Code Inspection, identify and correct objective errors only, explicitly distinguished from subjective analytical decisions. The distinction between objective errors and analytical choices is central to this criterion. Strictly manual.
#113 Let’s talk about it: making scientific computational reproducibility easier · tacit
“This means that after we uploaded the zip file with all the experiment’s content, our tool was able to automatically infer all the necessary configurations and execute the provided code. In each case, the code ran error-free, successfully producing results.” (p.7)
#115 Similarity-based assessment of computational reproducibility in jupyter notebooks · injunctive
“Since the focus of this work is to propose the similarity-based metric, to keep it simple, we chose notebooks that do not have any cell generating error outputs. It can be said that we assumed runnability as a prerequisite of reproducibility here.” (p. 7)
Note: Strongest explicit prerequisite-type statement in the paper, but framed as a sampling choice for their own study to test their proposed algorithm.
#116 AutoReproduce: automatic AI experiment reproduction with paper lineage · tacit
“Since not all generated code can be executed, the P_agent,i is set to 0 for the non-executable instances, resulting in a maximum performance gap of 1.0” (p.7, lines 21–22)
#118 Tooling for reproducible research: considerations for the past and future of data analysis · tacit
“If there were a change to the data or the code, re-weaving the source file would update the results and confirm the reproducibility of the analysis (or not, if there were some error in the code)” (p. 3)
#128 The reliability of replications: a study in computational reproductions · injunctive
“For curating, we attempted to fix mistakes in a team’s code only if the following two conditions were met: it had to be obvious that it was a mistake and, if so, that we could determine a counterfactual scenario of what the authors would have done instead had they become aware of the mistake. We only changed code when we did not have to make any decisions.” (p. 9, Methods — Curated section)
#131 Guide for Accelerating Computational Reproducibility in the Social Sciences · injunctive
Whenever code is available in the reproduction package, you should be able to debug those scripts. There are at least five types of debugging that can improve the reproduction package:
Code cleaning: Simplify the instructions (e.g., by wrapping repetitive steps in a function or a loop) or remove redundant code (i.e., old code that was commented out) while keeping the original output intact.
Performance improvement: Replace the original instructions with new ones that perform the same tasks but take less time (e.g., choose one numerical optimization algorithm over another while still obtaining the same results).
Adding unit tests: Add if/then statements after a code chunk or section (more or less every 100 lines of code) where you test that variables or statistics are computing as expected. If the value has changed, create a warning message that mentions what object has changed.
Environment set up: Modify the code to include correct paths to files, specific versions of software, and instructions to install missing packages or libraries.
Correcting errors: A coding error will occur when a section of the code in the production package executes a procedure that is in direct contradiction with the intended procedure expressed in the documentation (i.e., paper or code comments). For example, an error will occur if the paper specifies that the analysis is performed on a population of males, but the code restricts the analysis to females only..
#134 PaperRepro: automated computational reproducibility assessment for social science papers · injunctive
“Score 2 reflects minor inconsistencies or errors in the provided code that do not change major findings.” (§4.1.1, p.6)
#136 Artisan: agentic artifact evaluation · injunctive
“If the method judge classifies the submitted script as either last-mile reproduction or full reproduction, Artisan accepts the submitted script as the final reproduction script. In contrast, scripts classified as copied-results are rejected.” (p. 8).
Note: a two-tier rule for what counts as tolerable vs. intolerable errors during execution (static/runtime errors → feedback loop; last-mile → accepted; copied-results → rejected).
#138 Reproducibility and robustness of economics and political science research · tacit
“Computational reproducibility pertains to the ability of the provided replication folder to reproduce the exhibits and statistics displayed in the research… Reproducers may be able to reproduce all exhibits exactly as they appear (computationally reproducible), but the exhibits may have been constructed with coding errors or discrepancies.” (p.155, “Coding errors and recoding”)
#141 Understanding cultural persistence and change: A replication of Giuliano and Nunn (2021) · injunctive
“The remedies we propose to address the uncovered discrepancies are straightforward: doing so simply needs to adhere to the precise and unambiguous descriptions provided in the original article.” (p. 24)
#144 Investigating the reproducibility of the social and behavioural sciences · injunctive
“the reproduction analyst makes an error or did not spend enough time troubleshooting… If those are the sole reasons why the reproduction test failed, then the original outcome may have been reported precisely despite the independent failure to reproduce them.” (p.131)