Code deposit

Tier 1 · Available - Bronze core endorsement

All code needed to go from the data to every reported result, including data cleaning and preparation as well as the analysis, is publicly available with the paper in human-readable form, preferably in a trusted repository with a persistent identifier.

Rating

Met

All scripts/codes required to reproduce all analyses reported in the paper and appendix are present in human-readable form and publicly available, preferably in a trusted repository with a persistent identifier (e.g. a DOI). If only raw data are included, the code to clean them into the analysis data must be included as well.

Supported by: #12 code for all principal results in human-readable form + code from raw to analysis data, #78 extraction/scraping/cleaning/analysis code, #109 cleaning + analysis code as checklist items, #131 ACRE levels, #141 missing appendix code, #93 ‘all files necessary’, #70, #20, #10.

Partially met

Code is available but incomplete: e.g. the cleaning code is missing although only raw data are shared, or the code for some results (e.g. appendix tables, robustness checks) is missing.

Supported by: #131 L3/L6 (partial code), #125 and #100 cleaning code ‘Partial’ vs. analysis code ‘Fully’, #141 code for the appendix missing.

Not met

No scripts/codes are available.

Supported by: #109 ‘not computationally reproducible if code is missing’, #86 ‘no shared code’ = failure, #98 rating 1, #131 L1/L2, #70.

Not applicable

Never.

Prerequisites

Check order: 1

Builds on

License (2 sources)

Prerequisite for

Numerical output match (17 sources)
Table and figure match (9 sources)
Dependency resolution (7 sources)
Computing environment (7 sources)
Path resolution (6 sources)
Coding error handling (6 sources)
Codes consistent with claims (5 sources)
Code annotation (1 source)
Dynamic code inspection (1 source)
Workflow/reproduction tree (1 source)

In the corpus

Sources	Injunctive	Descriptive	Tacit	Automatable	Manual check	Automation unclear	LLM sources
103	80.6%	6.8%	12.6%	1%	8.7%	90.3%	1

Text evidence (104 quotes)

Show quotes

#1 An introduction to docker for reproducible research · injunctive

“Until such code is available in the first place, one cannot even begin to encounter the problems that the approaches discussed here set out to solve.” (p. 1)

#3 Reproducing statistical results · injunctive

“Availability of computer source code, input data and output data, with some reasonable level of documentation” — item 8 of the ICERM “recommended inclusions” list (p. 11)

#5 ReproZip: computational reproducibility with ease · injunctive

“researchers should create a compendium that encompasses all the components that are required to reproduce the experiment’s results, including data, code, software, library dependencies, and information about the source computational environment” (p. 2085)

#6 A manifesto for data sharing in social media research · injunctive

A researcher “could mostly fulfil the reproducibility requirements by sharing the tweet IDs at a conference website or in a repository, making available code for accessing, cleaning, processing and analyzing the data in a repository… and explain her/his methodology in as much detail as possible” (p.169, Sec 3.3.2)

#7 Tools and techniques for computational reproducibility · injunctive

“no amount of narrative is an adequate substitute for providing the actual commands that were used” (p.3); “a better alternative is to store them in a public repository with a permanent URL” (p.3); “It is most useful when scientists provide direct access to an archived version of the code via a uniform resource locator (URL)” (p.3)

#10 Sustainable computational science: the ReScience initiative · injunctive

“A submission must contain … open-source code that produces the replication results” (p.7) and “Missing code and/or data, undocumented dependencies, and inaccurate or imprecise description appear to be characteristic of much non-reproducible work” (p.8)

#11 Sciunits: reusable research objects · tacit

“By intercepting all calls, AV auditing determines all program binaries, libraries, scripts, and environment variables that a user program is dependent on.” (Sec. III)

#12 Reproducibility: a primer on semantics and implications for research · injunctive

“Methods: Computer code underlying figures, tables, and other principal results reported is made available in a human-readable form.” (Table 4); also “Making available the code that transforms the raw data into the analytic data set” and “Making the statistical code and documentation to reproduce the analysis obtainable” (four-components list) (p. 45-46)

#14 An empirical analysis of journal policy effectiveness for computational reproducibility · injunctive

“All computer codes involved in the creation or analysis of data must also be available to any reader of Science.” (p.2584)

#17 Transparency, reproducibility, and the credibility of economics research · injunctive

“Authors of accepted papers… must provide to the Review, prior to publication, the data, programs, and other details of the computations sufficient to permit replication.” (p. 934) and “…data must be posted to a trusted repository, and reported analyses will be reproduced independently prior to publication” (p. 964)

#18 Reproducibility confidentiality data access · injunctive

“Write code in a way that facilitates release!” / “Create code archives in public areas” (Slide: “Our recommendations to researchers”) and “Systematically release the programs used to generate the output” / “Code archives for replication inside confidential areas” (Slide: “Our recommendations to ADRFs”)

#19 The ReproRubric: evaluation criteria for the reproducibility of computational analyses · injunctive

Code is in a public, software version control system (e.g., git, Github, Gitlab, BitBucket); exact version is reported; The repository is made available in an archivable format (Fig. 2, p.3)

#20 Successes and struggles with computational reproducibility: lessons from the fragile families challenge · injunctive

“manuscripts needed to be accompanied by open-source code that would allow another researcher to reproduce all the figures and tables”; “The code must take the Fragile Families Challenge data files as an input and produce (1) all the figures and tables… and (2) your final predictions” (p.6, footnote 12) and “we recommend that authors include a single ‘run-all’ script that contains a sequence of commands that reproduces all the results” (p.10)

Note: The second quote probably refers to Master Script?

#23 Creating and sharing reproducible research code the workflowr way · injunctive

“Every commit to an (Rmd) analysis file is associated with a commit to the results file generated by that analysis file. 2. An analysis file is only published and committed if it runs successfully” (p. 6)

#24 Computational reproducibility: the elephant in the room · injunctive

“…reproduce every table, diagram, and statistical result using code, for which the source and means to run it are provided” (p. 138)

#25 Toward computational reproducibility: a doctoral student’s story of passing the baton · injunctive

“To test CR requires access to the raw data and code underpinning the original analysis.” (p. 688)

#28 A synthetic dataset primer for the biobehavioural sciences to promote reproducibility and hypothesis generation · tacit

“The R script and datasets to reproduce the analyses described in this paper are available online at https://github.com/dsquintana/synthpop-primer” (p. 1)

#30 Advancing computational reproducibility in the dataverse data repository platform · descriptive

“Publishing data and code as a replication package in a data repository is considered to be a best practice for enabling research reproducibility and transparency” (p.1, Introduction)

#32 Beyond open data: a model for linking digital artifacts to enable reproducibility of scientific claims · injunctive

“2) software that allows access to the data and the regeneration of those claims that rely on the version of the data” (p. 9)

#34 Reproducibility and replicability in economics · injunctive

“Typically, they required that data and code be submitted to the journal, for publication as ‘supplementary materials.’” (p. 8)

#35 Trust, but verify: an investigation of methods of verification and dissemination of computational research artifacts for transparency and reproducibility · injunctive

“In all seven cases, assessment… requires access to the code, programs, or other software artifacts used to generate the results” (p.139)

#37 The four possibilities of reproducible scientific computations · injunctive

“all the source code that can possibly have an impact on the results… is required to allow independent verification”

#38 Ten simple rules for writing dockerfiles for reproducible data science · injunctive

“you should publish all files COPYied into the image, e.g., test data or files for software installation from source… in the same public repository as the Dockerfile, e.g., in a research compendium”

#39 How to make a pie: reproducible research for empirical economics and econometrics · injunctive

“any researcher, when acting as a referee, should ask for data and code at the early stages of the publication process so that the article can be evaluated in light of its sources” (p. 1136) and “we believe that scientific journals and institutions financing research… should at least provide clear incentives or, even better, impose minimal mandatory data and code policies for funded research programs” (p. 1136)

#41 Reproducibility in accounting research: views of the research community · injunctive

“the computations sufficient to permit replication…Data and Code Repository” [Final Analysis Code: Required] — AER policy, p. 542

#42 AEA data and code availability policy · injunctive

“Authors of accepted papers that contain empirical work, simulations, or experimental work must provide, prior to acceptance, information about the data, programs, and other details of the computations sufficient to permit replication” (p. 818) and “A master script is strongly encouraged.” (p. 819)

#43 Dawning of a new age? Economics journals’ data policies on the test bench · injunctive

“the program code (e.g. of statistical analyses in econometric papers, experiments or simulations). Without having the program code of a statistical analysis, the results of a paper and their robustness cannot be assessed satisfactorily.” (p. 8)

#44 Active maintenance: a proposal for the long-term computational reproducibility of scientific results · tacit

“In three of the 20 studies, raw data and associated cleaning files were not deposited… we did not classify the resulting errors as indications of failed computational replication” (p. 463)

#45 Toward reusable science with readable code and reproducibility · injunctive

“only original code provides enough detail to allow for research replication” and “Sharing code enables a higher level of research transparency as anyone can review, examine and scrutinize published results in detail” (p. 1, Introduction). Also: “While a scientific finding can be clearly described in publications to provide a gist of its research methodology, only original code provides enough detail…” (p. 1)

#46 Reproducible research in R: a tutorial on how to do the same thing more than once · injunctive

“the sharing of such digital research products is a necessary condition for reproducibility, it is not a sufficient one” (p. 838)

Note: via usethis::use_github()), though the public/private decision is manual

#47 The science of open (communication) science: toward an evidence-driven understanding of quality criteria in communication research · injunctive

“reproducibility can indeed be elusive if analysis materials are not made available” (p. 690)

#48 Challenges of curating for reproducible and FAIR research output · injunctive

“Authors not making the data, software, workflow, or other digital objects available” (p.20)

#51 A reproducible data analysis workflow with R Markdown, Git, Make, and Docker · injunctive

“The workflow ensures meeting the primary goals that… 3) changes at any time… are tracked, tagged, and documented while earlier versions of both data and code remain accessible” (p.1)

#52 CODECHECK: an open science initiative for the independent execution of computations underlying research articles during peer review to improve reproducibility · injunctive

“The author submits their manuscript along with the code and data to the publisher” (Step 1, p. 3)

#54 WORCS: a workflow for open reproducible code in science · injunctive

“the TOP-guideline of sharing analysis code can be met simply by committing the source code of the manuscript to the Git repository, and making this remote repository Public” (p. 43)

#55 Computational reproducibility: a practical framework for data curators · injunctive

“everything from data cleaning and file conversion to analyses should be scripted” (p.7)

#56 Reproducibility badging and definitions · injunctive

“author-created digital objects used in the research (including data and code) are permanently archived in a public repository that assigns a global identifier and guarantees persistence, and are made available via standard open licenses that maximize artifact availability” (p.3)

#58 Improving research quality: the view from the UK reproducibility network institutional leads for research improvement · injunctive

“…such that all stages of the research pipeline (including research design, data collection processes, the resulting datasets, and analysis code etc.) are made openly available in a manner that allows for re-use.” (p.1)

#59 Understanding experiments and research practices for reproducibility: an exploratory study · injunctive

“The survey findings identify a reproducibility crisis and a strong need for sharing data, code, methods, steps, and negative and positive results.” (Abstract)

#60 Reproducibility in the social sciences · descriptive

“one promising alternative is to provide prepublication code and data review. Multiple journals (Christian et al. 2018, Colaresi 2016) now do a simple code review, requiring authors to submit the code and data required to produce key figures and results” (p.78)

#62 Containerization for creating reusable model code · tacit

“Noting that publishing model code facilitates the replication of its results, CoMSES Net developed a public repository where model source code and documentation can be archived and easily accessed” (~p.2)

#64 Reproducibility signals in science: a preliminary analysis · tacit

“More significantly, the readability of a scholarly article and accessibility of software artifacts, either as code repositories, psuedo-code, or algorithms, could be considered strong indicators for reproducibility.” (p. 143)

#66 Reproducible research publication workflow: a canonical workflow framework and FAIR digital object approach to quality research output · injunctive

“compendium containing data, code, software, and documentation” required after conditional acceptance (p.309)

#67 Tools and recommendations for reproducible teaching · injunctive

“We recommend hosting source files for your teaching materials in public repositories for other instructors to have access to” (p. 256)

#69 The interdependence of data producers and data users: how researchers’ behaviors can support or hinder each other · injunctive

“Share annotated software code to read the data source, to describe processing that created the data source… and to replicate findings cited in articles” (Table 2, p.23)

#70 The Stark realities of reproducible statistically orientated sociological research: Some newer rules of the sociological method · injunctive

“If you did not make your code (including tests) available, fail.” (Table 1, p.210)

#71 Teaching for Large-Scale Reproducibility Verification · injunctive

“the authors of conditionally accepted manuscripts are required to submit a replication package, consisting of all code used to process and analyze data, any data not available in an existing trusted repository, and a ‘README’” (p.276) and “publish papers only if the data and code used in the analysis are clearly and precisely documented and access to the data and code is non-exclusive to the authors” (p.276)

#72 Care to share? Experimental evidence on code sharing behavior in the social sciences · injunctive

“In practice, reproductions hinge on the availability of an original study’s code and data. Performing a replication solely based on the body of an article is tedious at best and impossible at worst.” (p. 2) and “To ensure transparency and bolster scientific credibility, requiring open code as part of any research submission should become the institutional standard” (p. 12)

#73 A backend platform for supporting the reproducibility of computational experiments · injunctive

“However, they do not make available the experiment code, so we cannot reproduce the same experiment.” (p. 14)

#75 Reinforcing reproducibility and replicability: an introduction · descriptive

“increasingly more stringent journal policies surrounding data and code availability, including active review and verification of replication packages” (p. 2)

#76 Making provenance work for you · tacit

“‘scripts’ - a directory containing a copy of the scripts for which provenance was collected” (p. 146).

#78 CRRESS - session 4 - reproducibility and confidential data · injunctive

“Full provision of all code – Data extraction code / Data scraping code / Data cleaning code / Analysis code – Except for small pieces that capture confidential information” (Slide 27, “Some better practices”)

#79 A review of containerization for interactive and reproducible analysis · injunctive

“it is now commonplace for underlying analysis code and data to be proffered alongside journal publications… most of the journals sponsored by the International Statistical Association and American Statistical Association require data and code be posted along with analysis” (p.2)

#80 Replication of the morning after: report from the nottingham replication games · injunctive

“JOP requires authors to provide data and code to reproduce all research results, which are uploaded on JOP’s Dataverse.” (p. 3)

#81 statacons: An SCons-based build tool for Stata · descriptive

“Public posting of data and code… are all becoming more common, both in general and with Stata users specifically” (p.148/149)

#82 Supporting reproducible research · injunctive

“Reproducibility thus provides one of the strongest justifications for open scholarly communication practices such as data and code sharing… Reproducibility requires that we adopt open and transparent methods, not just as an abstract good but as a fundamental part of research practice.” (p. 172)

#83 Reproducibility and transparency versus privacy and confidentiality: Reflections from a data editor · injunctive

“Authors of accepted papers that contain empirical work, simulations, or experimental work must provide, prior to acceptance, the data, programs, and other details of the computations sufficient to permit replication, as well as information about access to data and programs.” (slide “Key elements”)

#84 Replications in agricultural economics · injunctive

“a code availability statement is required in the article, indicating whether the code is accessible and, if so, where it can be found” (p. 361) and “mandatory code sharing is implemented, requiring the code to be posted to a trusted repository, with any exceptions to be identified when the article is submitted” (p. 361)

#85 The Role of Third-Party Verification in Research Reproducibility · injunctive

“checking whether the submitted material (all code, data, readme file, manuscript) complies with a set of guidelines” (p. 3–4)

#86 What makes computational communication science (ir)reproducible? · injunctive

p.6: “(1) No shared code” defined as a failure event; “Criterion 1 determines whether we can conduct the same analysis.” p.7: of 30 empirical articles, only 14 provided both code and data; 16 failed Criterion 1 and/or 2.

#87 Measuring transparency in the social sciences: political science and international relations · injunctive

“require authors to ‘delineate clearly the analytic procedures upon which their published claims rely, and where possible to provide access to all relevant analytic materials’”; also “It is relatively straightforward for researchers using statistical inference to release the very data and code that were necessary to produce the results in their papers” (p. 3)

#88 Reproducibility in Management Science · injunctive

Journal policy: “Authors of accepted papers…must provide…the data, programs, and other details of the experiment and computations sufficient to permit replication.” (p. 1344)

#89 Computational reproducibility in computational social science · injunctive

“if researchers are not willing to share their code, they should be explicit about the consequences and label their work as not computationally reproducible” (p.9)

#90 A guide for social science journal editors on easing into open science · injunctive

“journals could instead implement policies to make acceptance conditional upon direct links to open data, code, and materials” (p. 6)

#91 CORE-bench: fostering the credibility of published research through a computational reproducibility agent benchmark · tacit

Capsule folders: “data — Folder with data necessary to run the code,” “code — …Script to run code with appropriate arguments” (p. 3, Fig. 2)

#93 Unintended code errors and computational reproducibility · injunctive

“Reproducibility package includes all files necessary to reproduce the results” (Appendix A, FILES, p.8)

#94 Navigating the landscape of reproducible research: a predictive modeling approach · descriptive

“Typically, a paper’s reproducibility is inferred based on the availability of artifacts such as code, data, or supplemental information, often without extensive empirical investigation.” (Abstract)

#95 Computational reproducibility in finance: evidence from 1,000 tests · injunctive

“if the code is unavailable, fR(d,∅)=∅≠β” and “insufficient data/code sharing turns out to be a primary source of failure” (p. 3561)

#97 Why and how we share reproducible research at yale university’s institution for social and policy studies · injunctive

“ISPS invites affiliated researchers to deposit data, computer code, and associated materials” (p.3)

#98 Mass reproducibility and replicability: a new hope · injunctive

“a rating of 1 signifies the incapacity to reproduce results due to the absence of data or code” (p. 22)

Note: This paper explicitly rules out dependency and path resolution: “we do not document trivial coding errors such as versioning issues and missing packages/paths. Those coding errors are typically easily fixed by the replicators.” (p. 24)

#99 Introducing reproducible research standards at the world bank · injunctive

“The public package includes … all analytical scripts” (p.4)

#100 Computational and robustness reproducibility of “UN peacekeeping and democratization in conflict-affected countries” · tacit

Table 1: cleaning code = Partial, analysis code = Fully (p.13)

#101 Replicability of simulation studies for the investigation of statistical methods: the RepliSims project · injunctive

“providing open data and code poses no ethical barriers and thus should be required for all published simulation studies” (p.3); reinforced: “Journals that publish simulation studies should require code and data to be publicly available” (p.14)

#103 The Challenges of Replicating Volatile Platform-Data Studies: Replicating Schatto-Eckrodt et al. (2020) · tacit

“Considering that our study utilizes components from prior research—including code, data, and documentation—to reproduce initial findings, we classified our investigation as a computational reproduction (Ziemann et al., 2023)” (p. 4); “Schatto‑Eckrodt et al. provided the computational environment and analysis code in an OSF project.” (p. 5)

#104 Reproduce to validate: a comprehensive study on the reproducibility of economics research · injunctive

“data set(s) and programs used to run the final models” (p.964)

#105 Promoting reproducibility and replicability in political science · injunctive

“they ensure that the data and codes necessary for a computational reproducibility are available” (p.3)

#106 The data availability policies of political science journals · injunctive

“journals must (continue to) adopt and enforce standards that ensure researchers share the data and code necessary [to] reproduce their results” (p. 6)

#107 Teaching constructive replications in the behavioral and social sciences using quantitative data · tacit

“Particularly at the bachelor level, replication courses will be more successful when focusing on studies where data and code are available.” (p. 118) and “Where code is not available, students with advanced analytical skills could attempt to reproduce data handling and analytical steps described in the associated manuscript.” (p. 119)

#108 Checklists for computational reproducibility in social sciences: insights from literature and survey evaluation · injunctive

“Sharing the code as a public repository, e.g., on GitHub… a DOI is assigned… for persistent sharing” (Checklist 3)

#109 A protocol for structured robustness reproductions and replicability assessments · injunctive

“The precondition to conduct reproductions is the availability of data and code.” (p. 3); Table 1 Appendix (p. 9): “Cleaning code provided” and “Analysis code provided” are checklist items; “We consider a paper not computationally reproducible if code is missing.” (p. 4)

Note: Missing code is an absolute failure criterion.

#110 Data and code availability in political science publications from 1995 to 2022 · injunctive

“the APSA has a formal code of ethics that requires sharing data and/or code to reproduce the results (absent special circumstances)” (p. 342); King’s “researchers should share data and code with a professional archive… it should be made publicly available” (p. 341)

#113 Let’s talk about it: making scientific computational reproducibility easier · injunctive

“The integrity and reproducibility of research depend on the availability and accessibility of key artifacts like code, datasets, and dependencies.” (p. 3)

#117 Computational reproducibility of R code supplements on OSF · injunctive

“A fundamental aspect in achieving computational reproducibility is the provision of all necessary materials, including the raw data and the code used to generate the results.” (p.1)

#118 Tooling for reproducible research: considerations for the past and future of data analysis · descriptive

“many scientific journals… changed their article submission policies to encourage or require the submission of data and code along with their manuscripts” (p. 2)

#119 “Works on my machine”: a case study of replicability challenges inComputational humanities research · injunctive

“The most immediate variable concerns the availability of source code, as this forms the basis for further analysis” (p.144); “source code availability… [is] a necessary condition for the replication of computational research” (p.143)

#120 Reproducibility spectrum for applied social sciences: conceptual taxonomy and recommended tools · tacit

Code column defined as “Version control and sharing of analysis scripts”; “share on GitHub” listed as step 2

#121 Statistical perspectives on reproducibility: definitions and challenges · injunctive

“the requirement for reproducible research is that the documentation, data and code used for the analysis are available to others, so that they can verify the published results” (p. 5–6)

#124 Perceived problems, causes, and solutions of finance research reproducibility and replicability: a pre-registered report · descriptive

“One response to such concerns has been code and data sharing policies of journals that aim to improve transparency and provide third parties with the ability to reproduce the results using the exact same and applying the same method through code (Nagel, 2018).” (p. 1)

#125 A report on “corruption and Co-optation in autocracy: evidence from Russia” · injunctive

“Cleaning code provided [Partial]… some minor modifications required,” “Analysis code provided ✓ [Fully]” (Table 1, p.5)

#126 Automated Reproducibility Testing in R Markdown · injunctive

“the repository should contain both the Markdown files as well as the metadata files” (p. 3)

#128 The reliability of replications: a study in computational reproductions · injunctive

“The most important is transparency. A successful single computational reproduction of a given model is far more likely if the complete, well-annotated original code is available.” (p. 18)

#130 Reproducibility in the classroom · tacit

“Are code and data both readily available?” (p.10)

#131 Guide for Accelerating Computational Reproducibility in the Social Sciences · injunctive

Level 1 (L1): No data or code are available. Possible improvements include adding: raw data, analysis data, cleaning code, and analysis code. You may have detected papers without any reproduction materials at the Paper Selection stage, where you should have recorded them as abandoned candidate papers.

Level 2 (L2): Code scripts are available (partial or complete), but no data are available. Possible improvements include adding: raw data and analysis data.

Level 3 (L3): Analytic data and code are partially available, but raw data and cleaning code are missing. Possible improvements include: completing analysis data and/or code, adding raw data, and adding analysis code.

Level 4 (L4): All analytic data sets and analysis code are available, but the code fails to run or produces results inconsistent with the paper (not CRA). Possible improvements include: debugging the analysis code or obtaining raw data.

Level 5 (L5): Analytic data sets and analysis code are available and they produce the same results as presented in the paper (CRA). The reproducibility package may be improved by obtaining the original raw data.

Note: This is the highest level that most published research papers can attain currently. Computational reproducibility from raw data is required for papers that are reproducible at Level 6 and above.

Level 6 (L6): Cleaning code scripts are available (partial or complete), but raw data is missing. Possible improvements include: adding raw data.

Level 7 (L7): Cleaning code is available and complete, and raw data is partially available. Possible improvements: adding raw data.

Level 8 (L8): All the materials (raw data, analytic data, cleaning code, and analysis code) are available. However, the cleaning code fails to run or produces different results from those presented in the paper (not CRR) or the analysis code fails to run or produces results inconsistent with the paper (not CRA). Possible improvements: debugging the cleaning or analysis code.

Level 9 (L9): All the materials (raw data, analytic data, cleaning code, and analysis code) are available. The analysis code produces the same output as presented in the paper (CRA). However, the cleaning code fails to run or produces different results from those presented in the paper (not CRR). Possible improvements: debugging the cleaning code.

Level 10 (L10): All necessary materials are available and produce consistent results with those presented in the paper. The reproduction involves minimal effort and can be conducted starting from the analytic data ( CRA) and the raw data (CRR). Note that Level 10 is aspirational and may be unattainable for most research published today.

#132 Long-term effects of the targeting the ultra poor program—a reproducibility and replicability assessment of banerjee et al. (2021) · tacit

“the authors provide… all do-files to clean, prepare, and analyze the data alongside a master do-file that executes all do-files in the correct order” (p. 4–5)

#135 Automating computational reproducibility in social science: comparing prompt-based and agent-based approaches · injunctive

“all necessary code and data must be equally accessible” (p. 3)

#136 Artisan: agentic artifact evaluation · injunctive

“The reproduction script should contain all logic needed to reproduce the table from scratch, from running the artifact to formatting outputs into the final table.” (p. 3).

#137 Overcoming barriers to computational reproducibility · injunctive

“The code and data supplement should contain all materials necessary to independently reproduce the results… This typically includes the complete analytical code, the relevant dataset or datasets (where possible)” (p.7)

#138 Reproducibility and robustness of economics and political science research · injunctive

“Each of these journals has a data and code availability policy that requires authors to publicly share their materials upon publication” (p.151)

#139 PRIMAD-LID: a developed framework for computational reproducibility · injunctive

“versioned source code and a tagged release (e.g., Git commit hash and release archive)” (p. 2)

#140 How to Get There From Here? Barriers and Enablers on the Road Towards Reproducibility in Research · injunctive

“ML researchers noted that industry-funded research also presents a barrier, in that sharing of data, software and other materials is often not allowed by industry partners for reasons of competition. Researchers working in such collaborations would not be able to create reproducible research.”

#141 Understanding cultural persistence and change: A replication of Giuliano and Nunn (2021) · injunctive

“Comprehensive replication packages, including all necessary files, can greatly facilitate verification efforts and enhance research transparency” (p. 34) and (p. 23): “it does not include the raw data, the code necessary to transition from the raw data to the various estimation samples, and the codes that generate the results reported in the online Appendix of GN”).

#142 Teaching Reproducibility and Replicability While Teaching Econometrics in the Classroom · injunctive

“Not only must the code and data be available, but the documentation of how to replicate their analyses must be included.” (p. 116)

#143 What is Computational Reproducibility? · injunctive

“Executable scripts and code are the machine-readable implementation of models, methods, or processes, making the computational steps explicit and operational. This form facilitates practical testing and validation of methods, and consists of code or scripts that can be directly run on a computer.” (p. 3) an “Crucial measures for achieving this include rigorous documentation, the structured use of executable and algorithmic forms…” (p. 7).

#144 Investigating the reproducibility of the social and behavioural sciences · injunctive

“Without code provided, reproduction attempts may involve making reasonable inferences about how the data were prepared and analysed based on what is written in the paper.” (p.131)