Data deposit

Tier 1 · Available - Bronze core endorsement

The data needed to reproduce the reported results are publicly available with the paper, preferably in a trusted repository with a persistent identifier. This can be the raw data, the analysis data, or both; with analysis data only, results can be reproduced from the analysis data but not from the raw data.

Rating

Met

The data required to reproduce the analysis are provided, preferably in a trusted repository with a persistent identifier (e.g. a DOI). These can be either the raw data or the cleaned (analysis) data; whether the matching cleaning code is present is scored under code deposit. If the data cannot be shared (e.g. proprietary or confidential data), the criterion is also met when the README (1) states why the data cannot be shared (e.g. license, data use agreement, privacy law), (2) documents completely how an independent researcher can obtain the same data, to the standard of data provenance (provider, version/vintage, application procedure, costs, waiting times), and (3) the authors commit to preserving the data for a stated period (e.g. at least five years) or have deposited them in a secure archive.

Supported by: #109 raw vs. analysis data both accepted, raw needs cleaning code (note), #131 L5 (analysis data) vs. L10 (raw + analysis), #12 ‘analytical data set is available’, #100 analysis data only, #132 raw + analysis + cleaning code, #93.

Partially met

Some of the data required are missing: e.g. one of several data sources, the data behind some results (e.g. appendix tables), or only a subsample of the data used is provided. For data that cannot be shared: the reason is stated, but the access documentation is incomplete or there is no preservation commitment.

Supported by: Partial data: #131 L7 (raw data partially available), #15 ‘missing the data used for analysis’, #141 raw data missing.

Not met

No dataset is available, and for data that cannot be shared, neither the reason nor how to obtain the data is documented.

Supported by: #86 ‘no shared data’, #144 unavailability of data the main barrier, #98 rating 1, #131 L1/L2, #70 fail unless law prevents it.

Not applicable

A replication package is a simulation study (C1)

Prerequisites

Check order: 1

Builds on

License (3 sources)

Prerequisite for

In the corpus

Sources	Injunctive	Descriptive	Tacit	Automatable	Manual check	Automation unclear	LLM sources
94	76.6%	9.6%	13.8%	1.1%	7.4%	91.5%	0

Text evidence (95 quotes)

Show quotes

#3 Reproducing statistical results · injunctive

“Availability of computer source code, input data and output data, with some reasonable level of documentation” — item 8 of the ICERM “recommended inclusions” list (p. 11)

#5 ReproZip: computational reproducibility with ease · injunctive

“researchers should create a compendium that encompasses all the components that are required to reproduce the experiment’s results, including data, code, software, library dependencies, and information about the source computational environment” (p. 2085)

#6 A manifesto for data sharing in social media research · injunctive

A researcher “could mostly fulfil the reproducibility requirements by sharing the tweet IDs at a conference website or in a repository, making available code for accessing, cleaning, processing and analyzing the data in a repository… and explain her/his methodology in as much detail as possible” (p.169, Sec 3.3.2)

#11 Sciunits: reusable research objects · tacit

“Inclusion of data files is optional, which the user may or may not want to package based on the size of the dataset.” (Sec. III)

#12 Reproducibility: a primer on semantics and implications for research · injunctive

“Data: Analytical data set is available.” (Table 4, “Requirement” column) (p. 46)

#14 An empirical analysis of journal policy effectiveness for computational reproducibility · injunctive

“All data necessary to understand, assess, and extend the conclusions of the manuscript must be available to any reader of Science.” (p.2584)

#15 Coding be eR: assessing and improving the reproducibility of R-based research with containR · tacit

“many datasets were simply missing the data used for analysis, highlighting the need for a better system of documenting and including research-code dependencies” (Abstract, p.iii)

#17 Transparency, reproducibility, and the credibility of economics research · injunctive

“Authors of accepted papers… must provide to the Review, prior to publication, the data, programs, and other details of the computations sufficient to permit replication.” (p. 934) and “…data must be posted to a trusted repository, and reported analyses will be reproduced independently prior to publication” (p. 964)

#18 Reproducibility confidentiality data access · injunctive

“Accessible data” / “Available without any additional information or help from the authors” (Slide: “Key attributes of a replication archive”) and “Systematically require that data be in a trusted repository” (Slide: “Moving forward in economics”)

#19 The ReproRubric: evaluation criteria for the reproducibility of computational analyses · injunctive

Raw data generated during the project is shared; Materials/code are assigned a persistent ID and deposited in a trusted repository (Fig. 2, p.3)

#20 Successes and struggles with computational reproducibility: lessons from the fragile families challenge · injunctive

“authors were not supposed to include the Challenge data file in their computational reproducibility materials” (footnote 13, p.7)

#25 Toward computational reproducibility: a doctoral student’s story of passing the baton · injunctive

“To test CR requires access to the raw data and code underpinning the original analysis.” (p. 688)

#28 A synthetic dataset primer for the biobehavioural sciences to promote reproducibility and hypothesis generation · tacit

“Data and analysis scripts are available at the article’s Open Science Framework webpage https://osf.io/z524n/” (p.9)

#29 Journal data policies: exploring how the understanding of editors and authors corresponds to the policies themselves · descriptive

“Many of the requirements imposed by the policies we studied included a data sharing requirement alone, which is insufficient without accompanying code and other related materials to yield reproducible research.” (p. 11)

Note: This paper states that while sampled journal policies often impose the requirement of sharing dataset, this is often inadequate without the corresponding codes – amplifying the importance of code deposit.

#30 Advancing computational reproducibility in the dataverse data repository platform · descriptive

“Publishing data and code as a replication package in a data repository is considered to be a best practice for enabling research reproducibility and transparency” (p.1, Introduction)

#32 Beyond open data: a model for linking digital artifacts to enable reproducibility of scientific claims · injunctive

“Data-derived knowledge, represented by claims in the scholarly record, must persistently link to immutable versions of the digital artifacts from which it was derived, including 1) any data” (p. 9)

#34 Reproducibility and replicability in economics · injunctive

“Typically, they required that data and code be submitted to the journal, for publication as ‘supplementary materials.’” (p. 8)

#35 Trust, but verify: an investigation of methods of verification and dissemination of computational research artifacts for transparency and reproducibility · injunctive

“Two initiatives (AJPS, IS) require authors to deposit materials in a specific archival repository… Two initiatives (Biostatistics, SC) encourage the use of general archival repositories” (p.109–110)

#37 The four possibilities of reproducible scientific computations · injunctive

“the possibility to inspect all the input data… that can possibly have an impact on the results”

#38 Ten simple rules for writing dockerfiles for reproducible data science · injunctive

“you should publish all files COPYied into the image, e.g., test data or files for software installation from source… in the same public repository as the Dockerfile, e.g., in a research compendium” (p. 16)

#39 How to make a pie: reproducible research for empirical economics and econometrics · injunctive

“any researcher, when acting as a referee, should ask for data and code at the early stages of the publication process so that the article can be evaluated in light of its sources” (p. 1136) and “we believe that scientific journals and institutions financing research… should at least provide clear incentives or, even better, impose minimal mandatory data and code policies for funded research programs” (p. 1136)

#41 Reproducibility in accounting research: views of the research community · injunctive

“Papers are only published if the data used in the analysis are clearly and precisely documented and are readily available to any researcher for purposes of replication.” — AER policy, p. 542

#42 AEA data and code availability policy · injunctive

“Authors of accepted papers that contain empirical work, simulations, or experimental work must provide, prior to acceptance, information about the data, programs, and other details of the computations sufficient to permit replication” (p. 818)

Note: data isn’t required if it’s proprietary

#43 Dawning of a new age? Economics journals’ data policies on the test bench · injunctive

“the dataset(s) used to reach the findings of a paper. Without the data, empirical findings cannot be checked. Therefore, availability of data is essential for reproductions.” (p. 8)

#44 Active maintenance: a proposal for the long-term computational reproducibility of scientific results · tacit

“In three of the 20 studies, raw data and associated cleaning files were not deposited… we did not classify the resulting errors as indications of failed computational replication” (p. 463)

#45 Toward reusable science with readable code and reproducibility · injunctive

“After completing the form, the users upload their data and code.” (p. 4, Section III-B). Also: “leveraging previously published findings and reusing shared research data and code is critical for further scientific inquiries” (p. 1). The RE3 platform requires data upload as part of the workflow: “enables users to upload their data and code, and it incorporates both reproducibility tests and readability ratings.” (p. 3)

#46 Reproducible research in R: a tutorial on how to do the same thing more than once · injunctive

“the sharing of such digital research products is a necessary condition for reproducibility, it is not a sufficient one” (p. 838)

Note: via usethis::use_github()), though the public/private decision is manual

#47 The science of open (communication) science: toward an evidence-driven understanding of quality criteria in communication research · injunctive

“reproducibility can indeed be elusive if analysis materials are not made available” (p. 690)

#48 Challenges of curating for reproducible and FAIR research output · injunctive

“Authors not making the data, software, workflow, or other digital objects available” (p.20)

#51 A reproducible data analysis workflow with R Markdown, Git, Make, and Docker · injunctive

“The workflow ensures meeting the primary goals that… 3) changes at any time… are tracked, tagged, and documented while earlier versions of both data and code remain accessible” (p.1)

#52 CODECHECK: an open science initiative for the independent execution of computations underlying research articles during peer review to improve reproducibility · injunctive

“The author submits their manuscript along with the code and data to the publisher” (Step 1, p. 3)

#54 WORCS: a workflow for open reproducible code in science · injunctive

“the researcher should version control some indelible record of the raw data” (p. 40)

#55 Computational reproducibility: a practical framework for data curators · injunctive

“A copy of the raw, unprocessed data should be shared alongside the code…” (p.8)

#56 Reproducibility badging and definitions · injunctive

“author-created digital objects used in the research (including data and code) are permanently archived in a public repository that assigns a global identifier and guarantees persistence, and are made available via standard open licenses that maximize artifact availability” (p.3)

#58 Improving research quality: the view from the UK reproducibility network institutional leads for research improvement · injunctive

“…such that all stages of the research pipeline (including research design, data collection processes, the resulting datasets, and analysis code etc.) are made openly available in a manner that allows for re-use.” (p.1)

#59 Understanding experiments and research practices for reproducibility: an exploratory study · injunctive

“The survey findings identify a reproducibility crisis and a strong need for sharing data, code, methods, steps, and negative and positive results.” (Abstract)

#60 Reproducibility in the social sciences · descriptive

“one promising alternative is to provide prepublication code and data review. Multiple journals (Christian et al. 2018, Colaresi 2016) now do a simple code review, requiring authors to submit the code and data required to produce key figures and results” (p.78)

#65 Long-term availability of data associated with articles in PLOS ONE · injunctive

“PLOS became one of the first major journal publishers to enact a policy requiring authors to make the data underlying their publications publicly available and to include a Data Availability Statement to enable readers to locate the data” (p. 1)

#66 Reproducible research publication workflow: a canonical workflow framework and FAIR digital object approach to quality research output · injunctive

“compendium containing data, code, software, and documentation” required after conditional acceptance (p.309)

#67 Tools and recommendations for reproducible teaching · injunctive

“Raw data: Keep data in the raw form you find or collect it, and record any steps to process it to prepare it for teaching” (p. 253)

#70 The Stark realities of reproducible statistically orientated sociological research: Some newer rules of the sociological method · injunctive

“If you did not make your data available (and a law like FERPA or HIPPA doesn’t prevent it), fail.” (Table 1, p.210)

#71 Teaching for Large-Scale Reproducibility Verification · injunctive

“the authors of conditionally accepted manuscripts are required to submit a replication package, consisting of all code used to process and analyze data, any data not available in an existing trusted repository, and a ‘README’” (p.276) and “publish papers only if the data and code used in the analysis are clearly and precisely documented and access to the data and code is non-exclusive to the authors” (p.276)

#72 Care to share? Experimental evidence on code sharing behavior in the social sciences · injunctive

“In practice, reproductions hinge on the availability of an original study’s code and data. Performing a replication solely based on the body of an article is tedious at best and impossible at worst.” (p. 2)

#73 A backend platform for supporting the reproducibility of computational experiments · injunctive

“for E17, the necessary dependencies have been installed, but the data needed to run the experiment is not available.” (p. 14)

#75 Reinforcing reproducibility and replicability: an introduction · descriptive

“increasingly more stringent journal policies surrounding data and code availability, including active review and verification of replication packages” (p. 2)

#76 Making provenance work for you · tacit

“‘data’ - a directory containing copies of input and output files, URLs, plots created, and snapshot files” (p. 146); “rdtLite can save the values of larger containers in separate snapshot files” (p. 145).

#77 Report of the AEA data editor · injunctive

“authors can be compliant with the policy without providing a copy of data used, as long as the reason for the inability to provide the data is acceptable, correct, and documented” (p.855)

Note: Here, data deposit can be ruled out with certain conditions

#79 A review of containerization for interactive and reproducible analysis · injunctive

“it is now commonplace for underlying analysis code and data to be proffered alongside journal publications… most of the journals sponsored by the International Statistical Association and American Statistical Association require data and code be posted along with analysis” (p.2)

#80 Replication of the morning after: report from the nottingham replication games · injunctive

“JOP requires authors to provide data and code to reproduce all research results, which are uploaded on JOP’s Dataverse.” (p. 3)

#81 statacons: An SCons-based build tool for Stata · descriptive

“Public posting of data and code… are all becoming more common, both in general and with Stata users specifically” (p.148/149)

#82 Supporting reproducible research · injunctive

“Reproducibility thus provides one of the strongest justifications for open scholarly communication practices such as data and code sharing… Reproducibility requires that we adopt open and transparent methods, not just as an abstract good but as a fundamental part of research practice.” (p. 172)

#83 Reproducibility and transparency versus privacy and confidentiality: Reflections from a data editor · injunctive

“Authors of accepted papers that contain empirical work, simulations, or experimental work must provide, prior to acceptance, the data, programs, and other details of the computations sufficient to permit replication, as well as information about access to data and programs.” (slide “Key elements”) and “If data or programs cannot be published in an openly accessible trusted data repository, authors must commit to preserving data and code for a period of no less than five years following publication…and to providing reasonable assistance to requests for clarification and replication.” (slide “Key elements,” policy excerpt)

Note: data isn’t required if it’s proprietary and suggested another strategy of preservation for authors to follow

#84 Replications in agricultural economics · descriptive

“The practice of making code and data available has numerous advantages for the scientific community, in addition to allowing for an easy reproduction of the published results.” (p. 357)

#85 The Role of Third-Party Verification in Research Reproducibility · injunctive

“checking whether the submitted material (all code, data, readme file, manuscript) complies with a set of guidelines” (p. 3–4)

#86 What makes computational communication science (ir)reproducible? · injunctive

p.6: “(2) No shared data”; p.6: “Criterion 2 determines whether we can conduct the same analysis on the same data.”

#87 Measuring transparency in the social sciences: political science and international relations · injunctive

“a commitment by journal editor signatories to increase the availability of data ‘at the time of publication through a trusted digital repository’” (p. 3)

#88 Reproducibility in Management Science · injunctive

Journal policy: “Authors of accepted papers…must provide…the data, programs, and other details of the experiment and computations sufficient to permit replication.” (p. 1344)

#89 Computational reproducibility in computational social science · injunctive

“researchers should consider the option to archive their data…in a secure data archive that will ensure the long-term accessibility of data for other researchers” (p.6)

#90 A guide for social science journal editors on easing into open science · injunctive

“journals could instead implement policies to make acceptance conditional upon direct links to open data, code, and materials” (p. 6)

#91 CORE-bench: fostering the credibility of published research through a computational reproducibility agent benchmark · tacit

Capsule folders: “data — Folder with data necessary to run the code,” “code — …Script to run code with appropriate arguments” (p. 3, Fig. 2)

#93 Unintended code errors and computational reproducibility · injunctive

“Reproducibility package includes all files necessary to reproduce the results” (Appendix A, FILES, p.8)

#94 Navigating the landscape of reproducible research: a predictive modeling approach · descriptive

“Typically, a paper’s reproducibility is inferred based on the availability of artifacts such as code, data, or supplemental information, often without extensive empirical investigation.” (Abstract)

#95 Computational reproducibility in finance: evidence from 1,000 tests · injunctive

“For each sharable data set, mention whether it is directly included in the replication kit or available elsewhere… For each nonsharable data set… provide the following relevant information on how to obtain it” (p. 3589, Guideline R1)

#97 Why and how we share reproducible research at yale university’s institution for social and policy studies · injunctive

“ISPS invites affiliated researchers to deposit data, computer code, and associated materials” (p.3)

#98 Mass reproducibility and replicability: a new hope · injunctive

“a rating of 1 signifies the incapacity to reproduce results due to the absence of data or code” (p. 22); “we thus see a lack of raw data provided in a replication package as a significant barrier to reproducibility and replicability” (p. 32)

#99 Introducing reproducible research standards at the world bank · injunctive

“The public package includes … all data that can be redistributed … and a README” (p.4)

#100 Computational and robustness reproducibility of “UN peacekeeping and democratization in conflict-affected countries” · tacit

“only the analysis data were included” re: replication package (p.4); Table 1: raw data = No, analysis data = Fully (p.13)

#101 Replicability of simulation studies for the investigation of statistical methods: the RepliSims project · injunctive

“providing open data and code poses no ethical barriers and thus should be required for all published simulation studies” (p.3); reinforced: “Journals that publish simulation studies should require code and data to be publicly available” (p.14)

#103 The Challenges of Replicating Volatile Platform-Data Studies: Replicating Schatto-Eckrodt et al. (2020) · injunctive

“One crucial step behind every reproduction is the reacquisition of the data from the original study… If a study’s original data are not available to reproducers, initial sample data… must be re‑collected, which frequently involves substantial initial expenditures… as exact recollection of primary data is not guaranteed.” (p. 3)

#104 Reproduce to validate: a comprehensive study on the reproducibility of economics research · injunctive

“Data availability policies require authors to make available all data … necessary to reproduce their study” (p.963)

#105 Promoting reproducibility and replicability in political science · injunctive

“they ensure that the data and codes necessary for a computational reproducibility are available” (p.3)

#106 The data availability policies of political science journals · injunctive

“journals must (continue to) adopt and enforce standards that ensure researchers share the data and code necessary [to] reproduce their results” (p. 6)

#107 Teaching constructive replications in the behavioral and social sciences using quantitative data · tacit

“Particularly at the bachelor level, replication courses will be more successful when focusing on studies where data and code are available.” (p. 118, left col) and “Prescreened, available data is often best because seeking to acquire data directly from authors can take considerable time and may be a dead end.” (p. 119)

#108 Checklists for computational reproducibility in social sciences: insights from literature and survey evaluation · injunctive

“Providing sample input and output data to replicate for proof of concept” (Checklist 2)

#109 A protocol for structured robustness reproductions and replicability assessments · injunctive

“The precondition to conduct reproductions is the availability of data and code.” (p. 3); “replicators distinguish between the availability of raw data and analysis data.” (p. 3); Table 1 Appendix (p. 9): “Raw data provided” and “Analysis data provided” are explicit checklist items.

Note: Can be analytic or raw data, both must be present for a full reproduction.

#110 Data and code availability in political science publications from 1995 to 2022 · injunctive

“the APSA has a formal code of ethics that requires sharing data and/or code to reproduce the results (absent special circumstances)” (p. 342); King’s “researchers should share data and code with a professional archive… it should be made publicly available” (p. 341)

#113 Let’s talk about it: making scientific computational reproducibility easier · injunctive

“The integrity and reproducibility of research depend on the availability and accessibility of key artifacts like code, datasets, and dependencies.” (p. 3)

#117 Computational reproducibility of R code supplements on OSF · injunctive

“A fundamental aspect in achieving computational reproducibility is the provision of all necessary materials, including the raw data and the code used to generate the results.” (p.1)

#118 Tooling for reproducible research: considerations for the past and future of data analysis · descriptive

“many scientific journals… changed their article submission policies to encourage or require the submission of data and code along with their manuscripts” (p. 2)

#120 Reproducibility spectrum for applied social sciences: conceptual taxonomy and recommended tools · tacit

“Share data on OSF” listed under Platforms × Data; “upload to OSF” as step 1 of the ARTE progression

#121 Statistical perspectives on reproducibility: definitions and challenges · injunctive

“the requirement for reproducible research is that the documentation, data and code used for the analysis are available to others, so that they can verify the published results” (p. 5–6)

#124 Perceived problems, causes, and solutions of finance research reproducibility and replicability: a pre-registered report · descriptive

“One response to such concerns has been code and data sharing policies of journals that aim to improve transparency and provide third parties with the ability to reproduce the results using the exact same and applying the same method through code (Nagel, 2018).” (p. 1)

#125 A report on “corruption and Co-optation in autocracy: evidence from Russia” · tacit

“Raw data provided ✓ [Fully],” “Analysis data provided ✓ [Fully]” (Table 1, p.5)

#128 The reliability of replications: a study in computational reproductions · tacit

the paper uses “freely available data and code” as a criterion for selecting the Brady & Finnigan target paper (p. 7: “it is highly cited, offers freely available data and code”), and notes that inaccessible data introduces error (p. 19), but never explicitly states data deposit as a rule.

#130 Reproducibility in the classroom · tacit

“Are code and data both readily available?” (p.10)

#131 Guide for Accelerating Computational Reproducibility in the Social Sciences · injunctive

Level 1 (L1): No data or code are available. Possible improvements include adding: raw data, analysis data, cleaning code, and analysis code. You may have detected papers without any reproduction materials at the Paper Selection stage, where you should have recorded them as abandoned candidate papers.

Level 2 (L2): Code scripts are available (partial or complete), but no data are available. Possible improvements include adding: raw data and analysis data.

Level 3 (L3): Analytic data and code are partially available, but raw data and cleaning code are missing. Possible improvements include: completing analysis data and/or code, adding raw data, and adding analysis code.

Level 4 (L4): All analytic data sets and analysis code are available, but the code fails to run or produces results inconsistent with the paper (not CRA). Possible improvements include: debugging the analysis code or obtaining raw data.

Level 5 (L5): Analytic data sets and analysis code are available and they produce the same results as presented in the paper (CRA). The reproducibility package may be improved by obtaining the original raw data.

Note: This is the highest level that most published research papers can attain currently. Computational reproducibility from raw data is required for papers that are reproducible at Level 6 and above.

Level 6 (L6): Cleaning code scripts are available (partial or complete), but raw data is missing. Possible improvements include: adding raw data.

Level 7 (L7): Cleaning code is available and complete, and raw data is partially available. Possible improvements: adding raw data.

Level 8 (L8): All the materials (raw data, analytic data, cleaning code, and analysis code) are available. However, the cleaning code fails to run or produces different results from those presented in the paper (not CRR) or the analysis code fails to run or produces results inconsistent with the paper (not CRA). Possible improvements: debugging the cleaning or analysis code.

Level 9 (L9): All the materials (raw data, analytic data, cleaning code, and analysis code) are available. The analysis code produces the same output as presented in the paper (CRA). However, the cleaning code fails to run or produces different results from those presented in the paper (not CRR). Possible improvements: debugging the cleaning code.

Level 10 (L10): All necessary materials are available and produce consistent results with those presented in the paper. The reproduction involves minimal effort and can be conducted starting from the analytic data ( CRA) and the raw data (CRR). Note that Level 10 is aspirational and may be unattainable for most research published today.

Note: Can be analytic or raw data. If raw data is submitted, then cleaning codes must be provided.

#132 Long-term effects of the targeting the ultra poor program—a reproducibility and replicability assessment of banerjee et al. (2021) · tacit

“BDS meet all criteria by providing analysis code, data, cleaning code, and raw data” (p. 4–5)

#135 Automating computational reproducibility in social science: comparing prompt-based and agent-based approaches · injunctive

“all necessary code and data must be equally accessible” (p. 3)

#137 Overcoming barriers to computational reproducibility · injunctive

“The code and data supplement should contain all materials necessary to independently reproduce the results… This typically includes the complete analytical code, the relevant dataset or datasets (where possible)” (p.7)

#138 Reproducibility and robustness of economics and political science research · injunctive

“Each of these journals has a data and code availability policy that requires authors to publicly share their materials upon publication” (p.151)

#140 How to Get There From Here? Barriers and Enablers on the Road Towards Reproducibility in Research · injunctive

“ML researchers noted that industry-funded research also presents a barrier, in that sharing of data, software and other materials is often not allowed by industry partners for reasons of competition. Researchers working in such collaborations would not be able to create reproducible research.”

#141 Understanding cultural persistence and change: A replication of Giuliano and Nunn (2021) · injunctive

“it does not include the raw data, the code necessary to transition from the raw data to the various estimation samples” (p. 23); “Comprehensive replication packages, including all necessary files, can greatly facilitate verification efforts” (p. 34)

#142 Teaching Reproducibility and Replicability While Teaching Econometrics in the Classroom · injunctive

“Not only must the code and data be available, but the documentation of how to replicate their analyses must be included.” (p. 116)

#143 What is Computational Reproducibility? · injunctive

“Whenever feasible, openly sharing data, code, and configurations of computational environments enables the verification and reuse of research by other researchers.” (p. 6) and “Ensuring that the datasets are available and can be processed in the same way as the original study did is crucial for reproducibility.” (p. 5).

#144 Investigating the reproducibility of the social and behavioural sciences · injunctive

“The most substantial barrier to observing reproducibility in our sample of social and behavioural science papers was the unavailability of author data, preventing reproduction attempts.” (p.131, Discussion)