Documentation
Tier 1 · Available - Bronze core endorsement
Rating
A README file and a codebook are present. A README file contains at least: 1) a list and descriptions of all files in the replication package; 2) instructions on how to reproduce the results. A codebook should minimally contain: 1) variable labels and descriptions of what those mean; and 2) allowed values of each variable in the dataset.
Supported by: #42 README lists all files + purpose/format/provenance + how to replicate; codebook gives allowed values per variable. #48 codebook: all variables, definitions, values, value labels. #95 D3 codebook. #70 ‘no adequate data dictionary -> fail’. #55/#67/#137 README explains structure and contents. #19, #57, #69, #120, #97.
Either README or codebook is present, or both are present but their elements required in the “criterion_met” are incomplete.
Supported by: #14 (91% ‘data documented’, 57% ‘software documented’), #88 ‘insufficient documentation, missing information’, #133 ‘insufficient detail in README files’, #30 README present but context missing.
Never. When a replication package is a simulation study (C1), evaluate this criteria on the README alone.
Prerequisites
Check order: 0
Builds on
No core prerequisites.
Prerequisite for
- Dependency resolution (2 sources)
- Coding error handling (2 sources)
- Computing environment (2 sources)
- Numerical output match (2 sources)
- Table and figure match (2 sources)
- Path resolution (1 source)
- Workflow/reproduction tree (1 source)
In the corpus
| Sources | Injunctive | Descriptive | Tacit | Automatable | Manual check | Automation unclear | LLM sources |
|---|---|---|---|---|---|---|---|
| 83 | 77.1% | 3.6% | 19.3% | 13.3% | 7.2% | 79.5% | 1 |
Text evidence (83 quotes)
#1 An introduction to docker for reproducible research · injunctive
“Documentation for complex software environments is stuck between two opposing demands… to make things easier on novice users, documentation must explain details relevant to factors like different operating systems.” (p. 3)
#3 Reproducing statistical results · injunctive
“Complete statements of or references to every algorithm employed… Salient details of software used in the computation… Salient details of data reduction and statistical analysis methods… Instructions for repeating computations described in the paper” (p. 11)
#4 Storing reproducible results from computational experiments using scientific python packages · injunctive
“The lack of reproducibility of computational results is oftentimes attributed to missing information about critical computational details such as library versions, parameter values, or precise descriptions of the exact code that was run” (p.46)
#6 A manifesto for data sharing in social media research · injunctive
“Detailed documentation of the dataset and the provision of code and syntax allow everyone to check how collection, cleaning and analysis were performed.” (p.168, Sec 3.1) and A researcher “could mostly fulfil the reproducibility requirements by sharing the tweet IDs at a conference website or in a repository, making available code for accessing, cleaning, processing and analyzing the data in a repository… and explain her/his methodology in as much detail as possible” (p.169, Sec 3.3.2)
#7 Tools and techniques for computational reproducibility · injunctive
“narratives should indicate the exact software versions used, the order in which they were executed, and all non-default parameters that were specified” (p.2–3)
#9 Using ReproZip for reproducibility and library services · injunctive
“the configuration file…contains all of the required information for reproducing the work including commands, arguments, working directory, environment variables, operating system information, input files, output files, and software packages.” (pp.3–4)
#12 Reproducibility: a primer on semantics and implications for research · injunctive
“Documentation: Adequate documentation of the computer code, software environment, and analytical data set is available to enable others to repeat the analyses and to conduct other similar analyses.” (Table 4) (p. 46)
#14 An empirical analysis of journal policy effectiveness for computational reproducibility · tacit
“Data documented to clearly explain what each part represents” — 91% compliant; “Software documented well enough to run it and what it ought to do” — 57% compliant (Table 3, p.2586)
#17 Transparency, reproducibility, and the credibility of economics research · injunctive
“…the data used in the analysis are clearly and precisely documented and are readily available…” (p. 934) and “The data that was submitted was often an unlabeled and undocumented mess, a problem that has persisted with recent data-sharing policies…” (p. 934)
#18 Reproducibility confidentiality data access · injunctive
“Accompanying documentation is needs to detail creation of results” (Slide: “Key insight,” Lagoze & Vilhuber 2017)
#19 The ReproRubric: evaluation criteria for the reproducibility of computational analyses · injunctive
Materials/code are accompanied by rich metadata as well as a codebook or data dictionary; Code is well documented (Fig. 2, p.3)
#20 Successes and struggles with computational reproducibility: lessons from the fragile families challenge · injunctive
“we recommend that authors produce code headers that contain the following information: Purpose… Inputs… Outputs… Machine used… Expected runtime” (p.11); “We recommended that authors include a file listing all of the software and add-on packages used, including the exact version numbers” (p.11)
#23 Creating and sharing reproducible research code the workflowr way · injunctive
“It records information about the computing environment, including the operating system, the version of R used, and the packages that were used to produce the results.” (p. 5).
#25 Toward computational reproducibility: a doctoral student’s story of passing the baton · injunctive
“It is also critical to document the software environment and versions to ensure reusability and reproducibility of research results.” (p.689)
#26 Facilitating reproducible project management and manuscript development in team science: the projects R package · tacit
“a call to the projects function save_session_info(), which saves the R session information to a dated file, thereby documenting the crucial information necessary to reproduce the script” (p.5)
#30 Advancing computational reproducibility in the dataverse data repository platform · injunctive
“Even when this information is documented by the original authors in an instructions file (like readme), contextual information still might be missing, which could make the process of research verification and reuse hard or impossible” (p.1, Introduction)
#31 Reproducibility, preservation, and access to research with ReproZip and ReproServer · tacit
“it provides a configuration file that is machine-readable and lists all the information about the computational environment and dependencies necessary to recreate it” (p. 4–5)
#33 Documenting computing environments for reproducible experiments · injunctive
“authors must also share a description of computing environments” (p. 756)
Note: via automated trace log
#35 Trust, but verify: an investigation of methods of verification and dissemination of computational research artifacts for transparency and reproducibility · injunctive
“Three initiatives (AEA, AJPS, JASA-ACS) provide specific guidelines for data documentation, including codebooks or similar metadata, as well as requirements for accurate variable labeling” (p.139)
#38 Ten simple rules for writing dockerfiles for reproducible data science · injunctive
“It is often helpful to provide usage instructions, i.e., how to docker build and docker run the image, within the Dockerfile” (p. 16)
#39 How to make a pie: reproducible research for empirical economics and econometrics · injunctive
“We also suggest, as a good practice, the inclusion of a specific section in the final publication describing the computing environment” (p. 1142) and “it is useful to indicate, at the top of any program, the information needed to understand it, for example, the date, description, goal of the code, version… input (and output) data files, input parameters, the version of the software… packages used (and their version)…” (p. 1151)
#42 AEA data and code availability policy · injunctive
“Authors of accepted papers that contain empirical work, simulations, or experimental work must provide, prior to acceptance, information about the data, programs, and other details of the computations sufficient to permit replication” (p. 818) and “authors must provide a README file listing all included files and documenting the purpose, format, and provenance of each file provided, as well as instructing a user on how replication can be conducted” (p. 819) and “Codebooks or similar metadata should describe the allowed values and their meaning for each variable.” (p. 819)
#43 Dawning of a new age? Economics journals’ data policies on the test bench · injunctive
“descriptions of the data (e.g. data dictionary, codebooks, documentations) and/or of the entire research process (e.g. in readme-file and/or by including the instructions of experiments). Without descriptions of the data, it often is very difficult to make use of the data.” (p. 8)
#44 Active maintenance: a proposal for the long-term computational reproducibility of scientific results · injunctive
“‘clean,’ well-documented code with proper metadata will be easier to maintain and should be a matter of routine” (p. 465)
#45 Toward reusable science with readable code and reproducibility · tacit
“we ask the researcher about the used R version” (p. 4); “A quick text-parsing analysis revealed that the used R version was documented in deposited files in the format ‘R x.y.z’ in 44 out of 207 (21%) replication packages. Considering that the R version is essential for successfully re-executing old code files, this outcome is quite low. In the RE3 platform, we directly ask the depositor to specify the version as part of the submission system.” (p. 5–6)
#46 Reproducible research in R: a tutorial on how to do the same thing more than once · descriptive
“In practice, dependencies between project files (e.g., information on what script uses which data file and what script needs to be run first) or on external software (e.g., system libraries or components of the programming language, such as other R packages) are frequently unmentioned or not exhaustively and unambiguously documented” (p. 840)
#47 The science of open (communication) science: toward an evidence-driven understanding of quality criteria in communication research · injunctive
“While reproducibility can indeed be elusive if analysis materials are not made available or if data are not prepared for third-party use (e.g., when they lack annotations or clear variable labels)” (p. 690)
#48 Challenges of curating for reproducible and FAIR research output · injunctive
“We required the author to specify in their README that secondary users wishing to verify the results themselves had to use the specific version of that package, otherwise the code would fail” (p. 11) and “The Codebook provides context for the analysis data and must include all variables, variable definitions, values and value labels.” (p.11)
#52 CODECHECK: an open science initiative for the independent execution of computations underlying research articles during peer review to improve reproducibility · injunctive
“Ideally, the author is expecting the CODECHECK and prepares for it… providing a set of instructions on how to re-run the computational workflow.” (p. 3–4)
#54 WORCS: a workflow for open reproducible code in science · injunctive
“we recommend users to elaborate on the default codebook by adding variable description and categories” (p. 44)
#55 Computational reproducibility: a practical framework for data curators · injunctive
“an organized file directory should always contain a readme file… that explains the sub-directory structure, file contents, and processes” (p.7)
#56 Reproducibility badging and definitions · injunctive
“these research objects should contain standard and complete metadata” (p.5)
#57 Principles for data analysis workflows · injunctive
“researchers should confirm that it includes descriptive README files and field descriptions and also ensure that all abbreviations and coded entries are defined” (p.18) and “this includes documenting the specific version of any software that we used, as well as its relevant dependencies and compatibility constraints” (p.13)
#58 Improving research quality: the view from the UK reproducibility network institutional leads for research improvement · injunctive
“Even when data are made available, they can be unusable due to a lack of meta-data and accompanying executable analysis code” (p.2)
#59 Understanding experiments and research practices for reproducibility: an exploratory study · injunctive
“Every small detail of the experiment must be documented in order to repeat an experiment (Ioannidis et al., 2009; Kaiser, 2015).” (p.18) and “According to our survey, scientists consider sharing metadata and a clear description of raw data, negative results, measurements, settings, experimental setup, people involved, software parameters, methods, steps, and results very important to reproduce published results.” (p.18)
#61 A collaborative semantic-based provenance management platform for reproducibility · injunctive
“The lack of documentation of computational experiments along with their results and the ability to reuse parts of code are some of the issues hindering reproducibility in script-based environments” (p.4)
#62 Containerization for creating reusable model code · injunctive
“we recommend making dependencies and requirements explicit as the model is developed… This task can be facilitated by keeping a modelling notebook, analogous to a laboratory notebook, that follows the TRACE structure” (~p.6)
#63 Toward practical transparent verifiable and long-term reproducible research using guix · injunctive
“While names and version labels are a step toward transparency, they are insufficient to allow any future user to rebuild the computational environment on which these software are running.” (p.1); and p.3: “describing the version of source code is already better than nothing, but it does not provide all the useful information for redeploying the computational environment”
#66 Reproducible research publication workflow: a canonical workflow framework and FAIR digital object approach to quality research output · injunctive
“compendium containing data, code, software, and documentation” required after conditional acceptance (p.309) and “such as a codebook, README, and other supporting documentation and metadata” (p.307)
#67 Tools and recommendations for reproducible teaching · injunctive
“Data files used in teaching materials should be documented to reflect both the contents of the datasets and their provenance” (p. 256); “The course folder as well as each of its top level sub-folders should have a README file that explains what is included in that folder” (p. 256); “A simple solution for software documentation is to include the list of packages being used and their version numbers as well as the version of R” (p. 256)
#69 The interdependence of data producers and data users: how researchers’ behaviors can support or hinder each other · injunctive
“Provide metadata with detailed descriptions of the contents of the data source, including definitions associated individual variables and the scope of the data” (Table 2, p.22–23)
#70 The Stark realities of reproducible statistically orientated sociological research: Some newer rules of the sociological method · injunctive
“If you did not provide an adequate data dictionary, fail.” (Table 1, p.210)
#71 Teaching for Large-Scale Reproducibility Verification · injunctive
“the authors of conditionally accepted manuscripts are required to submit a replication package, consisting of all code used to process and analyze data, any data not available in an existing trusted repository, and a ‘README’” (p.276)
#73 A backend platform for supporting the reproducibility of computational experiments · injunctive
“Making available all the information to allow computational reproducibility is essential to reproduce the same results; such information includes the values of the parameters used in the execution of the experiment.” (p. 16)
#75 Reinforcing reproducibility and replicability: an introduction · injunctive
“A key component of the current landscape, and what will be a recurring topic in this column, are ‘replication packages,’ which here are defined as those materials (data, computer code, and instructions) linked to a specific publication that facilitate the replication of the manuscript’s results by others, but should also be computationally reproducible.” (p. 2)
#76 Making provenance work for you · tacit
“The ENVIRONMENT section shows… what version of R was used, what hardware and operating system were used…” (p. 145); framed as valuable “when including a script and its results in a paper, or when sharing a script with a colleague” (p. 145)
#77 Report of the AEA data editor · injunctive
“We now require that authors provide the information as per the Social Science Data Editors’ template ‘README’” (p.852)
#83 Reproducibility and transparency versus privacy and confidentiality: Reflections from a data editor · injunctive
“…papers only if the data used in the analysis are clearly and precisely documented…” (slide “Key elements”)
#85 The Role of Third-Party Verification in Research Reproducibility · injunctive
“checking whether the submitted material (all code, data, readme file, manuscript) complies with a set of guidelines” (p. 3–4)
#88 Reproducibility in Management Science · tacit
“Insufficient documentation, missing information” (7.4%) (p. 1350); also “information about the content of the replication package (readme, data, code, etc.) and their quality” was assessed by reviewers (p. 1347)
#89 Computational reproducibility in computational social science · injunctive
“3°CR can be achieved by following best practice guides to write reproducible code, using a research compendium, and by providing a declarative description of the computational environment” (p.8)
#91 CORE-bench: fostering the credibility of published research through a computational reproducibility agent benchmark · descriptive
“Contains a README file… most papers in the real world do [have one]” (p. 5, Table 2)
#92 Audio-as-data tools: replicating computational data processing · injunctive
“Researchers must also be transparent in their processing approach, including providing information about the tools they used, the version of that tool (if relevant), and the specific steps they conducted in data preparation” (p.12); “reproduction studies should use the same processing approaches, if possible… being explicit about the specific tool, software, programming language, or package/library used” (p.12)
#93 Unintended code errors and computational reproducibility · injunctive
“Descriptive information about computing environment is present” / “…installation of dependencies is present” (Appendix A, DOCUMENTATION, p.8)
#95 Computational reproducibility in finance: evidence from 1,000 tests · injunctive
Guideline R3: “Provide information on any required software (and its version), packages/libraries (and their versions)… hardware specifications… runtime” (p. 3589); Guideline D3: “Provide a variable dictionary (codebook)… Describe all variables” (p. 3589)
#97 Why and how we share reproducible research at yale university’s institution for social and policy studies · descriptive
“better documentation of data (provision of a README file or a codebook)” (p.4–5) and “ISPS publishes a curator README document as part of the replication package” (p.4)
#99 Introducing reproducible research standards at the world bank · injunctive
“reproducibility packages using only Excel [are accepted] if … there is sufficient step-by-step documentation in the README for the reviewer to reproduce the exact results” (p.4)
#101 Replicability of simulation studies for the investigation of statistical methods: the RepliSims project · injunctive
“Sharing of simulation code ideally gives readers access to details regarding simulation parameter settings, coding environments and dependencies (including their versions), random-number generator seeds, and implementation of algorithms for data generation” (p.14); also “reporting software-specific features such as definitions of improper solutions and version numbers facilitates assessment of reproduction” (p.14)
#102 In-Class Data Analysis Replications: Teaching Students While Testing Science · tacit
“missing details about the parameters used in the modeling (e.g., size of the random forest model), little information on the data preprocessing steps… explicit mistakes of the authors in reporting the method details” (p. 12–13)
#103 The Challenges of Replicating Volatile Platform-Data Studies: Replicating Schatto-Eckrodt et al. (2020) · tacit
“The shared materials comprised individual files for each analysis, a README file with usage instructions, an R session info file, and all shareable data compliant with X’s terms of service.” (p. 5)
#104 Reproduce to validate: a comprehensive study on the reproducibility of economics research · injunctive
“if the data used in the analysis are clearly and precisely documented and are readily available” (p.964, quoting AEA 2008)
#106 The data availability policies of political science journals · injunctive
“this archive should usually include the data and computer code to reproduce the results reported in the article and a README explaining the details” (p. 1)
#107 Teaching constructive replications in the behavioral and social sciences using quantitative data · tacit
“Transparent documentation of materials and construction of a reusable data package including a codebook and well-commented code are also excellent grading possibilities.” (p. 120)
#108 Checklists for computational reproducibility in social sciences: insights from literature and survey evaluation · injunctive
“a well-written README… should provide all necessary details to recreate the environment and reproduce results” (Checklist 3) and”Documenting the research setup and model configurations” (Checklist 2)
#113 Let’s talk about it: making scientific computational reproducibility easier · injunctive
“The integrity and reproducibility of research depend on the availability and accessibility of key artifacts like code, datasets, and dependencies. These artifacts must be well-documented, easily accessible, and replicable across different environments.” (p. 3)
#117 Computational reproducibility of R code supplements on OSF · injunctive
“This includes explicitly listing all software dependencies and their versions, using files like requirements.txt or including detailed information about their used environment using commands such as sessionInfo() in R.” (p. 2)
#119 “Works on my machine”: a case study of replicability challenges inComputational humanities research · injunctive
Projects which do not precisely describe their computational environment may therefore cease to function in the near future (p.143)
#120 Reproducibility spectrum for applied social sciences: conceptual taxonomy and recommended tools · tacit
“Write READMEs and data dictionaries”; “Create data dictionaries. Include metadata (variable definitions, units, collection methods)”
#121 Statistical perspectives on reproducibility: definitions and challenges · injunctive
“Donoho [50] advised to create a single R script that generates all the results, figures and tables, for a particular paper” (p. 16) and “Schwab et al. [131] described ReDoc, a simple software system where authors deposit all the documentation, data, and code, that allows readers to reproduce computational results” (p. 16–17)
#126 Automated Reproducibility Testing in R Markdown · tacit
“The metadata contains information about the R version used to generate the metadata, so future versions of our package could easily be adapted to this change.” (p. 8)
#128 The reliability of replications: a study in computational reproductions · injunctive
“Gary King refers to provision of original code as the most elegant way to engage in reproducible research. Our results demonstrate that truly elegant code should be well-structured, well-documented and comprehensible for researchers who do not use the software.” (p. 19, Discussion)
#130 Reproducibility in the classroom · tacit
“students are also explicitly taught on reproducibility practices and evaluated on these… naming files…, writing meaningful commit messages, saving session information (e.g., operating system information and versions of R packages), writing informative README files” (p.12–13)
#131 Guide for Accelerating Computational Reproducibility in the Social Sciences · tacit
“A reproduction package is the collection of materials that make it possible to reproduce a paper. This package may contain data, code, or documentation.” or “Separate your reproduction package in two: (1) data and (2) code and documentation.”
#132 Long-term effects of the targeting the ultra poor program—a reproducibility and replicability assessment of banerjee et al. (2021) · tacit
“the authors provide all questionnaires, a read-me document…” (p. 4–5)
#133 From guidelines to practice: evaluating the reproducibility of methods in computational social science · tacit
“Participants reported unclear distinctions between mandatory and optional steps, insufficient detail in README files, and missing guidance for environment setup and file handling.” (4.2.1)
#136 Artisan: agentic artifact evaluation · injunctive
“the instructions for reproducibility are clearly documented in the README. If this condition is met, the reproduction task becomes significantly easier” (p. 13, Section 4.2.2). Also: the artifact download mechanism “locates the README file by searching for files whose names begin with ‘README’” (p. 6).
#137 Overcoming barriers to computational reproducibility · injunctive
“A README file is of central importance. It should give a brief overview of all files… For long-term reproducibility, the software environment must also be clearly documented, including the versions of all additional packages used.” (p.7)
#138 Reproducibility and robustness of economics and political science research · injunctive
“Data editors ensure that replication packages include data and codes, and that the documentation (for example, a readme file) is complete.” (Methods, “Journal policy”)
#139 PRIMAD-LID: a developed framework for computational reproducibility · injunctive
data documentation includes “exact parameter and configuration files” (p. 2); “functionality, human roles, licensing, and metadata” (p. 14); and includes “reasoning, decisions, and choices underlying their experiment elements” (p. 14)
#140 How to Get There From Here? Barriers and Enablers on the Road Towards Reproducibility in Research · injunctive
“A particular challenge named in this context was the missing possibility of linking between scientific platforms, partly attributed to a lack of agreements on meta-data standards and guidelines.”
#141 Understanding cultural persistence and change: A replication of Giuliano and Nunn (2021) · tacit
“By closely examining the documentation and the data, we were able to reverse-engineer certain aspects of the data processing, even in the absence of access to the original code.” (p. 23)
#142 Teaching Reproducibility and Replicability While Teaching Econometrics in the Classroom · injunctive
“Not only must the code and data be available, but the documentation of how to replicate their analyses must be included.” (p. 116)
#143 What is Computational Reproducibility? · injunctive
“Research materials should be well-structured and include all necessary components, such as data, code, and documentation.” (p. 6), “Comprehensive documentation of every step of the research process, including data collection, preprocessing, analysis, and interpretation, is vital.” (p. 6) and “Adopting clear and consistent naming conventions for files and variables enhances clarity and avoids confusion.” (p. 6).