Computing environment
Tier 2 · Executable - Silver core endorsement
Rating
The README (or a session-info/colophon file) documents the operating system and version, the statistical software or programming language and version, and, where relevant, hardware (CPU, RAM, cores, GPU/HPC) and expected runtime. Alternatively or additionally, the environment is captured so it can be rebuilt (e.g. a Dockerfile or container image with a pinned base, a virtual machine, or a cloud capsule such as Binder or Code Ocean) and it builds and runs on another machine.
Environment information is present but incomplete (e.g. software named without version, operating system missing, no runtime given for long computations); or a container/VM/capsule is provided but cannot be built or run at the point of reproduction.
No information on the computing environment is given and no captured environment is provided.
Based on: #19 three levels (documented with versions / built with a script / VM or container), #95 R3 OS + hardware specs, #14 hardware/system software/processors, #55 in code or README, #71 resources summarized in README, #128 ‘session info’, #40 pinned base image.
Never.
Prerequisites
Check order: 3
Builds on
- Code deposit (7 sources)
- Dependency resolution (7 sources)
- Documentation (2 sources)
- Data deposit (1 source)
- License (1 source)
- Path resolution (1 source)
Prerequisite for
- Numerical output match (5 sources)
- Table and figure match (3 sources)
In the corpus
| Sources | Injunctive | Descriptive | Tacit | Automatable | Manual check | Automation unclear | LLM sources |
|---|---|---|---|---|---|---|---|
| 69 | 58% | 2.9% | 39.1% | 40.6% | 4.3% | 55.1% | 2 |
Text evidence (69 quotes)
#1 An introduction to docker for reproducible research · injunctive
“Docker defines the software environment to a particular operating system and suite of libraries… To address this concern, one must archive a binary copy of the image used at the time the research was first performed.” (p. 4)
#2 Towards reproducibility in scientific workflows: an infrastructure-based approach · injunctive
“The equipment used in a scientific experiment plays a key role on its reproducibility. Without the proper set of tools, it is hard to ensure the execution of the same process obtaining consistent results. In order to guarantee it, we need to document and describe the tools involved (types, brand, provider, version, etc.) and the information for setting it up (calibration, configuration, handling, etc.)” (p.3)
#3 Reproducing statistical results · tacit
“[ResearchCompendia] extends the ideas of Gentleman & Temple Lang (2007) to facilitate reproducibility in computational science by persistently linking the data and code that generated published findings to the article and by executing the code in the cloud to validate or certify those findings” (p. 12)
#5 ReproZip: computational reproducibility with ease · injunctive
“researchers should create a compendium that encompasses all the components that are required to reproduce the experiment’s results, including data, code, software, library dependencies, and information about the source computational environment” (p. 2085)
#7 Tools and techniques for computational reproducibility · injunctive
“virtual machines, which can encapsulate an entire operating system and all software, scripts, code, and data necessary to execute a computational analysis” (p.6–7)
#9 Using ReproZip for reproducibility and library services · tacit
“ReproZip packages are also highly portable, in that it automatically creates a virtual machine for the user…allowing research to be reproduced across different operating systems.” (p.3)
#10 Sustainable computational science: the ReScience initiative · injunctive
“the reviewers [must be] able to run the proposed implementation on their computers” (p.7–8) and “Missing code and/or data, undocumented dependencies, and inaccurate or imprecise description appear to be characteristic of much non-reproducible work” (p.8)
#11 Sciunits: reusable research objects · tacit
“the sciunit… uses application virtualization (AV) to automatically create a container of an executable application… it can be run on any compatible machine without installation, configuration, or root permissions.” (Sec. I)
#12 Reproducibility: a primer on semantics and implications for research · injunctive
“the software environment necessary to execute that code is available [to other investigator teams]” (Table 4, Methods row) (p. 46)
#14 An empirical analysis of journal policy effectiveness for computational reproducibility · injunctive
“Nearly impossible to reproduce (specialized hardware, intense computation requirements, sensitive data, human study, or other unavoidable reasons)” — 14% of sample (p.2587); ICERM item “Salient details of the test environment, including hardware, system software, and number of processors used” — 13% compliant (Table 2, p.2586)
#16 ReproServer: making reproducibility easier and less intensive · tacit
“From the .rpz file, ReproServer builds a Docker image and caches it on a private Docker registry for later execution… reprounzip automatically sets up the environment” (p.3-4)
#17 Transparency, reproducibility, and the credibility of economics research · tacit
“…nonlinear maximization methods from different software packages often produced wildly different estimates [and]… not a single AER article had tested their solution across different software packages…” (p. 934)
#19 The ReproRubric: evaluation criteria for the reproducibility of computational analyses · injunctive
Documentation of the computing environment with versions is provided; The environment is built with a script; A virtual machine, container, or emulated environment is provided (Fig. 2, “Environment” rows, p.3)
#20 Successes and struggles with computational reproducibility: lessons from the fragile families challenge · injunctive
“Containers prevent dependency hell. They do this by making an appropriately configured minicomputing environment…” (p.4)
#23 Creating and sharing reproducible research code the workflowr way · injunctive
“It creates a clean R session for executing the code. This is critical for reproducibility—results should not depend on the current state of the user’s R environment, and all objects necessary to run the code should be defined in the code or loaded by packages.” (p. 5)
#25 Toward computational reproducibility: a doctoral student’s story of passing the baton · tacit
“The majority of errors were caused by discrepancies between operating systems and versions of the software… the initial work was run on Linux with Python 2.7, whereas the replication experiment was conducted in a Windows environment using Python 3.6.” (p.689)
#27 Rebooting reproducibility: From re-execution to replication · injunctive
“Any framework that wants to open up the processes left of the line must account for access to potentially massive data and the implementation of expensive and highly technical computing infrastructure” (§“What Separates Difficult From Easy?”)
#28 A synthetic dataset primer for the biobehavioural sciences to promote reproducibility and hypothesis generation · tacit
“…recreates the complete computational environment used for this manuscript (i.e., the R version and R package versions used)” (p. 2)
#30 Advancing computational reproducibility in the dataverse data repository platform · injunctive
“the way data repositories store research data and code cannot fully facilitate reproducibility due to the absence of a runtime environment needed for the code execution” (p.1, Abstract) and “the researcher will be prompted to confirm that their Dockerfile does not include any of the common reproducibility errors” (p.3, Section 3.2)
#31 Reproducibility, preservation, and access to research with ReproZip and ReproServer · tacit
“…creates a single, distributable bundle that can be used to reproduce the entire experiment in another environment (e.g., on Linux, Windows, or macOS) (Chirigati et al., 2016).” (p. 3) and “It captures all dependencies at the OS level, including software, data files, databases, libraries, environment variables, parameters, and OS and hardware information.” (p. 3–4)
#32 Beyond open data: a model for linking digital artifacts to enable reproducibility of scientific claims · injunctive
“3) computational environment information including input parameters, function invocation sequences, and resource details” (p. 9)
#33 Documenting computing environments for reproducible experiments · injunctive
“reproducibility of an experiment often requires repeating the experiment in its native computing environment” (Abstract)
Note: via automated trace log
#35 Trust, but verify: an investigation of methods of verification and dissemination of computational research artifacts for transparency and reproducibility · injunctive
“Using the wrong version of software or data may result in irreproducibility of results… it is important to document in sufficient detail the computational environment used” (p.142–143)
#37 The four possibilities of reproducible scientific computations · injunctive
“the possibility to run the code on a suitable computer of one’s own choice in order to verify that it indeed produces the claimed results”
#38 Ten simple rules for writing dockerfiles for reproducible data science · injunctive
“for security, transparency, and reproducibility, you should only use images where you have access to the Dockerfile” (p. 9)
#39 How to make a pie: reproducible research for empirical economics and econometrics · injunctive
“it is also important to ensure the reproducibility of computation… keeping track of the software (and its version), the packages used (and their version), and the computer used (CPU, operating system) helps to avoid such disagreements” (p. 1151)
#40 The role of containers in reproducibility · injunctive
“In the paper, we posit that containers can play a critical role in aiding in reproducibility.” (p. 19) and “Use a well defined base image that is carefully tagged. The most reliable way is to use the hash tag reference for the image” (p. 22)
#44 Active maintenance: a proposal for the long-term computational reproducibility of scientific results · tacit
“The replication archive and the computing environment as it existed at the moment of deposit… are placed in a virtual container guaranteeing that even if the code is run on different machines, the same results will be reproduced” (p. 464)
#45 Toward reusable science with readable code and reproducibility · tacit
“All configurations for the Docker containers, the code and data files are saved to ensure a reusable, standardized environment that can guarantee the code will execute as intended in the future.” (p. 5–6)
#46 Reproducible research in R: a tutorial on how to do the same thing more than once · injunctive
“computational steps are only executed when the software environment is set up” (p. 862)
#48 Challenges of curating for reproducible and FAIR research output · tacit
“we usually resolve this issue by running on the same operating system and asking the authors to provide text in the README highlighting this in case secondary users attempt to verify on the wrong OS” (p.11, ID26)
#51 A reproducible data analysis workflow with R Markdown, Git, Make, and Docker · injunctive
“all computer code… is executed in a virtual environment that guarantees exact reproduction of results independent of the host operating system, the locally installed R version, and installed package versions” (p.3–4)
#52 CODECHECK: an open science initiative for the independent execution of computations underlying research articles during peer review to improve reproducibility · injunctive
“A cloud-based infrastructure allows for the codechecker and author to collaboratively improve the code and enforces a complete definition of the computing environment” (p. 4)
#55 Computational reproducibility: a practical framework for data curators · injunctive
“explicit documentation of both the execution environment used to run the software… should be clearly noted in the code and/or readme file” (p.7)
#61 A collaborative semantic-based provenance management platform for reproducibility · tacit
“The different execution environments play a role in computational experiments which is shown with the help of ProvBook” — execution of the same cell took 41.3ms, 2min35s, and 3min55s across three environments (p.19–20)
#62 Containerization for creating reusable model code · tacit
“Containers are standalone lightweight packages of the original model software and their dependencies that can be run independent of the platform” (Abstract, p.1) and “Being able to produce a controlled running environment that is lightweight and easily shared facilitates the production of consistent results from its corresponding model, irrespective of current versions of any software dependencies or the operating system of the host machine used to run it” (~p.4)
#63 Toward practical transparent verifiable and long-term reproducible research using guix · injunctive
“Reproducibility of the computational environment is driven by transparency. To be fully reproducible one computational environment must be defined by (i) the source code of the software and libraries used (ii) the various options or configuration are required at build-time or run-time and (iii) the complete graph that fixes all the dependencies.” (p.8)
#67 Tools and recommendations for reproducible teaching · tacit
“the renv package for R… and the venv… package for creating virtual environments for Python… provide tooling for someone else to reproduce your work using the same computational environment” (p. 256)
#69 The interdependence of data producers and data users: how researchers’ behaviors can support or hinder each other · tacit
“we’re moving toward making everything containerized so it’s re-executable in virtual environments” — Producer/User #9 (p.14); authors’ own gloss: “Using containers to create a virtual environment enabled users to reproduce results… without downloading the data or setting up the proper computational and storage environment” (p.14–15)
#71 Teaching for Large-Scale Reproducibility Verification · injunctive
“Does the Lab have access to the computing resources (specific OS, high-performance computing)?… Finally, the computational resources are summarized, if reported in the README” (p.278)
#73 A backend platform for supporting the reproducibility of computational experiments · injunctive
“we need to recreate the same environment using the same frameworks, code, data sources, the same PLs, dependencies, and the same version of the dependencies” (p. 10)
Note: via Docker
#74 Reproducibility as a service · tacit
“Retroactive reproducibility is the process of reproducing a computational experiment without the initial computational environment. Proactive reproducibility is the method of ensuring the future reproducibility of an experiment that is currently executable within its initial computational environment” (p. 1545)
#76 Making provenance work for you · tacit
“This is a common R programming error that can lead to unexpected results if the script is run again in a different environment” (p. 146) and “automatically constructs a computational environment in a Docker container for a given set of R scripts and the data they analyze” (p. 156)
#79 A review of containerization for interactive and reproducible analysis · injunctive
“we still recommend that before sharing notebooks they are re-run sequentially to ensure they work as intended… and containerizing the analysis provides a good opportunity to do this” (p.11–12). Supporting: “The goal of containerization is to ensure that if the code worked when containerized, it will work when the image is run by a third party” (p.5)
#82 Supporting reproducible research · injunctive
“computational reproducibility, which is the ability to rerun a pipeline using the original computational environment and dependencies… This is harder than expected, as it requires using the precise version of the software originally used with the exact parameters and input data, and occasionally even particular hardware configurations.” (p. 178)
#85 The Role of Third-Party Verification in Research Reproducibility · injunctive
“he recreates the same computing environment as the authors (e.g., same operating system, versions of the software, libraries)” (p. 3–4)
#86 What makes computational communication science (ir)reproducible? · tacit
Docker container based on ROCKER, pinned R 4.3.0/Python 3.11.3; “The goal of this dockerized pipeline is to fully automate the code execution in a standardized computational environment.” (p. 7-8)
#88 Reproducibility in Management Science · tacit
“Issues with software/hardware requirements” (2.9%); elaborated as “software could not be installed, or the code would run for an untenable amount of time” (p. 1349)
#89 Computational reproducibility in computational social science · injunctive
“provide a reproducible computational environment where at least the four components from the last section are clearly documented and can be automatically regenerated” (p.7); reinforced by “the reproducible materials must be nonrestrictive” for 3°CR (p.9).
#91 CORE-bench: fostering the credibility of published research through a computational reproducibility agent benchmark · injunctive
CORE-Bench-Hard: agent must “install all required libraries and dependencies, determine (and run) the correct command to reproduce the code” (p. 6, Table 3)
#95 Computational reproducibility in finance: evidence from 1,000 tests · injunctive
Guideline R3: OS/version, hardware specs (RAM, processor, cores, clusters) (p. 3589)
#99 Introducing reproducible research standards at the world bank · injunctive
“For virtual verifications, authors are provided instructions to set up a ‘clean slate environment,’ and the reviewer and author join a virtual meeting, where the author screenshares and the reviewer observes that the package runs” (p.4)
#102 In-Class Data Analysis Replications: Teaching Students While Testing Science · tacit
“Some students suspected that the discrepancy between the tool used for the replication and the originally used tool may play a role… potential differences in the model and optimizer implementations available in Python, R, and Stata” (p. 14, “Tool Differences” theme)
#103 The Challenges of Replicating Volatile Platform-Data Studies: Replicating Schatto-Eckrodt et al. (2020) · tacit
“The reproduction process was conducted in both the original computational environment… and a contemporary updated environment… By reproducing the study in the original computational environment, we ensured that the results were consistent with the conditions under which the original findings were generated.” (p. 5)
#104 Reproduce to validate: a comprehensive study on the reproducibility of economics research · descriptive
“the journal does not require authors to specify the required software, operating system and versions thereof, and most articles were silent on the topic” (p.970)
#108 Checklists for computational reproducibility in social sciences: insights from literature and survey evaluation · injunctive
“Deploy software setup to isolate and preserve the research environment, e.g., dockerizing it” (Checklist 2) and “Integrating the research resources with execution environment… e.g., MyBinder for easy and quick proof of concept” (Checklist 3)
#114 SciConv: a conversational tool for reproducibility · tacit
“A container is generated based on this inference and used to build a containerized environment.” (p. 458); “the tool packages the environment and outputs it into a reproducible archive… ensuring computational reproducibility across different systems.” (p. 458)
#117 Computational reproducibility of R code supplements on OSF · injunctive
“Finally, to support execution in automated or containerised environments, scripts should run non-interactively and avoid GUI-based functions in favor of programmatic alternatives, enabling full execution without manual intervention.” (p. 4)
#119 “Works on my machine”: a case study of replicability challenges inComputational humanities research · injunctive
“These images… already include all the programs and dependencies required for replicating findings, and are termed self-contained and self-executable research artefacts” (p.146)
#120 Reproducibility spectrum for applied social sciences: conceptual taxonomy and recommended tools · tacit
Containerization defined as an asset (Docker, Podman); “Master Quarto… Develop automated workflows” listed for experienced researchers
#123 Why computational reproducibility matters · injunctive
“What you want is a precisely specified and reproducible environment”; “bit-for-bit reproducibility is… a matter of bookkeeping”
#125 A report on “corruption and Co-optation in autocracy: evidence from Russia” · tacit
“These challenges occurred on both Windows and MacOS operating systems and with different R versions (4.4.1 and 4.4.3)” (p.5)
#128 The reliability of replications: a study in computational reproductions · injunctive
“differences between operating systems, packages and processors can produce different results […] Default settings often change across versions or software packages.” (p. 19, Discussion). Also: “sharing of environment and dependencies within the software (sometimes known as a ‘colophon’ or ‘session info’) theoretically gives all necessary information to remove intra-software and package variation in results.” (p. 19–20, Discussion)
#131 Guide for Accelerating Computational Reproducibility in the Social Sciences · tacit
Set up a computing capsule that executes the entire reproduction in a web browser without needing to install any software. For examples, see Binder and Code Ocean.
#133 From guidelines to practice: evaluating the reproducibility of methods in computational social science · tacit
“Environment standardization addresses system-level fragility. Condition C virtually eliminated environment-related failures…” (5.3, Implications bullet)
#136 Artisan: agentic artifact evaluation · descriptive
“We exclude 42 papers whose artifacts are not packaged using Docker, which yields 72 remaining papers. This reflects the fact that Docker and related virtualization technologies are now standard in artifact evaluation.” (p. 9, Section 3.2).
#139 PRIMAD-LID: a developed framework for computational reproducibility · injunctive
“A container or virtual machine image capturing the execution environment (e.g., Docker/Singularity recipe)” (p. 2).
#143 What is Computational Reproducibility? · injunctive
“Utilizing Docker and similar virtualization tools helps create consistent and reproducible computational environments by encapsulating the software dependencies and configurations needed to run code.” (p. 6) and “successful computational reproducibility depends on the whole software-stack and potentially also on the hardware set-up.” (p. 3) and “Differences in computational environments can significantly impact reproducibility.” (p. 4–5).
#144 Investigating the reproducibility of the social and behavioural sciences · tacit
“For 52.4 (64.5%…) of these papers, we were able to reproduce the findings with minimal effort other than executing the code on the data, a standard known as push-button reproducibility.” (p.129)