Dependency resolution

Tier 2 · Executable - Silver core endorsement

Every add-on package or library the code needs is declared with its exact version, ideally in a machine-readable file (e.g. renv.lock in R) or an installation script (e.g. a setup.do in Stata), so that a user can install them on a new machine and the code finds them.

Rating

Met

The package contains a dependency management file or installation script (e.g. renv.lock or groundhog in R, a setup.do in Stata) that installs all required packages/libraries with their exact versions and runs successfully, or the README lists all packages/software used with their exact versions, and these can be successfully installed at the point of reproduction.

Supported by: Installer/lock file: #19 ‘installed/updated by a script’, #108 requirements.txt, #120 renv/conda/pip, #134 single installation script, #79 renv/checkpoint/groundhog. Listing with versions: #95 R3, #117, #40, #38 pin versions, #70, #48, #57, #7.

Partially met

A dependency management file, installation script, or README list of dependencies is available, but it is incomplete (some packages or their versions are missing) or it cannot be successfully executed/installed at the point of reproduction (e.g. a package has been removed from its repository).

Supported by: #44 packages removed from CRAN, #103 deprecated packages, #61 version change breaks code, #143.

Not met

No dependency management file, installation script, or README information on dependencies is available.

Supported by: #10 ‘undocumented dependencies’, #73 ‘no information about how to install the project’, #77 manifest files ‘remain rare’.

Not applicable

Never. If no add-on packages are used, met when the README says so.

Prerequisites

Check order: 2

Builds on

Code deposit (7 sources)
Documentation (2 sources)
Data deposit (1 source)

Prerequisite for

In the corpus

Sources	Injunctive	Descriptive	Tacit	Automatable	Manual check	Automation unclear	LLM sources
59	55.9%	1.7%	42.4%	47.5%	5.1%	47.5%	5

Text evidence (59 quotes)

Show quotes

#1 An introduction to docker for reproducible research · tacit

“A Docker based approach works similarly to a virtual machine image in addressing the dependency problem by providing other researchers with a binary image in which all the software has already been installed, configured and tested.” (p. 3)

#2 Towards reproducibility in scientific workflows: an infrastructure-based approach · tacit

“we query the Software Stacks Dataset to recursively calculate the Requirement Dependency Graph, a directed graph that relates Software Stacks with their dependent ones” (p.7); graph may contain circular dependencies, handled by tracking visited nodes (p.7)

#5 ReproZip: computational reproducibility with ease · injunctive

“researchers should create a compendium that encompasses all the components that are required to reproduce the experiment’s results, including data, code, software, library dependencies, and information about the source computational environment” (p. 2085)

#7 Tools and techniques for computational reproducibility · injunctive

“it is essential to explicitly document any software dependencies and input data that are required for each step” (p.3); “it is critical to indicate which dependencies (and versions thereof) must be installed” (p.4)

#9 Using ReproZip for reproducibility and library services · injunctive

“The dependencies required to reproduce research can be exceptionally hard to track…thus impossible to install on a new machine, which means the percentage of reproducible research remains low.” (p.1, Abstract)

#10 Sustainable computational science: the ReScience initiative · injunctive

“the reviewers [must be] able to run the proposed implementation on their computers” (p.7–8) and “Missing code and/or data, undocumented dependencies, and inaccurate or imprecise description appear to be characteristic of much non-reproducible work” (p.8)

#11 Sciunits: reusable research objects · tacit

dependencies are identified and packaged automatically during package. (Sec. III)

#15 Coding be eR: assessing and improving the reproducibility of R-based research with containR · tacit

“This error commonly occurs when attempting to run R code written on a different computer, and indicates that the package the user tried to load (in the example case, QMSS) has not yet been installed.” (p.16)

#16 ReproServer: making reproducibility easier and less intensive · injunctive

“these may have many nested library dependencies, of which different versions may give entirely disparate results or not even run… researchers ideally have to create a compendium that encapsulates all the dependencies needed for reproducibility” (p.1)

#19 The ReproRubric: evaluation criteria for the reproducibility of computational analyses · injunctive

Documentation of exact versions of software and dependencies used; Software is installed/updated by a script; Dependencies are archived, bundled, or captured in an image or container for preservation (Fig. 2, “Software” rows, p.3)

#20 Successes and struggles with computational reproducibility: lessons from the fragile families challenge · injunctive

“the entire process will fail if just one package fails, and the probability that at least one package fails is 1 – pⁿ” (p.8)

#25 Toward computational reproducibility: a doctoral student’s story of passing the baton · tacit

“an external software package that was required was not mentioned and needed to be installed, specifically html2text…” (p.689)

#27 Rebooting reproducibility: From re-execution to replication · injunctive

“the difficulty in reconstructing a work’s original library dependencies and software versions” (para. 1, opening)

#31 Reproducibility, preservation, and access to research with ReproZip and ReproServer · tacit

“ReproZip is an open-source tool that automatically captures all the dependencies of a research product or application… ReproZip then automatically sets up the environment and allows users to rerun the application.” (p. 3) and “It captures all dependencies at the OS level, including software, data files, databases, libraries, environment variables, parameters, and OS and hardware information.” (p. 3–4)

#35 Trust, but verify: an investigation of methods of verification and dissemination of computational research artifacts for transparency and reproducibility · injunctive

“In some cases, reproduction requires installing software dependencies on a new target system while others allow for access to the original target system” (p.137)

#38 Ten simple rules for writing dockerfiles for reproducible data science · injunctive

“In some cases (e.g., Conda) the package manager is also able to make decisions about what versions to install, which is likely to lead to a non-reproducible build. For this reason, it is necessary to pin the dependency versions.” (p. 15)

#39 How to make a pie: reproducible research for empirical economics and econometrics · injunctive

“it is also important to ensure the reproducibility of computation… keeping track of the software (and its version), the packages used (and their version), and the computer used (CPU, operating system) helps to avoid such disagreements” (p. 1151)

#40 The role of containers in reproducibility · injunctive

“Any packages installed via a package manager should specify the exact versions to be installed… any dependencies must also be explicitly specified with versions.” (p. 22)

#44 Active maintenance: a proposal for the long-term computational reproducibility of scientific results · tacit

“three studies relied on packages that had been removed from CRAN (e.g., Design) or updated in ways that made it impossible to run the original syntax (e.g., Zelig)” (p. 464)

#45 Toward reusable science with readable code and reproducibility · injunctive

“One of the major challenges in computational reproducibility and reuse is capturing code dependencies necessary for its execution. Code dependencies can be software meant to build or run the code, external libraries, and system requirements needed to run the code correctly. These dependencies are currently not captured for much of the published research code, which is one of the major causes of non-reproducibility reported across fields.” (p. 5) and “One of the best reproducibility practices is to test research code in a clean environment before publishing it or sharing it online. Doing that helps the researcher (or code depositor) identify common errors such as fixed file paths, missing dependencies, and missing files.” (p. 5)

Note: The RE3 platform automates dependency installation via Docker/Conda

#46 Reproducible research in R: a tutorial on how to do the same thing more than once · injunctive

“Dependency tracking helps automatically resolve dependencies between project files. In essence, researchers provide a collection of computational recipes…. The concept of computational recipes is central to our understanding of reproducibility because it enables a unified way to reproduce a project automatically.” (p. 840)

#48 Challenges of curating for reproducible and FAIR research output · injunctive

“We required the author to specify in their README that secondary users wishing to verify the results themselves had to use the specific version of that package, otherwise the code would fail” (p.11, ID25)

#50 Fixing dependency errors for python build reproducibility · tacit

“PyDFix analyzes a total of 2,702 builds, identifying 1,921 (71.1%) of them to be unreproducible due to dependency errors” (Abstract, p.439); operationalized via error-pattern table (p.444)

#51 A reproducible data analysis workflow with R Markdown, Git, Make, and Docker · tacit

“Handling such dependencies is easy with Make because it allows you to manage dependencies by creating (computational) recipes to create or recreate files” (p.8)

#54 WORCS: a workflow for open reproducible code in science · injunctive

“All solutions to computational reproducibility require users to make their dependencies explicit” (p. 34)

#55 Computational reproducibility: a practical framework for data curators · injunctive

“explicit documentation of both the execution environment… and all requirements and dependencies of systems, software, and libraries should be clearly noted” (p.7)

#57 Principles for data analysis workflows · injunctive

“this includes documenting the specific version of any software that we used, as well as its relevant dependencies and compatibility constraints” (p.13)

#61 A collaborative semantic-based provenance management platform for reproducibility · tacit

“The first run of the eigenfaces Jupyter Notebook gave ModuleNotFoundError for User 1… The problem occurred because of the version change of the scikit-learn module” (p.19)

#62 Containerization for creating reusable model code · injunctive

“To reproduce the correct environment, it is vital to specify all the necessary settings and the required packages, extensions, or any other dependencies that are not included in the base image, as well as the right version of each of these additions” (~p.6)

#70 The Stark realities of reproducible statistically orientated sociological research: Some newer rules of the sociological method · injunctive

“If you did not record and report the versions of the software you used (including library dependencies), fail.” (Table 1, p.210)

#71 Teaching for Large-Scale Reproducibility Verification · injunctive

“Authors of replication packages often neglect to identify such packages, and if not available, code will fail” (p.277)

#73 A backend platform for supporting the reproducibility of computational experiments · tacit

“we were able to successfully install the required dependencies, execute, and create a reproducibility package for 20 of 25 (80%)”; failure case: “there is no information about how to install the project… some files were missing”

#74 Reproducibility as a service · injunctive

“Datasets need access to the original language packages they used in the initial computational environment” / “Language packages often need access to the original system libraries used in the initial computational environment” (p. 1550)

#77 Report of the AEA data editor · descriptive

“Sufficiently precise descriptions of the required auxiliary packages or libraries, as ‘manifest’-like files in R, Python, and Julia, or as ‘setup.do’-like programs in Stata, remain rare” (p.851)

#79 A review of containerization for interactive and reproducible analysis · injunctive

“One major source of trouble is ensuring correct code dependencies… installing a single package may actually require a large network of interrelated packages to be configured” (p.3–4); tools like renv/checkpoint/groundhog “enhance reproducibility by maintaining a local archive of packages” (p.4)

#86 What makes computational communication science (ir)reproducible? · tacit

“we automatically resolved the dependencies based on the shared code… the software dependencies of the shared code were automatically installed inside the dockerized environment,” using RENV/PIPREQS/REMOTES to scan R and Python packages. (p. 9)

#88 Reproducibility in Management Science · tacit

“Issues with software/hardware requirements” (2.9%); elaborated as “software could not be installed, or the code would run for an untenable amount of time” (p. 1349)

#89 Computational reproducibility in computational social science · injunctive

“provide a reproducible computational environment where at least the four components from the last section are clearly documented and can be automatically regenerated” (p.7) — the four components are OS, system components, language version, library versions (p.6).

#91 CORE-bench: fostering the credibility of published research through a computational reproducibility agent benchmark · injunctive

“The agent must install libraries, packages, and dependencies and run the code.” (p. 1, Fig. 1 caption)

#95 Computational reproducibility in finance: evidence from 1,000 tests · injunctive

Guideline R3: “any required packages/libraries (and their versions)” (p. 3589)

#103 The Challenges of Replicating Volatile Platform-Data Studies: Replicating Schatto-Eckrodt et al. (2020) · tacit

“Only minor code adjustments due to depreciated package dependencies for the exporting of results had to be made in the updated environment.” (p. 5)

#105 Promoting reproducibility and replicability in political science · tacit

“fixing file paths and libraries” — SSRP improvement step (p.6)

#107 Teaching constructive replications in the behavioral and social sciences using quantitative data · tacit

“We observed that students invest considerable effort in changing file paths or directories, installing packages, moving forward or backward in software or package version history, and identifying and debugging errors.” (p. 120)

#108 Checklists for computational reproducibility in social sciences: insights from literature and survey evaluation · injunctive

“Preserving the working environment… required libraries, packages, and their version, e.g., by generating requirements.txt” (Checklist 3)

#112 REPRO-bench: can agentic AI systems assess the reproducibility of social science research? · tacit

“Type 3 failures occur when agents cannot correctly install libraries.” (p.23624)

#113 Let’s talk about it: making scientific computational reproducibility easier · injunctive

“The integrity and reproducibility of research depend on the availability and accessibility of key artifacts like code, datasets, and dependencies.” (p. 3)

#114 SciConv: a conversational tool for reproducibility · tacit

“SCICONV automates the detection of dependencies and programming languages, and packages experiments into portable artifacts with minimal manual input.” (Abstract, p. 457)

#117 Computational reproducibility of R code supplements on OSF · injunctive

“First, all required packages should be explicitly declared and sourced from reliable repositories such as CRAN, Bioconductor, or GitHub, with version information when relevant.” (p. 4)

#118 Tooling for reproducible research: considerations for the past and future of data analysis · tacit

“users could always be assured that the code and the output were up to date… and that the code had executed, because otherwise the package could not have been built in the first place” (p. 3)

#119 “Works on my machine”: a case study of replicability challenges inComputational humanities research · injunctive

“A project’s dependencies were marked as reproducible only if the versions of all declared dependencies were explicitly and unambiguously specified… Even a single undeclared version… was sufficient for the project to be marked as non-reproducible.” (p.145)

#120 Reproducibility spectrum for applied social sciences: conceptual taxonomy and recommended tools · tacit

“Package/dependency management (renv, conda, pip requirements.txt, packrat)”; “Document computational environment (Dockerfile, renv.lock)”

#125 A report on “corruption and Co-optation in autocracy: evidence from Russia” · tacit

“we also had trouble reproducing the Oster (2019) statistics due to package issues” (p.5)

#131 Guide for Accelerating Computational Reproducibility in the Social Sciences · tacit

Environment set up: Modify the code to include correct paths to files, specific versions of software, and instructions to install missing packages or libraries.

#133 From guidelines to practice: evaluating the reproducibility of methods in computational social science · tacit

“These issues included dependency conflicts, missing files, version mismatches, and broken or deprecated APIs” (4.2.2)

#134 PaperRepro: automated computational reproducibility assessment for social science papers · tacit

“it identifies required dependencies from scripts and documentation, then consolidates them into a single installation script” (§3.1, Setup Agent, p.3)

#136 Artisan: agentic artifact evaluation · injunctive

“The agent must first download the artifact file before it can proceed with reproducing the results. However, we found that agents often struggle to find the correct way to download the actual artifact file given the artifact URL.” (p. 6). Also: excluding papers that rely on non-public APIs (p. 9) implicitly treats dependency resolvability as a requirement.

#138 Reproducibility and robustness of economics and political science research · tacit

“Fixing paths, missing packages and software requirements were not considered failures of computational reproducibility. In those instances, we fixed paths and added missing packages and software requirements.” (p.152)

#139 PRIMAD-LID: a developed framework for computational reproducibility · injunctive

“Longevity emerged from recognising that experiments must remain reproducible well beyond their initial creation, tackling challenges such as software decay, deprecated dependencies, or lost data.” (p. 9)

#143 What is Computational Reproducibility? · injunctive

“The use of external libraries and tools, which may update or deprecate over time, can also hinder reproducibility. If the specific version of a library used in the original study is no longer available, reproducing the exact computational environment becomes challenging.” (p. 5).