Data provenance
Tier 1 · Available - Bronze core endorsement
Rating
Every data source used in the analysis is cited with its version or vintage (or a persistent identifier such as a DOI) and mapped to the corresponding files in the replication package. For data that cannot be shared (e.g. proprietary or confidential data), the README states how, where, and under what conditions an independent researcher can obtain the original source data, including any restrictions, requirements, payments, and processing delays. For data collected via an API or web scraping, the README also states the API or query used, the date of collection, and the configuration settings.
Supported by: Access info for non-shareable data: #42 (how, where, conditions, restrictions, payments, delays), #95 R1 (provider, DOI, vintage, application procedure), #43, #105, #138, #71, #54. API/scraped data: #108 (API, date, configuration), #89 (API version), #34, #26 ‘frozen’ dataset. PID: #55, #65, #110, #70.
Data sources are cited, but the information is incomplete: e.g. some sources are not cited, the version/vintage or identifier is missing, or the access conditions for non-shareable data or the collection details for API-extracted data are only partly given.
Supported by: #14 (79% cite all data; 70% give data location), #30 personal-website deposits lack persistent citation, #63 no integrity check on repository, #102 outdated API.
A replication package is a simulation study (C1)
Prerequisites
Check order: 0
Builds on
No core prerequisites.
Prerequisite for
- Third-party certification (1 source)
In the corpus
| Sources | Injunctive | Descriptive | Tacit | Automatable | Manual check | Automation unclear | LLM sources |
|---|---|---|---|---|---|---|---|
| 48 | 70.8% | 2.1% | 27.1% | 10.4% | 10.4% | 79.2% | 0 |
Text evidence (48 quotes)
#4 Storing reproducible results from computational experiments using scientific python packages · tacit
“Chases… are metadata describing the specific code that was used in the computation and how it was called, i.e. they chase the provenance of the results” (p.48)
#6 A manifesto for data sharing in social media research · injunctive
Current sharing practices are criticized because “data are thus not presented in a way that advances the social media research field or reproducibility more generally; data are mostly not presented in a sustainable format that can be referenced consistently (e.g. via a DOI)… and datasets lack (standardized) metadata” (p.168, end Sec 2)
#9 Using ReproZip for reproducibility and library services · tacit
“ReproZip keeps strict provenance data. For instance, by observing which files were read and by using the software manager in the operating system, ReproZip can identify the software libraries on which the research depends.” (p.3)
#12 Reproducibility: a primer on semantics and implications for research · injunctive
“Distribution: Standard methods of distribution are used for others to access the software, data, and documentation.” (Table 4) (p. 46)
#14 An empirical analysis of journal policy effectiveness for computational reproducibility · tacit
“Proper citation of all code and data used, including that generated by the authors” — 79% compliant (Table 2, p.2586); “Data location provided in the acknowledgements” — 70% compliant (Table 3, p.2586)
#18 Reproducibility confidentiality data access · injunctive
“Clear data provenance” (Slide: “Key attributes of a replication archive”) and “Document the access protocol clearly and in a citable, persistent fashion. Ensure access protocol is transparent and predictable” (Slide: “Our recommendations to ADRFs”) and “Provenance of input data to which the researcher has access (data citation, metadata)” (Slide: “Our recommendations to ADRFs”)
#19 The ReproRubric: evaluation criteria for the reproducibility of computational analyses · injunctive
Raw data is accompanied by rich metadata including pronevance information
#26 Facilitating reproducible project management and manuscript development in team science: the projects R package · injunctive
“a reproducible analytic workflow should if possible incorporate a ‘frozen’ dataset that reflects a given set of queries or data collection forms at a specific point in time” (p.3)
#30 Advancing computational reproducibility in the dataverse data repository platform · tacit
personal-website deposits do “not natively provide a standardized persistent citation for referencing and accessing the research materials, nor sufficient metadata to make it discoverable” and lack “long-term accessibility” (p.2, Related Work)
#31 Reproducibility, preservation, and access to research with ReproZip and ReproServer · tacit
“…with detailed metadata and provenance information, which ensures high-fidelity, with respect to original functionality and utility.” (p. 7)
#32 Beyond open data: a model for linking digital artifacts to enable reproducibility of scientific claims · injunctive
“Links between digital scholarly objects imply the need for immutable versioning, for example through the assignment of a DOI, that are persistent and persistently connected both to claims in the scholarly record” (p. 12)
#34 Reproducibility and replicability in economics · injunctive
“As of this writing, it is not possible through said interface to define the precise revision of the underlying database, the version of the query system used, and the exact query used.” (re: IPUMS) (p. 17-18) and “…only if researchers other than the original author can access the nonpublic data can an attempt at reproducibility even be made, if it at some cost.” (p. 13)
#39 How to make a pie: reproducible research for empirical economics and econometrics · tacit
“Citing data with DOI identification from open trusted repositories can also facilitate the reusability of data sets” (p. 1141)
#42 AEA data and code availability policy · injunctive
“The data availability statement shall provide detailed information on how, where, and under what conditions an independent researcher can access the original source data… and it must be explicit and accurate about any restrictions, requirements, payments, and processing delays.” (p. 819) and “authors must provide a README file listing all included files and documenting the purpose, format, and provenance of each file provided, as well as instructing a user on how replication can be conducted” (p. 819)
#43 Dawning of a new age? Economics journals’ data policies on the test bench · injunctive
“journals need a procedure that regulates these cases. The goal is to permit reproductions in principle, even if the original data cannot be shared for legal reasons… Such a procedure might include providing an identifier of the data used, a contact address from where to obtain the data, information on the availability/access conditions of data, and the program code of the analysis.” (p. 9)
#48 Challenges of curating for reproducible and FAIR research output · injunctive
“We also require full data citations for any original source data used in the analysis data.” (p.11)
#51 A reproducible data analysis workflow with R Markdown, Git, Make, and Docker · injunctive
“To use checksum validation, checksums for all data files must be created and stored at the time of the original analysis. At the time of reproduction, the current checksum must be compared with the stored checksum to ensure data integrity” (p.19–20)
#54 WORCS: a workflow for open reproducible code in science · tacit
“the worcs function closed_data() adds a paragraph to the project readme, indicating how readers can contact the author for access” (p. 44)
#55 Computational reproducibility: a practical framework for data curators · injunctive
“If data was gathered from an online repository, a permanent link to the source should be provided” (p.8); “should therefore be deposited to a well-established repository that registers persistent identifiers” (p.8)
#59 Understanding experiments and research practices for reproducibility: an exploratory study · injunctive
“Paper recommends use of ORCID for researchers and persistent DOIs for datasets, software, and results as identification/provenance practice” (p.18)
#61 A collaborative semantic-based provenance management platform for reproducibility · injunctive
“A key factor to support scientific reproducibility is the provenance information that tells about the origin or history of the data. Recording and analysis of provenance data… play a vital role for scientists to know the methods and steps taken to generate the output, to reproduce own results or other scientist’s results” (p.2)
#63 Toward practical transparent verifiable and long-term reproducible research using guix · injunctive
“Data had to be fetched manually because of the lack of public application programming interface (API) on Flowrepository. In addition, Flowrepository does not provide file integrity check so there is no guarantee that the repository remained unchanged after the publication.” (p.8)
#65 Long-term availability of data associated with articles in PLOS ONE · injunctive
“authors should consider how to ensure long-term viability of the data by ensuring that the URL in the Data Availability Statement will be maintained over time” and “Requiring that researchers include a URL or DOI in documentation such as Data Availability Statements… may help increase the extent to which data are FAIR and accessible on a long-term basis” (p. 10-11)
#66 Reproducible research publication workflow: a canonical workflow framework and FAIR digital object approach to quality research output · tacit
“Assign an ID number to the package as a unique record (persistent identifier, or PID, including the version)” (Step 3, p.311–312)
#67 Tools and recommendations for reproducible teaching · injunctive
“A simple and recommended way of storing this information is a plain text README file that includes both the provenance and the variable description information” (p. 256)
#69 The interdependence of data producers and data users: how researchers’ behaviors can support or hinder each other · injunctive
“Provide information on methods used (e.g., original data sources, data collection methods, physical materials, instrumentation) that complete the description of the provenance for the shared data source” (Table 2, p.23)
#70 The Stark realities of reproducible statistically orientated sociological research: Some newer rules of the sociological method · injunctive
“Clearly identify the version of the dataset and its origins (i.e. where and when it was obtained) using a persistent identifier such as a digital object identifier (DOI).” (Newer Rule ii, p.217)
#71 Teaching for Large-Scale Reproducibility Verification · injunctive
“When data are not provided, data provenance is even more important, as it must completely describe how the student can obtain the data” (p.278)
#75 Reinforcing reproducibility and replicability: an introduction · injunctive
“The verification of replication packages, which includes not just checks of the computational reproducibility of the provided materials but also documented data provenance and completeness of such materials” (p.2–3)
#76 Making provenance work for you · tacit
“Data provenance also confers or adds value—as trustworthiness—to data, but data provenance can do more: it can be used to reproduce computational analyses and validate scientific conclusions” (p. 141–142). Concretely operationalized as: “The INPUTS and OUTPUTS sections list all input and output files, the date they were last modified, and their hash values” (p. 145–146); get.input.files returns “location - the path to the original file” (p. 152); version history via “the provenance is stored in a unique, time-stamped directory, allowing provenance from multiple executions to be analyzed and compared” (p. 146).
#77 Report of the AEA data editor · injunctive
“all authors are required to attest… that they have ‘legitimate access to and permission to use the data used’” (p.857) and “We continue to verify that authors cite all datasets they have used and accessed, as required by the AEA DCAP and the AEA’s citation requirements” (p.855)
#78 CRRESS - session 4 - reproducibility and confidential data · injunctive
“Full description of access path – The one you took – The one others might take, if different → Reproducibility of Access” (Slide 27)
#83 Reproducibility and transparency versus privacy and confidentiality: Reflections from a data editor · injunctive
“Authors are required to include a citation to the article sent to the AEA.” (slide “Tools and Guidance” — Depositing Data section)
#88 Reproducibility in Management Science · injunctive
“In these cases [licensed/proprietary/NDA data], detailed descriptions of data provenance and data set creation are expected.” (p. 1345–1346)
#89 Computational reproducibility in computational social science · injunctive
“When versioning of the API is available, the information should be documented” (p.6)
#93 Unintended code errors and computational reproducibility · injunctive
“Data citation information is present” / “Data source is provided” (Appendix A, DOCUMENTATION & DATA, p.8)
#95 Computational reproducibility in finance: evidence from 1,000 tests · injunctive
Guideline R1(ii): “provide… data provider, database identifier (name, DOI, vintage), application and registration procedures… instructions on which range and variables to pick” (p. 3589)
#100 Computational and robustness reproducibility of “UN peacekeeping and democratization in conflict-affected countries” · tacit
“We used the replication package provided on APSR’s Dataverse” (p.4)
#102 In-Class Data Analysis Replications: Teaching Students While Testing Science · tacit
“data release that does not entirely match the description in the paper or… an incomplete release of the data necessary to reproduce all the results… especially if depending on an outdated automated programming interface (API)” (p. 13)
#104 Reproduce to validate: a comprehensive study on the reproducibility of economics research · tacit
“clear references to the original data provenance and a description of how to construct the initial data sets” (p.968)
#105 Promoting reproducibility and replicability in political science · descriptive
“the data editors typically require the authors to provide detailed information describing how other researchers may obtain the data” (p.3)
#108 Checklists for computational reproducibility in social sciences: insights from literature and survey evaluation · injunctive
“Documenting the data collection process, i.e., API used, date, and configuration settings…” (Checklist 1) and “Sensitive information needed to recreate the study is provided on request…” (Checklist 3)
#110 Data and code availability in political science publications from 1995 to 2022 · injunctive
“we encourage authors to instead use a permanent archive (e.g., OSF or Dataverse) and supply a persistent DOI for the archive with their published article” (p. 343); King’s standard that a reference “be made in the original publication (usually in the first footnote)” (p. 341)
#121 Statistical perspectives on reproducibility: definitions and challenges · tacit
“following the FAIR (Findable, Accessible, Interoperable and Reproducible) principles [151] for data management can lead to higher reproducibility” (p. 17–18)
#131 Guide for Accelerating Computational Reproducibility in the Social Sciences · injunctive
In the paper you chose, find references to all data sources used in the analysis. A data source is usually described in narrative form. For example, if in the body of the paper, or the appendix, you see text like “(…) for earnings in 2018 we use the Current Population Survey”, the data source should be recorded as “2018 Current Population Survey”. If the first reference to this data source is found on page 1 of the appendix, you should record its location as “A1”. Do this for all data sources mentioned in the paper. Each row represents a unique data source.
Data sources also vary by unit of analysis, with some sources matching the same unit of analysis used in the paper (as in previous examples). In contrast, others may be less clear, e.g., “our information on regional minimum wages comes from the Bureau of Labor Statistics.” You should record such data source as “regional minimum wages from the Bureau of Labor Statistics.”
Next, look at the reproduction package and map the data sources mentioned in the paper to the data files in the reproduction package. In the Location column, record their folder locations relative to the main reproduction folder1. In addition to looking at the existing data files, we recommend that you also review the first lines of all code files (especially cleaning code), looking for lines that call the datasets. Inspecting these scripts may help you understand how different data sources are used, and possibly identify any missing files from the reproduction package. Whenever a data source contains multiple files, enter them in the same cell, separated by semicolon (;).
#138 Reproducibility and robustness of economics and political science research · injunctive
“In the event that the authors cannot share some or all the data, they request that information is provided on how other researchers could obtain the datasets.” (Methods, “Journal policy”)
#139 PRIMAD-LID: a developed framework for computational reproducibility · injunctive
“dataset identifiers, checksums, and preprocessing steps” p. 2; “data provenance and versioning” p. 2
#143 What is Computational Reproducibility? · injunctive
“Documenting methodologies, data sources, and analytical procedures clearly and thoroughly from the start will facilitate all later efforts to reproduce the research.” (p. 5–6) and “Many CSS studies rely on data and services accessed through APIs. These dependencies can create barriers when the behavior or the accessibility of these APIs change.” (p. 5)