Numerical output match

Tier 3 · Reproducible - Gold core endorsement

Re-running the code on the deposited data reproduces the numerical results reported in the paper and appendix: estimates, standard errors, test statistics, sample sizes and numbers in the text. Sources differ on the tolerance, from exact agreement to small stated deviations.

Rating

TipMet

Re-running the code on the deposited data reproduces all numerical results in the paper and appendix (estimates, standard errors, test statistics, p-values, sample sizes, numbers in the text) exactly, or with differences only due to rounding or display format.

Based on #99/#103/#109, <1e-10 #20, within 1% #128, <=10% #95, +/-15% & p<=0.05 #144, direction + significance #107/#128, rounding/formatting = minor #134. CRA vs. CRR split: #138.

WarningPartially met

All results are obtained, but some differ beyond rounding while the conclusions are unchanged: same direction and statistical significance, and deviations within the tolerance (e.g. <=10% of the original value, or <=15% and p-values differing by no more than 0.05); or only part of the results can be reproduced (e.g. main tables but not the appendix).

Based on:exact #99/#103/#109, <1e-10 #20, within 1% #128, <=10% #95, +/-15% & p<=0.05 #144, direction + significance #107/#128, rounding/formatting = minor #134. CRA vs. CRR split: #138.

ImportantNot met

The numerical results cannot be reproduced, or they deviate beyond the tolerance, or the direction or statistical significance of a focal result changes.

Based on exact #99/#103/#109, <1e-10 #20, within 1% #128, <=10% #95, +/-15% & p<=0.05 #144, direction + significance #107/#128, rounding/formatting = minor #134. CRA vs. CRR split: #138.

NoteNot applicable

Manuscript has no numerical results (C4)

Prerequisites

Check order: 4

Builds on

Recommended:

Prerequisite for

In the corpus

Sources Injunctive Descriptive Tacit Automatable Manual check Automation unclear LLM sources
49 69.4% 8.2% 22.4% 16.3% 10.2% 73.5% 4

Text evidence (50 quotes)

#10 Sustainable computational science: the ReScience initiative · injunctive

“obtain the same results as indicated in the accompanying paper” (p.7–8) and “consider these results sufficiently close to the ones reported in the original paper being replicated” (p.8)

#14 An empirical analysis of journal policy effectiveness for computational reproducibility · tacit

“we were able to replicate the results in the publication for all but 1” of 22 articles (p.2585)

#18 Reproducibility confidentiality data access · injunctive

“Catalog the result files, possibly certify that they were generated from the programs… Provide a provable chain for the results in journals” (Slide: “Our recommendations to ADRFs”)

#20 Successes and struggles with computational reproducibility: lessons from the fragile families challenge · injunctive

“compare the results created within our computing environment with the results shown in the manuscript”; tolerance <10⁻¹⁰ (p.7)

#24 Computational reproducibility: the elephant in the room · injunctive

“…reproduce every table, diagram, and statistical result using code, for which the source and means to run it are provided” (p. 138)

#28 A synthetic dataset primer for the biobehavioural sciences to promote reproducibility and hypothesis generation · injunctive

“Confirming general utility is a necessary step for making inferences from the synthetic dataset and is especially important for data exploration in the synthetic dataset” (p.3) and “Higher overlap is indicative of greater specific utility, which suggests that inference from fitted synthetic models is valid” (p. 3)

#35 Trust, but verify: an investigation of methods of verification and dissemination of computational research artifacts for transparency and reproducibility · injunctive

“An article is designated as reproducible if the AER succeeds in executing the code on the data provided and produces results matching those that the authors claim are reproducible” (Table 6.2, p.138)

#46 Reproducible research in R: a tutorial on how to do the same thing more than once · descriptive

“results are often manually copied from a statistical computing language into a text processor; if a given analysis is re-run and results are manually updated in the text processor, this may inadvertently lead to inconsistencies between the reported result and the reproduced result” (p. 838)

#51 A reproducible data analysis workflow with R Markdown, Git, Make, and Docker · injunctive

“The workflow ensures meeting the primary goals that 1) the reporting of statistical results is consistent with the actual statistical results (dynamic report generation)” (p.1)

#52 CODECHECK: an open science initiative for the independent execution of computations underlying research articles during peer review to improve reproducibility · injunctive

“we do not require results to be identical for a CODECHECK to pass but simply that the code runs and generates output files that the author claims” (p. 12)

#56 Reproducibility badging and definitions · injunctive

“an additional step was taken or facilitated by the badge issuer… to certify that an independent party has regenerated computational results using the author-created research objects, methods, code, and conditions of analysis” (p.4, ROR-R)

#58 Improving research quality: the view from the UK reproducibility network institutional leads for research improvement · injunctive

“…it is important that journals and publishers use the review process to ensure that research is reported in a transparent manner, and—if data and analysis code are both provided—that the results can be reproduced.” (p.2)

#60 Reproducibility in the social sciences · descriptive

“one promising alternative is to provide prepublication code and data review. Multiple journals (Christian et al. 2018, Colaresi 2016) now do a simple code review, requiring authors to submit the code and data required to produce key figures and results” (p.78)

#61 A collaborative semantic-based provenance management platform for reproducibility · tacit

“ProvBook provides a provenance difference module to compare the different executions of each cell of a notebook, thus helping the users either to (1) repeat and confirm/refute their results or (2) reproduce and confirm/refute others results” (p.12)

#66 Reproducible research publication workflow: a canonical workflow framework and FAIR digital object approach to quality research output · tacit

Curator “Confirms source code reproduces reported results” (Fig. 2, p.308–309)

#71 Teaching for Large-Scale Reproducibility Verification · injunctive

“The student assigned to conduct the computational reproducibility check then downloads any necessary data, and follows instructions in the README to run the programs provided”…“It is the student’s job to reproduce the manuscript’s results following the authors’ instructions as closely as possible” (p.278)

#80 Replication of the morning after: report from the nottingham replication games · tacit

“The materials produced the exact same results as reported in the paper and appendix… the reproduction was seamless.” (p. 3)

#85 The Role of Third-Party Verification in Research Reproducibility · injunctive

“compares the numerical values of all original and regenerated parameters presented in tables” (p.4)

#86 What makes computational communication science (ir)reproducible? · injunctive

  1. Technically executable, but results with major deviations”; “Criterion 4 determines whether the answer is indeed the same.” (p. 6)

#88 Reproducibility in Management Science · tacit

“Reproduction yields (partly) different results” (4.4%) (p. 1350)

#89 Computational reproducibility in computational social science · injunctive

“A result is reproducible when the same analysis steps performed on the same dataset consistently produce the same answer. This reproducibility of the result can be checked by the original investigators and other researchers with the nonrestrictive reproducible materials.” (p. 3)

#91 CORE-bench: fostering the credibility of published research through a computational reproducibility agent benchmark · injunctive

“the agent’s reported results for all questions fall within a 95% prediction interval for every task question of the results from the three manual runs” (p. 7)

#93 Unintended code errors and computational reproducibility · injunctive

“Code produces reported results: Article Figures / Article Tables / Appendix Figures / Appendix Tables / In-text findings” (Appendix A, CODE, p.8)

#95 Computational reproducibility in finance: evidence from 1,000 tests · injunctive

Table 2: difference “small (≤10% of the original result)” vs. “large (>10% of the original result)” defines the pass/fail threshold (p. 3568)

#97 Why and how we share reproducible research at yale university’s institution for social and policy studies · injunctive

“compare the code output with the numerical results reported in the manuscript” (p.4)

#98 Mass reproducibility and replicability: a new hope · injunctive

SSRP L5: “analytic data sets and analysis code are available, and they produce the same results as presented in the paper”; L10: “all necessary materials are available and produce consistent results with those presented in the paper” (pp. 58–59)

#99 Introducing reproducible research standards at the world bank · injunctive

reviewer verifies “the empirical findings presented in the paper can be exactly reproduced” (p.4); “b) stable, producing the exact same outputs every time it is run” (p.4)

#100 Computational and robustness reproducibility of “UN peacekeeping and democratization in conflict-affected countries” · tacit

“The materials produced the exact same results as reported in the paper… The reproduction in Stata was seamless” (p.4); “We obtain the exact same results with regards to coefficients, standard errors, and statistical significance levels as reported in (Blair et al. 2023) except for Models 1 and 2… which show one fewer observation… and the coefficient… is slightly bigger” (p.4–5)

#103 The Challenges of Replicating Volatile Platform-Data Studies: Replicating Schatto-Eckrodt et al. (2020) · injunctive

“We viewed a reproduction as ‘successful’ when we could reproduce the original result quantitatively… We chose the exact quantitative agreement of results, as it is the strictest criterion for assessing reproduction success.” (p. 4)

#104 Reproduce to validate: a comprehensive study on the reproducibility of economics research · injunctive

“Assessors were instructed to categorize articles as fully reproducible if all numbers and figures matched, up to some decimals” (p.974)

#105 Promoting reproducibility and replicability in political science · descriptive

“an independent verification process that computationally reproduces numerical results for accepted articles” — Odum Institute / AJPS (p.3)

#106 The data availability policies of political science journals · injunctive

“The journal has a posted policy that requires authors to share their data and/or code with the published article and the journal verifies the results.” (p. 4)

#107 Teaching constructive replications in the behavioral and social sciences using quantitative data · tacit

“When evaluating inconsistencies in coding and analyses, students can focus on whether the direction, strength, and statistical significance of the focal effects are consistent with the findings of the original study.” (p. 120)

#109 A protocol for structured robustness reproductions and replicability assessments · injunctive

“The decisive criterion for a study to be computationally reproducible is whether the results obtained via re-running the original code match those presented in the paper. If they do not, we consider the paper to be not computationally reproducible.” (p. 4)

#110 Data and code availability in political science publications from 1995 to 2022 · descriptive

“At some journals, an editorial assistant reruns the analysis and confirms that the results match those reported in the article (Brodeur et al. 2024)” (p. 340)

#112 REPRO-bench: can agentic AI systems assess the reproducibility of social science research? · injunctive

“agents are required to assess the validity of research findings by checking the consistency between reproduced and reported results” (p.23617); rounding example, p.23620

#113 Let’s talk about it: making scientific computational reproducibility easier · tacit

“This means that after we uploaded the zip file with all the experiment’s content, our tool was able to automatically infer all the necessary configurations and execute the provided code. In each case, the code ran error-free, successfully producing results. Subsequently, we verified that the outputs matched the expected outcomes, confirming their correctness.” (p. 7)

#123 Why computational reproducibility matters · injunctive

Argues bit-for-bit exact matching over tolerance-based “good-enough-for-me” matching; explicit dependency: “Bob would have run Alice’s environment, confirmed the result of 0.9 nm”

#126 Automated Reproducibility Testing in R Markdown · injunctive

“Check whether the results that are reported in text, tables, and figures are identical to the results of the original analysis.” (p. 6)

#128 The reliability of replications: a study in computational reproductions · injunctive

“we define a stricter dichotomy of Exact Replication where results must be within 1% of the original” (p. 8, Methods). Also: “Directional Reproduction, a dichotomy where results simply point in the same direction and match a null hypothesis significance test that the coefficient is exactly zero with a cut-off of p < 0.05” (p. 8, Methods).

#130 Reproducibility in the classroom · tacit

“Does running the provided code exactly produce the study results?” (p.10)

#131 Guide for Accelerating Computational Reproducibility in the Social Sciences · injunctive

All necessary materials are available and produce consistent results with those presented in the paper. The reproduction involves minimal effort and can be conducted starting from the analytic data ( CRA) and the raw data (CRR).

#132 Long-term effects of the targeting the ultra poor program—a reproducibility and replicability assessment of banerjee et al. (2021) · tacit

“For BDS-Table 4, we obtain some negligible discrepancies in effect sizes (affecting the third decimal place) and number of observations (two observations)” (p. 4–5)

#134 PaperRepro: automated computational reproducibility assessment for social science papers · injunctive

“Score 3 reflects minor differences in presentation, such as rounding or formatting.” (§4.1.1, p.6) and “These discrepancies require careful alignment between reproduced artifacts and reported results, rather than direct equality checks.” (§2, Scenario 3, p.2)

#136 Artisan: agentic artifact evaluation · injunctive

“the output judge runs the submitted script in a fresh container environment and classifies the result into four categories… Success: The submitted script runs to completion without any error, and the results match the expected values.” (p. 7).

#138 Reproducibility and robustness of economics and political science research · injunctive

“Computational reproducibility from raw data is required for papers that are reproducible at Level 6 and above.” (Methods, “Computational reproducibility,” L5 note); also L8/L9 explicitly separate CRA achievement from CRR failure: “the analysis code produces the same output as presented in the paper (CRA). However, the cleaning code fails to run or produces different results… (not CRR)”

#139 PRIMAD-LID: a developed framework for computational reproducibility · injunctive

“The success or failure of a reproducibility study is evaluated by whether the results are consistent with the previous ones, rather than identical” (p. 4).

#141 Understanding cultural persistence and change: A replication of Giuliano and Nunn (2021) · tacit

“we investigate how the identified issues affect the statistical precision, p-values, and magnitude of the original estimates” (p. 31)

#143 What is Computational Reproducibility? · injunctive

“Reproducibility means obtaining consistent results, ideally with identical numerical findings, using the same data and computational analysis methods as the original study.” (p. 2), and “computational reproducibility…can be defined as the ability to consistently arrive at numerical comparable findings using the same data and methods as the original research.” (p. 2).

#144 Investigating the reproducibility of the social and behavioural sciences · injunctive

“Approximate reproducibility was defined a priori as achieved if at least one statistical outcome was not precisely reproduced and all outcomes for a claim were reproduced within ±15% of what was originally reported and, for P values, a difference of no more than 0.05.” (p.128); “Precise reproducibility was achieved if all of the statistical outcomes of the reproduction were the same as originally reported.” (p.128)