Table and figure match

Tier 3 · Reproducible - Gold core endorsement

The code regenerates every table and figure in the paper and appendix, and the regenerated items match the published ones in structure and appearance; for figures, this means the levels, shapes and positions of curves and data points.

Rating

TipMet

The code regenerates every table and figure in the paper and appendix, and each matches the published one in structure (rows, columns, panels) and content; for figures, the levels, shapes and positions of curves and data points are the same. Only cosmetic differences remain (fonts, colors, layout).

WarningPartially met

Some display items are regenerated and match, while others are missing (no code produces them, e.g. made by hand) or deviate; or items are regenerated with minor deviations that do not change what they show (e.g. slightly shifted points because no random seed was set, a difference of a few observations).

ImportantNot met

The code regenerates none of the tables and figures, or the regenerated items differ substantially from the published ones (different structure, trends or shapes).

Based on: #85 level, shapes and positions of curves/data points, #20/#93 all figures and tables incl. appendix, #99 match exactly, #104 match up to some decimals, #109 ‘Fully’ identical vs. ‘Partially’ slight deviation, #103 random seed for figures, #132 negligible discrepancies (two observations).

NoteNot applicable

Manuscript has no table or figure (C3)

Prerequisites

Check order: 5

Builds on

Recommended:

Prerequisite for

No criterion builds on this one.

In the corpus

Sources Injunctive Descriptive Tacit Automatable Manual check Automation unclear LLM sources
24 54.2% 4.2% 41.7% 16.7% 20.8% 62.5% 1

Text evidence (24 quotes)

#14 An empirical analysis of journal policy effectiveness for computational reproducibility · tacit

“The reproduction procedure started by recording all relevant figures, numerical and analytical conclusions, and the figure captions… to reproduce the figures and conclusions” (p.2588)

#20 Successes and struggles with computational reproducibility: lessons from the fragile families challenge · injunctive

“regenerate… (2) all the figures that were generated by the author’s code, and (3) all the tables that were generated by the author’s code” (p.7)

#24 Computational reproducibility: the elephant in the room · injunctive

“…reproduce every table, diagram, and statistical result using code, for which the source and means to run it are provided” (p. 138)

#46 Reproducible research in R: a tutorial on how to do the same thing more than once · tacit

“This allows the creation of (conditionally) formatted text, statistical results, and figures that are guaranteed to be up-to-date because they are created anew every time the document is rendered” (p. 839)

#52 CODECHECK: an open science initiative for the independent execution of computations underlying research articles during peer review to improve reproducibility · injunctive

“The codechecker writes a certificate stating how the code was run and includes a copy of outputs (figures or tables) that were independently generated.” (Step 4, p. 4)

#60 Reproducibility in the social sciences · descriptive

“one promising alternative is to provide prepublication code and data review. Multiple journals (Christian et al. 2018, Colaresi 2016) now do a simple code review, requiring authors to submit the code and data required to produce key figures and results” (p.78)

#61 A collaborative semantic-based provenance management platform for reproducibility · tacit

“We evaluated the provenance capture and difference module in ProvBook with different output types, including images. The results show that ProvBook can handle different output types” (p.21); “Using nbdime, ProvBook provides diffing of notebooks based on the content and renders image-diffs correctly” (p.13)

#85 The Role of Third-Party Verification in Research Reproducibility · injunctive

“the level, shapes, and positions of all curves or data points depicted in figures” (p.4)

#86 What makes computational communication science (ir)reproducible? · injunctive

  1. Technically executable, but results with major deviations”; “Criterion 4 determines whether the answer is indeed the same.” (p. 6)

#88 Reproducibility in Management Science · injunctive

“individual reproducibility assessment of all results tables and figures and other results reported in the manuscript” (p. 1347)

#93 Unintended code errors and computational reproducibility · injunctive

“Code produces reported results: Article Figures / Article Tables / Appendix Figures / Appendix Tables / In-text findings” (Appendix A, CODE, p.8)

#95 Computational reproducibility in finance: evidence from 1,000 tests · tacit

AEA/RES subsample described as “818 items (tables and figures)” subject to reproduction attempts (p. 3574–3575)

#99 Introducing reproducible research standards at the world bank · injunctive

“c) consistent with the paper, meaning that the tables and figures reproduced match exactly those included in the paper” (p.4)

#100 Computational and robustness reproducibility of “UN peacekeeping and democratization in conflict-affected countries” · tacit

“In terms of reproducibility, we successfully reproduced all regression tables in the paper (Tables 2 through 9) using their codes in Stata and could not detect any coding errors in the data manipulation part.” (p. 3)

#103 The Challenges of Replicating Volatile Platform-Data Studies: Replicating Schatto-Eckrodt et al. (2020) · tacit

“it is recommended that a seed be determined for illustration methods, incorporating a randomness component to ensure consistent and reproducible visualizations” (p. 5) and “26 empirical claims in the original paper overall, including text passages, figures, and tables… All tables from the original study were reproduced.” (p. 5)”

#104 Reproduce to validate: a comprehensive study on the reproducibility of economics research · injunctive

“Assessors were instructed to categorize articles as fully reproducible if all numbers and figures matched, up to some decimals” (p.974)

#109 A protocol for structured robustness reproductions and replicability assessments · injunctive

Table 1 (Appendix, p. 9): “Reproducible from raw data” and “Reproducible from analysis data” rows operationalize whether the reproduced output matches the paper’s reported results. The instructions specify: “Mark as ‘Fully’ if… the reproduced results are identical to the paper’s results. Mark as ‘Partially’ if… there is slight deviation.” (Appendix, p. 9)

#126 Automated Reproducibility Testing in R Markdown · injunctive

“Check whether the results that are reported in text, tables, and figures are identical to the results of the original analysis.” (p. 6)

#131 Guide for Accelerating Computational Reproducibility in the Social Sciences · injunctive

All necessary materials are available and produce consistent results with those presented in the paper. The reproduction involves minimal effort and can be conducted starting from the analytic data ( CRA) and the raw data (CRR).

#132 Long-term effects of the targeting the ultra poor program—a reproducibility and replicability assessment of banerjee et al. (2021) · tacit

“We exactly reproduce BDS-Tables 1–3 as well as BDS-Fig. 1” (p. 4–5)

#136 Artisan: agentic artifact evaluation · injunctive

“Artisan, an automated LLM agent for reproducing research results given a paper and its artifact… The reproduction script s should contain all logic needed to reproduce the table from scratch… to formatting outputs into the final table.” (p. 3).

#138 Reproducibility and robustness of economics and political science research · tacit

“Computational reproducibility pertains to the ability of the provided replication folder to reproduce the exhibits and statistics displayed in the research (that is, manuscripts and appendices).” (main text, opening of “Coding errors and recoding”)

#139 PRIMAD-LID: a developed framework for computational reproducibility · tacit

“Outcomes refer to the experiment’s output” which includes tables and figures.

#141 Understanding cultural persistence and change: A replication of Giuliano and Nunn (2021) · tacit

“For each targeted specification, we generate a figure that compares the original estimation with the corrected results.” (p. 31)