Codes consistent with claims

Tier 3 · Reproducible - Gold core structural

The code does what the paper says it does: data preparation, samples, model specifications and estimators match the description in the paper, and results are computed by the code rather than copied in. This can be checked by reading the code against the paper, without running it (i.e., static code analysis).

Rating

TipMet

Reading the code against the paper (without running it) shows that data preparation (sample restrictions, exclusions, variable construction), model specifications and estimators (including controls and the level of standard-error clustering) match the paper’s description for all reported results, and every result is computed by the code rather than hard-coded or copied in.

WarningPartially met

The code matches the paper for the main results, but minor discrepancies are found that do not change the main findings (e.g. an undocumented outlier rule or control variable), or some results (e.g. in the appendix) cannot be traced to any code.

ImportantNot met

The code indisputably does something different from what the paper states for one or more main results (e.g. a different subsample, standard errors clustered at a different level, a different estimator), or results are hard-coded or copied in instead of computed. Differences in defensible analytical choices count as discrepancies only if the paper describes something else.

Based on: #17 ‘indisputably does not correctly carry out the test as described’, #98 clustering example, #109 check description vs. code, objective errors vs. analytical choices, #134 minor errors not changing major findings, #136 copied results rejected, #141 citations of descriptions vs. code.

NoteNot applicable

Never.

Prerequisites

Check order: 3

Builds on

Recommended:

Prerequisite for

In the corpus

Sources Injunctive Descriptive Tacit Automatable Manual check Automation unclear LLM sources
17 47.1% 0% 52.9% 23.5% 5.9% 70.6% 3

Text evidence (19 quotes)

#10 Sustainable computational science: the ReScience initiative · tacit

“They did not check for correctness of the code (i.e., does the code actually implement what is advertised in the paper), nor the reproducibility of the results” (p.3)

#12 Reproducibility: a primer on semantics and implications for research · injunctive

“Verification. This category entails checking that the computer code that the original investigators describe correctly solves the mathematical problem it claims to solve. Those verifying such code also need to examine whether the code is doing what it is supposed to be doing.” (p. 64-65)

#17 Transparency, reproducibility, and the credibility of economics research · injunctive

“…it also includes errors in coding. If an author describes a statistical test in the paper, but the code indisputably does not correctly carry out the test as described, this is also considered a verification failure.” (p. 942)

#19 The ReproRubric: evaluation criteria for the reproducibility of computational analyses · injunctive

Components of the report are easily related to what has been computed (Tier III, Workflow/Reporting, Fig. 2, p.3)

#35 Trust, but verify: an investigation of methods of verification and dissemination of computational research artifacts for transparency and reproducibility · injunctive

“The AEA, AJPS, and JASA-ACS initiatives require that the relationship between provided code/data and the figures and tables in the paper be specified (i.e., results provenance)” (p.141)

#35 Trust, but verify: an investigation of methods of verification and dissemination of computational research artifacts for transparency and reproducibility · injunctive

JASA-ACS: “[W]ithout having run the code, do you have any concerns that the code would not reproduce the key results?” (Table 6.2, p.138)

#66 Reproducible research publication workflow: a canonical workflow framework and FAIR digital object approach to quality research output · tacit

Curator “Confirms source code builds and executes” (Fig. 2, Review Code column, p.308–309)

#71 Teaching for Large-Scale Reproducibility Verification · tacit

Figure 1 (p.278) shows a “No data” branch leading to “Offline code assessment” rather than execution — code-only, no-data-required review path. (Evidence is diagram-based; supporting prose is thin — flagging for verification against the linked supplementary workflow guide, footnote 11)

#78 CRRESS - session 4 - reproducibility and confidential data · injunctive

“This package (data, code, tables, figures) and the results presented in the paper (tables, figures) are consistent.” (Slide 10)

#83 Reproducibility and transparency versus privacy and confidentiality: Reflections from a data editor · injunctive

“Program code and command files should be annotated to facilitate replication and ensure clear correspondence between code and figures.” (slide “Tools and Guidance” — Depositing Data section)

#89 Computational reproducibility in computational social science · tacit

“Literate Programming techniques can also be used to make sure the reporting in the manuscript is perfectly aligned with the data analysis” (p.7)

#98 Mass reproducibility and replicability: a new hope · tacit

“there might be important discrepancies between what the article states and what the programs compute… a study may report clustering the standard errors at the state level, but clustering at a different level in the programs.” (p. 24)

#109 A protocol for structured robustness reproductions and replicability assessments · injunctive

“Carefully review all code sections that are directly related to generating the main outcomes… Additionally, check whether the description in the paper matches the code.” (Appendix, p. 10)

#116 AutoReproduce: automatic AI experiment reproduction with paper lineage · tacit

“we conduct comparative analyses between the generated code, the key content described in the paper and the human-curated reference code implementation” (p.6, lines 44–45)

#126 Automated Reproducibility Testing in R Markdown · tacit

“fingerprints of the entire code chunk syntax are stored. Users are informed by a warning in the reproducibility report” when code changes between runs (p. 7)

#134 PaperRepro: automated computational reproducibility assessment for social science papers · tacit

“It also performs a lightweight semantic inspection to assess code quality and detect potential logical or coding errors, consulting the paper 𝑃 when needed.” (§3.1, Execution Agent, p.4)

#136 Artisan: agentic artifact evaluation · injunctive

“LLM-based method judge… classifies the reproduction method into three categories: Copied results: The submitted script just copies the expected values from the checked-in results or the paper without any meaningful reproduction… Last-mile reproduction… Full reproduction: The submitted script performs the most complete reproduction possible.” (p. 8).

#138 Reproducibility and robustness of economics and political science research · injunctive

“Of note, a paper might be fully reproducible, but the programs may contain coding errors. Similarly, there might be important discrepancies between what the article states and what the programs compute, while remaining computationally reproducible” (p. 155)

#141 Understanding cultural persistence and change: A replication of Giuliano and Nunn (2021) · tacit

“Our replication provides detailed citations from the original article to document the precise and unambiguous descriptions of the specifications used in the six equations estimated with individual-level data. We then identify any inconsistencies between these textual descriptions and the actual code or estimation sample used.” (p. 24)