How to use the rubric

Tiers and badges

The rubric has 21 criteria in three tiers. Each tier corresponds to a badge, and a criterion’s tier follows from its coded dimension:

Tier Dimension Criteria The criteria ask whether…

Tier 1 · Available - Bronze

 Availability 6 (5 core) the reproduction package contains everything needed for a reproduction, or documents how to obtain what cannot be shared

Tier 2 · Executable - Silver

 Executability 11 (4 core) code runs on another machine and fully functional

Tier 3 · Reproducible - Gold

 Consistency 4 (3 core) reproduced outputs exactly match what is reported in the article, apart from rounding and formatting

Each criterion is either core (required for its tier) or recommended (good practice, but not required for obtain a badge).

Core criteria also show why they are core, endorsement or structural.

The full verification process is available on the badge pipeline page.

Evidence on the criterion page

In the corpus

This section provides information regarding a criterion extracted from 145 papers in the scoping review, and covers:

  • Sources: the number of sources that mention the criterion.
  • Injunctive: the share of sources mentioning the criterion that frame it as a rule/requirement or a barrier to reproduce, using prescriptive or proscriptive languages.
    • key words: must, should, require, etc.
    • examples: “Authors must provide all data and code necessary to replicate all reported analyses”, “Code must execute without errors when run sequentially from the master script”, or “unavailable source codes is a critical barrier to reproduce results”
  • Descriptive: the share of sources mentioning the criterion that describe it as a common practice.
    • key words: most journals, most researchers, typically, common practice, widespread norms, etc.
    • examples: “Researchers typically include a master script to facilitate automated re-execution” or “Most CCS journals now expect replication packages to be deposited upon acceptance”
  • Tacit: inferred by the coder that the author(s) require(s) this criterion as a part of reproducibility check, but it is not stated explicitly as a rule.
    • example: a paper outlining a reproducibility attempt flagged hardcoded path errors during execution but doesn’t explicitly mention path resolution as a criterion.
  • Automatable: the share of sources mentioning the criterion, which state that it can be checked automatically.
  • Manual check: the share of sources mentioning the criterion, which state it should be verified manually.
  • Automation unclear: no clear information available in the text regarding the automation of that criterion.
  • LLM sources: among sources confirming automation potential (Automatable), how many of them do involve large language model.

Text evidence

the quotes from those papers that the definition is synthesized from. Click “Show quotes” to expand them.