# SocEnRep – Scoping Review

This folder contains all materials for the scoping review conducted as part of the SocEnRep project. The review covers reproducibility criteria, checklists, and verification practices in economics and the social sciences.

**Search period:** 2015–2026-03-12 (initial search); 2026-03-24–27 (backward snowballing, during full-text screening); 2026-03-30 (forward snowballing); 2026-04-01–02 (grey literature & recommendations)
**Protocol:** See `protocol_logbook.docx` for the full step-by-step protocol and decisions log.

---

## Directory Structure

```
litsearch/
├── codes/                        R scripts
├── coding/                       Codebooks, coding sheets, and Google Form/Sheet for Step 5 coding
├── dataset/                      Analysis-ready datasets exported from the coding process
├── output/
│   ├── search_output/            Literature search pipeline outputs, numbered by step (01–11)
│   ├── analysis_output/          Tables and figures from the descriptive analysis
│   └── cleaning_log/             Data-quality flags from the Part A response cleaning
└── protocol_logbook.docx         Protocol and decisions logbook
```

---

## `codes/` — R Scripts

| File | Description |
|------|-------------|
| `get_descriptive_coding.R` | Downloads the live Part A Google Form responses, cleans them, and writes the three `dataset/` files plus `descriptive_codebook.csv`. The sheet is link-shared, so the CSV export endpoint needs no authentication (`USE_GOOGLESHEETS4 <- TRUE` switches to an authenticated pull if link sharing is ever turned off). Overwrites the `dataset/` files on every run — re-run it whenever the sheet changes. Data-quality flags go to `output/cleaning_log/`. |
| `get_criteria_extraction.R` | Reads each coder's Part B workbook (`coding/criteria extraction/<coder>/`, sheets `Part_B1` and `Part_B2`), cleans them, and writes Amelia's two full-corpus files, the two shared-subsample files for IRR, and a codebook (`criteria_codebook.csv` field reference + `criteria_codebook_values.csv` observed values, since Part B has no Google Form to read field types and question text off of the way Part A does). Handles a header-only workbook, so it runs unchanged while Gunther's sheet is still empty — re-run once he has coded and his rows appear automatically. Overwrites the `dataset/` Part B files on every run. |
| `litsearch.R` | Runs 15 OpenAlex queries (6 Tier 1 core concept, 9 Tier 2 concept × discipline) and produces the deduplicated record pool. `DATE_TO` is hardcoded to `"2026-03-12"` for reproducibility — do not change to `Sys.Date()`. **Do not re-run** without consulting the project team; it would overwrite `01_deduplicated_results.csv`, which is the reproducibility artifact. |
| `forward_snowballing.R` | Runs Step 4 forward snowballing via the OpenAlex `cites` filter. Reads `07_fulltext_screened.ris` as the seed list, fetches all works citing each seed, deduplicates internally and against `01_deduplicated_results.csv`, and outputs `08_*` files. `DATE_TO` is hardcoded to `"2026-03-30"`. **Note (openalexR 3.0.1):** the `cites` filter must be passed as a direct named argument (`cites = seed_id`), not inside a `filter = list(...)` — the list approach triggers HTTP 400 "request line too large" (GitHub issue #360). |
| `descriptive_analysis_coding.R` | Descriptive analysis of Amelia's Step 5 coding (Parts A, B1, B2). Produces counts, distributions and cross-tabulations only — no IRR, no inferential statistics. Writes CSV tables to `output/analysis_output/tables/` and PNG figures to `output/analysis_output/figures/`, and prints the same tables to the console as a meeting handout. Paths use `here()`, so it runs from any working directory inside the repo. |
| `coding_sample.R` | Draws two proportionally stratified IRR samples from `10_final_set.xlsx`. Excludes BLOG and ART types (too little extractable information). Outputs `11_irr_sample_descriptive.csv` (n = 45, for RAs) and `11_irr_sample_criteria.csv` (n = 15, for Gunther). Both samples are independent draws, stratified by publication type using largest-remainder rounding, with `set.seed(2026)`. **Do not re-run.** The seed alone does not make the draw reproducible: it depends on the row order of `10_final_set.xlsx`, which was edited after the samples were drawn on 12.05.2026, so a re-run now returns a *different* 45 papers. The script has an `OVERWRITE_SAMPLES` guard that stops before overwriting the saved CSVs — those CSVs are the artifact the coders actually worked from. |

---

## `output/search_output/` — Pipeline Outputs

Files are numbered to reflect the order of the pipeline steps.

### Step 1 — Initial OpenAlex Search

| File | Description |
|------|-------------|
| `01_deduplicated_results.csv` | **Main pool.** 5,294 unique records after 3-pass deduplication (OpenAlex ID → DOI → normalised title). Generated by `litsearch.R` on 2026-03-12. Includes an `AN` column (= OpenAlex ID) to survive the downstream ASReview → Zotero → RIS export pipeline. |
| `02_overlap_log.csv` | Shows which queries retrieved the same records — useful for PRISMA flow and understanding query overlap. |
| `03_search_log.csv` | Summary of record counts per query. Input for the PRISMA flow diagram. |
| `04_pass3_removed_titles.csv` | Records removed by Pass 3 deduplication (normalised title matching). Inspect if > 50 records were removed. |

### Step 2 — Title/Abstract Screening (ASReview Lab, initial search)

Screening was done in ASReview Lab using model ELAS u4. Stopping rule: 265 consecutive irrelevant records (5% of pool). Screening completed 2026-03-22.

| File | Description |
|------|-------------|
| `05a_asreview_relevant.csv` | **174 records** labelled relevant by ASReview. Basis for full-text review. |
| `05b_asreview_relevant+irrelevant.csv` | 1,756 labelled records (174 relevant + 1,582 irrelevant). |
| `05c_asreview_relevant+irrelevant+not_seen.csv` | All 5,294 records with ASReview labels where available and unlabelled for the rest. |
| `ASReview charts_initial search/` | Progress charts exported from ASReview: `progress_density_chart.png`, `progress_recall_chart.png`, `not_relevant_waves_chart.png`, `preliminary_categorization.csv`. |

### Step 3 — Full-Text Screening

174 records loaded into Zotero; RA cleaned metadata and downloaded PDFs. Full-text screening completed 2026-03-27. During screening, 5 backward-snowballed records were identified and added manually (Whitehouse et al. 2019; Vilhuber et al. 2022; Hayden et al. 2023; Knöpfle & Schatto-Eckrodt 2024; Horbach et al. 2026). Details in the [Google Sheets screening log](https://docs.google.com/spreadsheets/d/1igEM7VPpJ41-t9kMSmy5gkYYK-knBBG8?rtpof=true&usp=drive_fs).

| File | Description |
|------|-------------|
| `06_05a_imported_to_Zotero.ris` | RIS export of the 174 ASReview-relevant records, formatted for Zotero import. Generated 2026-03-23. |
| `07_fulltext_screened.ris` | RIS export of the **124 included records** after full-text screening (48 excluded). This file is the seed list for forward snowballing. Exported from Zotero 2026-03-27. |
| `PRISMA_flow_numbers.xlsx` | Running PRISMA flow numbers across all steps. Update after each step. |
| `PRISMA_2020_flow_diagram_updated_SRs_v2.docx` | PRISMA 2020 flow diagram template (populated with step counts). |

### Step 4 — Forward Snowballing

Run completed 2026-03-30 using `forward_snowballing.R` against 124 seeds from `07_fulltext_screened.ris`.

| File | Description |
|------|-------------|
| `08_snowballing_log.csv` | Per-seed log: how many citing works were retrieved for each of the 124 seed papers. |
| `08_forward_snowballing_raw.csv` | All unique citing works retrieved across all seeds, after internal 3-pass deduplication. |
| `08_forward_snowballing_new.csv` | **2,445 new records** not already in the original pool (`01_deduplicated_results.csv`). These went into ASReview for title/abstract screening. |

### Step 4 — Title/Abstract Screening of Forward Snowballing (ASReview Lab)

Screening of the 2,445 forward snowballing records in ASReview. Completed 2026-03-30.

| File | Description |
|------|-------------|
| `09_asreview_relevant_Forward_Snowball.csv` | **18 records** labelled relevant from the forward snowballing pool. Imported into Zotero for full-text review; **12 included** after full-text screening (6 removed as content unsuitable). |
| `09_asreview_relevant+irrelevant+not_seen_Forward_Snowball.csv` | Full ASReview output with labels for all 2,445 records. |
| `09_forward_snowballing_imported_to_zotero.ris` | RIS export of the 18 relevant records, imported into Zotero. |
| `ASReview charts_forward snowball/` | Progress charts from the forward snowballing ASReview run: `progress_density_chart.png`, `progress_recall_chart.png`, `not_relevant_waves_chart.png`, `preliminary_categorization.csv`. |

### Step 4b–c — Backward Snowballing, Grey Literature and Recommendations

SSOAR GESIS searched on 2026-04-01 (0 new records). Four records added via recommendations: Miske et al. (2026) and Nosek et al. (2025) via social media/newsletter; Bleier (2025) and ACRe/BITSS (2022) via expert referral.

| File | Description |
|------|-------------|
| `10_final_set.xlsx` | The **145-paper final corpus**. Combines: 124 (initial search) + 12 (forward snowballing) + 5 (backward snowballing) + 4 (recommendations: 2 social media/newsletter, 2 expert referral). Verified against Zotero subcollection RIS exports in `check this/`. Columns: No., Authors, Year, Title, Source, DOI, URL, Type, Source/Query, OpenAlex ID, Notes. Sheet name: `Literature`. |
| `10_final_set.ris` | RIS export of the 145-paper final corpus. |
| `check this/` | Zotero subcollection RIS exports used to verify `10_final_set.ris`: `02 Relevant List w-o Snowballed Rec.ris` (125), `Backward.ris` (5), `Forward.ris` (12), `Recommendations.ris` (4). |

### Step 5 — Coding

| File | Description |
|------|-------------|
| `11_irr_sample_descriptive.csv` | **45 papers** drawn for descriptive coding IRR. Proportionally stratified by publication type. |
| `11_irr_sample_criteria.csv` | **15 papers** drawn for criteria extraction IRR. Proportionally stratified by publication type. Generated by `coding_sample.R` with `set.seed(2026)`. The sample of record is `coding/criteria extraction/Gunther/sample.xlsx`; the two currently agree. |

---

## `coding/` — Coding Materials

Step 5 coding is split into two parts (see `protocol_logbook.docx` for the full workflow):

- **Part A — Descriptive coding:** surface-level information (discipline focus, contribution type, definitions, workflows/tools, audit practice). Coded by Amelia (all 145 papers) + three RAs on the 45-paper IRR sample. IRR target: Krippendorff's α ≥ 0.70.
- **Part B — Criteria extraction:** B1 (how papers operationalize reproducibility success: classification, unit, coverage, statistical criterion) and B2 (specific criteria mapped to Availability/Executability/Consistency, with normative framing, dependency, and automation codes). Coded by Amelia (all 145 papers; **completed 2026-07-24**); Gunther codes the 15-paper IRR sample.

### `coding/descriptive/`

| File | Description |
|------|-------------|
| `v1_codebook_descriptive.docx` | Version 1 codebook for descriptive coding. |
| `v2_codebook_descriptive.docx` | **Current codebook** (2026-06-23), revised after the trial coding round and RA sync meeting. |
| `trial_coding_results.xlsx` | Results of the trial coding round (all coders coded Bauer et al. 2025 to calibrate; 2026-06-08). |
| `amelia_coding_Bauer et al.docx` | Amelia's worked example of coding Bauer et al. (2025) for the trial round. |
| `sync_meeting.pptx` | Slides from the RA sync/training meeting (2026-06-17). |
| `sample_descriptive.xlsx` | **The descriptive IRR sample of record.** One sheet per RA, each listing the same 45 papers with a `Done` tick box. `get_descriptive_coding.R` reads this file to define the subsample. |
| `Coding Litreview SocEnrep.gform` | Google Form used to collect coding responses. |
| `Coding Litreview SocEnrep (Responses).gsheet` | Google Sheet with all coding responses (linked to the Google Form). |

### `coding/criteria extraction/`

| File | Description |
|------|-------------|
| `Amelia/Amelia_coding-sheet_criteria_extraction.xlsx` | Amelia's B1/B2 coding sheet for all 145 papers. Completed 2026-07-24. Source of `dataset/b1_criteria_extraction.csv` and `dataset/b2_criteria_extraction.csv`. |
| `Gunther/Gunther_coding-sheet_criteria_extraction.xlsx` | Gunther's coding sheet for the 15-paper criteria IRR sample. |
| `Gunther/sample.xlsx` | The 15-paper criteria IRR sample. |

*Note*. `.gsheet`/`.gform` files and sampled paper PDFs are *.gitignored* (the PDFs for licensing reasons) and thus not available in the repository.

---

## `dataset/` — Analysis-Ready Datasets

Exports from the coding process, prepared for IRR calculation and synthesis. **Every file here is generated** and overwritten on each run — edit the source, not these CSVs. Part A comes from the Google Sheet via `codes/get_descriptive_coding.R`; Part B from the coding workbooks via `codes/get_criteria_extraction.R`.

All files use `paper_id` (the `No.` from `10_final_set.xlsx`) and a short `coder` name as their key columns.

Part A columns are the codebook variable codes (`B1`, `B1A`, `C1`, … `F3`) rather than the full Google Forms question text; `descriptive_codebook.csv` maps them back. Every row also carries `paper_id`, the `No.` from `10_final_set.xlsx`.

| File | Description |
|------|-------------|
| `descriptive.csv` | All Part A coding responses from the Google Form: Amelia (145 papers) + the three RAs (45–46 each). 282 rows after duplicate submissions are resolved. |
| `descriptive_coding.csv` | Amelia's Part A coding of the full 145-paper corpus. |
| `descriptive_shared_subsample_IRR.csv` | The shared IRR subsample as a complete grid: 45 papers × 4 coders = 180 rows, of which 179 were actually submitted. |
| `descriptive_codebook.csv` | Variable code → Google Forms question text, with the field type (`single_select` / `multi_select` / `free_text` / `derived`). |
| `b1_criteria_extraction.csv` | Part B1: reproducibility operationalizations (classification, scope/unit, coverage, statistical criterion) with text evidence. One row per operationalization (452 rows across 141 papers). |
| `b2_criteria_extraction.csv` | Part B2: specific reproducibility criteria mapped to dimensions, with normative framing, prerequisite/dependency relations, and automation/LLM codes. One row per criterion (678 rows across 132 papers). |
| `b1_shared_subsample_IRR.csv` | Part B1 rows for the 15-paper criteria IRR sample, all coders stacked (57 rows; Amelia only until Gunther codes). |
| `b2_shared_subsample_IRR.csv` | Part B2 rows for the 15-paper criteria IRR sample, all coders stacked (98 rows; Amelia only until Gunther codes). |
| `criteria_codebook.csv` | Field reference for the B1/B2 columns (20 fields): variable name, part, field type (`categorical` / `free_text` / `identifier` / `derived`), and a description paraphrased from the PART B scheme in `protocol_logbook.docx`. |
| `criteria_codebook_values.csv` | Observed values for the controlled-vocabulary B1/B2 fields (e.g. `criterion label`, `normative framing`, `can be automated`), with row and paper counts — computed from the actual coded data, not hardcoded, so drift from the protocol's controlled list is visible directly. |
| `sample_descriptive.xlsx` | Copy of the descriptive IRR sample with coder assignments. |

---

## `output/analysis_output/` — Analysis Outputs

Generated by `codes/descriptive_analysis_coding.R` from the `dataset/` files; re-running the script overwrites everything here.

| Output | Description |
|--------|-------------|
| `tables/00_*` | Coding coverage, papers with no criteria extracted, and data-quality flags. |
| `tables/01_*` – `08_*` | Part B2 criteria: per-paper counts, by dimension, by label, new criteria, normative framing, automation/LLM, dependency edges, criteria by year. |
| `tables/10_*` – `12_*` | Part B1 operationalizations, how many of the three axes are specified, and the verbatim statistical criteria. |
| `tables/20_*` – `30_*` | Part A descriptive coding (single- and multi-select fields) and corpus composition. |
| `figures/*.png` | Publication-ready versions of the main tables, 300 dpi. |

Tables whose name contains `flag` are **data-quality lists, not results**: free-text entries and near-duplicate spellings that should be harmonised before the synthesis (see Current Status below). Flag tables are written on every run even when empty, so an empty file means "checked, nothing found" rather than a stale leftover.

---

## `output/cleaning_log/` — Part A Cleaning Flags

Written by `codes/get_descriptive_coding.R`. None of these are results; they record what the cleaning step found and what it decided.

| File | Description |
|------|-------------|
| `flag_coder_name_variants.csv` | Free-text "Coder name" answers that were harmonised onto the canonical coder list. |
| `flag_duplicate_submissions.csv` | Submissions dropped because the same coder submitted the same paper twice, with the fields that differed between the two. The most recent submission is kept. |
| `flag_doi_mismatch.csv` | Coder-entered DOIs that differ from the DOI in `10_final_set.xlsx`. The corpus DOI is kept. |
| `flag_irr_sample_renumbered.csv` | IRR-sample rows whose `No.` no longer points at the same paper in `10_final_set.xlsx` (the file was renumbered after the sample was drawn). Papers are matched on title instead. Currently empty. |
| `flag_irr_missing_submissions.csv` | Coder × paper cells in the IRR sample that were never submitted. |
| `flag_marked_done_not_submitted.csv` | Papers an RA ticked `Done` in `sample_descriptive.xlsx` but never submitted through the form — a genuine missing submission, as opposed to a paper they were never assigned. |
| `flag_ra_papers_outside_sample.csv` | Papers the RAs coded that are not part of the 45-paper IRR sample (the trial-round paper and any coded by arrangement). Kept in `descriptive.csv`, excluded from the IRR file. |
| `flag_b2_blank_category.csv` | Individual `criterion category` cells left blank in a Part B2 sheet, with the workbook row to fix and the value recoverable from the criterion label (a label that maps to exactly one category everywhere else). The rest of each row is coded. |
| `flag_criteria_irr_coverage.csv` | Rows contributed per paper per coder in the 15-paper criteria IRR sample. A paper with no rows from a coder is ambiguous — either not yet coded, or coded and found to yield nothing — so it is listed rather than treated as agreement. |

---

## `protocol_logbook.docx`

The protocol and decisions logbook for the entire literature review. Documents all methodological decisions, screening criteria, the Step 5 coding scheme (Part A descriptive codes; Part B1/B2 criteria extraction codes), and step-by-step progress notes. **This is the primary reference for understanding how and why each step was conducted.** Last substantively modified: 2026-07-24.

---

## Workflow Overview

```
OpenAlex API (15 queries)
        |
        v
01_deduplicated_results.csv   [5,294 records]
        |
        v
ASReview title/abstract screening (Step 2)
        |
        v
05a_asreview_relevant.csv     [174 relevant]
        |
        v
Full-text screening in Zotero (Step 3)
        |
        v
07_fulltext_screened.ris      [124 included]
        |
   _____|___________________________
  |           |                     |
  v           v                     v
Backward    Forward snowballing   Recommendations
snowballing  via OpenAlex API     (grey lit, social
[5 records]  cites filter          media, expert)
  |               |                [4 records]
  |               v                     |
  |     08_forward_snowballing_new.csv  |
  |              [2,445 new]            |
  |               |                     |
  |               v                     |
  |     ASReview screening              |
  |               |                     |
  |               v                     |
  |     09_asreview_relevant_           |
  |     Forward_Snowball.csv            |
  |          [18 relevant]              |
  |               |                     |
  |     Full-text review                |
  |       [12 included]                 |
  |_______________|_____________________|
                  |
                  v
        10_final_set.xlsx  [145 papers]
                  |
                  v
        coding_sample.R (stratified IRR sampling)
                  |
         _________|_________
        |                   |
        v                   v
11_irr_sample_         11_irr_sample_
descriptive.csv        criteria.csv
[45 papers]            [15 papers]
        |                   |
        v                   v
Part A: descriptive    Part B: criteria
coding (Google Form)   extraction (sheets)
Amelia [145] + RAs     Amelia [145, done
[45, IRR]              2026-07-24] +
        |              Gunther [15, IRR]
        v                   |
dataset/*descriptive*.csv   v
                       dataset/b1_*.csv
                       dataset/b2_*.csv
```

---

## Current Status (2026-07-26)

- Part A descriptive coding: Amelia's full-corpus coding and RA IRR coding collected via Google Form. The export is now automated — `codes/get_descriptive_coding.R` pulls the live sheet, cleans it and rewrites the `dataset/` files (2026-07-26). The first automated pull picked up 13 cells that had been harmonised in the sheet since the manual 24.07 export (C1/C2 free-text contribution types folded into `Conceptual/review paper`, `Opinion` and `Educational/training resource`; F2 `AI agent`/`AI agents` → `AI Agents`).
- Part B criteria extraction: Amelia completed all 145 papers on 2026-07-24. The export is automated — `codes/get_criteria_extraction.R` reads the coding workbooks and writes Amelia's two files plus the two shared-subsample files (2026-07-26). Gunther's workbook is still header-only, so the IRR files hold Amelia's rows alone for now. 
- Descriptive analysis of Amelia's coding run (`codes/descriptive_analysis_coding.R`, 2026-07-26). Headline numbers: **679 criterion mentions** across 132 papers (13 papers yielded none), median 5 per paper; **Availability 372 / Executability 210 / Consistency 97**; 70.8% of mentions are framed injunctively; 12 criteria were newly coined during coding, of which `version control` (11) and `artifacts longevity/sustainability` (9) are frequent enough to consider promoting to the codebook.
- **Next:** compute Krippendorff's α for the descriptive IRR subsample (45 papers × 4 coders) and the criteria IRR sample (Gunther); reconciliation meeting if discrepancies are high; finalize PRISMA flow diagram; resolve the Brodeur et al. duplicate (working paper #98 vs. published version #138 — see logbook note of 2026-07-08).

---

## Notes

- **Reproducibility:** `DATE_TO` is hardcoded in both R scripts. Re-running the scripts will not produce identical results because OpenAlex is a live database that backfills records continuously — the saved CSVs are the reproducibility artifacts, not the scripts.
- **openalexR version:** 3.0.1 (R 4.5). Known bug with `filter = list(cites = ...)` causing HTTP 400 — see `forward_snowballing.R` comments and [GitHub issue #360](https://github.com/ropensci/openalexR/issues/360).
- **Git repo:** `https://github.com/rameliaz/litsearch`