Matrikulasi Program Studi S3 Psikologi

Dr. Rizqy Amelia Zein (Research Data and Methods Team, Center for Advanced Internet Studies & Department of Psychology, Universitas Airlangga)

How Research is Made Reproducible

A first-person perspective on open and reproducible science

AGENDA

Replication Crisis and Why Does Reproducibility Matters
Data Management Plan
Pre-Registration
Literate Programming
Registered Reports
Conclusions

Replication Crisis — and Why Reproducibility Matters

Let’s Start With a Thought Experiment

You read a fascinating finding in a top journal and want to build your dissertation on it.

Question 1: If another team ran the same study again with new participants, would they find the same result?
Question 2: If you took the original data and the original analysis code, would you get the same numbers reported in the paper?
Intuitively, we assume the answer to both is “yes, of course it should be! It’s published science in a top Q1 journal!”
Over the last 15 years, researchers actually checked. The answers were… rather uncomfortable.

How It Started (early 2010s)

2011 — Bem’s “precognition” paper: a top journal (Journal of Personality and Social Psychology - JPSP) published evidence that people can feel the future (Bem, 2011) by using standard methods and standard statistics. If the method can “prove” the impossible, what does that say about the method?
2011 — The Stapel fraud case: a prominent social psychologist was found to have fabricated data in dozens of published papers.
2011 — “False-Positive Psychology” (Simmons, Nelson, & Simonsohn): showed that ordinary, small, undisclosed analytic choices can make anything statistically significant — they “demonstrated” that listening to a song makes people feel younger.
These events triggered a field-wide question: how much of our literature can we actually trust?

But the problem started much earlier than that

The Multi-Site Replication Projects

Large teams systematically re-ran published studies with new data, and here is what came back:

36%

of 100 psychology studies replicated — Open Science Collaboration (2015)

61%

of 18 economics lab experiments replicated — Camerer et al. (2016)

62%

of 21 social-science studies from Nature & Science replicated — Camerer et al. (2018)

49%

of 164 papers across 12 disciplines replicated — COS SCORE project (Nature, April 2026)

Even when effects did replicate, they were on average only about half as large as originally reported (i.e., “The Decline Effect”) — SCORE found the same pattern again in 2026.
SCORE is the largest check yet: 865 researchers assessed 3,900 papers (2009–2018, 62 journals, spanning economics, psychology, political science, sociology, and more). Reproducing published analyses on the original data succeeded only 54% of the time.
This pattern, across psychology, economics, and beyond, now confirmed at scale, is what the media dubbed the “replication crisis.”

Three Words You Must Not Confuse

Following Barba (2018) and The Turing Way, people mix these up all the time:

Reproducibility (🇮🇩 “reka ulang”)

Same data + same analysis → same result. Someone else (or future you) re-runs your analysis on your data and gets the numbers in your paper.

Replicability (🇮🇩 “coba ulang”)

New data + same method → similar result. An independent team repeats the study with new participants and finds a comparable effect.

Robustness (🇮🇩 “keandalan”)

Same data + new analysis → same result or at least, similar conclusion. The finding remains stable when different analytical strategies are used.

Notice the Differences

The Really Uncomfortable Part

Let’s forget collecting new data (i.e., replication) for a moment. Can we even reproduce the original numbers from the original data? (i.e., “reproducibility”)

Hardwicke et al. (2018) tried to reproduce results from papers with mandatory open data in the journal Cognition: only 11 of 35 papers were reproducible without help from the original authors.
Artner et al. (2021) could reproduce only ~70% of major statistical claims from published psychology articles and some only after considerable detective work.
Crüwell et al. (2023) checked “open data” badges in Psychological Science: only 1 out of 14 articles was rated fully computationally reproducible.
And this directly affects people’s lives: the famous Reinhart & Rogoff austerity paper, which is widely used to justify real economic policies, contained a simple Excel error discovered only when a graduate student got the spreadsheet (Herndon et al., 2014).

Why Does This Happen?

It’s mostly not fraud, but rather ordinary incentives acting on ordinary humans.

Publication bias: journals prefer novel, positive, “clean” results. Therefore, null results are often being kept into the file drawer.
Publish or perish: careers are built on the number of publications, not on their substantive quality.
Questionable research practices (QRPs): a grey zone between good practice and misconduct (John et al., 2012):
- p-hacking: trying many analyses and reporting the one that “works”
- HARKing: Hypothesizing After the Results are Known — presenting an exploratory finding as if predicted (Kerr, 1998)
- Selective reporting of measures, conditions, or outcomes
The garden of forking paths: even without bad intent, every dataset offers countless defensible analysis choices, thus flexibility in analytical choices can inflate false positives (Gelman & Loken, 2014).

An article about computational science in a scientific publication is not the scholarship itself, it is merely advertising of the scholarship. The actual scholarship is the complete software development environment and the complete set of instructions which generated the figures.

Buckheit & Donoho (1995), paraphrasing Jon Claerbout

Why Should You Care?

Ensuring that our results are reproducible gives us more confidence in our work.

Your closest collaborator is you, six months ago and past-you does not answer Whatsapp chats. A reproducible workflow means you can always retrace your own steps.
Errors get caught early, before they end up in your dissertation, your defense, or a retraction.
International reviewers, supervisors, and committees increasingly expect it. Funders and journals now routinely require data, code, and analysis plans.
It makes you more efficient. Re-running an analysis after your supervisor/reviewers ask “please just add this one covariate” takes one click instead of one week.
It builds trust in your work. Reproducible research gets reused, cited, and extended.

A “Reproducibility Ladder”🪜

Start with the practice you really need, not the one you want. The four practices in this talk are not supposed to be a to-do list to finish this semester, or even the whole PhD program. They are a ladder you climb over years of your future professional career.

Rung 1 — Data Management Plan: lowest barrier, most direct benefit. A folder that anyone, including future you, can navigate.
Rung 2 — Pre-Registration: a planning discipline — write the hypothesis down before you know the answer.
Rung 3 — Literate Programming: a technical habit — let the document compute itself instead of copying numbers by hand.
Rung 4 — Registered Reports: the biggest commitment — review before results exist, over years.
This is the order I personally climbed it, over about 15 years of my scientific career. I’ll show you what each rung actually looked like from where I stood.

Data Management Plan

What Is a Data Management Plan (DMP)?

A short, living document, written before you collect data, describing how data will be collected, documented, stored, protected, and shared.

Think of it as answering, in advance: “If I disappeared tomorrow, could someone else find, understand, and use my data?”
Increasingly required by funders (DFG, ERC, NIH, and many national funders) and by ethics committees.
It forces the right questions at the right time, before problems become unfixable (e.g., a consent form that forbids data sharing).
Free tools with templates and guidance: DMPTool and DMPonline.
These are some good examples: Example 1, Example 2, Example 3, DCC UK DMP Database.

From My Own Work: Writing the DMP Habit Into a Live Project

I’m working on a scoping review of reproducibility practices in economics and social science, which is a part of a larger project I’m participated in. The structure of the working folder looks like this:

litsearch/
├── codes/                        R scripts
├── coding/                       Codebooks, coding sheets, and Google Form/Sheet
├── dataset/                      Analysis-ready datasets exported from coding
├── output/
│   ├── search_output/            Literature search pipeline outputs (01–11)
│   ├── analysis_output/          Tables and figures from the analysis
│   └── cleaning_log/             Data-quality flags from response cleaning
└── protocol_logbook.docx         Protocol and decisions logbook

Every file also gets one sentence of description — e.g. 10_final_set.xlsx: “The 145-paper final corpus… verified against Zotero subcollection RIS exports.” See the README here.
Benefit #1 — future me: I come back to this folder after months on other tasks and know exactly where to look.
Benefit #2 — future readers: when this becomes a published paper (hopefully!), reviewers and readers can verify anything I did.

What Goes Into a DMP?

Documentation

What data will you collect and how? Which formats? A codebook describing every variable (name, question wording, coding, missing values) so the data are interpretable without you.

Storage & Ethics

Where do the data live, how are they backed up (3-2-1 rule), who has access? How are participants protected, including informed consent, anonymization or pseudonymization, legal requirements?

Which data can be shared, where, under what license, and when? What is archived long-term (typically 10 years) and in which trusted repository?

FAIR Data

The guiding standard for shared research data (Wilkinson et al., 2016), data should be FAIR:

Findable — deposited with a persistent identifier (DOI) and rich metadata

Accessible — retrievable via standard protocols; access conditions are clear

Interoperable — open formats (CSV, not only SPSS), standard vocabularies

Reusable — clear license, provenance, and enough documentation to reuse

Important: FAIR ≠ open. The principle is “as open as possible, as closed as necessary.” Sensitive data can be FAIR while under restricted access.

Practical Habits That Cost Practically Nothing

File naming: machine-readable, human-readable, sortable — 2026-07-08_study1_raw.csv, not data baru FINAL (2).sav. No spaces, no special characters, dates as YYYY-MM-DD.
Raw data are sacred: store them read-only, in one place; all changes happen via scripts into data/processed/.
Write the codebook while you build the survey.
Choose a repository, not a personal website: OSF, Zenodo, PsychData, RIN Dataverse (BRIN) — they provide DOIs and long-term preservation.
Pick a license so others know what they may do: e.g., CC0/CC-BY for data and materials, MIT/GPL for code.
Plan consent for sharing: add data-sharing language to your informed consent now, as you cannot retroactively ask 800 anonymous participants.

Pre-Registration

The Problem Pre-Registration Solves

Remember the garden of forking paths: with enough flexibility, some significant result can almost always be found.
Statistics can only do its job if the distinction between two modes of research is honest:
- Confirmatory research: hypotheses specified before seeing the data → p-values are meaningful tests.
- Exploratory research: patterns discovered in the data → valuable for generating hypotheses, but p-values no longer mean what they claim.
Both are legitimate! The problem is presenting exploration as confirmation (HARKing).
Pre-registration restores the distinction: you write down your hypotheses, design, and analysis plan before data collection, in a time-stamped, read-only registry (Nosek et al., 2018).

From My Own Work: Locking In the Hypothesis Before the Data

One of my clearer examples: “Science and Religion in the Times of Crisis”, pre-registered on OSF.

The study tests the “dilution effect”: whether offering more explanations for a crisis reduces how useful people find each explanation (Jackson et al., 2024).
I wrote the hypothesis down, in a time-stamped, public record, before I knew whether the data would support it.
It greatly helped me to plan my study better and defend my design choice when discussing it with my supervisor.
Whatever came back, I could not simply reframe the study afterward as “this is what I predicted all along.” The time-stamp made that impossible and that helps keeping me honest with myself.
I have pre-registered every study I have run since 2021.
- It is now simply normal part of designing a study.

What Goes Into a Pre-Registration?

Free registries with templates: OSF Registries and AsPredicted. Typical contents:

Hypotheses & Design

Directional hypotheses, IVs and DVs with exact operationalizations, study design, conditions, and procedure.

Sample & Exclusions

Target sample size with justification (e.g., power analysis), stopping rule, and exclusion criteria.

Analysis Plan

The exact statistical model for each hypothesis, handling of missing data and outliers, inference criteria plus what you will do if assumptions fail.

Common Worries (and Why They’re Mostly Unfounded)

“It’s a prison — I can’t deviate.” You can! Deviations are fine; you just report them transparently (“we deviated from the plan because…”). The registry only makes deviations visible.
“It kills exploration.” No. You can explore freely; results are simply labelled as exploratory. Many templates even have a section for planned exploratory analyses.
“Someone will scoop my idea.” Registrations can be embargoed (kept private) until you publish.
“It’s too much work.” An AsPredicted registration is 9 questions. And most of it is text you need for your ethics application and methods section anyway. You are basically writing it earlier.
Caveat: pre-registration is not magic. Many pre-registrations are vague, and deviations are common and often unreported (van den Akker et al., 2024). The value comes from specific plans and transparent reporting, which is a skill you can practice from your very first study.

Literate Programming

The Copy-Paste Workflow (a.k.a. How Errors Are Born)

The traditional workflow in psychology looks like this:

Run the analysis in SPSS by clicking through menus…
…copy the numbers into a Word document by hand…
…format the table manually…
…then a reviewer asks you to exclude three participants and you do all of it again, hoping you click the same buttons.
Every manual step is a place where errors sneak in and nobody (including you) can later verify what exactly was done.
- Even if someone were to check, it would take a tremendous amount of effort, time, and energy to do so.
Roughly half of published psychology papers contain at least one inconsistent statistic — e.g., a p-value that does not match its own test statistic (Nuijten et al., 2016).

Let us change our traditional attitude to the construction of programs: Instead of imagining that our main task is to instruct a computer what to do, let us concentrate rather on explaining to human beings what we want a computer to do.

Donald Knuth (1984), Literate Programming

Literate Programming: One Document to Rule Them All

Idea: weave your narrative text, your analysis code, and its output (tables, figures, statistics) into a single document that regenerates itself.

You write text and code together; one click (“Render”) runs all code and produces the finished report, so numbers, tables and figures are inserted automatically.
Change the data or the analysis → re-render → every number in the document updates itself without having to manually copy-paste the numbers.
The document is the audit trail: anyone can see exactly which code produced which result.
Fun fact: these very slides are literate programming😉 This is a Quarto document rendered to a presentation.

From My Own Work: Where the Ladder Bent

I pre-registered diligently for years before I learned Quarto. Literate programming was the rung I picked up much after and it changed how I write my papers.

Every project folder is also a Git repository — a full history of every change (version control), so I (or a reviewer) can see exactly how a number came to be.
- Example of version control for this presentation.
The document pulls numbers directly from the analysis.
- I stopped hand-typing results into Word, which is exactly the step where copy-paste errors used to creep in.
Real example: grm-tutorial-paper, the repository behind a published paper written entirely as a literate document.
A pleasantly suprising bonus😉: editors and reviewers comment on how organized the project looks. That’s actually not the point of literate programming, but it doesn’t hurt your reputation either.

The Tools

All free, all open source, all with huge communities and tutorials:

Quarto/R Markdown

Text (Markdown) + chunks of R/Python code → papers, slides, websites, dissertations. Together with R and RStudio/Positron. quarto.org

Jupyter Notebooks

Interactive notebooks for Python (and R, Julia). Common in data science and computational modelling. jupyter.org

Git & GitHub/OSF

Version control: a full history of every change to your code and text, which “track changes” for your whole project, so no more disertasi_final_v7_BISMILLAH.docx.

What It Looks Like

Instead of copying numbers, you write:

```{r}
data <- read.csv(here::here("data", "processed", "study1.csv"))
model <- lm(wellbeing ~ social_media_use + age, data = data)
```

And in your text you write:

Social media use was negatively associated with wellbeing,
b = `r round(coef(model)[2], 2)`.

When rendered, `r ` is replaced by the actual value computed from the data so it is always current, never mistyped.
Packages like papaja and apaquarto (APA manuscripts) and apaTables format entire result sections for you.

Beyond the Document: A Reproducible Project

Literate programming works best inside a well-organized project:

One folder per project, with a standard structure: data/raw/ (read-only!), data/processed/, scripts/, output/, README.md.
Relative paths, never absolute ones: here::here("data", "raw", "survey.csv") instead of C:/Users/amelia/Desktop/... — so it runs on any computer, not just yours.
Set a seed (set.seed(42)) whenever randomness is involved, so results are identical on every run.
Record your environment: package versions change! Tools like renv () or renv + pyenv () snapshot the exact versions you used.
Never edit raw data by hand — every cleaning step happens in a script, so the path from raw data to final table is fully documented. Always keep your raw data untouched.

Computational Empathy

The mindset behind all of these habits is what The Turing Way calls it computational empathy: anticipating the needs of the person who will try to run your work without you in the room.

That person does not have your folder structure, your installed packages, your operating system or the crucial detail that lives only in your head (“oh, you have to run script 2 before script 1”).
Ask yourself: could a stranger on a different computer, with no chance to email me get from my raw data to my results?
In practice, computational empathy means:
- a README that says what the project is, what is where, and in which order to run things;
- stating requirements up front: software, package versions (renv!), expected run time;
- commenting the “why” in your code;
- avoiding hidden dependencies — no absolute paths, no “manually delete row 13 first,” no files that exist only on your desktop.
The kindest test: hand your project to a colleague and watch them try. Every question they ask is a missing piece of documentation.
And remember: the stranger you are being kind to is usually future you.

Registered Reports

Registered Reports: Pre-Registration with Superpowers🦸‍♀️🦸

A Registered Report (RR) is a journal article format where peer review happens before the results exist (Chambers & Tzavella, 2022).

Stage 1: you submit introduction, hypotheses, and detailed methods before data collection. Reviewers critique the design while you can still fix it.
If accepted, you receive In-Principle Acceptance (IPA): the journal commits to publishing your paper regardless of how the results turn out.
Stage 2: you run the study exactly as approved, submit the full paper; reviewers only check that you followed the plan and interpreted results sensibly.
Publication no longer depends on getting “exciting” results → publication bias is cut off at the root, and p-hacking loses its purpose.

From My Own Work: The Rung That Took ~2.5 Years

I have now been part of three published Registered Reports. This is the one I first-authored, and the one I’m proudest of.

From first conceiving the design to the paper appearing in print: about two and a half years.
Published as a Registered Report; postprint available open-access on PsyArXiv.
It was slow, and at several points frustrating.
- Stage 1 review sends you back to redesign before you’ve collected a single data point.
- That is uncomfortable, and it is the entire benefit: I fixed problems early on paper, not after running several studies with ~300 participants.
I would not have attempted this as my first reproducibility practice. It only made sense once pre-registration and literate programming were already regular habits.

Does It Work?

96%

of standard psychology articles report positive (hypothesis-confirming) results — a statistically implausible success rate

44%

of Registered Reports report positive results (Scheel, Schijen, & Lakens, 2021)

300+

journals now offer the RR format, incl. Nature Human Behaviour, Cortex, and many top psychology journals (COS list)

RRs are also rated as more rigorous and higher quality than comparable standard papers (Soderberg et al., 2021).

Registered Reports and Your PhD

The reviewer feedback arrives when it is useful, before you spend a year collecting data.
IPA is a publication guarantee you can plan a dissertation around: no more praying for significant results in study 3 of 3.
Null results become publishable findings, not wasted years, which is honest science and good for your track record.
Timing matters: Stage 1 review takes time, so RRs suit studies planned well in advance (perfect for a PhD proposal!), but less suited for quick pilot work.
In some cases, reviewers may request a pilot study first to justify the design proposed in the RR.
Some journals offer fast-track or scheduled review for student projects; Peer Community In RR (PCI RR) reviews Stage 1 protocols journal-independently.
Talk to your supervisor and co-supervisor early: an RR-based dissertation chapter is increasingly common and highly valuable for one’s scientific career.
- Probably, not yet a common practice in Airlangga, but it is getting common in many parts of the world.

Conclusions

Where I Started

Start with the transparency you need, not the transparency you want. Here is the order I actually climbed:

Data Management Plan — lowest barrier, most immediate payoff. I started here because it directly helped the study I was running right then.
Pre-registration — once organizing data was routine, planning the analysis in advance was the natural next commitment.
Literate programming — a technical skill I built once pre-registration had already made me think in terms of “the plan, written down.”
Registered Report — the hardest rung, and the one that took 2.5 years. I would not have started here.

All of these happened over roughly 15 years, started from one habit at a time.

The Big Picture

Plan → write a data management plan; pre-register your hypotheses and analysis plan — or go all-in with a registered report.
Do → organize your project in one folder; keep raw data read-only; analyze with literate programming so every result is regenerable with one click.
Share → deposit data (FAIR!), code, and materials in a trusted repository with a DOI and a license; link everything from the paper, with enough computational empathy that a stranger can actually run it.
These practices only require habits, and habits are easiest to build now, at the start of your PhD, before you have legacy chaos to untangle.
Reproducibility is a spectrum, not a purity yes/no test: every step (a codebook here, a script there) already makes your work more trustworthy than without doing anything.

Your Starter Kit for Your 1st Year

Concrete first steps:

This Week

Create a free Zenodo account. Set up your first project with a clean folder structure and a README. Rename your files properly (yes, all of them).

This Month

Learn how to use and Quarto or R Markdown by trying out available tutorials (Intro to , R4DS, Quarto TPG). Then, write your next assignment as a literate document. Draft a one-page DMP for your dissertation project.

This Year

Pre-register your first study on OSF or AsPredicted. Discuss with your supervisor whether a chapter could become a Registered Report.

Where to Learn More

The Turing Way — a free, community-written handbook on reproducible research (start with the “Guide for Reproducible Research”).
FORRT — Framework for Open and Reproducible Research Training: glossaries, curated resources, summaries of the key literature.
OSF support pages — hands-on guides for projects, registrations, and preprints.
quarto.org — official tutorials for literate programming with R and Python.
Riverbank reading: Munafò et al. (2017), A manifesto for reproducible science — the best single overview paper to start with.
Find your local open science community (e.g., Airlangga Open Science) — reproducibility is much easier, and more fun, together.

Your closest collaborator is you six months ago, but you don’t reply to emails.

Mark Holder (popularized by Karl Broman) — make life easy for future you.