Does It Reproduce?

www.doesitreproduce.com

Does It Reproduce?

Autonomous AI-driven re-runs of published computational biomedical pipelines

Please read this first

These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.

At a glance

643 of 1,273 assessed publications have been reproduced so far (score ≥ 75) — browse them.

Score distribution

Reproducibility scores across all assessed papers, with a fitted normal curve — filter to see how journal, year, topic or impact reshapes it.

How the database grows

Reproduced over time

1,273 publications worked through

last 7 days

Cumulative studies processed — full and partial reproductions (a partial is reproduced, just not fully). Dashed line = the fully-reproduced subset.

processed (total) (1,273) fully reproduced (643)

1273 processed total · 24/day average over 53 days

Experimental results indexed over time

4,659 papers indexed

3535

last 7 days

Cumulative papers whose full text we extracted into structured claims, setups, results and statistics — the searchable backbone of the corpus. Dashed line = high-completeness extractions.

indexed (total) (4,659) high completeness (4,526)

4659 papers indexed · 103.5/day average over 45 days

35,075 core claims 27,052 experimental setups 32,439 key results 34,197 statistics

structured, searchable fields extracted from the indexed full texts — growing with every paper

Dataset quality assessment over time

319,965 datasets with a quality score

88204

last 7 days

Cumulative data accessions (GEO / SRA / ENA / …) that carry an actual quality score. Dashed line = scored from measurements on the data itself, not from the deposit record.

scored (total) (319,965) measured from the data (59,528)

319,965 datasets scored · 6955.8/day average over 46 days — a further 1,888,274 accessions are catalogued but not yet scored

🤝

Reproduced automatically — and fairly

Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.

Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.

How this works

Autonomous reproduction

Every reproduction runs autonomously on the

Agents read each paper, fetch its data, rerun the analysis on the brainbox compute, and grade what reproduces — end to end, without a human in the loop.

The pipeline, live

Pipeline paused ♥ Support the compute

Reproductions run continuously. Here is what just finished — and the papers readers have asked for next.

1 reproduced (24 h) 0 open reader requests

Recently reproduced

Requested by readers — up next

nothing yet

How it works

Each publication is read on two independent axes. The Level (L1–L4) says how rigorously it was examined; the 0–100 reproducibility score is the outcome of the attempt. Both are shown together (e.g. L3 · 82). A score is never a statement about the authors.

Level — how it was curated

L1Fully automated, AI-curated re-run of the bioinformatic pipeline
L2Human scientist plausibility check of the L1 AI curation
L3Scientist-curated partial re-run of the bioinformatic pipeline
L4Scientist-curated full re-run of the bioinformatic pipeline

Score — the reproducibility verdict

A 0–100 reproducibility score from hands-on re-running of the paper's computational claims — the share that held up, colour-banded:

<50 · did not hold 50–74 · mixed ≥75 · held up

No computationPure wet-lab / in-vivo paper — nothing computational to recompute. Neutral: neither pass nor fail.
InconclusiveThe reproduction could not be completed or scored (e.g. data unavailable, pipeline error). No conclusion is drawn.
⚑A separate, human-reviewed flag — independent of the score and never assigned automatically.

Every score stores an auditable, ACMG-style worksheet: each item that counted toward it, the exact part of the reproduction it came from, its points, and any decisive gate. Open any record to inspect it line by line.