Analytical code sharing practices in biomedical research.
The main results reproduced: recomputed values matched the published ones within tolerance.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Same input data as the authors
- ✓Reported values were directly comparable
- ✓The central claim held under reproduction
- 🔴A deviation arose in the data or preprocessing
- 🔴A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The deviation was non-trivial in magnitude
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
DESCRIBED WELL ENOUGH -> clean 1:1 reproduction. Meta-research/bibliometric paper whose entire computational result is one notebook (notebooks/Figures.ipynb) over a small shipped curated TSV (data/240_B.txt, 453 rows x 29 cols) + journal_data.csv in github.com/Mangul-Lab-USC/code-availability @ 30c574d (all three input SHA256 match the deposit). Reproduced FRESH on «our HPC» (SLURM «job», partition std, node n093, 00:01:34) by building the AUTHORS' EXACT pinned env (python 3.9.7, pandas 1.4.1, numpy 1.26.1, scipy 1.11.1, matplotlib 3.8.1, seaborn 0.13.0, matplotlib-venn 0.11.9, pySankey2 2.0.1) on node-local disk and executing the authors' real Figures.ipynb end-to-end with nbconvert -- ZERO cell errors (every numeric AND plotting cell ran). Cross-checked by an independent re-computation (repro_stats.py) and a C5 probe (probe_statement.py). 15 of 17 graded claims reproduce EXACTLY to 3 sig figs from the authors' own code+data: N=453; 49.9% code-sharing (226/227); statement 209/453=46.1% and 179/209=85.6% shared; primary 36.3% vs secondary 61.7% (chi2=28.1); code+data primary chi2=44.2; data 69.6% of 204; both/data-only/code-only/neither 29.9/39.7/6.4/24.0%; temporal cumulative 7.3->49.9%; platforms GitHub 76.9% etc; data repos GEO 28.8%/SRA 17.8%; journal mandatory OR 2.3x (p=1.9e-5) and chi2=17.9. ONE MISMATCH/FLAG (C5): the reported statement-vs-availability 'Pearson chi2=42.8, p=2.7e-9' is NOT computed by any cell in the shipped notebook, is internally inconsistent (chi2=42.8 at df=1 -> p6e-11, not 2.7e-9), and the natural 2x2 from the shipped data gives chi2196-198 (p~1e-44) -> flagged as possibly mis-stated/non-reproducible for a HUMAN reviewer, NOT asserted as fabrication. NOT ATTEMPTED: C14 Fig 2 link-stability (manual/scripted temporal link-checking, out of scope, not in shipped table) and exact EPS/PNG figure regeneration (cosmetic). No fabrication beyond the single C5 flag; honesty over coverage.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 92assessed: 2026-06-16 ⛓ 5d8405f254f7
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-22
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-16no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetThe study investigates the extent to which analytical code (and accompanying data) is shared in biomedical research publications, and whether factors like code availability statements, analysis type (primary vs. secondary), and journal policy influence actual code sharing rates.
- ★ Nearly half (49.9%) of 453 examined biomedical manuscripts failed to share the analytical code used to generate their results finding
- ★ Presence of a code availability statement is significantly associated with actual code availability finding
- ★ Studies performing secondary analyses were more likely to share code than studies performing primary analyses finding
- ★ Data sharing is a more prevalent practice than code sharing in biomedical research finding
- ★ Code sharing practices have increased over time, from 7.3% of articles in 2016 to 49.9% in 2021 finding
- ★ Journal code/data sharing policies (mandatory vs. encouraged) positively impact code sharing rates finding
- Fewer than 1% of code-sharing links were non-functional/unstable, indicating good archival stability where code is shared finding
- A curated dataset and Python analysis code documenting code/data sharing status of 453 manuscripts is publicly available resource
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| Bibliometric/full-text survey using Python ElementTree parsing of XML manuscripts | 453 biomedical research manuscripts (2016-2021) from 8 journals (Nature Biotechnology, Genome Medicine, Nature Methods, Genome Biology, Bioinformatics, BMC Bioinformatics, Nucleic Acids Research, Nature Genetics) | none | presence/location of code and data availability, availability statements, code/data hosting repository, link functionality, citation counts | Python (ElementTree) |
| Chi-square test of independence | 453 manuscripts stratified by primary vs. secondary analysis and by presence/absence of code and data availability statements | none | statistical association between availability statements and actual sharing, and between analysis type and code sharing | scipy 1.11.1 |
| Odds ratio calculation with Fisher's exact test | 8 journals classified by code/data sharing policy (Mandatory, Encouraged/Mandatory, No policy) | journal policy type | odds ratio and significance of policy effect on code/data availability | scipy 1.11.1 |
| Manual link verification | Subset of manuscripts (n=453) with shared code/data links | none | functional vs. non-functional (broken) code-sharing URLs | — |
- – 49.9% of 453 studies did not share analytical code 49.9%
- – Of shared code, 76.9% was hosted on GitHub, 13.7% on other platforms, 5.9% on Zenodo, 2.6% as Supplemental Material, 0.8% on GitLab 76.9%
- ▲ Code availability statement significantly associated with code sharing chi2=42.8, p=2.7e-9
- ▲ 61.7% of secondary analysis articles shared code vs. 36.3% of primary analysis articles chi2=28.1, p=1.15e-7
- – Only 29.9% of primary analysis articles shared both code and data; 24.0% shared neither; data-only sharing (39.7%) far exceeded code-only sharing (6.4%) chi2=44.16, p=3.02e-11
- ▲ Cumulative code sharing rose from 7.3% (2016) to 49.9% (2021)
- – GEO was the most common data repository (28.8% of shared data), followed by SRA (17.8%) 28.8%
- ▲ Data availability statement significantly associated with data sharing chi2=125.1, p=6.1e-27
- pvalue p = 2.7 × 10^-9 (association between code availability statement and code sharing (chi2=42.8))
- pvalue p = 1.15 × 10^-7 (code sharing rate between secondary vs primary analysis studies (chi2=28.106))
- pvalue p = 3.02 × 10^-11 (code and data sharing across all studies (chi2=44.16))
- pvalue p = 6.1 × 10^-27 (data availability statement association with data sharing (chi2=125.1))
- other 49.9% (proportion of studies not sharing analytical code)
- other 209 (46.1%) (manuscripts using a code availability statement)
- other 179 (85.6%) (manuscripts with code availability statement that actually shared code)
- count 480 sampled, 453 analyzed (27 excluded) (manuscripts randomly sampled from 12,603 PMC open-access articles across 8 journals, 2016-2021)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This cross-sectional bibliometric study examined analytical code and data sharing practices across 453 randomly selected manuscripts from eight biomedical journals (2016–2021). Associations between categorical variables—such as code sharing, data sharing, availability statements, analysis type, and journal policy—were tested primarily with Pearson's chi-square tests of independence; journal policy effects were assessed with standard odds ratios and Fisher's exact tests. Results were reported as proportions and percentages with exact p-values, using Python/scipy for all tests.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Pearson's chi-square test of independence | Association between presence of code availability statement and actual code sharing (Fig. 1B) | 453 | not stated |
| Pearson's chi-square test of independence | Code sharing rate by analysis type: primary vs. secondary analysis (Fig. 1C) | 453 | not stated |
| Pearson's chi-square test of independence | Code and data sharing status among primary analysis studies (Fig. 1D) | 204 | not stated |
| Pearson's chi-square test of independence | Association between data availability statements and data sharing | 453 | not stated |
| Standard odds ratio with Fisher's exact test | Effect of journal policy (mandatory vs. encouraged) on code and data availability | — | not stated |
-
Multiple Pearson's chi-square tests were conducted across related hypotheses (code sharing, data sharing, statement presence, analysis type, journal policy) without any multiplicity correction↳ Could also: Apply a false discovery rate correction (e.g., Benjamini-Hochberg) or Bonferroni adjustment across the family of tests — When several correlated hypotheses are tested in the same study, a correction procedure bounds the expected proportion of spurious findings; reporting both raw and adjusted p-values gives readers a basis for assessing which associations remain notable under stricter thresholds
-
Effect magnitude for chi-square tests was not reported beyond the chi-square statistic value itself↳ Could also: Report Cramér's V (or phi for 2×2 tables) alongside each chi-square result — Cramér's V standardises the chi-square statistic relative to both sample size and table dimensions, conveying practical magnitude independent of n; with n = 453, even small associations can reach low p-values, so a standardised effect-size metric helps readers gauge substantive importance separately from statistical significance
-
Key proportions (e.g., 49.9% code-sharing rate, 61.7% secondary-analysis sharing rate) were reported as point estimates without confidence intervals↳ Could also: Add 95% confidence intervals around primary proportions using Wilson score or Clopper–Pearson intervals — Confidence intervals communicate the estimation uncertainty around observed proportions and let readers directly assess precision; they are especially informative when comparing rates across subgroups of unequal size
-
Year-over-year trends in code and data sharing (2016–2021) were characterised descriptively by listing annual cumulative percentages↳ Could also: Apply the Cochran–Armitage test for trend, or fit a logistic regression with calendar year as a continuous predictor — A formal trend test would quantify whether the observed monotonic increase is statistically supported and yield an effect estimate (e.g., odds ratio per year); logistic regression could additionally adjust for journal or analysis type simultaneously
-
The odds ratio for journal policy effects was reported without an accompanying confidence interval↳ Could also: Report a 95% confidence interval around the odds ratio (e.g., via the Woolf log-linear method or the exact conditional method paired with Fisher's test) — A confidence interval conveys both the direction and the precision of the estimated association and is the standard complement to an odds ratio in epidemiological and survey reporting
-
Articles were sampled using a stratified design (10 per journal per year), but analyses treated all 453 observations as if drawn from a simple random sample↳ Could also: Use survey-weighted chi-square tests (e.g., Rao–Scott correction) or mixed-effects logistic regression with journal as a random effect — Observations within the same journal may share characteristics (editorial culture, author community) that induce clustering; explicitly modelling that structure yields standard errors and p-values that reflect the actual sampling design and can affect conclusions for outcomes that vary substantially across journals
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
No assessed neighbours yet — the network grows as more papers are assessed.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-38983240 "Analytical code sharing practices in biomedical research"
Sharma et al., PeerJ Comput Sci 2024. DOI 10.7717/peerj-cs.2066. Repo: https://github.com/Mangul-Lab-USC/code-availability Data: zenodo 10.5281/zenodo.10108745
Nature of the paper
This is a meta-research / bibliometric study, NOT a wet-lab or omics-pipeline
paper. The authors manually curated 480 randomly-sampled biomedical articles
(genomics-heavy) and annotated, per article, whether analytical code / data were
shared, where, and author/journal metadata. The single computational artifact is
notebooks/Figures.ipynb (pandas + scipy + matplotlib/seaborn) run over the
curated annotation table data/240_B.txt (TSV) and data/journal_data.csv.
In scope (pipeline-derived = the notebook's deterministic statistics)
All headline numbers are deterministic value_counts / groupby / chi2 over the
shipped curated table — fully reproducible by re-running the notebook on the data.
See claims.tsv C1-C13, C15. These are the core reproducible results:
- C1/C2 Fig 1A: N=453, code-sharing rate 49.9%.
- C3-C5 Fig 1B: code-statement counts + chi2.
- C6-C7 Fig 1C: primary vs secondary + chi2.
- C8-C10 Fig S2/1E: data sharing & combined code+data in primary.
- C11 Fig 1F: temporal trend.
- C12-C13 Table S1 / Fig S8: hosting / data-repository distributions.
- C15 Fig 3C: journal-policy odds ratio (needs journal_data.csv join; attempt last).
Out of scope
- C14 Fig 2 link-stability (<1% unstable): produced by manual/scripted link-checking of URLs over time, not derivable from the shipped annotation table -> not attempted.
- Regenerating the exact EPS/PNG figure files is cosmetic; we reproduce the NUMBERS the figures encode, which is what matters for fabrication detection.
- The upstream PMC-scraping / sampling scripts (scripts/*.sh) that built the annotation table are not re-run: the curation is manual human annotation, not a reproducible pipeline. We take the shipped curated table as the input (the paper's own data), per brief rule P16/2.
Method
Clone repo + build the pinned conda env (python 3.9.7, pandas 1.4.1, scipy 1.11.1) on «infra» inside a «our HPC» compute job; execute Figures.ipynb with nbconvert and ALSO run a focused verification script computing C1-C13 directly from 240_B.txt. Compare both against the reported values. Pure-CPU, seconds of compute, tiny data.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
This is essentially a 1:1 reproduction of a bibliometric meta-research paper: 15 of 17 claims reproduce exactly from the authors' own pinned environment, code, and shipped 453-row TSV, with only trailing-digit rounding differences. The single substantive deviation is C5 (statement×availability chi2=42.8, p=2.7e-9), which is not derivable from the shared data, is internally inconsistent (chi2=42.8 at df=1 ⇒ p≈6e-11), has no computing cell in the deposited notebook, and the natural 2×2 gives chi2≈196–198 — so the deviation sits on the authors' side (a likely mis-stated/typo'd statistic). However the association itself still holds (even stronger), so the central conclusion is unaffected; C14 link-stability is out-of-scope (snapshot 0.43% is consistent with the reported <1%). Net: solid reproduction with one flagged non-reproducible secondary statistic → overall yellow.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.