Analyzing biomarker discovery: Estimating the reproducibility of biomarker sets.
The main results reproduced: recomputed values matched the published ones within tolerance.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Reported values were directly comparable
- ✓The central claim held under reproduction
- 🟡Could not use the authors’ exact input data
- 🟡A deviation arose in the data or preprocessing
- 🟡A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The deviation was non-trivial in magnitude
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
Described well enough to reproduce, but NOT from the cited code: the authors' repo github.com/amirfrz/BMDA ships only LICENSE + a 160-byte README (single 2021 commit, no analysis code). Per brief P16 I re-implemented the fully-specified method (BD_{t,0.05,BH} = two-sample t-test + BH, biomarker iff adj p<0.05; reproducibility = Jaccard; uRS = disjoint balanced half-splits, oRS = duplicate-subjects balanced splits, k=50) and ran it on the assigned dataset GSE7390 (curated breastCancerTRANSBIG = TRANSBIG/Desmedt 2007; outcome = e.dmfs metastasis event) on «our HPC» SLURM. Result is honest 1:1 PARTIAL: uRS = 0 ± 0 % matches the paper EXACTLY (only ~1 probe survives BH on full data, so half-splits never overlap); oRS = 1.64 ± 1.44 % vs reported 5.15 ± 2.81 % — same near-zero single-digit-% regime, numerically lower (likely t-test variant Student-vs-Welch, exact overbound duplication/split scheme, or seed). The paper's central GSE7390 claim — BD_{t,0.05,BH} finds essentially zero reproducible biomarkers — is reproduced. NOT attempted (80/20): the other ~24 Table-1 datasets (incl. controlled-access Metabric/EGA and TCGA), SNP variants, survival-median binarization, and the Shiny web tool. Audit flag: Table 1 is not regenerable from the shipped repo (empty); values credible from the described method, not flagged as fabrication.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 83assessed: 2026-06-15 ⛓ 05e8b9f60fd5
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-15
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-15no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusCan the reproducibility of a biomarker set produced by a fixed univariate biomarker discovery process be quantified and reliably estimated from a single labeled dataset, so researchers can judge whether their proposed biomarkers would recur in future comparable studies?
- ★ A Reproducibility Score, RS(D,BD), defined as the average Jaccard overlap between biomarker sets found by the same discovery process on comparable datasets from the same distribution, quantifies biomarker reproducibility on a 0-1 scale method
- ★ RS cannot be computed directly from one dataset but can be reliably bracketed by an algorithmic overbound and underbound for the case of univariate hypothesis testing on dichotomous groups method
- ★ The overbound and underbound approximations are meaningful and match known reproducibility results across many real microarray, mRNAseq, and SNP datasets finding
- ★ MCC correction and stringent p-value thresholds (e.g. p<0.05), while controlling false positives, can reduce the reproducibility of discovered biomarker sets finding
- ★ Reproducibility increases with sample size; small datasets yield low reproducibility finding
- ★ A publicly available web app and code compute RS estimates for any labeled dataset with binary class labels and continuous or discrete features resource
- The t-test was chosen as the main biomarker discovery process over fold-change because prior simulation work showed t-tests have higher recall for true biomarkers method
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| Gene expression microarray (continuous features), univariate t-test biomarker discovery | Breast cancer microarray dataset GSE7390 and other GDS datasets | none | Reproducibility Score over/under bounds; Jaccard overlap of biomarker sets; t-test p-values vs binary outcome (e.g. metastasis) | NCBI GDS browser datasets |
| mRNAseq gene expression (continuous features), univariate t-test biomarker discovery | TCGA-BRCA (breast cancer) and TCGA-KIPAN (pan-kidney) tumor data | none | Reproducibility Score approximations | GDC portal / LinkedOmics |
| SNP / GWAS-style discrete-feature biomarker discovery | SNP dataset (EGAD00010000268) | none | Reproducibility Score approximations for discrete features | EGA archive dataset |
- – Overbound and underbound approximations of RS match known reproducibility results across many datasets
- – Breast cancer signatures of van't Veer (70 genes) and Wang (76 genes) shared only 3 genes in common 3 genes
- – Only 17 genes overlapped between Sorlie's 456-gene list and van't Veer's 231-gene list; only 2 genes shared between Sorlie and Ramaswamy lists 17 and 2 genes
- – Only 6 of 53 published cancer-biology findings could be confirmed (~10% reproducibility) 6/53 (~10%)
- ▼ Reproducibility score decreases for datasets with smaller sample sizes
- – PAC-sorting worst-case analysis implies thousands of patients per dataset are needed to guarantee 50% gene-list overlap for breast cancer 50% overlap requires several thousand patients
- pvalue p<0.05 (Threshold for declaring a feature a biomarker via t-test (after MCC correction))
- count 3 genes in common (out of 70 and 76) (Overlap between van't Veer and Wang breast cancer signatures)
- count 17 of 456 vs 231 (Overlap between Sorlie and van't Veer gene lists)
- count 6 of 53 (Begley and Ellis confirmable published cancer findings)
- other 70% of researchers failed to reproduce others' experiments; 52% perceived a reproducibility crisis (2016 Nature survey of over 1500 scientists)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This methodological paper proposes a Reproducibility Score (RS) for biomarker discovery processes, defined as the expected Jaccard similarity between biomarker sets produced from two independently drawn, comparable datasets from the same distribution. Rather than conducting primary inferential comparisons between biological groups, the authors develop resampling-based algorithms that produce over-bounds and under-bounds on this score for a given dataset and biomarker discovery process. Empirical validation is conducted across multiple public microarray and SNP datasets, using per-feature univariate t-tests (with MCC correction) as the primary biomarker discovery mechanism, with the RS bounds serving as the main reported quantitative outputs.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Two-sample univariate t-test (per feature) | Applied to each feature independently to identify differentially expressed genes or SNPs between binary outcome groups (case vs. control); used as the primary biomarker discovery process BD(·) across all empirical datasets | — | not stated |
| Jaccard similarity coefficient | Measuring overlap between pairs of biomarker sets discovered from comparable dataset draws or subsamples; forms the basis of the Reproducibility Score definition (Eq. 1) | — | na |
| Multiple Comparison Correction (FDR and/or FWE) | Applied to per-feature t-test p-values before declaring biomarkers at p < 0.05 threshold; effect of MCC type and stringency on RS is also studied in supplementary experiments | — | not stated |
-
The Jaccard index (|A∩B|/|A∪B|) is used to measure overlap between two discovered biomarker sets, forming the core of the Reproducibility Score↳ Could also: The Sørensen–Dice coefficient (2|A∩B|/(|A|+|B|)) or the overlap coefficient (|A∩B|/min(|A|,|B|)) could also quantify biomarker-set similarity — Dice weights shared elements more heavily relative to set size, and the overlap coefficient is insensitive to size asymmetry between sets; either may be preferred when MCC thresholds produce biomarker sets of highly variable or unequal size across subsamples, which is a documented phenomenon with stringent corrections
-
Resampling-based over-bounds and under-bounds are used as paired approximations to the true RS↳ Could also: Bootstrap confidence intervals around a central RS point estimate could also characterize estimation uncertainty — Bootstrap CIs provide a single central estimate with a symmetric margin of error, which maps onto familiar inferential reporting conventions and may be more interpretable to practitioners comparing RS values across datasets or discovery algorithms
-
Univariate t-tests are used as the sole primary biomarker discovery process in all empirical experiments↳ Could also: Non-parametric alternatives such as the Wilcoxon rank-sum (Mann-Whitney U) test could also serve as the univariate discovery process within the same RS framework — Wilcoxon rank-sum does not assume normality of feature distributions, which may matter for skewed gene expression data; applying the RS framework with a rank-based test would demonstrate whether the RS bounds generalise across discovery methods with differing distributional assumptions
-
Standard BH-FDR and Bonferroni-type FWE corrections are discussed as the MCC options, applied to correlated gene-expression features↳ Could also: Permutation-based FWE control (e.g., Westfall-Young) could also be applied, accounting for the actual dependence structure among features — Standard BH-FDR assumes independence or positive dependence among tests, whereas gene expression features are often strongly correlated; permutation-based procedures use the observed correlation structure and could produce different biomarker sets, with downstream effects on the RS that would be informative to characterise
-
Empirical validation of the RS approximations relies entirely on real public datasets where ground-truth biomarkers are unknown↳ Could also: Simulation studies with known ground-truth biomarker sets (as used by Klebanov et al., which the paper cites approvingly) could also benchmark the RS bounds — Simulated data with defined true positives would allow direct measurement of bound tightness against a gold standard and would support reporting of sensitivity and specificity of the over- and under-bounds, complementing the empirical validation on real data
-
Reproducibility is defined and measured at the level of individual feature identities (exact gene or SNP match between two discovered sets)↳ Could also: Reproducibility could also be assessed at the pathway or gene-set enrichment level, measuring whether discovered features map to the same biological pathways even when individual identifiers differ — Many correlated features may be equivalently associated with an outcome, so individual feature lists vary across studies while the underlying biological signal is stable; a pathway-level RS would capture this and could be a complementary or more biologically interpretable measure, particularly for multi-factorial diseases the paper explicitly discusses
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
No assessed neighbours yet — the network grows as more papers are assessed.
Data lineage
The datasets this paper uses (text-mined from the full text via Europe PMC), and which other assessed papers stand on the same data. A shared dataset is a factual link — not a judgement.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-35901020
Paper: Forouzandeh, Rutar, Kalmady, Greiner. "Analyzing Biomarker Discovery: Estimating the Reproducibility of Biomarker Sets." PLoS One 2022. PMID 35901020, PMC9333302.
What the paper does (pipeline-derived): Defines a reproducibility score
(RS) for biomarker-discovery (BD) algorithms = expected Jaccard
J(A,B)=|A∩B|/|A∪B| between biomarker sets found on two independent datasets.
RS is bounded by two computable estimators on a single dataset D with k=50 reps:
- uRS (underbound): J between BD on two disjoint outcome-balanced halves.
- oRS (overbound): duplicate subjects → 2n pool, J between BD on two
outcome-balanced size-n subsets.
Primary BD algorithm
BD_{t,0.05,BH}: per-probe two-sample t-test, BH-adjust, biomarker iff adjusted p<0.05. Reported per dataset in Table 1 (uRS, oRS).
In scope (attempted)
- GSE7390 row of Table 1 (the RU's assigned dataset). Reported:
uRS = 0 ± 0 %,oRS = 5.15 ± 2.81 %. Fully pipeline-derived, public data, method fully specified in Methods → reproducible.
Out of scope (not attempted) — and why
- Other 24 datasets in Table 1 (GDS968, Metabric, TCGA-BRCA/KIPAN, 18 GEO sets,
7 SNP sets): same method, but 80/20 — one clear data point chosen. Metabric and
EGA (
EGAD00010000268) are controlled-access (data_restricted). TCGA needs separate large downloads. - SNP-based and survival-binarization variants, the Shiny web tool (biomarker.shinyapps.io/BiomarkerReprod), and figures beyond Table 1.
Code-availability note (audit-relevant)
The cited code repo github.com/amirfrz/BMDA contains only LICENSE + a
160-byte README (single "Initial commit", 2021-08-10); no analysis code was
ever committed. Per brief rule P16, a faithful re-implementation of the
fully-specified method on the paper's own data is an equally valid reproduction —
that is what was done here. The empty repo is flagged so a human reviewer knows
the reported numbers are not runnable from the shipped artifact.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
The central GSE7390 claim reproduces: BD_{t,0.05,BH} finds essentially zero reproducible biomarkers, with uRS matching exactly (0±0%, only ~1 probe survives BH so half-splits never overlap). The oRS deviates ~3× (1.64±1.44% reproduced vs 5.15±2.81% reported) but stays in the same near-zero regime, with the direction and qualitative conclusion intact. The discrepancy sits on our methodology side (underspecified t-test flavor, duplication/split scheme, label/seed) in a method the paper does not fully pin down, not on the authors' data. A real audit flag — not fabrication — is that the cited repo is empty (LICENSE+README only), so Table 1 is not regenerable from the shipped code, only from the described method; values remain credible.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.