Prognostic biomarker discovery in pancreatic cancer through hybrid ensemble feature selection and multi-omics data.
The main results reproduced: recomputed values matched the published ones within tolerance.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Same input data as the authors
- ✓Reported values were directly comparable
- ✓No relevant deviation in data/preprocessing
- ✓No authors-side cause for any deviation
- ✓Reported values are derivable from the shared data
- ✓Any deviation was negligible
- ✓The central claim held under reproduction
- ✓Overall, the reproduction was clean
- Every checked point held up.
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
DESCRIBED WELL ENOUGH -> 1:1 reproduced. The paper (Zobolas et al., BioData Mining 2026) benchmarks hybrid ensemble feature selection (hEFS) for PDAC multi-omics survival prediction across 3 cohorts (cao2021=CPTAC n125, wissel2023=TCGA-PAAD n81, osipov2024=MolTwin n71). Its own MIT repo (bblodfon/pdac-efs-bench2024 @bc174b2) SHIPS the preprocessed data AND the pipeline result objects (result.rds, result_auc.rds, fs.rds, timings.rds); only the step-1 hEFS efs/ EnsembleFSResult objects are withheld (by-request, most-expensive step). Two phases, both on «our HPC». PHASE A: recomputed every reported Fig 3/4/5 + text number directly from the shipped pipeline outputs (minimal R env) -- all 14 reported values match (6 exact, 8 within-tol): per-cohort C-index ranges, multi-omics feature counts (CoxLasso 200-300, hEFS ~50), per-omic counts (<15), Clinical+GEX C-index 0.64, 12-month Uno AUC 0.67(TCGA)/0.63(CPTAC), and ALL per-method training timings (hEFS-9 ~14/8/11 min, EFS-CoxLasso ~15s, etc.). This is an anti-fabrication check: every printed number is derivable from the shipped data -> NO fabrication signs. PHASE B: a GENUINE re-run of bench/run_mm_bench.R (byte-faithful copy of its mm_bench task/learner/scoring) consuming the shipped fs.rds, regenerating the per-fold Harrell C-index over 100 MC-CV splits for the 34 core (dataset x config x FS) cells, and comparing to the shipped result.rds -> 34/34 within 0.01 C-index, 16 EXACT (deterministic Cox/CoxLasso cells, delta=0), RSF cells differ <=0.0053 from random-forest stochasticity. Required pinning the repo's exact mlr3 stack (mlr3 0.22.1, mlr3proba 0.7.1@91052a0, mlr3extralearners 1.0.0@ea83b0d, renv.lock) because the shipped mlr3 task_list.rds objects fail to deserialize under newer mlr3. NOT ATTEMPTED (hard 20%, by design): (1) regenerating the step-1 hEFS efs/ objects from scratch (not shipped, by-request) and hence run_fs.R / the exact hEFS feature rankings -- we consumed the shipped fs.rds; (2) the BlockForest integration model and the Uno-AUC/Brier measures in Phase B (kept the R env light); (3) figure typesetting and the Nogueira-stability / redundancy-FDR secondary panels.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 92assessed: 2026-06-16 ⛓ c2fe55473eff
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-16
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-16no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusCan a hybrid ensemble feature selection (hEFS) framework—combining data subsampling with multiple embedded and wrapper-based survival models, voting-theory-inspired ranking, and Pareto-front optimization—identify sparse, stable, and predictive prognostic biomarkers from high-dimensional multi-omics survival data without relying on user-defined thresholds?
- ★ The hEFS framework integrates data subsampling with multiple prognostic models (embedded and wrapper-based), aggregates feature rankings via a voting-theory-inspired approach, and selects the optimal feature subset via Pareto front optimization, eliminating user-defined thresholds. method
- ★ Across all three PDAC cohorts, hEFS selected substantially fewer and more stable features than CoxLasso (~10 vs. ~60 per modality) with lower variance. finding
- ★ hEFS maintains discrimination performance comparable to clinical baselines (C-index ~0.54–0.60) despite selecting far fewer features. finding
- ★ Prognostic performance depended mainly on modality choice rather than integration model, with gene expression plus clinical variables achieving the highest discrimination (C-index ~0.64). finding
- ★ Feature redundancy was low and ensemble subsampling further improved stability. finding
- ★ The hEFS methodology is openly implemented in the mlr3fselect R package built on the mlr3 ecosystem. resource
- ★ The framework is extended to multi-omics data via a late-fusion strategy and benchmarked against conventional late-fusion CoxLasso. method
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| Multi-omics survival benchmarking (feature selection comparison hEFS vs CoxLasso) | Three pancreatic ductal adenocarcinoma (PDAC) patient cohorts | none | sparsity, stability, redundancy, discrimination (C-index), computational cost | mlr3fselect / mlr3 R ecosystem |
| Gene expression-based survival modeling (with clinical variables) | PDAC patient cohorts | none | discrimination performance (C-index ~0.64) | mlr3fselect R package |
| Monte Carlo cross-validation evaluation | PDAC multi-omics datasets | data subsampling/resampling | predictive performance and feature selection stability across 100 iterations | — |
- ▼ hEFS selected far fewer features per modality than CoxLasso (~10 vs ~60) ~10 vs ~60 features
- – hEFS maintained discrimination comparable to clinical baselines C-index ~0.54–0.60
- ▲ Gene expression plus clinical variables achieved the highest discrimination power C-index ~0.64
- – hEFS yielded lower variance and improved stability via ensemble subsampling, with low feature redundancy
- count ~10 vs. ~60 features per modality (hEFS vs CoxLasso) (sparsity comparison across all cohorts)
- other C-index ~0.54–0.60 (hEFS discrimination comparable with clinical baselines)
- other C-index ~0.64 (gene expression plus clinical variables, highest discrimination)
- count 100 Monte Carlo cross-validation iterations (benchmarking iterations)
- count 3 PDAC cohorts (pancreatic ductal adenocarcinoma cohorts benchmarked)
- count nine survival prediction models (models combined in the hEFS ensemble)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This methods and benchmarking paper introduces a hybrid ensemble feature selection (hEFS) framework for high-dimensional multi-omics survival analysis, evaluated against a CoxLasso baseline using 100 Monte Carlo cross-validation (MCCV) iterations across three pancreatic ductal adenocarcinoma (PDAC) cohorts. Performance was assessed across five dimensions: sparsity (number of selected features), selection stability, inter-feature redundancy, discrimination (concordance index, C-index), and computational cost. Results are reported as approximate aggregate C-index values (~0.54–0.64) by cohort and omics modality combination; the full results and inference details are in sections not available in the provided text excerpt, which terminates mid-Methods.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Concordance index (Harrell's C-index) | Discrimination performance of hEFS vs CoxLasso across three PDAC cohorts and multiple omics modalities/integration strategies | — | not stated |
| 100-iteration Monte Carlo cross-validation (resampling-based benchmarking framework) | All evaluation metrics — sparsity, stability, redundancy, C-index, computational cost — compared between hEFS and CoxLasso | — | not stated |
| Pareto front optimization (multi-objective, non-inferential) | Determining the optimal feature subset size by balancing sparsity and predictive performance within hEFS | — | na |
-
Discrimination was summarized with the C-index (Harrell's concordance statistic) as the primary predictive performance metric↳ Could also: The Integrated Brier Score (IBS) or time-dependent AUC (td-AUC) could also be used, either alongside or instead of C-index — The C-index is a rank-based summary collapsed across all time points and does not assess calibration; IBS captures both discrimination and calibration over time, while td-AUC allows time-specific assessment — both are informative for heterogeneous oncology cohorts where event timing matters
-
Method comparison across the 100 MCCV iterations is described in the abstract with approximate summary values, without mention of formal hypothesis tests↳ Could also: A paired Wilcoxon signed-rank test or a linear mixed-effects model on the per-iteration metric distributions could also formalize the comparison between hEFS and CoxLasso — Since both methods are evaluated on the same set of MCCV splits, a paired test on the 100 iteration-level metric values would provide inferential evidence (p-value, effect size with CI) for whether observed differences exceed chance variation, complementing descriptive summaries
-
Feature stability was evaluated within the hEFS framework using internal voting-frequency-based measures across subsamples and models↳ Could also: Standardized external stability indices such as the Jaccard similarity coefficient, the Pearson correlation of selection-frequency vectors, or the stability metric of Nogueira et al. (2018) could also be reported — External, method-agnostic stability metrics facilitate direct comparison with published feature selection benchmarks and enable meta-analytic synthesis across studies, which is harder when stability is defined and measured internally to a single framework
-
Multi-omics data were integrated using a late-fusion strategy (separate per-modality models with predictions combined post-hoc)↳ Could also: Early fusion (concatenating all omics layers into a single feature matrix before modeling) or structured regularization approaches such as priority-Lasso or IPF-Lasso could also serve as integration baselines — Including early fusion as an additional comparator would quantify whether modality-specific modeling in late fusion provides systematic benefits over simple concatenation — a practically important reference point for practitioners selecting an integration strategy
-
The optimal feature subset size was selected via a Pareto front optimization balancing sparsity and C-index, designed to eliminate user-defined thresholds↳ Could also: The one-standard-error rule applied to the cross-validated C-index curve, or information criteria (e.g., AIC adapted for penalized Cox models) could also be used as threshold-free subset-size selection strategies — The one-SE rule and information criteria are widely established and interpretable; including them as comparators would contextualize the Pareto approach within the broader methodological landscape and clarify when each strategy differs in practice
-
Benchmarking used 100 Monte Carlo cross-validation iterations (random train–test splits repeated independently) as the evaluation framework↳ Could also: Repeated stratified k-fold cross-validation or the 0.632+ bootstrap estimator could also be used as evaluation frameworks — MCCV, repeated k-fold CV, and the 0.632+ bootstrap each carry different bias-variance trade-offs for performance estimation; k-fold CV uses data more efficiently in small-n settings, while 0.632+ bootstrap is known to produce lower optimism bias — reporting sensitivity to the resampling scheme would strengthen generalizability claims
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
Assessed papers, coloured by verdict. Click a node to open it.
- No assessed neighbours yet — the network grows as more papers are assessed.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
scope.md — pmid-41957754
Paper: Zobolas et al., Prognostic biomarker discovery in pancreatic cancer
through hybrid ensemble feature selection and multi-omics data, BioData Mining
2026. DOI 10.1186/s13040-026-00546-0.
Code: https://github.com/bblodfon/pdac-efs-bench2024 (own authors' repo,
MIT, commit bc174b298fdd32292c6d7e97eee420b9e3178085, 2026-04-14).
Data: zenodo 10.5281/zenodo.18375793 — but the repo ships all preprocessed
data + intermediate result objects directly under data/ and bench/.
Dataset name mapping (repo → paper)
| repo dir | paper cohort | n | omics |
|---|---|---|---|
cao2021 |
CPTAC-PDAC | 125 | GEX, CNV, proteo, phospho, glyco (5) |
wissel2023 |
TCGA-PAAD | 81 | GEX, CNV, mutation, methyl, RPPA (5) |
osipov2024 |
MolTwin | 71 | SNV, CNV, INDEL, pathology (4) |
Pipeline (3 steps in bench/)
run_efs.sh/efs.R— hybrid ensemble feature selection (hEFS): 9 base survival learners × B=100 inner subsamples → EnsembleFSResult objects inbench/efs/. Most expensive step. Intermediateefs/objects are NOT in the repo (by-request only).run_fs.R— per-omic feature selection per 100 MC-CV iterations; consumesbench/efs/(step 1) + CoxLasso. Outputbench/fs.rds(SHIPPED).run_mm_bench.R— late-fusion multi-omics integration + model training (CoxPH, CoxLasso, RSF, BlockForest) on 100 MC-CV splits; consumes the SHIPPEDfs.rds+task_list.rds+subsampling.rds. Outputbench/result.rds(SHIPPED, 27 MB) andbench/result_auc.rds(SHIPPED). Re-runnable without the expensive step-1efs/objects.
In scope (pipeline-derived, attempted)
- A. Recompute reported aggregate numbers from the SHIPPED pipeline outputs
(
result.rds,result_auc.rds,fs.rds,timings.rds) and check they match the paper's figures — anti-fabrication check that the printed numbers are derivable from shipped data:- Fig 5a: per-cohort C-index ranges (TCGA 0.54–0.56, MolTwin 0.60–0.62, CPTAC 0.58–0.60).
- Fig 3b: multi-omics feature counts (CoxLasso 200–300; hEFS ≈ 50).
- Sup Fig 10 / text: Clinical+GEX C-index ≈ 0.64; 12-mo Uno AUC TCGA 0.67, CPTAC 0.63.
- Fig 5b–c: per-method training timings.
- B. Genuine partial re-run of
run_mm_bench.Rfrom the shippedfs.rds(reduced grid / model subset) → fresh C-index → compare to shippedresult.rds.
Out of scope (not attempted — the hard 20%)
- Step 1 hEFS
efs/objects are not shipped (by-request, "most time consuming part"). Sorun_fs.R(which needs them) and the exact hEFS feature rankings cannot be regenerated from scratch. We instead consume the shippedfs.rds. → recorded, not chased. - Stability (Nogueira), redundancy FDR, and the full figure typesetting are secondary; computed only if cheap.
- Wet-lab / biological interpretation: none (paper is fully computational).
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
Exemplary reproduction. The authors' own repo ships the preprocessed multi-omics input data (also on Zenodo) and the pipeline result objects, so Phase A could recompute all 14 reported Fig 3/4/5 + text numbers directly from shipped data (anti-fabrication PASS) and Phase B could genuinely re-run run_mm_bench.R over 100 MC-CV splits, matching the shipped result.rds to ≤0.0053 C-index (16/34 cells exact). No deviation sits on the authors' or our methodology side — residuals are pure RSF stochasticity and figure-band aggregation. The central claim (multi-omics FS does not beat clinical-only; +GEX reaches C-index ~0.64) is fully confirmed. Only the by-request step-1 hEFS objects were withheld, which does not affect input-data identity.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.