Comparative analysis of molecular signatures reveals a hybrid approach in breast cancer: Combining the Nottingham Prognostic Index with gene expressions into a
Provisional — an automated or curator check raised a specific concern and points reviewers here. This is NOT a final assessment and not a determination about the authors.
The main result did not reproduce in this reproduction attempt. Where our recomputation produced values that differ from the published ones, those discrepancies are listed below. This is a single automated attempt — not peer review and not a finding of error or misconduct — and differences can also arise from data access, undocumented parameters or the computing environment. The verdict can be contested via “report an error”.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- Nothing in this column.
- 🔴Could not use the authors’ exact input data
- 🟡Reported values were only indirectly comparable
- 🔴A deviation arose in the data or preprocessing
- 🔴A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The deviation was non-trivial in magnitude
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
Described well enough only in PART. The repo (DiTscho/HybridSignature @42a1464, 100% R) ships the analysis scripts plus four small Rdata artifacts (selections, randomSignatures, patient_ids.training/test) but NOT the input matrix. The MAIN pipeline (hybrid_signature_analysis.R -> Tables 1-3, Test Set 1) requires Metabric.data, which is access-restricted (METABRIC/EGA DAC, EGAC00001000484), so the headline C-index/IAUC/SSS/HR numbers could NOT be regenerated. The paper's GSE96058 (SCAN-B, Test Set 2) external validation has NO shipped code at all, so although GSE96058 is public on GEO it was not attempted (would require re-implementing the whole external pipeline + a METABRIC-trained model + the exact n=440 event-ratio downsample -- the hard >20%). What WAS reproduced 1:1 on «our HPC» (SLURM 2177279, R 4.3.3) from the public shipped data: signature gene composition and the 100-random-signature metric distribution (suppl_100_random_signatures.R). Results: OncotypeDx=21 genes (exact match), EndoPredict=15 (exact), Hybrid=14 genes+NPI (count consistent); random-sig mean C-index 0.6240 (95% CI 0.6198-0.6282), Brier 0.1614, R2 0.0557, LogRank 39.6. TWO NOTABLE MISMATCHES flagged for human review: (a) shipped patient_ids give a 703/131 train/test split (834 total), not the paper's stated 883/379 (1262 total); (b) shipped selections$hybrid genes are DISJOINT from the paper's Table 1a Hybrid signature (only NPI shared). These could be a stale/placeholder commit rather than misreporting and are not resolvable without the restricted METABRIC data; they do not by themselves prove the paper's tables are wrong. NOT attempted: anything needing METABRIC (all Test Set 1 tables, Table 1 HRs) and the GSE96058 external validation.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 48assessed: 2026-06-15 ⛓ ca910f356fe3
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-15
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-15no human curator yet
- Last updated
- 2026-09-19
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetCan clinical/pathological characteristics (via the Nottingham Prognostic Index) be combined with gene expression data into a data-driven 'Hybrid signature' that provides prognostic information substantially better than random gene sets, and what is the true information gain of existing commercial molecular signatures over random signatures?
- ★ A Hybrid signature combining the NPI with 14 SIS-selected genes was constructed from METABRIC training data method
- ★ Random sets of genes are frequently significantly associated with breast cancer outcome, undermining naive interpretation of molecular signature validity finding
- ★ None of the tested signatures (Hybrid, EndoPredictGL, OncotypeDxGL, NPI, Random, NPI+Random) can be considered a definitive main candidate for providing prognostic information finding
- ★ Both the Hybrid signature and the OncotypeDx gene list identify ER+/Her2- patients who may not require adjuvant chemotherapy finding
- ★ Combining multiple signatures substantially improves identification of patients who do not need adjuvant chemotherapy finding
- Overlap among the six commercially available breast cancer gene signatures is almost non-existent (no single gene shared by all six) finding
- Concordance between MammaPrint and OncotypeDx is low (kappa=0.40) finding
- ★ The Signature Skill Score was introduced as a simple measure to assess a signature's improvement over random signatures method
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| Gene expression profiling (microarray) | METABRIC cohort, ER+/Her2- breast cancer patients, no chemotherapy (training set n=883, test set 1 n=379) | none (observational, in silico) | Gene expression levels used as predictors of survival outcome | Illumina HT-12 v3 |
| Gene expression profiling (external validation) | GSE96058 / SCAN-B cohort, ER+/Her2- breast cancer patients under 80y, no chemotherapy (test set 2, downsampled from n=1381) | none (observational) | Overall survival prediction from gene expression signatures | — |
| Sure Independence Screening (SIS) feature selection | METABRIC training set (n=883) | none | Ranking and selection of 15 most important genes (cox response, lasso penalty, bic tuning) forming the Hybrid signature | R package SIS |
| Cox proportional hazards regression | METABRIC training set (n=883); six signatures: Hybrid, EndoPredictGL, OncotypeDxGL, Random, NPI+Random, NPI | none | Hazard ratios, 95% CI, and p-values per gene/signature for survival association | — |
| Decision Tree analysis / survival analysis | Test set 1 (METABRIC) and test set 2 (GSE96058) | none | Patient stratification into risk groups and identification of patients not requiring adjuvant chemotherapy | — |
| Batch effect correction (ComBat) | METABRIC and GSE96058 combined datasets | none | Batch-adjusted gene expression values prior to analysis | R package sva |
- ▲ NPI as part of the Hybrid signature was significantly associated with outcome HR=1.54 (95% CI 1.32-1.80), p<0.001
- ▲ NPI alone (signature f) was significantly associated with outcome HR=1.65 (95% CI 1.43-1.90), p<0.001
- ▼ PREX1 gene in Hybrid signature strongly associated with better outcome HR=0.62 (95% CI 0.51-0.75), p<0.001
- ▼ PGR gene in OncotypeDxGL strongly associated with outcome HR=0.83 (95% CI 0.76-0.92), p<0.001
- ▲ BIRC5 gene in EndoPredictGL associated with worse outcome HR=1.61 (95% CI 1.25-2.08), p<0.001
- ▲ PLAU gene in Random signature significantly associated with outcome despite being randomly selected HR=1.38 (95% CI 1.15-1.65), p<0.001
- – Low concordance found between MammaPrint and OncotypeDx signatures kappa=0.40, 95% CI 0.30-0.49
- count 1262 ER+/Her2- patients selected from METABRIC (Study cohort after filtering for ER+, Her2-, no chemotherapy, known survival status)
- count 883 training / 379 test set 1 (70/30 split of METABRIC cohort)
- count 1381 patients selected from GSE96058/SCAN-B (External validation cohort (test set 2), ER+/Her2-, no chemo, age <80)
- correlation kappa=0.40, 95% CI: 0.30-0.49 (Concordance between MammaPrint and OncotypeDx from OPTIMA study)
- other probability of 0.9 (Venet et al. finding that any set of ≥100 randomly selected genes has 0.9 probability of significant association with outcome)
- count >150,000 papers on molecular signatures; <100 clinically validated (Background on molecular signature validation rate in medicine)
- other <25% (Proportion of patients with most frequent cancers who currently benefit from precision oncology)
- pvalue NPI HR=1.54 (1.32-1.80), p<0.001 (Cox regression of NPI within Hybrid signature, METABRIC training set)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
The paper uses the METABRIC cohort (n=1262 ER+/HER2- patients not receiving chemotherapy) split 70/30 into training and internal test sets; an independent external cohort (SCAN-B/GSE96058, n=1381 selected patients) serves as test set 2 via 1000 random downsamplings matched on event-to-patients-at-risk ratio. Sure Independence Screening (SIS) with a Cox-LASSO penalty and BIC tuning selected 15 features (genes plus NPI) from the training set to form the 'Hybrid signature', which was then compared with OncotypeDxGL, EndoPredictGL, NPI alone, Random, and NPI+Random signatures via Cox proportional hazards regression. Results are reported as per-gene hazard ratios with 95% confidence intervals and p-values; patient stratification is additionally assessed using decision trees and survival analysis.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Cox proportional hazards regression (per-gene and signature-level) | All six signatures (Hybrid, EndoPredictGL, OncotypeDxGL, NPI, Random, NPI+Random) fitted in METABRIC training set (Table 1) and applied to both test sets | 883 (training); 379 (test set 1); downsampled subset of 1381 repeated 1000 times (test set 2) | not stated |
| Sure Independence Screening (SIS) with Cox response, LASSO penalty, BIC tuning | Feature selection of 15 most important genes/covariates from training set | 883 | not stated |
| Decision Trees (survival stratification) | Patient risk-group stratification and survival outcome assessment across signatures | — | na |
| Survival analysis (specific test not named in available text) | Comparison of patient outcome groups stratified by signature risk categories | — | not stated |
-
Overall signature performance was assessed via per-gene Cox HRs in separate models rather than a single summary discrimination metric per signature↳ Could also: Harrell's concordance index (C-statistic) or time-dependent AUC with bootstrap confidence intervals could also be computed per signature — A discrimination metric directly quantifies each signature's overall ability to rank patients by risk, enabling straightforward head-to-head comparison between signatures — which is the paper's central research question
-
Individual gene-level p-values are reported across multiple Cox models (up to 21 genes per model, six models) without stated multiple testing correction↳ Could also: Benjamini-Hochberg FDR correction or Bonferroni correction applied within each signature model (or across all tests) could also be used — Adjusting for the number of simultaneous comparisons is a standard practice when reporting many p-values from the same dataset, and would make the gene-level thresholds more conservative
-
Feature selection used SIS with a single 70/30 training/test split↳ Could also: Repeated k-fold cross-validation or bootstrap-based stability selection could also be used to assess feature selection robustness — These approaches provide internal estimates of both model performance and the consistency with which each feature is selected, reducing sensitivity to the particular random split
-
External validation on SCAN-B used random downsampling to match event-to-patients-at-risk ratio, repeated 1000 times↳ Could also: Validation on the full external cohort using the C-statistic or calibration plots (e.g., observed vs. predicted survival curves) with bootstrap CIs could also be used — Using the complete external cohort avoids information loss from downsampling and provides a direct assessment of discrimination and calibration in an independent population
-
The proportional hazards assumption of the Cox models is not stated as having been verified↳ Could also: Schoenfeld residual tests or log-log survival plots could also be used to assess whether the hazard ratio remains constant over time for each covariate — With follow-up extending up to 30 years, time-varying effects are plausible for some genes; verifying this assumption is a standard diagnostic step and informs whether time-varying coefficient extensions are warranted
-
Standard decision trees were used for patient survival stratification↳ Could also: Survival-specific recursive partitioning (e.g., rpart with a survival splitting rule) or conditional inference trees could also be used — Survival-tailored tree methods optimize splits directly on the survival outcome and handle censoring natively, which may yield more accurate and statistically grounded risk strata than general-purpose decision trees
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
No assessed neighbours yet — the network grows as more papers are assessed.
Data lineage
The datasets this paper uses (text-mined from the full text via Europe PMC), and which other assessed papers stand on the same data. A shared dataset is a factual link — not a judgement.
Downstream reach in the literature
199 downstream papers · 2 datasetsHow widely the datasets deposited by this paper are reused across the whole literature (Europe PMC), beyond our assessed set. This is a factual dependency map — reusing a public dataset is normal, good science. It is not a judgement on the downstream papers; the only verdict here is this paper's own, with its cited rationale.
- Clinical Value of RNA Sequencing-Based Classifiers f... 2018 · 165 cites
- bc-GenExMiner 4.5: new mining module computes breast... 2021 · 142 cites
- Abundance of Regulatory T Cell (Treg) as a Predictiv... 2020 · 95 cites
- Bulk and single-cell transcriptome profiling reveal... 2021 · 89 cites
- A lncRNA prognostic signature associated with immune... 2020 · 86 cites
- Prognosis and Dissection of Immunosuppressive Microe... 2022 · 73 cites
- The genomic and transcriptomic architecture of 2,000... 2012 · 4,682 cites
- Cooperation between Constitutive and Inducible Chemo... 2019 · 677 cites
- Aerobic Glycolysis Controls Myeloid-Derived Suppress... 2018 · 364 cites
- Dynamics of breast-cancer relapse reveal late-recurr... 2019 · 321 cites
- Hypoxia promotes stem cell phenotypes and poor progn... 2014 · 189 cites
- Tumor mutational burden is a determinant of immune-m... 2018 · 189 cites
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
Only auxiliary claims were checkable from public shipped data and these reproduced exactly (OncotypeDx=21, EndoPredict=15 genes; random-signature mean C-index 0.6240). The study's headline result is untestable here because the main pipeline needs access-restricted METABRIC and the GSE96058 validation ships no code — that is a data-availability limitation, not proof the paper is wrong. However, the deposited artifacts internally contradict the paper (hybrid genes disjoint from Table 1a; 703/131 vs stated 883/379 split), which is a genuine flag on the authors'/deposit side that a human must resolve; it is plausibly a stale/placeholder commit and cannot be settled without the restricted data, so this is a flagged partial, not confirmed fabrication.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at [email protected].
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.