Prediction of prognostic signatures in triple-negative breast cancer based on the differential expression analysis via NanoString nCounter immune panel.
The main results reproduced, with only marginal, non-material deviations.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Reported values were directly comparable
- ✓No authors-side cause for any deviation
- ✓Reported values are derivable from the shared data
- ✓Any deviation was negligible
- ✓The central claim held under reproduction
- 🟡Could not use the authors’ exact input data
- 🟡A deviation arose in the data or preprocessing
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
DESCRIBED WELL ENOUGH? Mostly. Code (sungheep/nanostring, edgeR.R) and data (GEO GSE143222) are both public, but the repo hardcodes two input paths that are NOT shipped (nano_count.csv, Nanostring_all.csv) and does not state which sample subset or which edgeR test was used. I reconstructed the count matrix 1:1 from the 55 GEO RCC count files and the group labels from the GEO series matrix. OUTCOME = 1:1 to within software-version tolerance. The edgeR DEG pipeline reproduces: same method (TMM + classic exactTest, FDR<0.05), same data, same top DEGs, and p-values matching within ~2x (headline IL2RA pCR p=1.82E-07 vs reported 2.34E-07). DEG counts differ by 1-2 (relapse 14 vs 13; pCR 7 vs 9) and gene-set overlap is 11/13 (relapse) and 7/9 (pCR); the 2 missed pCR genes (HAMP, FCER1A) sit just above FDR 0.05. Critically, the paper's numbers match the CLASSIC exactTest on all 55 samples, NOT the glmQLFTest the repo script ends on (3 relapse/0 pCR DEGs) nor Pre-only samples (3/0) -- so a reproducer must infer method+subset from the data, which I documented. No fabrication signal: every reported value is derivable from the public data; residual gaps are consistent with edgeR version drift (paper ~2019 vs 4.0.16). NOT ATTEMPTED (the hard, stochastic 20%): Random Forest prediction AUCs (0.84 pCR / 0.88 relapse) and elastic-net/lasso AUCs -- both use SMOTE + random CV folds + random train/test split with no seed shipped, so not bit-reproducible; Cox PH HR tables (secondary, time-boxed); DAVID pathway enrichment (external manual web tool); all wet-lab steps (out of scope).
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 72assessed: 2026-06-15 ⛓ ef089e833991
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-15
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-15no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusCan immune-related gene expression quantified via the NanoString nCounter Immunology Panel, combined with edgeR differential expression analysis for feature selection, identify prognostic biomarkers predicting tumour relapse and pathologic complete response (pCR) in triple-negative breast cancer (TNBC) despite a small sample size?
- ★ edgeR identifies 9 DEGs associated with pCR and 13 DEGs associated with relapse from 579 immune genes in a small TNBC sample set (n=55) finding
- ★ Random Forest prediction models built on edgeR-selected DEGs outperform models built on Elastic Net-selected features in accuracy and type I error control finding
- ★ Three relapse-model DEGs (FCER1A, EDNRB, TGFBI) show prognostic significance for patient survival via Cox proportional hazards regression finding
- ★ edgeR's negative binomial model is suitable for feature selection from NanoString nCounter count data even with small sample sizes (n>=2) method
- An additive edgeR model adjusting for independent subject revealed 41 genes differentially expressed between pre- and post-treatment groups in the relapse group finding
- ★ The prediction models are robust as validated by a randomization/permutation test (empirical P=0.015 for pCR, P=0.018 for relapse) finding
- The approach provides a scalable framework using edgeR + Random Forest for biomarker discovery from the NanoString nCounter Immunology Panel resource
- NanoString nCounter gene expression highly correlates with IHC staining intensity for T and B cell markers finding
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| NanoString nCounter gene expression (Immunology Panel, 579 immune genes) | FFPE tissue biopsies from 55 TNBC patients (anthracycline/taxane neoadjuvant chemotherapy) | neoadjuvant chemotherapy (pre/post treatment); disease status pCR/relapse | mRNA expression counts | NanoString nCounter Analysis System, GX Human Immunology V2 kit (NanoString Technologies, Seattle, WA, USA) |
| Immunohistochemistry (anti-CD3, -CD8, -CD20 staining) | same samples from 56 patients | none | staining intensity of CD3+, CD8+, CD20+ cells correlated with gene expression | — |
| Differential expression analysis (edgeR, negative binomial model) | NanoString count matrix (55 pre-treatment, 14 post-treatment samples) | design matrix contrasts (pCR vs non-pCR; relapse vs non-relapse; PRE vs POST) | DEGs at FDR<0.05 | edgeR (R) |
| Random Forest prediction modelling | training/test split (7:3) of pCR and relapse datasets | SMOTE oversampling, GridSearchCV hyperparameter optimization, 5-fold CV | AUC and PPV for pCR/relapse prediction | sklearn (Python) |
| Cox proportional hazards regression survival analysis | TNBC patient cohort using prediction-model genes | none | hazard ratios and prognostic significance (P<0.05) for overall/recurrence-free survival | — |
| Functional enrichment analysis | edgeR-derived DEGs | none | functional annotation of signature genes | DAVID |
| Protein interaction network analysis | feature-derived DEGs | none | protein-protein interaction network | STRING |
- – 9 DEGs associated with pCR and 13 DEGs associated with relapse identified by edgeR at FDR<0.05
- – pCR Random Forest prediction model performance AUC=0.84, PPV=0.7
- – Relapse Random Forest prediction model performance AUC=0.88, PPV=0.69
- ▼ Elastic Net pCR model performed worse than edgeR-based model AUC=0.64, PPV=0
- ▼ Elastic Net relapse model performed worse than edgeR-based model AUC=0.68, PPV=0.23
- ▼ CD1A and FCER1A downregulated in pCR group and significant in Cox model (HR<1, non-risk factors) CD1A HR=0.103 (P=0.00755); FCER1A HR=0.475 (P=0.00128)
- ▲ FCER1A, EDNRB, TGFBI significantly upregulated in relapse group (P<0.05) in Cox model
- ▲ NanoString vs IHC staining correlation for T/B cell markers Spearman r=0.623-0.761
- pvalue empirical P = 0.015 (pCR model), empirical P = 0.018 (relapse model) (randomization/permutation test of prediction model robustness)
- correlation 0.623-0.761 (Spearman correlation between NanoString gene expression and IHC staining intensity)
- other AUC=0.84, PPV=0.7 (pCR Random Forest model performance)
- other AUC=0.88, PPV=0.69 (relapse Random Forest model performance)
- pvalue IL2RA 2.34E-07; CCL5 1.17E-06; FCER1A 7.49E-04 (top pCR DEG edgeR P-values)
- pvalue CCL5 9.49E-06; EDNRB 4.34E-04; TGFBI 1.07E-03 (relapse DEG edgeR P-values)
- pvalue HLA-DQA1 1.45E-25; HLA-DQB1 6.26E-19; CEACAM6 2.36E-04 (DEGs for relapse and POST treatment group)
- count 579 immune genes; n=55 samples (14 post-treatment, 6 pCR cases); 41 genes in PRE/POST additive model (study cohort and panel size)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
The study applied edgeR's negative binomial generalized linear model to NanoString nCounter count data from 55 TNBC patients to identify differentially expressed genes (DEGs) associated with pathological complete response (pCR) and relapse, using FDR < 0.05 as the selection threshold across 579 immune genes. Selected DEGs served as features for Random Forest classification models trained with SMOTE oversampling and 5-fold cross-validation, with model significance assessed via 1000-permutation randomization tests yielding empirical P-values. Survival analysis was performed using Cox proportional hazards regression applied to genes within each prediction model, with P < 0.05 used to identify prognostic markers. Model performance was summarized as AUC and positive predictive value (PPV), and hazard ratios with 95% confidence intervals were reported for Cox regression.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| edgeR negative binomial GLM with FDR correction | Differential expression of 579 immune genes: pCR vs non-pCR (9 DEGs identified) and relapse vs non-relapse (13 DEGs identified) | n=55 | stated |
| edgeR additive GLM adjusting for paired subject effects | Three pre/post-treatment comparisons stratified by relapse and pCR status (POST+pCR vs POST+relapse; POST+relapse vs POST+non-relapse; PRE/POST in relapse group); 41 DEGs identified in additive model | n=14 paired samples (12 in one comparison after 2 MDS outliers excluded) | stated |
| Random Forest classification with 5-fold cross-validation and hyperparameter optimization via GridSearchCV | Prediction of pCR and relapse using edgeR-selected DEGs as features; SMOTE applied for class-imbalance correction | n=55 (70:30 train/test split) | na |
| Permutation test (1000 permutations, empirical P-value) | Validation of pCR and relapse prediction model significance; AUC and PPV used as test statistics | n=55 | stated |
| Cox proportional hazards regression (individual gene models, P < 0.05 threshold) | Survival analysis using each of the 9 pCR-model genes and 13 relapse-model genes as individual predictors to identify prognostic markers | not explicitly stated | not stated |
| Spearman correlation | Concordance between NanoString nCounter expression of 5 genes (CD3D, CD3E, CD8A, CD8B, CD20) and IHC staining intensity for CD3+, CD8+, CD20+ cells | n=56 | na |
-
edgeR's negative binomial model was used for differential expression analysis of NanoString nCounter count data↳ Could also: DESeq2 (also a negative binomial GLM for count data) could also be applied to NanoString count data — DESeq2 offers an alternative shrinkage estimator for dispersion and log-fold changes (apeglm/ashr) and is equally validated for count-based expression data; running both tools and comparing DEG overlap is a common practice to assess robustness of calls in small-sample settings
-
A fixed 70/30 single train-test split was used to evaluate prediction model performance↳ Could also: Leave-one-out cross-validation (LOOCV) or repeated stratified k-fold cross-validation could also be used for performance estimation — With n=55, the 30% test set contains approximately 17 samples, making single-split AUC and PPV estimates highly variable; LOOCV or repeated CV uses all observations for evaluation and typically produces more stable performance estimates under small-sample constraints
-
Cox proportional hazards regression was applied to each of the 9 pCR-model genes and 13 relapse-model genes individually at an uncorrected P < 0.05 threshold↳ Could also: A LASSO-penalized Cox model or Benjamini-Hochberg correction applied across the gene set could also be used — Testing each gene individually without multiplicity adjustment across 9 or 13 simultaneous hypotheses raises the family-wise error rate; a penalized Cox model or FDR correction would also account for this when selecting prognostic genes from a pre-defined feature set
-
SMOTE oversampling was applied to the training set to address class imbalance before Random Forest training↳ Could also: Class-weighted (cost-sensitive) Random Forest or undersampling of the majority class could also address class imbalance — SMOTE generates synthetic minority-class samples by interpolation, which can introduce noise when the original sample size is small; class-weighted approaches operate on the original data distribution and avoid synthetic sample generation, offering an alternative with different bias-variance trade-offs
-
Model performance was reported using AUC and positive predictive value (PPV) alone↳ Could also: Sensitivity, specificity, negative predictive value (NPV), and Matthews Correlation Coefficient (MCC) could also be reported — PPV is sensitive to class prevalence and can be misleading under imbalance; reporting NPV, sensitivity, specificity, and MCC alongside AUC provides a more complete characterization of classifier behavior across both outcome classes, particularly relevant here given the unequal group sizes
-
Two samples (NS010 and NS032) were excluded from one pre/post comparison based on visual inspection of a multidimensional scaling plot↳ Could also: A sensitivity analysis retaining these samples, or edgeR's robust dispersion estimation, could also be reported — Reporting results both with and without the excluded observations, or applying robust dispersion shrinkage, would allow readers to assess how much the DEG calls in that comparison depend on the exclusion of these two data points
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
No assessed neighbours yet — the network grows as more papers are assessed.
Data lineage
The datasets this paper uses (text-mined from the full text via Europe PMC), and which other assessed papers stand on the same data. A shared dataset is a factual link — not a judgement.
- Repression of Divergent Noncoding Transcriptio... L1 No data access
- Repression of Divergent Noncoding Transcriptio... L1 No data access
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-33138797
Paper: Lim et al. 2020, BMC Cancer. "Prediction of prognostic signatures in
TNBC based on differential expression analysis via NanoString nCounter immune panel."
Code: https://github.com/sungheep/nanostring (own code, 3 R scripts + 3 notebooks)
Data: GEO GSE143222 — 55 NanoString nCounter Immunology-v2 count files
(GSM4253872..GSM4253926 = Patient/No 1..55), 579 genes each. Phenotype labels
(relapse, pCR, pre/post, 5-yr status) are in the GEO series matrix.
Reproduction strategy
The repo's edgeR.R reads two hardcoded local paths that are NOT shipped
(«path», Nanostring_all.csv). They are reconstructable
1:1 from the GEO deposit: the count matrix from the 55 *-count.txt.gz files and the
group labels from the series-matrix characteristics. We re-implement edgeR.R's exact
logic on that reconstructed input. (P16: re-running the paper's own pipeline on the
paper's own public data.)
IN SCOPE (pipeline-derived, clearly specified) — edgeR differential expression
Pipeline: edgeR (TMM norm + quasi-likelihood GLM, FDR<0.05), exactly as edgeR.R:
DGEList → filterByExpr-equivalent rowSums(cpm>1)>=2 → calcNormFactors →
estimateDisp/estimateCommonDisp/estimateTagwiseDisp → glmQLFit(~0+group) →
glmQLFTest(contrast=c(-1,1)) → decideTestsDGE(p.value=0.05).
| Result (reported) | Paper location | In scope |
|---|---|---|
| relapse DEGs = 13 | Results / Table 2 | YES — primary target |
| pCR DEGs = 9 | Results / Table 2 | YES — primary target |
| Specific DEG gene names + p-values (e.g. IL2RA pCR p=2.34E-07; CCL5 relapse p=9.49E-06) | Table 2 | YES — secondary |
| Pre/Post relapse comparison = 41 genes | Results | partial (different contrast; attempt if time) |
OUT OF SCOPE (stochastic / underspecified — the hard 20%, not attempted or best-effort only)
- Random Forest prediction (AUC pCR=0.84, relapse=0.88; PPV): uses SMOTE oversampling + 5-fold CV + random train/test split with no fixed seed shipped → not bit-reproducible; results depend on RNG. Pipeline: randomForest (Python notebooks). Best-effort only if edgeR lands cleanly.
- Elastic net / lasso comparison (AUC 0.64/0.68): random CV folds, no seed.
- Cox PH prognostic genes (Tables 5,6 HRs): derivable but secondary; attempt if time.
- DAVID pathway enrichment (Table 7): external web tool, manual — out of scope.
- Wet-lab (NanoString hybridization, IHC): out of scope by definition.
Primary deliverable
1:1 reproduction of the edgeR DEG counts (13 relapse / 9 pCR) and the Table-2 gene p-values from GSE143222, with sample-set sensitivity (Pre-only n=41 vs all n=55) documented honestly since the exact sample subset used is not stated in the repo.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
Solid, near-quantitative reproduction. The public GSE143222 data and edgeR/TMM/exactTest method regenerate Table 2's headline values almost exactly (IL2RA pCR p=1.82E-07 vs 2.34E-07; CCL5 relapse p=1.42E-05 vs 9.49E-06) with no fabrication signal — every reported value is derivable from shared data. Residual differences are on our/technical side: edgeR version drift and an underspecified method/cohort (the repo ends on glmQLFTest and ships no inputs, so the 55-sample matrix and the classic-exactTest choice had to be inferred). Deviations are minor — DEG counts off by 1–2, two borderline pCR genes (HAMP FDR=0.063, FCER1A) just above threshold — so the central prognostic-signature conclusion holds; predictive AUCs were not attempted (no seed shipped).
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.