Application of Machine Learning in Predicting Hepatic Metastasis or Primary Site in Gastroenteropancreatic Neuroendocrine Tumors.
Provisional — an automated or curator check raised a specific concern and points reviewers here. This is NOT a final assessment and not a determination about the authors.
The main results reproduced, with only marginal, non-material deviations.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Reported values were directly comparable
- 🟡Could not use the authors’ exact input data
- 🔴A deviation arose in the data or preprocessing
- 🔴A deviation was attributed to the published material
- 🔴Reported values were not (fully) derivable from the shared data
- 🟡The deviation was non-trivial in magnitude
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
Pure-ML paper (no code repo shipped; the scaffold's parallel-fastq-dump code pointer is a wrong enrichment artifact). Reproduced under P16 by re-implementing the described pipeline on the paper's own public data GSE98894: raw Entrez counts -> log2CPM -> Random Forest on the authors' reported marker genes, stratified 70:30 x 200 splits, on «our HPC». RESULT = PARTIAL. Described well enough to reproduce the data, cohort, and gene sets (cohort matches within 0-2 samples; pancreas counts exact). Random Forest TRAINING accuracy of 100% reproduces exactly but is uninformative (RF memorizes training data). The paper's headline claim of 100% TEST accuracy/sensitivity/specificity is NOT reproducible: hepatic-metastasis RF reaches ~90-91% held-out (never 100% across 200 random splits) and primary-site RF ~78-84% (the paper's own non-RF algorithms reported 81.25%, matching our RF mean). Most likely cause is feature-selection leakage (mRMRe gene selection apparently run on the full dataset before the train/test split) -> over-optimistic test metrics; flagged for human review as over-optimism, NOT fabrication, since the genes are genuinely discriminative. NOT attempted (hard 20%): re-running the stochastic 500-iteration mRMRe selection, the GSE118014 independent validation (different platform/units, label mapping unclear), the multi-label scikit-learn models, and the 6 non-RF algorithms.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 59assessed: 2026-06-14 ⛓ 19c8bb5825cf
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-14
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-15no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetMolecular markers for hepatic metastasis and primary site are unknown in gastroenteropancreatic neuroendocrine tumors (GEP-NETs); the study tests whether RNA-SEQ-derived gene expression signatures and machine learning models can accurately predict liver metastasis and the primary site (pancreas vs. small intestine).
- ★ Multi-gene random forest models classify primary tumor vs. liver metastasis samples with 100% accuracy in training/test cohorts and >90% accuracy in an independent validation cohort finding
- ★ Multi-gene random forest models identify the primary site (pancreas or small intestine) with 100% accuracy in training/test cohorts and >95% accuracy in an independent cohort finding
- ★ Multi-label RF models for concurrent prediction of hepatic metastasis and primary site achieve >98.42% training and >87.42% test accuracy finding
- mRMRe feature selection identified nine gene features (including HP, ALB, TBX20, BMP10, RBP4, SFRP2, NKX2-3, LMO3, PRRX2) relevant to hepatic metastasis classification method
- mRMRe feature selection identified twelve gene features (including SYT16, FAR2) relevant to primary site classification method
- Haptoglobin (HP) is the top-ranked feature for hepatic metastasis classification across RF, k-NN, SVM, and LDA models finding
- ★ WGCNA modules significantly correlated with tumor class and primary site contain the machine-learning-identified gene features with high module membership and gene significance scores mechanism
- ★ This is the first reported robust molecular signature to predict liver metastasis or primary site for GEP-NETs, potentially complementing clinical management resource
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| bulk RNA-SEQ | pancreatic and small intestinal NET tissue (primary tumor and liver metastasis samples) | none | gene expression counts for downstream classification/differential expression | STAR aligner v2.7.2, hg38 p.12, Ensembl GTF v101 |
| machine learning classification (LDA, RF, CART, SVM, k-NN, XGB, GBM) | RNA-SEQ gene expression data, NET tissue training/test cohort (n=182, GSM2626909) | none | classification accuracy, sensitivity, specificity for primary vs. hepatic metastasis | R caret package v6.0.90 |
| machine learning classification for primary site | liver metastasis samples of pNETs (n=30) and siNETs (n=26) | none | classification accuracy for primary site (pancreas vs. small intestine) | R caret package |
| multi-label random forest classification | RNA-SEQ gene expression data, all NET samples (n=182) | none | simultaneous prediction accuracy for hepatic metastasis and primary site | Python scikit-learn |
| differential expression analysis | primary tumor vs. liver metastasis NET tissue | none | differentially expressed genes | DESeq2 |
| independent validation classification | GSE118014 pNET RNA-SEQ dataset (primary n=25, liver metastasis n=7) | none | model classification accuracy vs. No Information Rate | — |
| weighted gene co-expression network analysis (WGCNA) | VST-normalized RNA-SEQ counts, 24,123 genes, 182 samples | none | module-trait correlation with tumor class and primary site, module membership, gene significance | R WGCNA package v1.7.0 |
- – Two 5-gene RF models achieved 100% accuracy, sensitivity, and specificity distinguishing primary from metastatic samples in training and test sets 100%
- – Both 5-gene RF metastasis models exceeded 90% accuracy on independent validation dataset, outperforming No Information Rate >90%; p<0.0182
- – LDA and k-NN achieved highest test-set accuracy using the 9-gene metastasis feature set 96.23%
- – RF, LDA, XGB, SVM models using 12 primary-site genes achieved 100% training accuracy but lower test accuracy 100% train / 81.25% test
- – Reduced feature-set RF, GBM, and XGB models achieved 100% accuracy in training, test, and independent datasets for primary site classification 100%
- ▲ Multi-label RF model outperformed individual models on test set for hepatic metastasis prediction 96.36% vs 92.45%
- ▲ Multi-label RF model outperformed individual models on test set for primary site prediction 90.9% vs 81.25%
- – All nine hepatic-metastasis and all twelve primary-site gene features were differentially expressed between primary and liver metastasis samples
- count n=214 (total RNA-SEQ NET tissue samples used across the study)
- count n=182 (primary=126, liver metastasis=56) (training/test dataset GSM2626909 for hepatic metastasis model)
- count 24,123 gene features (genes retained after expression-based filtering for analysis)
- pvalue p < 0.0182 (RF hepatic metastasis models accuracy vs. No Information Rate on independent validation)
- other 100% accuracy, sensitivity, specificity (two 5-gene RF models on training and test data for metastasis classification)
- correlation |r| ≥ 0.4, p < 1×10^-9 (WGCNA module-trait correlation with tumor class (primary vs. liver metastasis))
- correlation |r| ≥ 0.4, p < 2×10^-8 (WGCNA module-trait correlation with primary site (pNETs vs. siNETs))
- count n=32 (25 primary, 7 liver metastasis) (independent GSE118014 validation cohort for primary site models)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
The study applied machine learning to RNA-SEQ data from 182 NET tissue samples (primary tumors and liver metastases) to classify hepatic metastasis status and primary site (pancreas vs. small intestine). Gene features were selected via the mRMRe algorithm (500 independent runs), and seven ML algorithms were trained using repeated 6-fold cross-validation (100 repeats) on a 70/30 random train/test split of the main cohort. Model performance was reported as accuracy, sensitivity, and specificity, with external validation on an independent cohort (n=32).
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Repeated 6-fold cross-validation (100 repeats) with grid search; ROC used as accuracy metric during training | Training of all seven ML models (LDA, RF, CART, SVM, k-NN, XGB, GBM) for hepatic metastasis and primary site classification | ~127 (70% of 182) for metastasis training set; ~39 (70% of 56 liver metastasis samples) for primary site training set | not stated |
| DESeq2 (Wald test) differential expression analysis | Confirmation of differential expression of mRMRe-selected gene features between primary tumors and liver metastases | 182 | not stated |
| mRMRe (minimum redundancy maximum relevance) feature selection, 500 independent executions, intersection of consistently selected features | Feature selection from 24,123 VST-normalized, SV-adjusted genes for hepatic metastasis and primary site models | ~127 (metastasis training set); ~39 (primary site training set) | not stated |
| WGCNA module–trait bicor correlation with fixed thresholds (|r| ≥ 0.4; p < 1×10⁻⁹ for tumor class; p < 2×10⁻⁸ for primary site) | Association of co-expression modules with tumor class (primary vs. metastasis) and primary site (pNET vs. siNET) | 182 | not stated |
| No Information Rate (NIR) comparison, p-value reported as threshold | Assessment of RF model accuracy on independent validation cohort (GSE118014) | 32 | not stated |
-
Model performance was reported as single point-estimate percentages (accuracy, sensitivity, specificity) with no uncertainty quantification around those estimates↳ Could also: Bootstrap confidence intervals or nested cross-validation could also be used to quantify variability in performance estimates — With a relatively small overall n (182) and a 32-sample external validation set, single-split point estimates carry meaningful sampling variance; confidence intervals would convey the plausible range of true generalization performance
-
Hundreds of feature combination sets were evaluated on the same held-out test partition to select the best-performing models↳ Could also: A nested cross-validation framework (inner loop for feature/model selection, outer loop for performance estimation) could also be applied — Selecting the best model after observing test-set performance on many candidates introduces optimistic bias; nested CV provides an estimate of generalization that is independent of the model-selection step
-
Accuracy was the primary performance metric across all models and feature combinations↳ Could also: Area under the ROC curve (AUROC), balanced accuracy, or Matthews Correlation Coefficient (MCC) could also serve as primary metrics — When class sizes are unequal (126 primary vs. 56 metastasis samples), AUROC and MCC are less sensitive to class imbalance than raw accuracy and provide complementary discrimination information
-
Feature selection used mRMRe run 500 independent times, retaining only genes selected across all runs↳ Could also: LASSO-penalized logistic regression, elastic net, or Boruta with cross-validation could also be used to identify stable, low-redundancy feature subsets — These methods embed regularization and stability assessment within a single unified procedure and produce continuous importance scores, making it straightforward to compare feature-selection stability across alternative strategies
-
WGCNA module–trait associations were assessed using fixed thresholds on correlation magnitude (|r| ≥ 0.4) and raw p-values across 24 modules and 2 traits↳ Could also: FDR correction (e.g., Benjamini–Hochberg) applied across all module–trait tests, or permutation-based significance, could also be used — Testing 24 modules against 2 traits (48 comparisons) without multiplicity adjustment inflates the family-wise error rate; FDR or permutation methods would provide adjusted significance thresholds for this family of tests
-
DESeq2 was used to confirm differential expression of the selected gene features↳ Could also: edgeR or limma-voom could also be applied to RNA-seq count data for differential expression analysis — Both are widely validated alternatives for RNA-seq DE; agreement across two independent methods is a common additional evidence of robustness for nominated biomarker genes
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
No assessed neighbours yet — the network grows as more papers are assessed.
Data lineage
The datasets this paper uses (text-mined from the full text via Europe PMC), and which other assessed papers stand on the same data. A shared dataset is a factual link — not a judgement.
Downstream reach in the literature
18 downstream papers · 2 datasetsHow widely the datasets deposited by this paper are reused across the whole literature (Europe PMC), beyond our assessed set. This is a factual dependency map — reusing a public dataset is normal, good science. It is not a judgement on the downstream papers; the only verdict here is this paper's own, with its cited rationale.
- Pan-cancer Convergence to a Small-Cell Neuroendocrin... 2019 · 156 cites
- Genome analysis identifies differences in the transc... 2021 · 15 cites
- Long intergenic noncoding RNA profiles of pheochromo... 2020 · 14 cites
- Involvement of Neutrophils in Metastatic Evolution o... 2021 · 11 cites
- Drug perturbation gene set enrichment analysis (dpGS... 2021 · 8 cites
- The Interplay of Four Main Pathways Recomposes Immun... 2022 · 8 cites
- PTEN regulates invasiveness in pancreatic neuroendoc... 2022 · 14 cites
- Five Novel Genes Related to the Pathogenesis and Pro... 2019 · 11 cites
- Characterization of the Secretome, Transcriptome, an... 2022 · 11 cites
- Proteogenomic characterization of pancreatic neuroen... 2024 · 11 cites
- Altered CELF4 splicing factor enhances pancreatic ne... 2024 · 9 cites
- Molecular Subtypes of Pancreatic Neuroendocrine Tumo... 2025 · 2 cites
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-37887568
Paper: Padwal MK, Basu S, Basu B. Application of Machine Learning in Predicting Hepatic Metastasis or Primary Site in Gastroenteropancreatic Neuroendocrine Tumors. Curr Oncol 2023;30(10):8902-8918. PMID 37887568 · PMCID PMC10605255 · DOI 10.3390/curroncol30100668.
What kind of paper
A pure computational / machine-learning study. No wet-lab generation: the authors re-analyse public RNA-seq of GEP-NETs and train classifiers. This makes the whole paper, in principle, in-scope for reproduction.
Data (public)
- GSE98894 — discovery cohort (Alvarez et al. NET RNA-seq, 212 samples, raw Entrez
counts in
GSE98894_RAW.tar; sampletypeandoriginin the series matrix). - GSE118014 — independent validation (33 PanNETs, log2TPM matrix). Both downloaded onto «infra» inside «our HPC» (see manifest for path + SHA256).
Code availability
The paper ships no code repository (no GitHub/Zenodo link in the article; the
code.json pointer auto-filled at scaffold time — rvalieris/parallel-fastq-dump — is
an incorrect enrichment artifact and unrelated to this paper). Per brief rule P16,
reproduction proceeds by re-implementing the described pipeline (RNA-seq counts →
normalization → Random Forest on the authors' reported feature genes) on the paper's own
data. This is an equally valid reproduction.
In scope (pipeline-derived; what we attempt)
| # | Result | Pipeline | Reported location |
|---|---|---|---|
| C1 | Hepatic-metastasis RF classifier (primary vs liver-met) — train & test accuracy/sens/spec | counts→logCPM→RF on 5-gene set HM-RF1 {ALB,SFRP2,PRRX2,LMO3,NKX2-3} | Table 3 |
| C2 | Hepatic-metastasis RF classifier — HM-RF2 {ALB,SFRP2,PRRX2,LMO3,TBX20} | same | Table 3 |
| C3 | Primary-site RF classifier (pancreas vs small-intestine, liver-met samples) — train & test | counts→logCPM→RF on PS-RF1 {DPP6,GABBR2,SYT16,SGPP1} | Table 6 |
| C4 | Cohort composition (n=182: 126 primary + 56 liver-met; 83 pNET / 43 siNET primary; 30 pNET / 26 siNET liver) | sample-label parse of GSE98894 | Sec. 2 / Table 1 |
Out of scope / the hard 20% (not attempted, stated why)
- mRMRe 500-iteration feature selection that produced the gene sets. We do not re-run the stochastic feature selection; instead we test the authors' reported final gene sets directly on their data — a faithful test of the reported models, not of the selection search. (Re-running mRMRe to rediscover the exact 5 genes is the optional 20%.)
- Independent validation on GSE118014 (different platform/units, sample-label mapping unclear from the matrix) — attempted opportunistically only.
- Multi-label scikit-learn models and all 7-algorithm sweeps beyond RF (the headline result is the RF models; other algorithms are secondary).
Compute
All compute on «our HPC»-2 (SLURM, conda prefix env on «infra»). «host» holds results only.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
On the paper's own public data (GSE98894) the cohort (pancreas counts exact, SI off by 1-2), 100% training accuracy, and gene-set discriminativeness all reproduce, but the headline 100%/100%/100% TEST acc/sens/spec is not derivable — held-out RF reaches only ~90-91% (HM, never 100% in 200 splits) and ~78-84% (PS), the latter matching the paper's own non-RF algorithms at 81.25%. The most likely cause is on the authors' side: feature-selection (mRMRe) run before the train/test split, leaking test data into gene choice and inflating test metrics — a 'too perfect' over-optimism flag, not fabrication, since the genes genuinely discriminate. Deviation is moderate (magnitude/direction of predictive ability holds at ~90%) but the central perfect-classification claim does not reproduce.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.