Developing prognostic gene panel of survival time in lung adenocarcinoma patients using machine learning.
Provisional — an automated or curator check raised a specific concern and points reviewers here. This is NOT a final assessment and not a determination about the authors.
The main result did not reproduce in this reproduction attempt. Where our recomputation produced values that differ from the published ones, those discrepancies are listed below. This is a single automated attempt — not peer review and not a finding of error or misconduct — and differences can also arise from data access, undocumented parameters or the computing environment. The verdict can be contested via “report an error”.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- Nothing in this column.
- 🔴Could not use the authors’ exact input data
- 🟡Reported values were only indirectly comparable
- 🟡A deviation arose in the data or preprocessing
- 🔴A deviation was attributed to the published material
- 🔴Reported values were not (fully) derivable from the shared data
- 🔴The deviation was non-trivial in magnitude
- 🔴The central claim did not (fully) hold under reproduction
- 🔴Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
DESCRIBED WELL ENOUGH? Partially. The repo (ningshuishi/genedata @10aa9c15) ships 8 unmodified Python-2 modeling scripts but NO data, NO README, and NO preprocessing code; every script hardcodes an input matrix (with24_data_norm.csv) that is absent. The Table 1 headline numbers (NB acc 0.75 / AUC 0.81; RF 0.75/0.76; SVM 0.74/0.73) are computed on a TCGA RNA-seq+methylation+mutation INTEGRATED matrix (131 samples x 40540 features) which is neither shipped nor regenerable from shipped code -> the headline results are NOT independently reproducible (the hard >20% we deliberately did not chase). WHAT WE DID (different-but-valid, honest): ran the authors' OWN unmodified scripts (filter_relief.py -> GaussianNB/rf/svc) on the paper's PUBLIC validation cohort GSE72094 (398 samples x 60607 probes), in the paper's declared stack (py2.7 / numpy 1.14.3 / pandas 0.23.0 / sklearn 0.20.4), reconstructing the input matrix in the scripts' exact format. The pipeline RUNS and yields best accuracies 0.62-0.64 (NB 0.635, RF 0.642, SVM 0.619) with AUCs 0.55-0.66 -- markedly LOWER than Table 1, as expected for a different cohort/feature set/split. 1:1 vs DIFFERENT vs DROP: it is a DIFFERENT-data run of the authors' real code (P16-valid), graded partial, not a 1:1 of Table 1 (impossible by construction). TWO INTEGRITY FLAGS for the human auditor: (1) the entire Table 1 depends on an unshipped, unverifiable input matrix -> possible-fabrication/non-reproducibility note; (2) Methods claim '4-fold cross-validation' but the shipped code does a single hard-coded positional train/test split (rows 0:99 / 99:) with NO cross-validation, and the TCGA test set is only ~32 samples, making AUC 0.81 (CI up to 0.94) statistically fragile. NOT ATTEMPTED: TCGA download+integration+normalization (no code), the 22-gene panel selection, and the KM/mutation/TMB statistics (GraphPad/SPSS, manual, out of scope).
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 32assessed: 2026-06-15 ⛓ 781f89a6bcf2
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-15
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-15no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusThe authors hypothesize that genetic features (RNA expression) selected from transcriptome data correlate with and can predict the survival time of lung adenocarcinoma patients, enabling accurate classification of long-survival versus short-survival cohorts using machine learning.
- ★ A panel of 22 genetic features with Naïve Bayes can predict whether lung adenocarcinoma patient survival time is >3 years (accuracy=75%, AUC=0.81). finding
- ★ Naïve Bayes is the best and most stable algorithm across datasets for predicting LUAD survival time, outperforming other models on the external validation cohort. finding
- ★ The 22 selected genes affect the survival time of lung adenocarcinoma patients. finding
- ★ The coupled mutation number within the 22 genes is lower in the long-survival group (>6 years) than the short-survival group (<1 year). finding
- ★ Using the Relief algorithm to select top genetic features and machine learning to classify survival outperforms traditional Cox regression for LUAD prognosis. method
- The 22-feature panel comprises 13 protein-coding genes and 9 long non-coding RNAs, suggesting lncRNAs affect protein coding. resource
- No significant correlation was found between DNA methylation value and survival time for the selected genes. finding
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| bulk RNA-seq (FPKM) | TCGA-LUAD human lung adenocarcinoma patient tumors | none | gene expression for survival-time classification (>3 vs <3 years) | TCGA GDC portal data; R v3.4.3, Pandas v0.23.0, Numpy v1.14.3 |
| RNA expression microarray (log2, IRON normalized) | GEO GSE72094 human lung adenocarcinoma patients (n=174) | none | gene expression for model validation of survival classification | GSE72094 microarray |
| DNA methylation array | TCGA-LUAD human samples (Methylation 27 set has 200 samples) | none | methylation beta values vs survival time | Illumina Human Methylation 27 Beadchip and Illumina Human Methylation 450 Beadchip; limma in R |
| DNA mutation sequencing (level 3) | TCGA-LUAD human samples (131 samples; mutation analysis on short/long-survival groups) | none | mutation counts within 22 genes and total mutation burden vs survival group | Firebrowse (level 3 MAF data) |
- – Naïve Bayes with top 22 features classified >3-year survival on TCGA with accuracy 75% and AUC 0.81. AUC=0.81 (0.70–0.94), ACC=0.75
- – Naïve Bayes maintained stable performance on the external GEO validation cohort while Logistic Regression and SVR(poly) dropped. NB 69% vs LR 51%, SVR(poly) 57%
- – Naïve Bayes significantly distinguished the >3 and <3 year survival groups in the GEO confirmation cohort (Kaplan-Meier). P=0.0438
- ▼ Mutation counts within the 22 genes were significantly lower in long-survival (>6 years) vs short-survival (<1 year) group. P=0.031
- ▲ Ratio of mutations in 22 genes to total genes was higher in short-survival than long-survival group. P=0.026
- – Total mutation burden did not differ significantly between short- and long-survival groups. P=0.147
- – Poly SVR had the best regression performance with high concordance. ACC=0.77, C-index=0.69, MAE=1.92, RMSE=2.81
- – SULT1E1 was the most frequently mutated gene in the short-survival group, appearing 3 times in 28 samples. 3 mutations in 28 samples
- other AUC=0.81 (0.70–0.94) (Naïve Bayes classification of >3-year survival, TCGA)
- other accuracy=75% (0.68–0.81) (Naïve Bayes, 22 features, TCGA cohort)
- pvalue P=0.031 (Mann-Whitney U: mutation number in 22 genes, short vs long survival)
- pvalue P=0.0438 (Breslow test, Kaplan-Meier separation of groups in GEO cohort)
- pvalue P=0.026 (Mann-Whitney U: ratio of mutations in 22 genes to total)
- pvalue P=0.147 (Mann-Whitney U: total mutation burden, short vs long survival (not significant))
- count 131 cancer samples covering RNA-Seq, DNA-Seq and DNA methylation (TCGA-LUAD samples used)
- other C-index average=0.58 (traditional Cox regression baseline performance cited)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
The paper applies eight supervised machine learning algorithms with 4-fold cross-validation to classify TCGA lung adenocarcinoma patients into >3 or <3 year survival groups using RNA-seq features selected by the Relief algorithm; Naïve Bayes (accuracy=75%, AUC=0.81, 22 features) was selected as the best-performing model. The model was externally validated on an independent GEO cohort (n=174) with Kaplan-Meier curves and the Breslow test. Secondary analyses compared mutation counts between extreme survival subgroups using Mann-Whitney U tests, assessed gene expression differences with Student's t-tests, and evaluated DNA methylation with the limma package under FDR-controlled thresholds. Results are reported as classification accuracy, AUC, C-index, and exact p-values.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Naïve Bayes binary classifier evaluated by accuracy and AUC (4-fold cross-validation) | Primary survival prediction (>3 vs <3 years) using top 22 RNA-seq features; TCGA training and internal validation | 131 TCGA samples with complete RNA-Seq, DNA-Seq, and methylation data | not stated |
| Breslow (Gehan-Breslow-Wilcoxon) test | Kaplan-Meier survival curve comparison between predicted >3 and <3 year groups in GEO confirmation cohort (Figure 4, P=0.0438) | 174 (GEO dataset GSE72094) | not stated |
| Mann-Whitney U test | Mutation counts in 22 genes: short-survival (<1 year) vs long-survival (>6 years) groups (Figure 6A, P=0.031) | 28 samples stated for short-survival group; n for long-survival group not stated; drawn from 131 TCGA samples with mutation data | not stated |
| Mann-Whitney U test | Total mutation burden: short-survival vs long-survival groups (Figure 6B, P=0.147) | Same subgroups as Figure 6A; exact n per group not stated | not stated |
| Mann-Whitney U test | Ratio of mutations in 22 genes to total genes: short-survival vs long-survival (Figure 6C, P=0.026) | Same subgroups; exact n per group not stated | not stated |
| Student's t-test | RNA expression differences between survival groups for four selected genes (BBOX1, CRISP3, MARCH4, UNC5A; Figure 5, ***P<0.01) | not stated | not stated |
| limma linear model with FDR filter (|log2FC|>1, FDR<0.05) | Differential DNA methylation analysis versus survival time | Up to 200 samples (Illumina Human Methylation 27 dataset); exact n after filtering not stated | not stated |
| Cox proportional hazards model | Feature filtering (supplementary appendix only; not part of main model analysis) | not stated | not stated |
-
Survival time was binarized at a 3-year threshold and framed as a binary classification task↳ Could also: Regularized time-to-event modeling such as LASSO-Cox or elastic net Cox regression could also be applied to the same high-dimensional RNA-seq features — These approaches preserve the continuous time-to-event outcome including censoring, directly optimize a survival-relevant objective, and are the prevailing paradigm for genomic survival modeling; they can extract more information per sample than binary grouping
-
Feature selection via the Relief algorithm was performed on the full dataset before cross-validation↳ Could also: Nested cross-validation, in which feature selection is repeated independently within each training fold, could also be used — Performing feature selection inside each CV fold avoids potential information leakage from test-fold labels into feature weights; this is a standard practice for unbiased generalization estimates in high-dimensional settings
-
Three Mann-Whitney U tests were conducted on related mutation comparisons (mutation count in 22 genes, total burden, and ratio) without a stated multiple testing correction↳ Could also: A Bonferroni correction or Benjamini-Hochberg FDR could also be applied across this family of related tests — When multiple tests share a conceptual family — here, all comparing mutation characteristics between the same two groups — applying a correction controls the probability of at least one spurious result within that family
-
The Breslow (Gehan-Breslow-Wilcoxon) test was used to compare Kaplan-Meier survival curves↳ Could also: The log-rank test (Mantel-Cox) is also widely used for Kaplan-Meier comparisons and is the default in many survival analysis guidelines — The Breslow test weights early time points more heavily than later ones, while the log-rank test weights all time points equally; stating the rationale for the choice helps readers understand which part of the survival curve drives the reported significance
-
Student's t-tests were used to compare gene expression between survival groups (Figure 5)↳ Could also: The Mann-Whitney U test (already used elsewhere in this paper) or a limma moderated t-test could also be applied for the same expression comparisons — RNA-seq expression values — particularly in small subgroups — may not follow a normal distribution; Mann-Whitney U does not assume normality, and limma's moderated t-test borrows variance information across genes, which can increase stability in small samples
-
ML model performance (accuracy, AUC) was reported with min-max ranges across cross-validation folds↳ Could also: Bootstrap confidence intervals or the DeLong method for AUC confidence intervals could also be used to characterize uncertainty in performance estimates — Formal confidence intervals convey statistical uncertainty in a way that fold-range min-max values do not, and are commonly reported alongside AUC in clinical prediction model literature (e.g., following TRIPOD reporting guidelines)
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
No assessed neighbours yet — the network grows as more papers are assessed.
Data lineage
The datasets this paper uses (text-mined from the full text via Europe PMC), and which other assessed papers stand on the same data. A shared dataset is a factual link — not a judgement.
Downstream reach in the literature
100 downstream papers · 1 datasetsHow widely the datasets deposited by this paper are reused across the whole literature (Europe PMC), beyond our assessed set. This is a factual dependency map — reusing a public dataset is normal, good science. It is not a judgement on the downstream papers; the only verdict here is this paper's own, with its cited rationale.
- APOBEC3B and APOBEC mutational signature as potentia... 2018 · 205 cites
- MUC1-C integrates PD-L1 induction with repression of... 2017 · 104 cites
- Identification and Validation of a Novel Signature B... 2022 · 94 cites
- Single-cell analysis reveals prognostic fibroblast s... 2023 · 91 cites
- An LKB1-SIK Axis Suppresses Lung Tumor Growth and Co... 2019 · 90 cites
- Development and Validation of a Robust Ferroptosis-R... 2021 · 78 cites
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-35117753
Paper: Liu Y, et al. Developing prognostic gene panel of survival time in lung adenocarcinoma patients using machine learning. Transl Cancer Res 2020. DOI 10.21037/tcr-19-2739 · PMCID PMC8799101.
Repo: https://github.com/ningshuishi/genedata (master, last push 2019-10-12, 4 KB, Python, no README, no archive/private flags).
What the repo actually ships
8 Python (Python-2) modeling scripts, no data, no README, no preprocessing code:
filter_relief.py— Relief feature ranking; readswith24_data_norm.csv, writesgene_importance_relief_with24.csv.GaussianNB.py,rf.py,svc.py,lr_xunhuan.py,linear_regression_xunhuan.py,ridge_xunhuan.py,svr_xunhuan.py— classifiers/regressors. Each readswith24_data_norm.csv+gene_importance_relief_with24.csv, sweeps the top 1..200 Relief genes, printsacc, auc, feature_num, writeskfold/<model>4.xls.
Hardcoded contract (verbatim from the scripts):
- Input matrix
with24_data_norm.csv,index_col=0(integer row ids 0..N-1), columns = normalized features + analive_yearcolumn in years, plus two artifact columnsUnnamed: 0.1,Unnamed: 0.1.1thatfilter_relief.pydrops. - Relief class split:
alive_year <= 3vs> 3. Classifier label:0 if alive_year <= 2 else 1. Fixed split: train = rows 0:99, test = rows 99: (no randomisation).
In scope (pipeline-derived) vs out of scope
| Reported result | Pipeline | In scope? |
|---|---|---|
| Table 1 classifier acc/AUC/#features on TCGA (NB 0.75/0.81; LR 0.77/0.74; RF 0.75/0.76; SVM 0.74/0.73; SVR/Ridge…) | Relief → sklearn classifier (the shipped scripts) | Partially — code shipped, but the TCGA-integrated input matrix is NOT shipped and not regenerable (no preprocessing code). |
| 22-gene panel (13 mRNA + 9 lncRNA) | Relief top-k | Out — depends on the unshipped TCGA matrix + a manual cut. |
| GEO validation accuracy 69% on GSE72094 | trained model applied to GSE72094 | Partially — GSE72094 is public; the trained model/22-gene list is not shipped. |
| KM p=0.0438 (Fig 4); mutation p=0.031/0.026 (Fig 6); TMB p=0.147 | survival stats / mutation analysis (GraphPad/SPSS) | Out — wet-lab/manual stats tools, not in the repo. |
Core obstacle (the honest finding)
The shipped artifact is not self-contained: every script requires
with24_data_norm.csv (the authors' TCGA RNA-seq + DNA-methylation + mutation
integrated, normalized 131-sample × 40 540-feature matrix) and that matrix —
and the code that builds it — are absent. Reconstructing it exactly is the hard
20 % and is not derivable from anything shipped → the exact Table 1 (TCGA) numbers cannot be reproduced 1:1 from repo+data alone.
What we DO attempt (bounded, honest)
Run the authors' unmodified scripts (filter_relief.py → GaussianNB.py,
rf.py, svc.py) on the paper's public dataset GSE72094 (the validation
cohort), reconstructing with24_data_norm.csv in the exact format the scripts
expect (per-probe min-max normalized expression + alive_year in years + the two
artifact columns + the fixed 0:99 / 99: split). This exercises the real Relief →
classifier pipeline on the paper's real public data and lets us compare the
achievable NB/RF/SVM accuracy & AUC against Table 1.
Grade expectation: partial — same code & pipeline, but the modeling cohort is GSE72094 (validation set), not the TCGA matrix behind Table 1; comparison is indicative, not a 1:1 of the TCGA numbers.
Out of scope / not attempted: TCGA download+integration+normalization (no shipped code), the 22-gene panel selection, KM/mutation/TMB statistics (GraphPad/SPSS, manual).
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
The paper's entire Table 1 (NB acc 0.75/AUC 0.81, RF 0.75/0.76, SVM 0.74/0.73) is computed on a TCGA RNA+methylation+mutation integrated matrix (131x40540) that is not shipped and not regenerable — no data, no preprocessing code — so the headline numbers are not independently derivable (authors'-side reproducibility defect, not access restriction). Running the authors' own unmodified scripts on the public GSE72094 cohort yields acc 0.62-0.64 and AUCs collapsing toward chance (SVM AUC 0.564), far below the claims. A second integrity flag: Methods claim 4-fold CV but the code does a single hard-coded positional split, making the reported AUCs fragile. Net: a critical discrepancy resting on unverifiable inputs and a methods-vs-code mismatch — possible-fabrication/non-reproducibility concern.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.