Fast, accurate, and racially unbiased pan-cancer tumor-only variant calling with tabular machine learning.
Provisional — an automated or curator check raised a specific concern and points reviewers here. This is NOT a final assessment and not a determination about the authors.
Part of the results reproduced; minor but material deviations remained.
- Nothing in this column.
- 🔴Could not use the authors’ exact input data
- 🔴Reported values were only indirectly comparable
- 🟡A deviation arose in the data or preprocessing
- 🔴A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The deviation was non-trivial in magnitude
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
This paper has a computational component, but its primary data is legally or ethically access-restricted — identifiable patient cohorts, rare-disease genomes, or controlled-access biobanks that cannot be openly shared. The reproduction therefore could not be attempted. That is a neutral verdict: it does not mean the result is wrong or that the authors fell short — only that, for legitimate privacy reasons, it cannot be independently checked from public data. We deliberately do NOT assign a 0–100 score here, because a low number would wrongly read as a failed reproduction.
▸Reproduction agent’s raw note
DROP. Real code = AbbVie-GRC-Methods-Dev/new_normal (not the dreamquark tabnet lib in the registry); MIT, ships CODE ONLY. The method (engineer ~30 features per tumor-only variant -> TabNet/XGBoost/LightGBM somatic-vs-germline; matched-normal FILTER = truth) is described clearly and the feature_engineering/prob_somatic/eval code is readable, BUT it is not runnable on any obtainable data: (1) all training + headline-test data is TCGA WES = dbGaP CONTROLLED-ACCESS; (2) model inputs are MAFs from AbbVie's internal non-shipped 'fioSeq' pipeline carrying pre-computed annotations (COSMIC v85 license-gated, gnomAD pop_max, mappability, trinucleotide context, fpfilter, matched-normal truth) that the repo consumes but never computes; (3) no trained model (tabnet_trained_tcga.pkl), no engineered feature CSVs, no intermediates shipped (verified 404). The single public cohort (melanoma SRP067938: 47 WXS runs = 23 tumor + matched normals, present and well-formed) is downloadable but cannot reproduce R4 without the TCGA-trained model and the fioSeq annotation stack. README is one sentence. Not attempted as a from-scratch surrogate because a different caller + differently-trained model would not reproduce the reported numbers (would be a different experiment) and the brief forbids forcing/fabrication. NOT attempted: any compute job (no obtainable inputs). Verdict is provisional and human-checkable.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessmentassessed: 2026-06-19 ⛓ 6898243fccee
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-19
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-19no human curator yet
- Last updated
- 2026-09-19
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetThe authors hypothesized that supervised machine-learning algorithms (TabNet, XGBoost, LightGBM) trained on tumor-only-derived features could accurately classify variants as somatic or germline in tumor-only WES samples lacking a matched-normal, thereby improving TMB estimation and reducing racial bias caused by underrepresentation of minority germline variants in reference databases.
- ★ Tree-based (XGBoost, LightGBM) and deep-learning (TabNet) tabular ML classifiers achieve state-of-the-art somatic vs germline classification in tumor-only WES samples, outperforming PureCN. finding
- ★ Adding an ML classifier improves concordance between tumor-only and matched-normal TMB from R2=0.006 to 0.71-0.76, with LightGBM performing best. finding
- ★ The trained ML models generalize across cancer subtypes and exome-capture kits not seen in training, with a 100% call rate. finding
- ★ Tumor-only TMB estimates are extremely inflated for Black patients relative to white patients due to racial bias in germline variant databases, and this bias is reproduced in the study. finding
- ★ XGBoost and LightGBM classifiers eliminate the racial bias in tumor-only TMB estimation observed with standard tumor-only variant calling. finding
- ★ 30 mutation- and copy-number-specific features (germline database frequency, COSMIC counts, VAF, major allele frequency, trinucleotide/mutational signature context, local copy-number-derived VAF histogram features) were engineered from tumor-only data to train the classifiers. method
- Somatic/germline truth labels for training were derived from an independent matched-normal variant-calling pipeline. method
- An ensemble average of the three models' posterior probabilities was constructed for classification. method
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| Whole-exome sequencing (WES) variant calling, tumor-only pipeline | TCGA training set: 105 patients across BLCA, GBM, HNSC, LUAD, LUSC, OV, STAD (15 each) | none (tumor-only vs matched-normal comparison) | Somatic vs germline variant classification features (30 engineered features) | Agilent Custom V2 exome-capture kit, sequenced at Broad Institute |
| WES variant calling, matched-normal pipeline | Same TCGA training patients (matched-normal blood samples) | none | Ground-truth somatic (1) vs germline (0) labels | — |
| Copy-number variation (CNV) analysis | TCGA tumor-only samples (training, validation, test) | none | Local copy-number segmentation used to derive VAF-bin histogram features | — |
| WES variant calling, tumor-only pipeline (validation set) | TCGA COAD, DLBC, TGCT (15 each, n=45) | none | Model AUC, average precision for somatic/germline classification | SeqCap EZ HGSC VCRome kit, Baylor College of Medicine |
| WES variant calling, tumor-only pipeline (holdout test set 1) | TCGA BRCA, SARC, UCEC (15 each, n=45) | none | AUC, MCC, TPR, TNR, PPV, NPV, balanced accuracy, call rate for SNVs and indels | Nimblegen SeqCap EZ Exome v3, Washington University in St. Louis |
| WES variant calling, tumor-only pipeline (holdout test set 2) | Metastatic melanoma dataset (n=23 patients, ref. 39) | none | AUC, MCC, TPR, TNR, PPV, NPV, balanced accuracy for somatic/germline classification | Nimblegen SeqCap EZ Exome v3, sequenced by UCLA |
| Machine learning classification (TabNet, XGBoost, LightGBM, PureCN comparison) | All TCGA and melanoma cohorts (tabular feature matrices) | ML classifier applied vs standard tumor-only filtering (PureCN) | Tumor mutational burden (TMB) concordance with matched-normal TMB (R2) | — |
| Racial bias analysis of TMB estimation | Tumor-only cohort stratified by patient race (Black vs white) | none/ML classifier applied | Tumor-only TMB inflation relative to matched-normal TMB by race | — |
- ▲ Training AUCs: TabNet 0.96, LightGBM 0.98, XGBoost 0.99 0.96-0.99
- ▼ Validation set AUC dropped to 0.91-0.92 across models 0.91-0.92
- ▲ Holdout test (BRCA/SARC/UCEC) overall AUC: TabNet 0.942, XGBoost 0.946, LightGBM 0.949, vs PureCN 0.85 AUC 0.942-0.949 vs 0.85
- ▲ Holdout test (BRCA/SARC/UCEC) call rate 100% for all ML models vs 82.2% for PureCN 100% vs 82.2%
- ▲ Metastatic melanoma holdout overall AUC: TabNet 0.852, XGBoost 0.861, LightGBM 0.867, vs PureCN 0.824 AUC 0.852-0.867 vs 0.824
- ▲ TMB concordance (matched-normal vs tumor-only) improved from R2=0.006 (no classifier) to R2=0.71-0.76 with ML classifier addition, LightGBM best R2 0.006 to 0.71-0.76
- ▲ Tumor-only TMB inflation for Black patients relative to white patients reproduced, attributed to germline database racial bias
- – XGBoost and LightGBM eliminate the racial bias in tumor-only TMB estimation
- correlation R2 = 0.006 (baseline, no ML classifier) (Tumor-only vs matched-normal TMB concordance without classifier)
- correlation R2 = 0.71-0.76 (with ML classifier) (Tumor-only vs matched-normal TMB concordance with ML classifier, LightGBM best)
- other AUC > 94% (TCGA holdout test set (BRCA, SARC, UCEC))
- other AUC > 85% (Metastatic melanoma holdout test set)
- other LightGBM overall test set (BRCA/SARC/UCEC): AUC 0.949, MCC 0.766, TPR 0.874, TNR 0.892, PPV 0.886, NPV 0.88, balanced accuracy 0.883, call rate 100 (Table 1, overall 30,270 somatic / 31,359 germline variants)
- other PureCN overall test set (BRCA/SARC/UCEC): AUC 0.85, MCC 0.592, call rate 82.2 (Table 1, comparison baseline method)
- fold_change 2.2-16.9-fold inflation of tumor-only TMB vs matched-normal (cited from prior study) (Background citation on tumor-only TMB inflation)
- other 67% false positive rate without matched-normal sample (cited from prior study) (Background citation on tumor-only variant calling error rate)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
The paper describes a supervised machine-learning classification study rather than a classical hypothesis-testing design: TabNet, XGBoost, and LightGBM classifiers were trained on features from tumor-only variant calls (with truth labels from matched-normal calling) using a fixed train/validation/holdout-test split across TCGA cancer subtypes and an independent metastatic melanoma cohort. Model performance was quantified using discrimination and agreement metrics (AUC, MCC, sensitivity/specificity, PPV/NPV, balanced accuracy, call rate) benchmarked against PureCN, and concordance between matched-normal and tumor-only TMB estimates was summarized with R². Patient-level results per tissue type were also displayed as boxplots (median, quartiles, Tukey-defined outliers).
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Area under the ROC curve (AUC) and classification metrics (MCC, TPR/sensitivity, TNR/specificity, PPV, NPV, balanced accuracy, call rate) | Comparison of TabNet, XGBoost, LightGBM, and PureCN predictions vs. matched-normal truth labels on training, validation, and holdout test sets (Table 1, Fig. 2) | Training n=105 patients (15 per subtype x7); validation n=45 (15 per subtype x3); holdout test 1 n=45 (BRCA/SARC/UCEC, 15 each); holdout test 2 n=23 (metastatic melanoma) | not stated |
| Coefficient of determination (R²) | Concordance between matched-normal-derived and tumor-only-derived TMB estimates, with vs. without ML classifier | not stated in provided text | not stated |
| Descriptive boxplot summary (median, quartiles, 1.5xIQR outlier fences) | Patient-level performance metrics by tissue type, Fig. 2 | n=15 patients per tissue type as stated | na |
-
Classifier performance (AUC, MCC, sensitivity/specificity, PPV/NPV) is reported as point estimates on holdout test sets without confidence intervals.↳ Could also: Bootstrap resampling or analytic methods (e.g., DeLong's method for AUC) to compute 95% confidence intervals around these performance metrics — Confidence intervals would convey the precision of the estimated performance given the finite size of the holdout test sets, complementing the point estimates already reported.
-
Differences in performance among TabNet, XGBoost, LightGBM, and PureCN are compared via point estimates in Table 1.↳ Could also: A paired statistical comparison such as McNemar's test or DeLong's test for comparing correlated ROC curves on the same held-out variants — A formal paired test would help quantify whether observed differences between classifiers exceed what might be expected from sampling variability alone.
-
Concordance between matched-normal and tumor-only TMB estimates is summarized using R².↳ Could also: Bland-Altman agreement analysis or a concordance correlation coefficient (e.g., Lin's CCC) — These approaches directly quantify agreement and systematic bias between the two TMB estimation methods, which can complement the variance-explained interpretation of R².
-
Patient-level metrics per tissue type are summarized visually with boxplots (median, IQR, Tukey outlier fences) for relatively small per-group n (n=15-23).↳ Could also: Reporting means with SD or bootstrap confidence intervals alongside the boxplots — With small per-subtype sample sizes, an additional parametric or resampling-based summary could offer another view of central tendency and spread alongside the nonparametric boxplot description.
-
No multiplicity correction is described despite many comparisons being drawn across models, tissue subtypes, and metrics from overlapping datasets.↳ Could also: A false discovery rate procedure (e.g., Benjamini-Hochberg) if formal hypothesis tests were performed across these comparisons — Such a correction would control the proportion of false positives when many related comparisons are examined simultaneously.
-
Holdout test set sizes (e.g., n=45, n=23) were used without a described a priori sample-size or power calculation.↳ Could also: A pre-specified power analysis to determine holdout size needed to detect a target difference in AUC or MCC — This would provide a rationale for the chosen holdout sizes and an estimate of the study's sensitivity to detect meaningful performance differences.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-36611079
Title: Fast, accurate, and racially unbiased pan-cancer tumor-only variant calling with tabular machine learning. (McLaughlin et al., npj Precision Oncology 2023; PMID 36611079, PMC9825621, DOI 10.1038/s41698-022-00340-1)
Paper's code (real repo): https://github.com/AbbVie-GRC-Methods-Dev/new_normal
(MIT). NOTE: the registry/code_url listed dreamquark-ai/tabnet — that is only
the generic TabNet library the paper depends on, not the paper's code. The
paper's actual implementation is the AbbVie new_normal repo. We analysed the
latter.
What the paper does (one line)
Re-cast tumor-only somatic-vs-germline variant classification as a tabular ML
problem: run a somatic caller in tumor-only mode to get candidate variants,
engineer ~30 features per variant (germline population frequency, COSMIC counts,
VAF, copy-number, trinucleotide context, etc.), and train TabNet / XGBoost /
LightGBM to label each variant somatic vs germline. Truth labels come from the
matched-normal run of the same caller (FILTER==PASS ⇒ somatic).
In-scope (pipeline-derived) results
All headline numbers are pipeline-derived and therefore in scope in principle:
| id | result | pipeline | reported |
|---|---|---|---|
| R1 | AUC on TCGA holdout test (SNV+indel) | new_normal feature-eng + TabNet/XGB/LGBM | LGBM 0.949 / XGB 0.946 / TabNet 0.942 (PureCN 0.85) |
| R2 | MCC on TCGA test | same | LGBM 0.766 / XGB 0.757 / TabNet 0.762 |
| R3 | Sensitivity / Specificity / PPV on TCGA test | same | TabNet sens 0.931; LGBM spec 0.892, PPV 0.886 |
| R4 | AUC on melanoma external test (SRP067938) | trained model applied to Hugo-2016 | LGBM 0.867 / XGB 0.861 / TabNet 0.852 (PureCN 0.824) |
| R5 | TMB concordance R² (test) | same | TabNet 0.705 / XGB 0.725 / LGBM 0.759; naïve 0.006 |
| R6 | runtime | LightGBM vs PureCN | LGBM 55.4 s (1 core) = 21.9× faster than PureCN 1214 s (250 cores) |
| R7 | racial-bias removal | LGBM TMB Black 1.76 vs white 1.68 mut/Mb | p>0.05 |
| R8 | variant counts (TCGA test) | caller output | 30,270 somatic / 31,359 germline |
Out-of-scope
- Wet-lab / sequencing (TCGA + Hugo cohorts were sequenced by others) — external.
- Manual cohort curation, racial-ancestry annotation provenance — external.
Reproducibility assessment (the decisive part)
The repo ships code only. It does not ship any of the inputs the code needs, and the inputs are not independently obtainable:
- Model inputs are MAFs from an internal, non-shipped pipeline. Every entry
point (
gather_input_data.wes_pipeline_to_ml_input,1_format_mafs.py) reads pre-existing per-patient.maffiles from/fioSeq/projects/output/dna/...(input_pipeline='fioseq'). "fioSeq" is AbbVie's internal variant-calling + annotation pipeline; it is not in the repo and not described runnably. The input MAFs already carry annotation columns the downstream code consumes but never computes:max_cosmic_count(COSMIC v85),pop_max(gnomAD/ExAC/1000G),100mer_mappability,trinucleotide_context,mutation_change,fpfilter,ontology, plus the FILTER that encodes the matched-normal truth. Reproducing them needs a full annotation stack (incl. COSMIC v85, license-gated) that is not provided. - Training / validation / headline-test data is dbGaP controlled-access. The
105 training + 45 validation + 45 holdout-test samples (R1–R3, R5–R8) are TCGA
WES BAMs (subtypes BLCA/GBM/HNSC/LUAD/LUSC/OV/STAD; COAD/DLBC/TGCT;
BRCA/SARC/UCEC). TCGA controlled-access WES requires a dbGaP Data Access
Request the room does not hold.
data_splits.csvlists barcodes + internal paths, no usable data. - No trained model, no feature tables, no intermediates are shipped. The
melanoma eval (
5_eval_trained_tabnet.py) loads../tcga_pancancer/tabnet_trained_tcga.pklandmafs/engineered_plus_labels.csv— both 404 in the repo (verified by HEAD requests).
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at [email protected].
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.