Fast, accurate, and racially unbiased pan-cancer tumor-only variant calling with tabular machine learning.
Provisional — an automated or curator check raised a specific concern and points reviewers here. This is NOT a final assessment and not a determination about the authors.
Part of the results reproduced; minor but material deviations remained.
- Nothing in this column.
- 🔴Could not use the authors’ exact input data
- 🔴Reported values were only indirectly comparable
- 🟡A deviation arose in the data or preprocessing
- 🔴A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The deviation was non-trivial in magnitude
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
This paper has a computational component, but its primary data is legally or ethically access-restricted — identifiable patient cohorts, rare-disease genomes, or controlled-access biobanks that cannot be openly shared. The reproduction therefore could not be attempted. That is a neutral verdict: it does not mean the result is wrong or that the authors fell short — only that, for legitimate privacy reasons, it cannot be independently checked from public data. We deliberately do NOT assign a 0–100 score here, because a low number would wrongly read as a failed reproduction.
▸Reproduction agent’s raw note
DROP. Real code = AbbVie-GRC-Methods-Dev/new_normal (not the dreamquark tabnet lib in the registry); MIT, ships CODE ONLY. The method (engineer ~30 features per tumor-only variant -> TabNet/XGBoost/LightGBM somatic-vs-germline; matched-normal FILTER = truth) is described clearly and the feature_engineering/prob_somatic/eval code is readable, BUT it is not runnable on any obtainable data: (1) all training + headline-test data is TCGA WES = dbGaP CONTROLLED-ACCESS; (2) model inputs are MAFs from AbbVie's internal non-shipped 'fioSeq' pipeline carrying pre-computed annotations (COSMIC v85 license-gated, gnomAD pop_max, mappability, trinucleotide context, fpfilter, matched-normal truth) that the repo consumes but never computes; (3) no trained model (tabnet_trained_tcga.pkl), no engineered feature CSVs, no intermediates shipped (verified 404). The single public cohort (melanoma SRP067938: 47 WXS runs = 23 tumor + matched normals, present and well-formed) is downloadable but cannot reproduce R4 without the TCGA-trained model and the fioSeq annotation stack. README is one sentence. Not attempted as a from-scratch surrogate because a different caller + differently-trained model would not reproduce the reported numbers (would be a different experiment) and the brief forbids forcing/fabrication. NOT attempted: any compute job (no obtainable inputs). Verdict is provisional and human-checkable.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessmentassessed: 2026-06-19 ⛓ 6898243fccee
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-19
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-19no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusThe authors hypothesized that supervised machine-learning algorithms (TabNet, XGBoost, LightGBM) can effectively classify mutations as somatic vs germline in tumor-only solid tumor samples lacking a matched-normal, thereby improving TMB accuracy and removing racial bias in tumor-only variant calling.
- ★ Tabular ML classifiers (TabNet, XGBoost, LightGBM) trained on tumor-only-derived features achieve state-of-the-art somatic vs germline classification, with AUC>94% on TCGA holdout and AUC>85% on metastatic melanoma. finding
- ★ Adding an ML classifier improves concordance between matched-normal and tumor-only TMB from R2=0.006 to 0.71-0.76, with LightGBM performing best. finding
- ★ XGBoost and LightGBM eliminate the significant racial bias (extreme TMB inflation for Black patients) inherent in traditional tumor-only variant calling caused by germline database biases. finding
- ★ The ML models generalize across cancer subtypes and exome-capture kits with a call rate of 100%. finding
- ★ A supervised classification framework using 30 engineered mutation- and copy-number features from tumor-only variant calling, with truth labels from an independent matched-normal pipeline. method
- Biological differences between cancer tissue subtypes are more influential on classification performance than the choice of machine-learning model. finding
- ML-based classifiers achieve accuracy comparable or superior to PureCN with drastically reduced computational cost. finding
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| Whole-exome sequencing (WES) tumor-only variant calling | 105 TCGA tumor samples (BLCA, GBM, HNSC, LUAD, LUSC, OV, STAD) — training set | none | somatic vs germline variant classification / engineered features | Agilent Custom V2 exome-capture kit (Broad Institute) |
| WES matched-normal variant calling (truth-label generation) | TCGA patient-matched normal samples | none | somatic/germline ground-truth labels | — |
| WES tumor-only variant calling (validation) | 45 TCGA tumor samples (COAD, DLBC, TGCT) | none | somatic vs germline classification accuracy metrics | SeqCap EZ HGSC VCRome capture kit (Baylor College of Medicine) |
| WES tumor-only variant calling (holdout test 1) | 45 TCGA tumor samples (BRCA, SARC, UCEC) | none | AUC/MCC/TPR/TNR/PPV/NPV classification metrics | Roche Nimblegen SeqCap EZ Exome v3 kit (Washington University in St. Louis) |
| WES tumor-only variant calling (holdout test 2) | 23 metastatic melanoma patients (Hugo dataset) | none | somatic vs germline classification metrics | sequenced by UCLA |
| Copy-number variation (CNV) segmentation analysis | tumor-only samples (panel of normals, no patient-matched normal) | none | local copy-number features / VAF-binned histograms | — |
| Bayesian tumor-only somatic inference (benchmark comparison) | TCGA holdout and metastatic melanoma samples | none | somatic vs germline probability / classification metrics | PureCN |
| Tumor mutation burden (TMB) estimation | tumor-only vs matched-normal WES samples across cancer subtypes and racial groups | none | TMB (nonsynonymous somatic mutations per Mb); concordance R2 | — |
- ▲ LightGBM best AUC on BRCA/SARC/UCEC holdout (SNVs+indels overall) AUC=0.949
- ▲ LightGBM best AUC on metastatic melanoma holdout (overall) AUC=0.867
- ▲ Tumor-only vs matched-normal TMB concordance improves with ML classifier R2 from 0.006 to 0.71-0.76
- – All ML models classify variants with full call rate vs PureCN's partial rate 100% vs 73.6-88.5%
- – Training AUCs of the optimally trained models TabNet 0.96, LightGBM 0.98, XGBoost 0.99
- ▼ Validation-set AUC lower than training, indicating mild overfitting or harder tumor types AUC=0.91-0.92
- ▲ LightGBM best MCC, PPV, and balanced accuracy on TCGA holdout (overall) MCC=0.766, PPV=0.886, balanced acc=0.883
- – All models perform better on SNVs than indels; PureCN strong specificity on indels
- correlation R2 = 0.006 to 0.71-0.76 (tumor-only vs matched-normal TMB concordance before/after ML classifier)
- other AUC > 94% (TCGA holdout (BRCA, SARC, UCEC) test performance)
- other AUC > 85% (metastatic melanoma holdout test performance)
- fold_change 2.2-16.9 fold inflation (reported tumor-only-calculated TMB inflation depending on germline database filtering (ref 7))
- other 67% false positive rate (reported effect of missing matched-normal sample (ref 6))
- count 105 (TCGA training samples (15 each from 7 subtypes), Agilent Custom V2 kit)
- count 30,270 somatic / 31,359 germline (variant counts in BRCA/SARC/UCEC overall blind test set)
- count 15,813 somatic / 12,871 germline (variant counts in metastatic melanoma overall blind test set)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
The paper describes a supervised machine-learning classification study rather than a classical hypothesis-testing design: TabNet, XGBoost, and LightGBM classifiers were trained on features from tumor-only variant calls (with truth labels from matched-normal calling) using a fixed train/validation/holdout-test split across TCGA cancer subtypes and an independent metastatic melanoma cohort. Model performance was quantified using discrimination and agreement metrics (AUC, MCC, sensitivity/specificity, PPV/NPV, balanced accuracy, call rate) benchmarked against PureCN, and concordance between matched-normal and tumor-only TMB estimates was summarized with R². Patient-level results per tissue type were also displayed as boxplots (median, quartiles, Tukey-defined outliers).
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Area under the ROC curve (AUC) and classification metrics (MCC, TPR/sensitivity, TNR/specificity, PPV, NPV, balanced accuracy, call rate) | Comparison of TabNet, XGBoost, LightGBM, and PureCN predictions vs. matched-normal truth labels on training, validation, and holdout test sets (Table 1, Fig. 2) | Training n=105 patients (15 per subtype x7); validation n=45 (15 per subtype x3); holdout test 1 n=45 (BRCA/SARC/UCEC, 15 each); holdout test 2 n=23 (metastatic melanoma) | not stated |
| Coefficient of determination (R²) | Concordance between matched-normal-derived and tumor-only-derived TMB estimates, with vs. without ML classifier | not stated in provided text | not stated |
| Descriptive boxplot summary (median, quartiles, 1.5xIQR outlier fences) | Patient-level performance metrics by tissue type, Fig. 2 | n=15 patients per tissue type as stated | na |
-
Classifier performance (AUC, MCC, sensitivity/specificity, PPV/NPV) is reported as point estimates on holdout test sets without confidence intervals.↳ Could also: Bootstrap resampling or analytic methods (e.g., DeLong's method for AUC) to compute 95% confidence intervals around these performance metrics — Confidence intervals would convey the precision of the estimated performance given the finite size of the holdout test sets, complementing the point estimates already reported.
-
Differences in performance among TabNet, XGBoost, LightGBM, and PureCN are compared via point estimates in Table 1.↳ Could also: A paired statistical comparison such as McNemar's test or DeLong's test for comparing correlated ROC curves on the same held-out variants — A formal paired test would help quantify whether observed differences between classifiers exceed what might be expected from sampling variability alone.
-
Concordance between matched-normal and tumor-only TMB estimates is summarized using R².↳ Could also: Bland-Altman agreement analysis or a concordance correlation coefficient (e.g., Lin's CCC) — These approaches directly quantify agreement and systematic bias between the two TMB estimation methods, which can complement the variance-explained interpretation of R².
-
Patient-level metrics per tissue type are summarized visually with boxplots (median, IQR, Tukey outlier fences) for relatively small per-group n (n=15-23).↳ Could also: Reporting means with SD or bootstrap confidence intervals alongside the boxplots — With small per-subtype sample sizes, an additional parametric or resampling-based summary could offer another view of central tendency and spread alongside the nonparametric boxplot description.
-
No multiplicity correction is described despite many comparisons being drawn across models, tissue subtypes, and metrics from overlapping datasets.↳ Could also: A false discovery rate procedure (e.g., Benjamini-Hochberg) if formal hypothesis tests were performed across these comparisons — Such a correction would control the proportion of false positives when many related comparisons are examined simultaneously.
-
Holdout test set sizes (e.g., n=45, n=23) were used without a described a priori sample-size or power calculation.↳ Could also: A pre-specified power analysis to determine holdout size needed to detect a target difference in AUC or MCC — This would provide a rationale for the chosen holdout sizes and an estimate of the study's sensitivity to detect meaningful performance differences.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-36611079
Title: Fast, accurate, and racially unbiased pan-cancer tumor-only variant calling with tabular machine learning. (McLaughlin et al., npj Precision Oncology 2023; PMID 36611079, PMC9825621, DOI 10.1038/s41698-022-00340-1)
Paper's code (real repo): https://github.com/AbbVie-GRC-Methods-Dev/new_normal
(MIT). NOTE: the registry/code_url listed dreamquark-ai/tabnet — that is only
the generic TabNet library the paper depends on, not the paper's code. The
paper's actual implementation is the AbbVie new_normal repo. We analysed the
latter.
What the paper does (one line)
Re-cast tumor-only somatic-vs-germline variant classification as a tabular ML
problem: run a somatic caller in tumor-only mode to get candidate variants,
engineer ~30 features per variant (germline population frequency, COSMIC counts,
VAF, copy-number, trinucleotide context, etc.), and train TabNet / XGBoost /
LightGBM to label each variant somatic vs germline. Truth labels come from the
matched-normal run of the same caller (FILTER==PASS ⇒ somatic).
In-scope (pipeline-derived) results
All headline numbers are pipeline-derived and therefore in scope in principle:
| id | result | pipeline | reported |
|---|---|---|---|
| R1 | AUC on TCGA holdout test (SNV+indel) | new_normal feature-eng + TabNet/XGB/LGBM | LGBM 0.949 / XGB 0.946 / TabNet 0.942 (PureCN 0.85) |
| R2 | MCC on TCGA test | same | LGBM 0.766 / XGB 0.757 / TabNet 0.762 |
| R3 | Sensitivity / Specificity / PPV on TCGA test | same | TabNet sens 0.931; LGBM spec 0.892, PPV 0.886 |
| R4 | AUC on melanoma external test (SRP067938) | trained model applied to Hugo-2016 | LGBM 0.867 / XGB 0.861 / TabNet 0.852 (PureCN 0.824) |
| R5 | TMB concordance R² (test) | same | TabNet 0.705 / XGB 0.725 / LGBM 0.759; naïve 0.006 |
| R6 | runtime | LightGBM vs PureCN | LGBM 55.4 s (1 core) = 21.9× faster than PureCN 1214 s (250 cores) |
| R7 | racial-bias removal | LGBM TMB Black 1.76 vs white 1.68 mut/Mb | p>0.05 |
| R8 | variant counts (TCGA test) | caller output | 30,270 somatic / 31,359 germline |
Out-of-scope
- Wet-lab / sequencing (TCGA + Hugo cohorts were sequenced by others) — external.
- Manual cohort curation, racial-ancestry annotation provenance — external.
Reproducibility assessment (the decisive part)
The repo ships code only. It does not ship any of the inputs the code needs, and the inputs are not independently obtainable:
- Model inputs are MAFs from an internal, non-shipped pipeline. Every entry
point (
gather_input_data.wes_pipeline_to_ml_input,1_format_mafs.py) reads pre-existing per-patient.maffiles from/fioSeq/projects/output/dna/...(input_pipeline='fioseq'). "fioSeq" is AbbVie's internal variant-calling + annotation pipeline; it is not in the repo and not described runnably. The input MAFs already carry annotation columns the downstream code consumes but never computes:max_cosmic_count(COSMIC v85),pop_max(gnomAD/ExAC/1000G),100mer_mappability,trinucleotide_context,mutation_change,fpfilter,ontology, plus the FILTER that encodes the matched-normal truth. Reproducing them needs a full annotation stack (incl. COSMIC v85, license-gated) that is not provided. - Training / validation / headline-test data is dbGaP controlled-access. The
105 training + 45 validation + 45 holdout-test samples (R1–R3, R5–R8) are TCGA
WES BAMs (subtypes BLCA/GBM/HNSC/LUAD/LUSC/OV/STAD; COAD/DLBC/TGCT;
BRCA/SARC/UCEC). TCGA controlled-access WES requires a dbGaP Data Access
Request the room does not hold.
data_splits.csvlists barcodes + internal paths, no usable data. - No trained model, no feature tables, no intermediates are shipped. The
melanoma eval (
5_eval_trained_tabnet.py) loads../tcga_pancancer/tabnet_trained_tcga.pklandmafs/engineered_plus_labels.csv— both 404 in the repo (verified by HEAD requests).
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.