scATD: a high-throughput and interpretable framework for single-cell cancer drug resistance prediction and biomarker identification.
The main results reproduced, with only marginal, non-material deviations.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Same input data as the authors
- ✓Reported values were directly comparable
- ✓No relevant deviation in data/preprocessing
- ✓No authors-side cause for any deviation
- ✓Reported values are derivable from the shared data
- ✓Any deviation was negligible
- ✓The central claim held under reproduction
- ✓Overall, the reproduction was clean
- Every checked point held up.
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
Described well enough to reproduce without author contact; essentially 1:1 on the one pinnable scalar. scATD's OWN repo is doriszmr/scATD (the BRIEF's code_url, OSU-BMBL/scDEAL, is a baseline competitor in the paper, not the authors' code). Reproduced scATD-sf-dist (Dist-VAE) single-cell drug-response prediction on GSE140440 (docetaxel, prostate cancer, 324 cells) by PURE INFERENCE: the repo ships the preprocessed h5ad + labels, and figshare ships the authors' trained DAL checkpoint (checkpoint_fold5_final_epoch_150.pth, md5 verified). The reported MCC 0.496 reproduced to 0.4949 (within-tol, Δ=0.0011) and AUROC came out 0.821, consistent with the paper's 'highest AUROC' claim for this dataset. No fabrication concern: the reported value is fully derivable from the shipped weights+data. The heavy ~80% (scFoundation/Geneformer 100M-param LLM embedding extraction, GPU, multi-GB weights) was deliberately skipped because Dist-VAE is the distilled model that runs directly on gene expression and the authors ship all precomputed artifacts. NOT attempted: scATD-sf/gf per-dataset metrics (need per-drug DAL training), cross-dataset Table 3, silhouette Table 2, Integrated-Gradients biomarkers Figs 7-8, survival KM Fig 9, efficiency timings Fig 6d. The exact AUROC scalar is only a Fig-4/5 heatmap cell / paywalled Table S3, so AUROC is graded qualitatively rather than exact (the last 20%, not chased).
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 62assessed: 2026-06-14 ⛓ 47e2bbae59fe
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-14
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-15no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetThe paper tests whether a transfer-learning framework that combines large language model embeddings (scFoundation, Geneformer), bidirectional style transfer (Bi-AdaIN), and knowledge distillation can achieve high-throughput, accurate, and interpretable single-cell drug resistance prediction without requiring model retraining for each new patient dataset.
- ★ scATD enables high-throughput single-cell drug sensitivity prediction for new patients without model parameter retraining via bidirectional Bi-AdaIN style transfer method
- ★ scATD-sf-dist uses knowledge distillation from large LLMs into a Res-VAE backbone to improve efficiency and reduce resource requirements while preserving predictive accuracy method
- ★ scATD achieves state-of-the-art predictive accuracy, generalization, and efficiency across diverse single-cell drug response datasets compared to existing models finding
- ★ scFoundation's representational space closely aligns with both bulk and single-cell RNA-seq data despite not being pretrained on bulk data finding
- ★ Rigorous selection of reference background samples for feature attribution algorithms yields more meaningful gene-drug resistance relationship insights than default zero baselines finding
- ★ Local interpretability analysis reveals patient-specific and individual gene contribution patterns (linear or nonlinear) to drug resistance, beyond global gene importance rankings finding
- scATD integrates a Residual Variational Autoencoder (Res-VAE) architecture unifying bulk and single-cell data feature reconstruction method
- Identified biomarkers are validated through literature support, cross-dataset robustness testing, and perturbation of feature attribution algorithms resource
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| single-cell RNA-seq (LLM embedding extraction + VAE pretraining) | PanglaoDB (>1 million single-cell expression profiles) | none | reconstructed feature embeddings / latent representation | — |
| bulk RNA-seq drug response profiling | 1280 cancer cell lines (GDSC and CCLE databases, multiple cancer types) | drug/compound treatment (83 drugs/compounds) | drug sensitivity vs resistance label (source-domain training) | — |
| single-cell RNA-seq drug response classification | 14 GEO single-cell datasets across cell lines and tumor tissues (e.g. oral squamous cell carcinoma, NSCLC, prostate, pancreatic, breast, melanoma, CLL) | drug treatment (e.g. Cisplatin, Crizotinib, Docetaxel, Erlotinib, Gefitinib, Gemcitabine, Ibrutinib, Paclitaxel, Palbociclib, PLX4720) | sensitive vs resistant cell classification | — |
| single-cell RNA-seq interpretability perturbation experiment | GSE137829, 6 prostate cancer patients, tumor tissue | feature attribution algorithm perturbation | gene importance / feature attribution robustness | — |
| bulk RNA-seq with clinical prognosis data | TCGA: LUAD, PAAD, BRCA, SKCM, PRAD | none | biomarker identification and survival/prognosis risk stratification | — |
| bulk RNA-seq with RECIST-labeled clinical drug response | BRCA_RECIST and PAAD_RECIST datasets | drug treatment | clinically guided drug response prediction generalization | — |
- ▲ scATD demonstrates superior accuracy, generalization, and efficiency in benchmarking across diverse datasets compared to existing models
- – scFoundation embeddings, though not pretrained on bulk RNA-seq, align closely with both bulk and single-cell representational spaces
- – Using default zero baselines for feature attribution produces significant gene interpretation issues
- count 1280 cancer cell lines, 83 drugs/compounds (bulk source-domain training data from GDSC and CCLE)
- count over one million single-cell expression profiles (PanglaoDB pretraining dataset)
- count 14 single-cell datasets (GEO single-cell drug response datasets used)
- count 100 million parameters (scFoundation model size (xTrimoGene architecture))
- count over 5 million single-cell expression profiles (scFoundation pretraining dataset size)
- count 3072 (scFoundation feature embedding dimension)
- count 19264 genes (scFoundation) vs 25426 genes (Geneformer) (fixed vocabulary sizes of the two LLMs)
- count 6 prostate cancer patients (GSE137829 dataset used for interpretability perturbation experiments)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
scATD is a machine-learning benchmarking paper presenting transfer learning frameworks for single-cell drug resistance prediction. Primary evaluation uses multi-dataset benchmarking of model variants (scATD-sf, scATD-gf, scATD-sf-dist) against three comparator models across 14+ single-cell datasets spanning multiple cancer types and drugs. The Res-VAE component is assessed with 10-fold cross-validation during pretraining, and biomarker validity is examined via Kaplan-Meier survival analysis on TCGA data across five cancer types.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| 10-fold cross-validation | Res-VAE pretraining evaluation on PanglaoDB dataset | PanglaoDB: over one million single-cell expression profiles (exact training n not further specified in available text) | not stated |
| Kaplan-Meier survival analysis | Biomarker validation module; high- vs. low-risk group comparison using TCGA RNA-seq and clinical prognosis data across LUAD, PAAD, BRCA, SKCM, and PRAD | — | not stated |
-
Multiple models are compared across 14+ datasets with no mention of statistical testing or correction for multiple comparisons↳ Could also: Paired statistical tests (e.g., Wilcoxon signed-rank across datasets) combined with a multiplicity correction such as Benjamini-Hochberg FDR could also be applied — Formal testing with correction would quantify whether observed performance differences exceed what is expected by chance when comparing many models across many datasets, providing inferential grounding beyond descriptive metric tables
-
A single fixed random seed (42) was used across all experiments for reproducibility↳ Could also: Repeating experiments across multiple random seeds and reporting mean ± SD of performance metrics could also be applied — Results from a single initialization may not reflect variance attributable to random weight initialization or data splitting; averaging across seeds provides a more stable estimate of expected performance
-
Integrated Gradients was the sole feature attribution method used for biomarker identification↳ Could also: SHAP (SHapley Additive exPlanations) or LIME could also be applied alongside Integrated Gradients — Different attribution methods can emphasize different aspects of feature importance; comparing concordance across methods can strengthen confidence in identified biomarkers — a concern the authors themselves raise regarding over-reliance on single interpretability techniques
-
Survival analysis used Kaplan-Meier curves to compare dichotomized high- vs. low-risk groups↳ Could also: Cox proportional hazards regression could also be used to model survival as a continuous function of a risk score — Cox regression quantifies hazard ratios with confidence intervals, avoids arbitrary dichotomization of a continuous risk score, and can accommodate clinical covariates — complementing the visual KM approach
-
Each of the 14 single-cell datasets was evaluated separately for benchmarking↳ Could also: A meta-analytic aggregation (e.g., random-effects meta-analysis of AUROC across datasets) could also summarize performance — Meta-analysis would yield a pooled performance estimate with uncertainty quantification and could formally test for heterogeneity across datasets, datasets which vary substantially in cancer type, cell source, and size
-
10-fold cross-validation was used to evaluate the Res-VAE pretraining loss↳ Could also: Nested cross-validation (outer loop for performance estimation, inner loop for hyperparameter selection) could also be used when hyperparameters are tuned on the same data — Nested CV separates model selection from performance estimation, which can reduce optimistic bias in reported metrics when hyperparameters are optimized on cross-validation folds
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
Assessed papers, coloured by verdict. Click a node to open it.
- No assessed neighbours yet — the network grows as more papers are assessed.
Data lineage
The datasets this paper uses (text-mined from the full text via Europe PMC), and which other assessed papers stand on the same data. A shared dataset is a factual link — not a judgement.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-40501071 (scATD)
Paper: Zhou M, Luo Z, Yin YH, Liu Q, Wang G, Zhao Y. scATD: a high-throughput and interpretable framework for single-cell cancer drug resistance prediction and biomarker identification. Brief Bioinform 2025; 26(3):bbaf268. PMID 40501071.
Authors' own code: https://github.com/doriszmr/scATD (the BRIEF's code_url
pointed at OSU-BMBL/scDEAL, which is a baseline compared against in the paper,
not the authors' framework — corrected to the real scATD repo).
Data + models: figshare 27908847 (raw + preprocessed h5ad, precomputed
scFoundation/Geneformer embeddings, pretrained VAE checkpoints, trained DAL
checkpoints for the drug–dataset combinations).
What scATD is
A transfer-learning framework for single-cell drug-sensitivity prediction. Four stages: (1) feature embedding from a pretrained transcriptomic LLM (scFoundation or Geneformer); (2) Res-VAE reconstruction + knowledge distillation; (3) domain adaptation bulk↔single-cell (Bi-AdaIN style transfer, or MMD loss); (4) Integrated-Gradients biomarker attribution + survival analysis. Three variants:
- scATD-sf (VAE_sf, scFoundation backbone, parameter-free Bi-AdaIN)
- scATD-gf (VAE_gf, Geneformer backbone)
- scATD-sf-dist (Dist-VAE, distilled — runs directly on gene expression, no LLM at inference time)
In scope (pipeline-derived, computational) vs out of scope
| Reported result | Pipeline | In scope? |
|---|---|---|
| Single-cell drug-response prediction metrics (AUROC/AUPRC/F1/MCC) per GSE dataset (Figs 4–5, Tables) | scATD inference (Dist-VAE / VAE-sf) on provided embeddings/checkpoints | YES — primary target |
| Cross-dataset generalization (Table 3) | VAE-sf transfer + inference | partial (needs DAL training) — not primary |
| Silhouette scores of embeddings (Table 2) | embedding + clustering | possible, secondary |
| Computational-efficiency timings (Fig 6d) | wall-clock, hardware-dependent | out (not 1:1 reproducible across HW) |
| Integrated-Gradients biomarkers, SHAP plots (Figs 7–8) | attribution | out of 80/20 (heavy, qualitative) |
| Survival KM curves / p-values (Fig 9) | external TCGA survival | out (separate analysis) |
| scFoundation / Geneformer embedding extraction | 100M-param LLM, GPU, large weights | skipped — authors ship precomputed embeddings (the heavy 80% we deliberately do not redo) |
Primary reproduction target (80/20, lowest-risk, clearly specified)
scATD-sf-dist (Dist-VAE) single-cell prediction on GSE140440 (docetaxel, prostate cancer; DU145 & PC3; 162 sensitive + 162 resistant = 324 cells).
This is the repo's shipped worked example: Dist_VAE/inference/ ships
preprocessed_GSE140440.h5ad + label file, and figshare ships the authors'
trained DAL checkpoint checkpoint_fold5_final_epoch_150.pth. So this is pure
inference with the authors' own weights — no LLM, no retraining, deterministic
(seed 42). The code computes AUROC, AUPRC, F1, MCC, accuracy, precision, recall
with a fixed probability threshold (--PP_threhold 0.05, the shipped default).
Reported value to compare against: paper states for GSE140440 that scATD-sf-dist "achieved the highest AUROC and a slightly lower MCC (0.496) than scDEAL (0.5) but showed lower MCC variability." (Results / Table S3.) The exact AUROC is shown only in the Fig-4/5 heatmaps (not as a legible number in text), so MCC=0.496 is the pinnable scalar; AUROC is compared qualitatively ("highest") plus we report our reproduced value.
Known reproducibility gotchas (handled)
- Repo default branch is
master(notmain). - Inference imports
Dist_VAE.distillation_VAE_pretraining_model.{config, Dist_VAE_model_inference}, but those files ship one directory deeper — copied up at run time. - Inference only needs the DAL checkpoint; the 396 MB Panglao Dist-VAE pretrain checkpoint is NOT required at inference (config/model classes are code only).
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
On the single pinnable scalar — MCC = 0.496 for scATD-sf-dist on GSE140440 — reproduction lands at 0.4949 (Δ=0.0011, rounding-level), and AUROC = 0.821 is consistent with the paper's 'highest AUROC' claim. The value is fully derivable from the authors' shipped checkpoint + preprocessed h5ad, so there is no fabrication concern and any deviation is on the technical/version side, not the authors'. Caveats are scope-related, not quality-related: this was a pure-inference reproduction (training/embedding pipeline not re-run) covering only one of many claims, and AUROC could only be checked qualitatively because the paper prints it solely as a heatmap cell.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.