CanCellCap: robust cancer cell capture across tissue types on single-cell RNA-seq data by multi-domain learning.
The main result did not reproduce in this reproduction attempt. Where our recomputation produced values that differ from the published ones, those discrepancies are listed below. This is a single automated attempt — not peer review and not a finding of error or misconduct — and differences can also arise from data access, undocumented parameters or the computing environment. The verdict can be contested via “report an error”.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Same input data as the authors
- ✓Reported values were directly comparable
- 🔴A deviation arose in the data or preprocessing
- 🔴A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The deviation was non-trivial in magnitude
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
Described well enough to run. CanCellCap = PyTorch binary cancer-vs-normal per-cell classifier; reproduced the GSE109761 (357-cell) per-dataset metrics from Suppl. Add. file 2 Table 1. TWO findings: (L1) the repo's SHIPPED per-cell predictions reproduce all 5 reported metrics EXACTLY (diff <=7e-16) using the test_testing.py averaging convention -> no metric fabrication. (L2) Re-running the RELEASED pretrained my_model.pt on the RELEASED GSE109761 input via the documented infer.py (exact requirements.txt versions, eval mode) does NOT reproduce: 94% per-cell agreement, accuracy 0.910 vs 0.938, AUROC 0.899 vs 0.983; released-model probs saturate to 0/1 while shipped are soft. No mode (eval/dropout/MC-dropout) recovers the reported numbers. Conclusion: reported numbers are self-consistent with shipped predictions but those predictions are NOT regenerable from the released artifact (artifact/version mismatch, not hardware noise) -> partial reproduction, verifiability gap flagged. NOT attempted: other 32 datasets, 5 baselines, training-from-scratch, secondary figures, native-CUDA GPU run (would not change verdict).
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 42assessed: 2026-06-14 ⛓ 53eac22a7078
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-14
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-15no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusCan a multi-domain learning framework that disentangles tissue-common cancer patterns, tissue-specific patterns, and sequencing-platform effects robustly and accurately identify cancer cells in scRNA-seq data across diverse tissues, cancer types, and sequencing platforms?
- ★ CanCellCap, a multi-domain learning framework integrating domain adversarial learning and Mixture of Experts, identifies cancer cells across all tissues, cancers, and sequencing platforms by extracting tissue-common and tissue-specific gene expression patterns. method
- ★ A masking-reconstruction strategy simulating dropout events enables CanCellCap to handle scRNA-seq data from different sequencing platforms by eliminating platform effects. method
- ★ CanCellCap achieves 0.977 average accuracy in cancer cell identification across 13 tissue types, 23 cancer types, and 7 sequencing platforms. finding
- ★ CanCellCap outperforms five state-of-the-art methods across 33 benchmark datasets in accuracy, F1, and precision with statistical significance. finding
- ★ CanCellCap generalizes to unseen cancer types, tissue types, sequencing platforms, and across species (mouse). finding
- CanCellCap is applicable to spatial transcriptomics data, accurately identifying cancer spots. finding
- CanCellCap demonstrates strong computational efficiency, completing inference on 100,000 cells in a few minutes. finding
- Interpretability analyses reveal critical biomarkers and pathways relevant to cancer diagnosis and therapy. resource
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| scRNA-seq cancer cell classification (model training/validation) | 328,230 human cells across 13 tissue types from TISCH | none | cancer vs normal cell classification accuracy | TISCH database; 7 sequencing platforms |
| scRNA-seq benchmarking against SOTA methods | 33 testing datasets, 834,788–855,713 human cells, 15 tissue types, 25 cancer types | none | accuracy, F1, recall, precision, AUROC | 6 sequencing platforms |
| scRNA-seq generalization to unseen cancer/tissue types | 113,393 cells (4 unseen cancer types: CC, HB, SCLC, CMM); 89,922 cells (unseen tissues: soft tissue, prostate) | none | accuracy, F1, recall, precision, AUROC | — |
| scRNA-seq cross-species generalization | 63,734 cells from mouse (three mouse single-cell datasets) | gene mapping adaptation | cancer cell identification metrics | — |
| scRNA-seq sequencing-platform generalization | datasets from 6 platforms (Smart-seq2 5600, 10X Genomics 146,387, Microwell-seq 10,502, C1 Fluidigm 3134, Drop-seq 21,011, GEXSCOPE 86,486 cells) | none | average accuracy across platforms | Smart-seq2, 10X Genomics, Microwell-seq, C1 Fluidigm, Drop-seq, GEXSCOPE |
| scRNA-seq cancer origin identification | TISCH2 Testing Dataset, 122,269 cells from 7 tissues | none | cancer cell origin identification | TISCH2 |
| simulated dropout robustness test | Testing Dataset with masked expression values | random masking at dropout rates 0–98% | identification robustness to dropout | — |
| spatial transcriptomics cancer spot identification | human prostate cancer samples | none | accurate identification of cancer spots | — |
- – Average accuracy on Validation Dataset across 13 tissue types 0.9777
- – 12 of 13 tissue types exceeded 0.95 accuracy; bone tissue lowest bone=0.8630
- – Highest average accuracy across six sequencing platforms (outperforming all SOTA) 0.9229
- – Accuracy on unseen sequencing platforms C1 Fluidigm / Drop-seq / GEXSCOPE 0.956 / 0.9121 / 0.9393
- ▲ Performance on unseen cancer types (vs Cancer-Finder accuracy 0.8705, AUROC 0.8676) accuracy 0.9297, AUROC 0.9722
- – Performance on unseen tissue types (soft tissue, prostate) accuracy 0.7877, F1 0.7847, precision 0.8333, AUROC 0.8172
- ▼ Cancer-Finder accuracy on 10X Genomics platform (notably unstable, lower than CanCellCap) 0.8539
- – In t-SNE of final embeddings v_cat, cancer and normal cells separate more clearly than in raw expression X
- mean 0.9777 (Average validation accuracy across 13 tissue types)
- mean 0.977 (Average accuracy across 13 tissues, 23 cancers, 7 platforms (abstract))
- mean 0.9229 (Average accuracy across six sequencing platforms)
- mean 0.9297 (Accuracy on unseen cancer types)
- mean 0.9722 (AUROC on unseen cancer types)
- mean 0.7877 (Accuracy on unseen tissue types)
- count 328,230 cells (Training/Validation Dataset from TISCH, 13 tissue types)
- count 834,788 cells (Testing Dataset, 33 datasets, 15 tissue types, 25 cancer types)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
CanCellCap is evaluated as a deep-learning classifier for cancer cell identification in scRNA-seq data, benchmarked against five existing methods across 33 external testing datasets (~855,000 cells spanning 15 tissue types). Performance was quantified using accuracy, F1, recall, precision, and AUROC computed via scikit-learn. Inter-method differences in ranked performance across datasets were assessed with the Bonferroni-Dunn test and visualised with Critical Difference (CD) diagrams at a 95% confidence level.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Bonferroni-Dunn post-hoc test with Critical Difference diagrams (95% confidence level) | Comparison of CanCellCap vs. five SOTA methods on accuracy, F1, recall, precision, and AUROC across 33 testing datasets (Fig. 2B) | 33 testing datasets | not stated |
-
The Bonferroni-Dunn test was used to compare all methods against the top-ranked classifier across 33 datasets↳ Could also: The Nemenyi post-hoc test (also common in multi-classifier benchmarking, e.g., Demšar 2006) could also be used — Nemenyi covers all pairwise comparisons simultaneously rather than comparisons only against a single control, which would additionally characterise how lower-ranked baselines differ from each other — a useful complement when the paper discusses relative ordering among several methods
-
Performance on testing datasets was reported as point estimates (e.g., accuracy = 0.9777) without accompanying uncertainty quantification↳ Could also: Bootstrap confidence intervals around each performance metric could also be computed per dataset or across datasets — Bootstrap CIs would quantify sampling variability in performance estimates attributable to the specific cell composition of each test dataset, complementing the cross-dataset distributional view already provided by the boxplots
-
Training and validation used a single 80/20 random split of the available labeled data↳ Could also: Repeated stratified k-fold cross-validation could also be used to estimate validation performance — Multiple folds yield a distribution of validation performance estimates rather than a single point, providing a sense of split-to-split variance; for datasets of this scale a single split is commonly adopted for computational efficiency, so both are standard practice
-
Statistical significance of rank differences was communicated via CD diagrams without reporting exact p-values↳ Could also: Pairwise Wilcoxon signed-rank tests with Benjamini-Hochberg FDR correction could also be reported — Pairwise Wilcoxon tests on matched dataset scores provide exact p-values and effect-size estimates (e.g., rank-biserial correlation), which some readers find easier to interpret than CD diagram connectivity; both approaches are standard for matched multi-dataset comparisons
-
Dispersion across the 33 testing datasets is shown visually as boxplots↳ Could also: Tabular numerical summaries (e.g., median with IQR, or mean with SD) could also accompany the figures — Numerical tables allow precise citation of spread and facilitate meta-analyses or direct numerical comparisons; boxplots give a richer distributional picture but exact values are harder to extract — combining both is common in benchmarking papers
-
Model generalization to unseen cancer types, tissue types, and species was evaluated by reporting point-estimate metrics on held-out partitions↳ Could also: A permutation test (label-shuffling baseline) or a calibration analysis could also be reported for these out-of-distribution scenarios — A permutation baseline establishes the empirical null performance level for each partition, making it easier to interpret how much of the reported accuracy exceeds chance; calibration analysis (e.g., reliability diagrams) would additionally assess whether predicted probabilities are well-calibrated on out-of-distribution data
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
No assessed neighbours yet — the network grows as more papers are assessed.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-40739511 (CanCellCap)
Paper: Bai et al. 2025, BMC Biol 23:... "CanCellCap: robust cancer cell
capture across tissue types on single-cell RNA-seq data by multi-domain learning."
DOI 10.1186/s12915-025-02337-1 · PMCID PMC12312500.
Repo: https://github.com/Ying-Lab/CanCellCap @ commit
b4a0ae10d64dd07379963b321371654aeb22b1de (own code of the authors, P16 own-repo).
Data: GEO GSE109761 (one of 33 external testing datasets); model + all
preprocessed test parquets + pretrained weights on the authors' Google Drive.
What CanCellCap is
A PyTorch multi-domain-learning model for binary per-cell classification: cancer cell (1) vs normal cell (0) in scRNA-seq. Architecture = feature masking-reconstruction + domain-adversarial (GRL) tissue confuser + Mixture-of- Experts (one expert per tissue) + MLP classifier. Trained on TISCH1 (328,230 cells, 13 tissues), evaluated on 33 external datasets (~855k cells).
In scope (pipeline-derived, reproducible)
| result | pipeline | our target |
|---|---|---|
| Per-dataset test metrics for GSE109761 (accuracy, recall, weighted-F1, weighted-precision, AUROC) reported in Suppl. Additional file 2, Table 1 | test_testing.py / infer.py: pretrained my_model.pt → per-cell pred → sklearn metrics |
reproduce the 5 reported GSE109761 values |
Two layers:
- L1 (self-consistency): recompute the 5 metrics from the repo's shipped
per-cell predictions (
results/all_results/GSE109761_..._CanCellCap_result.csv) and compare to Suppl. Table 1. Detects fabricated/inconsistent reported numbers. - L2 (true re-run): run the pretrained model on the GSE109761 input parquet (from the Drive data archive), regenerate predictions, compare to (a) the shipped predictions and (b) the reported metrics. Detects whether the shipped numbers are actually reproducible from model+data.
GSE109761 chosen as the concrete RU per the BRIEF's data: geo:GSE109761, and
because it is the smallest test set (357 cells) → fast, fully auditable, CPU-runnable.
Out of scope (not attempted, with reason)
- Full 33-dataset benchmark + 5 baseline methods (CopyKAT/SCEVAN/ikarus/PreCanCell/ Cancer-Finder) — 80/20: requires GBs of data, extra checkpoints (cancer_finder.pt), and external R/Python tools; one dataset suffices to test the reported-number claim.
- Training from scratch (Fig 1B validation 0.9777) — needs full TISCH1 train set + GPU-days; we reproduce inference with the shipped weights instead.
- Ablations, dropout-robustness, spatial, cross-species, interpretability (SHAP), cancer-origin — secondary figures, not the core capture metric.
- Wet-lab / manual steps — none; paper is fully computational.
Pipeline / environment
torch==1.13.1, scikit-learn==1.4.1.post1, pandas==2.2.2, numpy==1.26.4,
anndata==0.10.6, scanpy==1.9.8, pyarrow==14.0.2 (repo requirements.txt).
Preprocessing inside the dataloader: sc.pp.normalize_total(target_sum=1e4) +
sc.pp.log1p, reindex to the model's HVG list (from checkpoint), fillna 0.
Inference run on «our HPC» (CPU, std partition; 357 cells is trivial — no GPU needed).
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
L1 is exact — all five Suppl. Table 1 metrics (acc 0.938375 … AUROC 0.983405) recompute to ≤7e-16 from the repo's shipped per-cell predictions, so there is no metric fabrication at the reporting step. L2 fails — re-running the released my_model.pt through the documented infer.py (exact pinned versions, all inference modes) disagrees on 6% of cells and gives acc 0.910 / AUROC 0.899, and released probabilities saturate to 0/1 vs soft shipped probs, ruling out fp/hardware noise. The defect is on the authors' artifact side (a different checkpoint/version produced the published predictions), located in the model-output computation, not in data or preprocessing. Net: directionally the classifier still performs well, but the exact published numbers are not independently reproducible from the released artifacts — a real verifiability gap, hence overall yellow.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.