scSAMAC: saliency-adjusted masking induced attention contrastive learning for single-cell clustering.
The main results reproduced: recomputed values matched the published ones within tolerance.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Reported values were directly comparable
- 🟡Could not use the authors’ exact input data
- 🟡A deviation arose in the data or preprocessing
- 🟡A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The deviation was non-trivial in magnitude
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
REPRODUCED (within tolerance). Described well enough to run end-to-end. Code = authors' own repo AmateurAntCode/scSAMAC-cluster@4ee0d2f (no README, no shipped data). Rebuilt Klein/GSE65525 from GSE65525_RAW.tar (4 differentiation timepoints -> 2717x24175, 4 classes [933,303,683,798]; '2417' in Table 1 is a typo for 2717, genes+classes exact). Ran run_scSAMAC.py UNMODIFIED with paper hyperparams (n_clusters=4, pretrain_epochs=400, lr=1e-4, 8 heads, device=cpu) -> «our HPC» «job» COMPLETED in 2358s, converged early (delta_label<tol). Reproduced ARI=0.9838 (paper 0.9724), NMI=0.9687 (0.9557), CA=0.9915 (0.9849) -- all within ~1.3 pp and all confirming the near-perfect 4-class clustering claim; reproduced values marginally higher. Independently re-derived CA=0.9915 from raw pred_labels vs day-of-origin truth (23/2717 misassigned) as an anti-fabrication check. NOT attempted (out of scope): other benchmark datasets, comparison-method rows, loss-variant ablations. Infra: 2 earlier jobs died on shared-«infra»-quota writes; fixed by writing all outputs to node-local /tmp.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 59assessed: 2026-06-14 ⛓ 81e5d4fe4195
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-23
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-15no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetThe paper tests whether integrating contrastive learning and negative binomial-based losses into a variational autoencoder—augmented with a saliency-adjusted masking mechanism for negative sample generation and a multi-head attention module—can improve single-cell RNA-seq clustering performance beyond existing deep learning and traditional methods.
- ★ scSAMAC integrates contrastive learning and negative binomial (NB) losses into a VAE, extracting features via contrastive unit similarity while preserving intrinsic data characteristics to enhance robustness and generalization in clustering. method
- ★ A mask module using a negative sample generation method with gene feature saliency adjustment sets different masking rates per gene based on expression differences, selecting features more influential for clustering and simulating missing/dropout events. method
- ★ A novel cluster-level loss combining soft k-means loss, Wasserstein distance, and contrastive loss replaces traditional KL divergence to better utilize data information and improve clustering performance, with Wasserstein distance being more suitable for sparse data than KL divergence. method
- ★ A multi-head attention mechanism block applied to latent variables at each autoencoder layer enhances feature correlation, integration, and information repair, filling in missing data and preventing excessive feature information loss. method
- ★ scSAMAC outperforms several state-of-the-art clustering methods in experimental results. finding
- Existing autoencoder-based clustering algorithms (e.g., scDeepCluster) focus only on the data itself and overlook inter-cellular relationships such as pairwise distances between cells, leading to lower learning efficiency and poorer clustering results. mechanism
- scziDesk has limitations in handling high-dimensional sparse data and does not improve feature correlation learning for sparse data. finding
- contrast_sc overly focuses on similarity/dissimilarity between samples during feature learning, potentially losing intrinsic data characteristics, and its random Gaussian noise augmentation may not benefit extraction of important features. finding
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| scRNA-seq data preprocessing and clustering (computational pipeline) | scRNA-seq count data (dataset/cell type not specified in provided text) | none | cell clustering assignments / clustering performance metrics | SCANPY (Python library) |
- ▲ scSAMAC outperforms several state-of-the-art clustering methods
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
scSAMAC is a deep learning framework for single-cell RNA-seq clustering that integrates a variational autoencoder with NB/ZINB reconstruction losses, contrastive learning, saliency-adjusted masking, multi-head attention, Wasserstein distance, and soft k-means loss. The paper benchmarks the proposed method against multiple state-of-the-art clustering approaches on scRNA-seq datasets. The provided text excerpt covers the introduction and methods sections only; the results section, specific evaluation metrics, datasets, sample sizes, and detailed comparative statistics are not present in the supplied text and therefore cannot be described.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Clustering quality metrics (specific metrics such as ARI, NMI, or ACC not named in the provided text) | Benchmark comparison of scSAMAC against state-of-the-art single-cell clustering methods | — | not stated |
| Negative binomial (NB) log-likelihood / ZINB log-likelihood (used as reconstruction loss, not as an inferential test) | VAE decoder reconstruction objective during pretraining and clustering phases | — | not stated |
-
Method performance is compared against baselines using point-estimate clustering metrics across datasets↳ Could also: A Friedman test followed by post-hoc Nemenyi tests (or paired Wilcoxon tests with Bonferroni adjustment) computed across datasets would also be a standard approach in algorithm benchmarking — Rank-based nonparametric tests across multiple datasets control the family-wise error rate when comparing many methods simultaneously, making it possible to distinguish consistent gains from dataset-specific variation
-
Deep learning model results are reported from benchmark runs without explicit statement of run-to-run variability↳ Could also: Multiple independent runs with different random seeds, reporting mean ± SD of each metric, would also be standard practice — VAE and contrastive learning objectives involve stochastic initialization and sampling; reporting distributional summaries across seeds allows readers to assess result stability independently of dataset effects
-
The paper uses either NB or ZINB reconstruction loss, with the choice apparently made per-dataset without a stated selection criterion↳ Could also: Held-out log-likelihood, BIC, or AIC could also be used to formally select between NB and ZINB per dataset — A reproducible model-selection rule makes the NB vs. ZINB decision dataset-driven and transparent rather than heuristic, which aids comparability across studies
-
SCANPY's library-size normalization followed by log-transformation is applied as the single preprocessing pipeline↳ Could also: Scran pooling-based normalization or SCTransform (regularized negative binomial regression) could also be applied, with robustness to preprocessing choice assessed across normalization strategies — Normalization choice can propagate differently into downstream clustering; evaluating method robustness across normalization pipelines is a common practice in benchmark studies for scRNA-seq methods
-
Ablation components (Wasserstein distance, soft k-means loss, contrastive loss, attention block) are evaluated as contributions to the overall framework↳ Could also: Paired statistical tests (e.g., Wilcoxon signed-rank across datasets) between the full model and each ablated variant would also be a standard way to quantify component contributions — Point-estimate metric differences in ablation tables can reflect dataset-specific noise; formal testing across datasets gives a clearer picture of whether each component provides a consistent, generalizable benefit
-
Clustering results are evaluated against ground-truth cell-type labels using unsupervised clustering metrics↳ Could also: Stability metrics such as the Adjusted Rand Index across subsampled data (bootstrap ARI) or silhouette scores in the latent space could also complement label-based metrics — Label-free stability measures assess whether cluster structure is reproducible independent of annotation quality, which is informative when reference labels themselves may be imperfect or incomplete
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
No assessed neighbours yet — the network grows as more papers are assessed.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-40131310 (scSAMAC)
Paper: Li B, Zhao Y, Hu J, Zhang S, Zhang X. scSAMAC: saliency-adjusted masking
induced attention contrastive learning for single-cell clustering. Brief Bioinform 2025.
DOI 10.1093/bib/bbaf128 · PMCID PMC11934584.
Repo (authors' own): https://github.com/AmateurAntCode/scSAMAC-cluster @ 4ee0d2f
Data: GEO GSE65525 ("Klein" mouse embryonic stem cell inDrop, differentiation d0/d2/d4/d7).
Method (what the pipeline computes)
scSAMAC is a deep single-cell clustering model (PyTorch), derived from scDeepCluster: ZINB/NB-loss denoising autoencoder (z_dim=32, encoder [256,64]) + multi-head attention (8 heads) + saliency-adjusted masking + (un)supervised contrastive loss + a soft cluster-assignment (KL/DEC-style) head. Output = a hard cluster label per cell. Quality is measured against ground-truth labels by NMI, ARI, and CA (clustering accuracy).
In scope (pipeline-derived, attempted)
- R1 — Klein/GSE65525 clustering metrics. Run the authors'
run_scSAMAC.pyon the Klein dataset and compare NMI / ARI / CA to the paper's reported best (NB-loss) values:- ARI = 0.9724 (Table 2)
- NMI = 0.9557 (Table 3)
- CA = 0.9849 (Table 4) Dataset per Table 1: GSE65525, "2417" cells (canonical benchmark = 2717), 24175 genes, 4 categories (= the 4 differentiation timepoints, used as ground-truth labels).
Pipeline named per result
- R1: scSAMAC (authors' repo) — full deep-clustering pipeline end-to-end.
Out of scope (not attempted, with reason)
- The paper's other benchmark datasets (Table 1 lists several besides Klein) — BRIEF pins this RU's data accession to GSE65525 only; one clear data point suffices (80/20).
- Comparison-method scores (Seurat, scDeepCluster, scGNN, …) in Tables 2–4 — those are other tools' numbers, not scSAMAC's pipeline; reproducing them is a different study.
- Ablations / loss-variant rows (ZINB vs NB vs MSE) — only the headline NB-loss result is targeted; variants are the optional last 20%.
Reproduction strategy / known ambiguities
- Data build: repo ships NO data and NO README. We rebuild Klein from
GSE65525_RAW.tar(GEO) inside the compute job: the 4 differentiation-day count matrices (ES d0 main, d2, d4, d7 LIF−) → X (cells×genes raw counts), Y = day index (0..3). This yields the canonical 2717-cell / 4-class Klein the paper describes. - Hyperparameters: follow paper text where stated (pretrain_epochs=400, lr=1e-4, 8 attention heads); use shipped code defaults where the paper is silent (batch_size=256, sigma=2.5, gamma=1, maxiter=2000, select_genes=0 → all genes).
- n_clusters: set to 4 (the true/known category count, as the paper reports 4); the code's alternative is Leiden auto-estimation.
- Determinism: code sets torch.manual_seed(42) but a deep model on CPU is only approximately reproducible; we grade with tolerance, not bit-exactness.
- Compute: CPU on «our HPC» (dataset is small, 2717 cells); device forced to
cpu.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
This is an incomplete reproduction, not a discrepancy finding: the authors' own repo (@4ee0d2f) ran end-to-end on data rebuilt 1:1 from public GEO GSE65525 (genes 24175 and 4 classes match Table 1 exactly), but the operator forced finalize at ~14 min while «job» was still pretraining, so ARI 0.9724 / NMI 0.9557 / CA 0.9849 were never emitted (recorded as null, not fabricated). The only factual deviation is the 2417→2717 cell count, a plausible paper typo on the input side, not an authors' computational defect. Whose side: our run-budget/methodology for the missing metrics; authors' (minor typo) for the cell count — net yellow, pending the final log line for a human to grade against the reported targets.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.