Cell-type annotation with accurate unseen cell-type identification using multiple references.
The main results reproduced, with only marginal, non-material deviations.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Same input data as the authors
- ✓Reported values are derivable from the shared data
- ✓Any deviation was negligible
- ✓The central claim held under reproduction
- 🟡Reported values were only indirectly comparable
- 🟡A deviation arose in the data or preprocessing
- 🟡A deviation was attributed to the published material
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
Described well enough to reproduce, and reproduced 1:1 on the authors' own code+data. mtANN is the authors' own tool (Zhangxf-ccnu/mtANN @0effbf8); ran its multi-reference NN-ensemble annotation + unseen-cell-type identification on the shipped Pancreas collection from Zenodo 7922657 (leave-one-dataset-out, query=Baron, 3 references, 24 base models, seed=5, CPU, «our HPC» SLURM «job»). Reproduced 0.88 annotation accuracy (in the paper's high pancreas range) and unassigned-fraction 5.90% vs true-unseen 6.63% (closely tracking, the method's core claim). Status=partial because the paper reports results as figure distributions with no printed scalar for this exact configuration, so exact/within-tol against a published number is impossible; grades are vs the paper's qualitative claims. DELIBERATE 80/20 deviation: gene selection used the authors' SHIPPED precomputed gene lists (data/panc/panc_genes/) rather than re-running the fragile live R(limma/Seurat v4)+giniclust3 step, so the heavy/finicky rpy2+R+Seurat+giniclust3 env was skipped; the deep-learning ensemble (the actual contribution) was fully recomputed. NOT attempted: full leave-one-cell-type-out benchmark sweeps vs all baselines (scmap/Seurat/ItClust/CHETAH), the COVID-19/GSE158055 application (Fig 6), and live re-execution of gene selection.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 68assessed: 2026-06-14 ⛓ f447b672d240
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-14
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-15no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetReference-based scRNA-seq cell-type annotation methods are biased toward known cell types and fail when query data contain cell types absent from the reference; the paper tests whether integrating multiple reference datasets with ensemble deep learning and a novel multi-aspect uncertainty metric can accurately annotate cell types while identifying these previously 'unseen' cell types.
- ★ mtANN integrates multiple reference datasets and eight gene selection methods via ensemble learning (multiple deep classification models + majority voting) to improve cell-type annotation accuracy method
- ★ mtANN defines a new unseen cell-type identification metric combining intra-model, inter-model, and inter-prediction uncertainty measures mechanism
- ★ mtANN uses a data-driven approach based on fitting a Gaussian mixture model to the uncertainty metric to adaptively determine the threshold for identifying unseen cell types method
- ★ mtANN outperforms state-of-the-art methods (scmap-clust, scmap-cell, Seurat v3, ItClust, scGCN entropy/enrichment, scANVI) in unseen cell-type identification as measured by AUPRC finding
- ★ The ensemble (mtANN, integrating all reference datasets and gene selection methods) consistently outperforms any single base classification model trained on one reference subset finding
- No single gene selection method (DE, DV, DD, DP, BI, GC, Disp, Vst) consistently outperforms the others across reference datasets finding
- ★ mtANN shows predictive power for annotation and unseen cell-type identification on a collection of COVID-19 datasets with patients of different symptom severities finding
- Competing methods like ItClust can misalign cell labels and clusters and overfit when the reference is missing cell types present in the query, causing inconsistent performance across data collections mechanism
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| scRNA-seq cell-type annotation benchmarking | Human PBMC (7 datasets: Celseq, Drops, inDrop, Seq-Well, Smart-seq2, 10X v2, 10X v3) | none (cross-technology reference vs query comparison) | annotation accuracy across base classification models and gene selection methods | — |
| scRNA-seq cell-type annotation benchmarking | Human pancreas (4 datasets: Baron, Muraro, Segerstolpe, Xin) | none (cross-technology reference vs query comparison) | annotation accuracy across base classification models and gene selection methods | — |
| scRNA-seq leave-one-cell-type-out unseen cell-type identification | PBMC collection (7 datasets, 50 tests) and Pancreas collection (4 datasets, 25 tests) | simulated unseen cell type via leave-one-cell-type-out | AUPRC score for distinguishing unseen vs shared cell types | — |
| scRNA-seq cell-type annotation and unseen cell-type identification | COVID-19 patient samples (GSE158055) | disease state (COVID-19 patients with different symptom severities) | annotation/unseen cell-type identification performance across 249 tests | — |
- ▲ mtANN's integrated performance (red line) is consistently higher than every individual base classification model/gene-selection-method combination on Celseq (PBMC) and Baron (Pancreas) query datasets
- ▲ mtANN achieves higher AUPRC than scmap-clust, scmap-cell, Seurat v3, ItClust, scGCN (entropy), scGCN (enrichment), and scANVI when 10X v3 is the query dataset in the PBMC collection
- – Performance of the eight gene selection methods varies across reference datasets, with no method universally best
- – scmap-clust, ItClust, and scGCN (enrichment) rank first on some datasets but show large performance drops on others across PBMC vs Pancreas collections
- – 75 total benchmark tests conducted (50 PBMC + 25 Pancreas) for unseen cell-type identification across technologies
- – 249 tests prepared using COVID-19 dataset collection to assess mtANN performance
- count 75 benchmark tests (total unseen cell-type identification tests across PBMC and Pancreas collections)
- count 50 tests (leave-one-cell-type-out tests within the PBMC collection)
- count 25 tests (leave-one-cell-type-out tests within the Pancreas collection)
- count 249 tests (tests prepared using the COVID-19 dataset collection)
- other AUPRC score (primary metric used to compare methods' ability to distinguish unseen from shared cell types)
- count eight gene selection methods (DE, DV, DD, DP, BI, GC, Disp, Vst) (gene selection methods applied to each reference dataset in Module I)
- count seven sequencing technologies (PBMC collection datasets: Celseq, Drops, inDrop, Seq-Well, Smart-seq2, 10X v2, 10X v3)
- count four datasets (Pancreas collection: Baron, Muraro, Segerstolpe, Xin)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a computational methods benchmarking paper introducing mtANN, a deep-learning/ensemble-learning approach for scRNA-seq cell-type annotation with unseen-cell-type identification. Performance is evaluated by AUPRC scores across 75 leave-one-cell-type-out benchmark tests (PBMC and Pancreas collections) and 249 COVID-19 tests, with results visualized as boxplots and method rankings (number of rank-first occurrences). Threshold selection for unseen-cell-type identification uses a data-driven Gaussian Mixture Model fit to the uncertainty metric distribution. No formal inferential hypothesis tests between methods are described in the text.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| AUPRC (Area Under the Precision-Recall Curve) score comparison | Unseen cell-type identification benchmarks across PBMC (50 tests) and Pancreas (25 tests) collections, and COVID-19 collection (249 tests) | 75 benchmark tests (PBMC + Pancreas); 249 COVID-19 tests | na |
| Gaussian Mixture Model (GMM) fitting for data-driven threshold determination | Module V: selecting the cutoff on the uncertainty metric to flag unseen cell types | — | stated |
| Majority voting (ensemble aggregation) | Module III: combining predictions from all base classification models to obtain metaphase annotation | — | na |
| Entropy-based uncertainty metric (intra-model, inter-model, inter-prediction) | Module IV: quantifying per-cell likelihood of belonging to an unseen cell type | — | na |
| Rank-first counting across benchmark tests | Summary comparison of method stability across PBMC and Pancreas collections (Fig 3B, S4 Fig) | 75 benchmark tests | na |
-
Method comparisons across 75+ benchmark tests are summarized by rank-first counting and visual boxplot inspection, without a formal statistical test for differences between methods↳ Could also: A Friedman test (non-parametric repeated-measures comparison) followed by a post-hoc Nemenyi or Wilcoxon signed-rank test with Holm or Benjamini-Hochberg correction could also be applied across the paired benchmark tests — When the same set of benchmark tasks is used for all methods (a paired/blocked design), a Friedman test explicitly models the blocking structure and provides a p-value for overall differences; post-hoc pairwise tests with correction then identify which pairs differ, complementing the descriptive rank-first summary
-
Unseen-cell-type identification performance is assessed solely with AUPRC↳ Could also: AUROC (Area Under the ROC Curve) could also be reported alongside AUPRC — AUPRC is particularly informative under class imbalance (which applies here, since unseen cells are a minority), while AUROC provides a complementary view of overall rank discrimination; reporting both gives readers a fuller picture of operating characteristics across thresholds
-
The threshold for unseen-cell-type identification is determined by fitting a Gaussian Mixture Model to the uncertainty metric and selecting cells in the high-uncertainty component↳ Could also: A fixed percentile-based cutoff, an extreme-value distribution fit, or a receiver-operating-characteristic-derived threshold on held-out labeled data could also be used — The GMM approach is data-driven and unsupervised (no ground truth needed at inference), which matches the real-world use case; alternative approaches such as percentile thresholds are simpler to interpret and less sensitive to GMM convergence assumptions, and ROC-derived thresholds can be tuned to a desired false-positive rate when some labeled unseen examples are available
-
Results across all benchmark tests are pooled into a single boxplot per method per query dataset, without stratifying by the identity of the left-out cell type or its frequency in the query↳ Could also: Stratified reporting by cell-type frequency or cell-type identity (e.g., rare vs. common unseen types) could also be presented — Identifying unseen cell types that are rare is harder than identifying those that are abundant; stratified analysis would reveal whether performance differences between methods are concentrated in difficult (rare unseen) or easy (abundant unseen) scenarios, providing more actionable guidance for practitioners
-
Accuracy of the base classification models is compared against mtANN (the ensemble) using a single-number metric (accuracy or AUPRC) per reference-subset pair, displayed as points vs. a red line↳ Could also: A paired Wilcoxon signed-rank test over the distribution of base-model scores vs. the ensemble score, with effect size (rank-biserial correlation), could also quantify the benefit of ensembling — The visual comparison (red line above all points) is intuitive, and a formal test would additionally quantify the probability that the observed gap arose by chance and the magnitude of the improvement, which is useful when the gap is small or inconsistent across datasets
-
Dispersion of AUPRC scores across benchmark tests is shown with boxplots (median and IQR)↳ Could also: Bootstrap confidence intervals around the mean AUPRC, or violin plots overlaid on the boxplots, could also convey the full score distribution — With a moderate number of benchmark tests (e.g., 50 in the PBMC collection), bootstrap CIs would show uncertainty around the central tendency, and violin plots would reveal multimodality that IQR-based boxes may obscure—particularly relevant if performance varies sharply by cell-type identity or reference composition
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
No assessed neighbours yet — the network grows as more papers are assessed.
Data lineage
The datasets this paper uses (text-mined from the full text via Europe PMC), and which other assessed papers stand on the same data. A shared dataset is a factual link — not a judgement.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
Reproduction ran the authors' own mtANN (commit 0effbf8) on their shipped Zenodo Pancreas data and fully confirmed both claims where tested: 0.8802 annotation accuracy sits inside the paper's 0.8–0.95 boxplot range, and predicted-unassigned 5.90% closely tracks true-unseen 6.63%. The only deviations are on our methodology/availability side, not the authors' — no printed scalar exists to match 1:1 (figures-as-distributions), gene selection used the authors' shipped precomputed lists, and CPU was used instead of GPU. Severity is negligible and no value is suspicious; the main limits are the untested comparative 'best-in-class' claim and the unreproduced full benchmark sweeps/COVID-19 application (out of scope), which keep the overall judgement at solid-yellow rather than green.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.