Hierarchical cell-type identifier accurately distinguishes immune-cell subtypes enabling precise profiling of tissue microenvironment with single-cell RNA-seque
The main results reproduced: recomputed values matched the published ones within tolerance.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Same input data as the authors
- ✓Reported values were directly comparable
- ✓No authors-side cause for any deviation
- ✓Reported values are derivable from the shared data
- ✓Any deviation was negligible
- ✓The central claim held under reproduction
- 🟡A deviation arose in the data or preprocessing
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
DESCRIBED WELL ENOUGH and reproduced 1:1 for the targeted dataset. HiCAT (the authors' own pip tool MarkerCount.hicat) ships a ready example dataset (Melanoma_5K_rev.h5ad = GSE72056, 4513 cells), the markers DB, and an example notebook with exact parameters. Running the shipped get_perf_ext metric on the shipped stored predictions reproduces the paper's headline Melanoma numbers: major-type match rate 98.92% vs reported 98.9% (EXACT), minor-type 93.64% vs reported 94.9% (within ~1.3 pts). The metric reading (match rate = Correct + Correctly-Unassigned) is independently validated because the paper's reported second-best value Manual-scType 71.3% reproduces exactly (71.35%). HiCAT is the most accurate of all 7 marker-based tools at both levels, reproducing the central headline. A fresh HiCAT re-run (authors' pinned dependency versions) reproduces the stored HiCAT predictions cell-by-cell at 96.74%/93.11%, confirming the pipeline itself is reproducible. NOTE: with current numpy-2/sklearn-1.7 the fresh run crashes in a GaussianMixture Cholesky step (a version regression, not a data issue); pinning numpy 1.26.4 / scipy 1.12.0 / scikit-learn 1.3.2 fixes it. NOT ATTEMPTED (the hard ~20%, per 80/20): the other 7 of 8 datasets (PBMC/CBMC/Lung x2/BALF/Colon/BRCA-100K incl. GSE176078) and the cross-dataset average match rates (85.6%/72.3%/64.5-59.2-56.7%) — large downloads + per-dataset manual-label curation + re-running 6 competitor tools each. The melanoma example exercises the identical HiCAT pipeline and is representative. Grades are PROVISIONAL for human audit.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 94assessed: 2026-06-14 ⛓ b3ae7c368f96
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-14
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-15no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetWhether structuring cell-type markers hierarchically (major-type, minor-type, subset) in a marker-based scRNA-seq annotation tool can overcome the limited performance of existing marker-based cell-type identifiers, especially for identifying immune-cell subsets.
- ★ HiCAT is a hierarchical, marker-based cell-type identifier that uses gene set analysis (GSA) scoring with markers structured in a three-level taxonomy tree (major-type, minor-type, subset) method
- ★ HiCAT outperforms existing marker-based identifiers in major-type identification across most tested datasets finding
- ★ HiCAT outperforms existing marker-based identifiers in minor-type identification in the majority of datasets finding
- ★ HiCAT produces the clearest immune-cell landscape for subset identification based on qualitative marker expression profile evaluation finding
- ★ HiCAT applied to ulcerative colitis scRNA-seq data discovered distinct macrophage and T-cell subset features not previously identifiable finding
- ★ Manual annotations may not be reliable enough as ground truth for evaluating minor-type and subset identification performance finding
- ★ Knowing a cell's broader (major/minor) type improves decisions on its finer (minor-type/subset) identity, especially when markers overlap across different lineages mechanism
- GMM-based correction improves major-type identification performance by leveraging proximity in gene expression feature space method
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| scRNA-seq (10X Genomics) | human PBMC (PBMC 68K, GSE93421) | none | major/minor-type cell identification accuracy | 10X Genomics |
| scRNA-seq (Drop-seq) | human cord blood mononuclear cells (CBMC 8K, GSE100866) | none | major/minor-type cell identification accuracy | Drop-seq |
| scRNA-seq (Smart-seq2) | human melanoma tumor tissue (Melanoma 5K, GSE72056) | none (tumor vs. non-tumor cell population) | major/minor-type identification and tumor-cell rejection accuracy | Smart-seq2 |
| scRNA-seq (10X v2) | human lung tissue (Lung 57K/ERP114453 and Lung 114K/SRP218543) | none | major/minor-type cell identification accuracy | 10X v2 |
| scRNA-seq (10X Genomics) | bronchoalveolar lavage fluid, COVID-19 patients (BALF 43K, GSE158055) | none | immune-cell profiling and identification accuracy | 10X Genomics |
| scRNA-seq (10X Genomics) | human colon tissue, including ulcerative colitis (Colon 365K, SCP259) | disease state (ulcerative colitis vs. normal) | major/minor-type identification and macrophage/T-cell subset profiling | 10X Genomics |
| scRNA-seq (10X Genomics) | human breast cancer tumor tissue (BRCA 100K, GSE176078) | none (tumor vs. normal epithelial cell population) | major/minor-type identification and tumor-cell rejection accuracy | 10X Genomics |
| Computational benchmarking (GSA scoring, PCA/clustering, GMM correction, kNN correction) | in silico, all eight scRNA-seq datasets | none | ROC/AUC and pairwise identifier match rate for major/minor-type calls | — |
- ▲ HiCAT had the best 'correct' assignment rate for major-type identification in 7 of 8 datasets; scSorter was slightly better only in PBMC 68K
- ▲ Excluding tumor/non-marker-DB cell types, HiCAT achieved greater than 95% accuracy in major-type identification in most datasets, except PBMC 68K and BRCA 100K 95%
- ▲ HiCAT was best in minor-type identification in 6 of 8 datasets; scType was best in PBMC 68K and Lung 57K
- ▲ HiCAT almost perfectly identified tumor cells in Melanoma 5K via correct unassignment
- – In BRCA 100K, most identifiers including HiCAT misclassified cancer epithelial cells as normal epithelial cells; scCatch performed worst overall
- ▲ Average best pairwise identifier match rate (85.6%) exceeded average best match rate with manual annotation (72.3%) across datasets for minor-type calls 85.6% vs 72.3%
- ▼ In CBMC 8K, HiCAT-DigitalCellSorter pair match rate (85.0%) was slightly below the scSorter-DigitalCellSorter pair (85.4%), the only dataset where HiCAT was not in the best pair 85.0% vs 85.4%
- ▲ GMM-based correction improved ROC/AUC separability of major-type GSA scores compared to before correction
- mean 72.3% (average best pairwise match rate with manual annotation for minor-type identification across datasets)
- mean 85.6% (average best pairwise identifier-to-identifier match rate for minor-type identification across datasets)
- other 85.4% (scSorter-DigitalCellSorter match rate in CBMC 8K minor-type identification)
- other 85.0% (HiCAT-DigitalCellSorter match rate in CBMC 8K minor-type identification)
- other >95% accuracy (HiCAT major-type identification accuracy in most datasets excluding PBMC 68K and BRCA 100K)
- count 365,492 cells (total cells in Colon 365K dataset)
- count 100,064 cells; 24,489 tumor cells (total and tumor cell counts in BRCA 100K dataset)
- other false positive rate ≤ 0.05 (threshold criterion used to set major-type rejection thresholds from GSA score distributions)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
HiCAT is a computational cell-type annotation tool evaluated across eight public scRNA-seq datasets using manual annotations as ground truth for major- and minor-type benchmarking. Performance is quantified via five categorical criteria (correct, error, correctly-unassigned, erroneously-assigned, erroneously-unassigned) and compared against seven competing marker-based identifiers. For subset-level evaluation, where no standardized ground truth exists, pairwise prediction match rates among identifiers and qualitative marker expression profiles are used instead. ROC curves and AUC values illustrate GSA score separability before and after GMM-based correction.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Gene set analysis (GSA) scoring | Primary statistical scoring of cell-type identity at major-type, minor-type, and subset levels for all datasets | — | not stated |
| Gaussian mixture model (GMM)-based correction | Post-scoring reassignment of major-type labels based on gene expression proximity in reduced feature space | — | not stated |
| k-nearest neighbor (kNN) rule correction (k=31) | Minor-type identification correction step across all eight datasets | — | not stated |
| ROC curve and AUC | Separability evaluation of GSA scores for major-type and minor-type identification, shown for Colon 365K and BRCA 100K as representative datasets | — | not stated |
| Pairwise prediction match rate | Cross-identifier agreement for minor-type and subset identification across all eight datasets (Table 2a/b) | — | na |
| Five-criterion performance evaluation (correct, error, CUA, EA, EUA) | Major-type and minor-type benchmarking against manual annotations across all eight datasets (Figures 2A, 3A) | — | not stated |
-
Performance across datasets and identifiers is compared by visual inspection of stacked bar charts showing percentage breakdowns of the five criteria (C, E, CUA, EA, EUA)↳ Could also: Aggregate scalar metrics such as macro-averaged F1 score or Matthews correlation coefficient (MCC) could also summarize multi-class identification accuracy into a single comparable value per method per dataset — A single scalar summary statistic makes cross-method ranking more systematic, amenable to statistical comparison, and easier to present in tabular form; both F1 and MCC are widely used in cell-type classification benchmarks
-
Pairwise raw match rates between identifier pairs are reported to assess minor-type and subset agreement in the absence of a reliable ground truth↳ Could also: Cohen's kappa or Krippendorff's alpha could also quantify inter-rater agreement while correcting for agreement expected by chance — Chance-corrected agreement statistics are more interpretable when class distributions differ across datasets or when the number of classes varies, making it easier to compare concordance levels on an equal footing
-
ROC curves and AUC are used to illustrate GSA score separability for selected datasets; rejection thresholds are set to keep per-type false positive rate ≤0.05↳ Could also: Precision-recall (PR) curves and area under the PR curve (AUPRC) could also be shown, particularly given the class imbalance typical in scRNA-seq where some immune cell types are rare — PR curves are generally more informative than ROC curves under class imbalance because they are sensitive to changes in the minority-class detection rate rather than the majority-class true-negative rate
-
No measure of variability is reported around performance estimates derived from the eight datasets; results are presented as point estimates per dataset↳ Could also: Bootstrap confidence intervals or summary statistics (mean ± SD) across the eight datasets could also convey the stability of each method's performance advantage — A central-tendency plus uncertainty summary helps distinguish consistent performance gains from gains that are driven by one particularly favorable dataset, and supports stronger comparative conclusions
-
Qualitative inspection of marker expression profiles is used for subset-level evaluation because no standardized annotated ground truth for immune-cell subsets exists↳ Could also: Quantitative per-cell enrichment scores (e.g., AUCell or Seurat AddModuleScore applied to canonical subset marker gene sets) could also provide a numeric summary for subset evaluation across identifiers — Numeric enrichment scores enable more reproducible, threshold-based comparisons across tools and datasets than visual inspection of expression violin or dot plots, and could be used to compute correlation or rank agreement between annotation methods
-
The eight benchmark datasets are treated independently, with no aggregation of performance across datasets into a formal meta-analytic summary↳ Could also: A random-effects meta-analysis or mixed-effects model treating dataset as a random factor could also pool accuracy estimates while accounting for heterogeneity across datasets and protocols — Formal pooling with a random-effects model would quantify both average performance differences between methods and the degree to which those differences vary by dataset, supporting more generalized conclusions
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
Assessed papers, coloured by verdict. Click a node to open it.
Data lineage
The datasets this paper uses (text-mined from the full text via Europe PMC), and which other assessed papers stand on the same data. A shared dataset is a factual link — not a judgement.
Downstream reach in the literature
98 downstream papers · 1 datasetsHow widely the datasets deposited by this paper are reused across the whole literature (Europe PMC), beyond our assessed set. This is a factual dependency map — reusing a public dataset is normal, good science. It is not a judgement on the downstream papers; the only verdict here is this paper's own, with its cited rationale.
- Temporal profiling of the breast tumour microenviron... 2022 · 159 cites
- Cellular architecture of human brain metastases. 2022 · 151 cites
- Identification of the novel exhausted T cell CD8 + m... 2024 · 111 cites
- Molecular mechanisms and therapeutic significance of... 2024 · 91 cites
- Context-dependent activation of STING-interferon sig... 2023 · 76 cites
- BIDCell: Biologically-informed self-supervised learn... 2024 · 63 cites
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-36681937 (HiCAT)
Paper: Lee et al. 2023, Briefings in Bioinformatics 10.1093/bib/bbad006. "Hierarchical cell-type identifier (HiCAT) accurately distinguishes immune-cell subtypes..."
Tool: HiCAT — a marker-based, hierarchical (major→minor→subset) cell-type
annotation tool for scRNA-seq, distributed as the pip package MarkerCount
(from MarkerCount.hicat import HiCAT). Repo: https://github.com/combio-dku/HiCAT.
This is the authors' own code AND ships a ready example, so reproduction =
running the shipped tool on the shipped/paper data per the documented parameters.
What the paper reports (pipeline-derived)
The paper benchmarks HiCAT against 6 other marker-based annotators (scSorter, Garnett, SCINA, DigitalCellSorter, scType, scCatch) across 8 scRNA-seq datasets, reporting match rate (%) with manual/ground-truth annotation at major-type (Table 2a / Fig 2) and minor-type (Table 2b / Fig 3) levels.
Datasets used (paper): PBMC 68K (GSE93421), CBMC 8K (GSE100866), Melanoma 5K (GSE72056), Lung 57K (ERP114453), Lung 114K (SRP218543), BALF 43K (GSE158055), Colon 365K (SCP259), BRCA 100K (GSE176078).
IN SCOPE (attempted — the clearly-specified, low-hanging 80%)
The repo ships Melanoma_5K_rev.h5ad — explicitly "one of the data we used
in our paper" (= GSE72056) — together with the markers DB
(cell_markers_rndsystems_hs.tsv) and an example notebook
(HiCAT_example_py_v03.ipynb) with exact parameters. Its .obs carries the
ground-truth celltype_major/celltype_minor plus stored predictions for HiCAT
and all 6 competitors. This makes two reported numbers directly reproducible:
- C1 — Melanoma major-type match rate (HiCAT vs manual): reported 98.9% (Table 2a).
- C2 — Melanoma minor-type match rate (HiCAT vs manual): reported 94.9% (Table 2b).
- C3 — qualitative headline: HiCAT > all 6 competing tools (computable from the
stored competitor predictions via the shipped
get_perf_ext). - C4 — pipeline self-consistency: does a fresh HiCAT re-run (notebook params)
reproduce the authors' stored
HiCAT_major/HiCAT_minorcolumns? (concordance %).
Pipeline: HiCAT (MarkerCount.hicat.HiCAT) on the count matrix with the shipped
markers; metric via the shipped get_perf_ext.
OUT OF SCOPE / NOT ATTEMPTED (the hard ~20%)
- The other 7 datasets (PBMC/CBMC/Lung×2/BALF/Colon/BRCA-100K incl. GSE176078): large downloads + per-dataset manual-label curation + re-running 6 competitor tools each. Not low-hanging; skipped per 80/20. The melanoma example exercises the identical HiCAT pipeline, so it is representative of the method's results.
- Re-running the 6 competitor tools from scratch (we trust the authors' stored competitor predictions in the h5ad for C3 rather than re-deriving them).
- Tissue-microenvironment / downstream biological-interpretation figures (qualitative, not a single pinnable number).
Compute
All heavy compute on «our HPC» («infra») via SLURM («host» ssh «host» ... sbatch), conda env built inside the job (compute nodes have internet);
repo + 124 MB h5ad fetched inside the job. «host» only orchestrates.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
On the single reproduced dataset (GSE72056 melanoma, shipped with the authors' own HiCAT repo) the reproduction is essentially 1:1: the headline major-type match rate 98.9% reproduces exactly (98.92%), the independent scType cross-check 71.3% reproduces exactly (71.35%), and a fresh re-run reproduces the stored predictions cell-by-cell. The only deviation is the minor-type rate (94.9%→93.64%, Δ1.26pts), plausibly explained by a marker-DB version difference (our/technical side, not an authors' defect). The central claim — HiCAT best of 7 marker-based tools — holds fully. The main limitation is scope: 7 of 8 datasets and all cross-dataset averages were not attempted, so the verdict is solid-with-explainable-deviation rather than a complete 1:1.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.