PanglaoDB: a web server for exploration of mouse and human single-cell RNA sequencing data.
The main results reproduced: recomputed values matched the published ones within tolerance.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Same input data as the authors
- ✓Reported values were directly comparable
- ✓Reported values are derivable from the shared data
- ✓Any deviation was negligible
- ✓The central claim held under reproduction
- 🟡A deviation arose in the data or preprocessing
- 🟡A deviation was attributed to the published material
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
PanglaoDB sample SRS2781556 (mouse microglia, 10x). Described WELL ENOUGH to reproduce the documented pipeline outputs from PanglaoDB's own shipped raw count matrix. 1:1 EXACT on all three per-sample QC summary statistics: cells=948, expressed genes=21771, median genes/cell=2590 (recomputed from the matrix after the documented >=1000-counts cell filter — the shipped matrix is the raw 70862-droplet matrix). The assigned tool Scrublet (0.2.3) was run on the 948-cell matrix and the paper's 'remove top 5%' policy reproduced deterministically (47 removed, 901 kept); noted that the fixed 5% policy is ~4x more aggressive than Scrublet's own automatic threshold (12 cells) here. NOT attempted: (a) full from-FASTQ count-matrix rebuild (HISAT2/featureCounts/UMI-tools on SRR6410935, ~43GB) because the 10x cell-calling/barcode whitelist is unspecified per sample, so the exact cell set is not faithfully recoverable; (b) the C4 removed-cell IDENTITY (PanglaoDB publishes no per-cell scores and pins no Scrublet version/params); (c) clustering (7 clusters) as resolution-dependent. No fabrication signal: reported QC metadata is bit-exactly derivable from the distributed data.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 80assessed: 2026-06-16 ⛓ ee2c01c8922f
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-16
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-16no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusThere is a strong need for a curated, pre-processed and easily accessible resource for published single-cell RNA sequencing data; the paper addresses whether a unified web platform can collect, standardize and enable exploration of mouse and human scRNA-seq studies along with automatic cell-type annotation.
- ★ PanglaoDB is a web server providing pre-processed and pre-computed analyses of >1054 single-cell experiments (>4 million cells) from mouse and human across many tissues and platforms. resource
- ★ A unified, protocol-agnostic bioinformatics pipeline remaps and recounts raw SRA reads to homogenize algorithms, reference genomes and annotations across studies. method
- ★ A community-curated cell-type marker compendium of >6000 gene-cell-type associations supports automatic annotation of cell types. resource
- ★ A Cell-Type Activity (CTA) score with frequency-based gene down-weighting infers the cell type of each cell cluster. method
- ★ The cell-type prediction method was validated against datasets with known biological cell types and against expected tissue cell types. finding
- PanglaoDB additionally provides regulon/co-expression network, gene set activity, cell cycle, differential expression and disease association analyses. method
- PanglaoDB improves on prior databases (scRNASeqDB, SCPortalen) by offering far larger scale plus pre-computed analyses and advanced visualization. finding
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| single-cell RNA-seq read mapping/quantification pipeline | human (GRCh38) and mouse (GRCm38) public SRA datasets | none | per-cell gene read counts (sparse matrix) | hisat v.2.1.0, samtools v.1.8/sambamba v.0.6.7, subread featureCounts v.1.6.2, UMI-tools v.0.5.3, sratoolkit v.2.9.0, GENCODE v.27 |
| cell clustering and dimensionality reduction | mouse and human scRNA-seq samples (845 mouse, 209 human) | none | cell clusters and t-SNE/UMAP embeddings | Seurat v.2.3.2 (FindClusters), Scrublet, t-SNE, UMAP |
| cell-type inference (CTA score) and validation | 17 homogeneous flow-sorted datasets and 13 whole-tissue samples | none | predicted cell type vs reported/expected cell type | — |
| regulon / co-expression network and motif enrichment | per cell cluster (scRNA-seq) | none | transcription-factor-centered regulons / motif AUC significance | GRNBoost2, fimo (MEME v.4.12.0), PWMs from JASPAR/HOCOMOCO/SwissRegulon/UniPROBE/CIS-BP, TFCat, FANTOM5 |
| gene set activity (GSA) | mouse and human cell clusters | none | gene set enrichment activity | MSigDB v.6.1, Ensembl BioMart orthologs |
| cell cycle phase assignment | single cells within clusters | none | G1/G2M/S phase assignment per cell | scran (cyclone) |
| differential expression and disease association | cell clusters | none | marker genes / gene-disease associations | Seurat FindMarkers, eDGAR database |
- – Integrated 845 mouse and 209 human single-cell samples (>1054 experiments) containing >4 million cells. 845 mouse + 209 human samples; >4 million cells
- – In all 17 flow-sorted homogeneous datasets, predicted cell type matched the reported biological cell type. 17/17
- – In all 13 randomly selected tissue samples, the expected prominent cell type matched a predicted cell type. 13/13
- – Marker compendium contains more than 6000 gene-cell-type associations covering >150 possible cell types. >6000 associations; >150 cell types
- – In sample SRS2781556 annotated as microglia, prediction found mostly microglia plus a small neutrophil cluster, attributed to shared myeloid origin.
- – Collected 4104 positional weight matrices representing motifs of 560 transcription factors for regulon analysis. 4104 PWMs; 560 TFs
- count >1054 single-cell experiments (experiments integrated in PanglaoDB)
- count >4 million cells (total cells across samples)
- count 845 mouse and 209 human samples (integrated single-cell samples)
- count >6000 gene-cell-type associations (marker compendium size)
- count 17 datasets, all matched (flow-sorted validation, predicted = biological cell type)
- count 13 tissue datasets, all consistent (tissue-based orthogonal validation)
- pvalue adjusted P > 0.05 set to 'Unknown'; AUC < 0.05 significance threshold (cell-type FDR cutoff and motif enrichment threshold)
- count 4104 PWMs from 560 transcription factors (motif collection for regulon inference)
Statistical methods review
Model: opusA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a database/resource paper describing PanglaoDB, a web server aggregating mouse and human single-cell RNA-seq data through a unified bioinformatics pipeline; rather than hypothesis-testing experiments, its statistical components are embedded in computational analyses (cell-type inference, regulon/motif enrichment and gene-set activity). Cell-type calls used a cell-type activity score followed by a one-sided Fisher's exact (hypergeometric) test with Benjamini–Hochberg FDR control, and gene-set activity used a one-sided Fisher's exact test with Bonferroni correction. Method validation was descriptive, reporting concordance counts (17/17 sorted samples and 13/13 tissue samples) between predicted and expected/known cell types rather than formal inferential statistics.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| one-sided Fisher's exact test (hypergeometric test) | cell-type inference: significance of the top-ranking cell type for each cell cluster, on genes expressed vs not expressed (expression >0) | — | not stated |
| one-sided Fisher's exact test | gene set activity (GSA) enrichment against MSigDB signatures | gene sets restricted to between 10 and 500 genes | not stated |
| kernel density estimate with integration over the curve (AUC-based significance), used in place of a Z-score | regulon analysis: significance of a transcription-factor motif's enrichment in top-ranking co-expressed genes vs genomic background (threshold AUC<0.05) | limited to top 200 ranking genes per motif | stated (noted the distribution was often non-Gaussian, motivating the KDE approach) |
-
Cell-type significance was assessed with a one-sided Fisher's exact (hypergeometric) test on genes called expressed (>0) vs not.↳ Could also: A permutation/empirical null built by shuffling gene-cell-type labels, or a rank-based enrichment such as a Mann-Whitney/AUCell-style test on continuous expression, could also be used. — A permutation or rank-based approach can incorporate expression magnitude rather than a binary expressed/not threshold, which may be informative when dropout makes the >0 cutoff sensitive to depth.
-
Multiplicity was handled with Benjamini–Hochberg FDR for cell-type inference and Bonferroni for gene set activity.↳ Could also: A single consistent framework (e.g. BH FDR throughout, or Holm for family-wise control) could also be applied across both analyses. — Using one correction family throughout can make error-rate guarantees uniform across the resource; BH controls FDR while Bonferroni/Holm control family-wise error, and stating which target applies where can aid interpretation.
-
Validation was reported as exact concordance counts (17/17 and 13/13) between predicted and expected cell types.↳ Could also: Accompanying these with a 95% confidence interval for the proportion (e.g. Wilson interval) could also convey the precision of the concordance estimate. — For small validation sets, an interval communicates the uncertainty around a perfect or near-perfect agreement rate that a raw count alone does not.
-
Motif enrichment significance was derived from a kernel density estimate integrated to an AUC, replacing the Z-score used in the source method because the distribution was often non-Gaussian.↳ Could also: An empirical permutation P value against a shuffled-background null could also quantify significance without assuming a parametric form. — A permutation null is distribution-free like the KDE approach and additionally yields a directly interpretable P value and a natural route to multiple-testing correction across motifs.
-
Clustering resolution (Seurat resolution = 0.8) was selected by manual evaluation to produce fewer, larger clusters.↳ Could also: A data-driven stability or silhouette-based criterion (e.g. scanning resolutions and selecting by cluster stability) could also guide the choice. — An automated stability metric can make the resolution choice reproducible and comparable across the many heterogeneous samples processed by the same pipeline.
Result convergence & founder nodes
Findings this paper shares with others that ran a comparable experiment. A node’s strength is how many independent papers report it (replication breadth) — not how often it is cited, so a heavily-replicated but under-cited founder still stands out.
-
Automated cell-type prediction (CTA score) matched the reported cell type in all 17 flow-sorted homogeneous datasets.scRNA-seq mouse human flow-sorted none 2019×1papers★ This paper is the founder (earliest)
-
Microglia-annotated sample SRS2781556 was predicted mostly as microglia with a small neutrophil cluster, consistent with shared myeloid origin.scRNA-seq mouse human microglia mixed 2019×1papers★ This paper is the founder (earliest)
-
The expected prominent cell type matched a predicted cell type in all 13 randomly selected whole-tissue samples.scRNA-seq mouse human whole-tissue none 2019×1papers★ This paper is the founder (earliest)
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
Assessed papers, coloured by verdict. Click a node to open it.
- No assessed neighbours yet — the network grows as more papers are assessed.
- Hierarchical cell-type identifier accurately dis... L1 94/100
- Mouse-Geneformer: A deep learning model for mous... L1 95/100
- scATD: a high-throughput and interpretable frame... L1 62/100
- Transcriptomic data meta-analysis reveals common... L2 90/100
- The mouse gastric surface epithelial cell and it... L1 100/100
- GeneSetCart: assembling, augmenting, combining,... L1 76/100 ⚑
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Reproduction scope — pmid-30951143 (PanglaoDB)
Paper: Franzén, Gan & Björkegren (2019). PanglaoDB: a web server for exploration of mouse and human single-cell RNA sequencing data. Database (Oxford), baz046. PMID 30951143 · PMCID PMC6450036.
Assigned reproduction unit: apply the third-party tool Scrublet (github.com/AllonKleinLab/scrublet — doublet detection) to one PanglaoDB sample, SRS2781556 (PanglaoDB internal id SRA641134; SRA run SRR6410935; Mus musculus brain microglia, 10x chromium, "Repop D5"). Per BRIEF rule P16, using an existing third-party tool on the paper's own data is a valid reproduction.
What PanglaoDB's pipeline does (Methods, PMC6450036)
- Map reads with HISAT2 v2.1.0 (MAPQ ≥ 60) to GRCm38/GRCh38.
- GENCODE v27 annotation, exons collapsed to "meta genes",
id =
[symbol]_[ENSEMBL]. - featureCounts (subread v1.6.2) assigns gene to BAM; UMI-tools v0.5.3 dedup + count (3' UMI protocols).
- Keep cells with ≥ 1000 uniquely mapped reads after UMI dedup.
- Doublet detection: "Cell doublets were predicted with the Scrublet software and the top 5% cells with the highest scores were removed."
- Cluster retained cells; keep clusters with > 10 cells.
In scope (pipeline-derived, attempted)
| # | Result (reported for SRS2781556) | Pipeline step | How reproduced |
|---|---|---|---|
| C1 | Number of cells = 948 (≥1000 reads) | QC step 4 | count columns of PanglaoDB's shipped raw count matrix |
| C2 | Number of expressed genes = 21,771 | quantification | count genes (rows) with non-zero total |
| C3 | Median expressed genes/cell = 2,590 | quantification | median of per-cell non-zero gene counts |
| C4 | Doublet removal = top 5% by Scrublet score | step 5 (assigned tool) | run Scrublet on the shipped raw matrix; rank scores; top 5% = 47 of 948 cells; also report Scrublet's own auto-threshold doublet rate |
C1–C3 are recomputed directly from PanglaoDB's own shipped per-sample raw
count matrix (SRA641134_SRS2781556.sparse.RData; "Counts are not normalized,
columns are cells, rows are genes"). This is an auditability check: do the
numbers PanglaoDB reports match the data they distribute? C4 re-runs the
assigned tool (Scrublet) exactly as the Methods describe.
Out of scope / hard 20% (not attempted, with reason)
- Full from-FASTQ rebuild of the count matrix (HISAT2 → featureCounts →
UMI-tools from SRR6410935, ~43 GB). Skipped as the costly, low-yield 20%:
the 10x cell-calling / barcode whitelist PanglaoDB used is not specified
per sample in the Methods, so the exact 948-cell set is not faithfully
recoverable from the paper alone — many degrees of freedom. Using PanglaoDB's
own shipped matrix removes that ambiguity and isolates the documented Scrublet
step. Recorded as
env/docs-limitedfor this sub-step, not a whole-RU drop. - Number of clusters = 7 (step 6): clustering method/resolution not pinned to an exact reproducible parameterisation; cluster count is resolution- dependent. Treated as optional; may be probed but not graded as exact.
- Per-cell doublet scores: PanglaoDB publishes no per-cell Scrublet scores, so C4 is graded on the procedure (top-5% → 47 cells) and the data the step consumes, not on a published per-cell ground truth.
Drop assessment
NOT dropped. Code (Scrublet) resolves and is public; data (SRS2781556) resolves on SRA/ENA and PanglaoDB ships the exact raw matrix; identifiable reported results exist (948 / 21,771 / 2,590 / top-5%). → eligible, attempted.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
The three directly comparable QC claims (948 / 21771 / 2590) reproduce bit-exactly from PanglaoDB's own distributed raw matrix, so the reported metadata is fully data-derived with no fabrication signal. The only gaps are on data-availability / underspecification: PanglaoDB ships no per-cell Scrublet scores (C4 verifiable at procedure level only — 47/948 removed) and pins no clustering resolution (C5 not attempted). These limits sit on the authors'/database side, not in our computation, and do not contradict any reported value — hence a solid but partial reproduction rather than a clean 1:1 across all five claims.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.