DeepGSEA: explainable deep gene set enrichment analysis for single-cell transcriptomic data.
The main result did not reproduce in this reproduction attempt. Where our recomputation produced values that differ from the published ones, those discrepancies are listed below. This is a single automated attempt — not peer review and not a finding of error or misconduct — and differences can also arise from data access, undocumented parameters or the computing environment. The verdict can be contested via “report an error”.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓No relevant deviation in data/preprocessing
- ✓Reported values are derivable from the shared data
- ✓Any deviation was negligible
- ✓The central claim held under reproduction
- 🟡Could not use the authors’ exact input data
- 🟡Reported values were only indirectly comparable
- 🟡A deviation was attributed to the published material
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
Described well enough to reproduce, and the pipeline reran 1:1 to COMPLETION this time. DeepGSEA is the authors' own PyTorch tool; I ran its shipped glioblastoma pipeline (run_glioblastoma.sh hyperparameters, seed 0, 5-fold CV with the REPO'S OWN shipped splits) on the paper's own PUBLIC data: GSE132172 (21209 genes x 134 cells) + MSigDB v2023.1.Hs GO-BP (1740 sets) + Reactome (1887 sets) + go-basic.obo, all fetched in-job onto «infra», env (torch 1.13.1 + scientific stack) rebuilt from scratch inside the compute job. The clean re-run (SLURM 2218736, COMPLETED exit 0, 1h32m on n095) finished BOTH GO and pathway concepts x 5 folds plus pvalue_real.py combination -- the full completion the prior force-finalized run lacked (it had only collected the GO output before the operator asked to finalize and «infra» was reclaimed). Determinism is confirmed: GO:0046717's per-fold p-values are bit-identical to the prior independent run with the env rebuilt. RESULTS for the three concepts the authors single out for interpretation (interpret.py phenotype/heatmap in run_glioblastoma.sh): (1) GO:0046717 'acid secretion' -> ENRICHED, combined-p 1.41e-28, rank 290/1740; (2) R-HSA-8866427 'VLDLR internalisation and degradation' -> STRONGLY ENRICHED, combined-p 1.56e-34, rank 13/1887 (top 0.7%); (3) R-HSA-444821 'Relaxin receptors' -> NOT ENRICHED, combined-p 1.0, rank 1883/1887 (bottom). So 2 of 3 author-interpreted concepts reproduce as enriched as expected, and the third (R-HSA-444821) reproduces as essentially the least enriched set -- recorded honestly and FLAGGED for human review, but NOT called fabrication because interpret.py renders a concept's figure regardless of its enrichment p-value and the exact Suppl. E.2 text (which might present R-HSA-444821 as a contrast/negative example) is paywalled and unobtainable. Overall status 'partial' (not exact/within-tol) for two honest reasons: the reported decimals are paywalled (strict numeric comparison impossible), and one of the highlighted concepts does not reproduce as enriched. Statistical caveat recorded: with only 134 cells, ~94% of gene sets have combined-p<0.05, so RANK is the meaningful discriminator, not bare significance. NOT ATTEMPTED: Alzheimer 'generation of neurons' P=1.6e-25 (data SCP1375 = Broad-login/Terra-gated => data_restricted); simulation Fig2/3 (stochastic Splatter, not a deterministic 1:1); influenza (supplement-only). Code-integrity frictions (don't invalidate results, hurt out-of-box reproducibility): pvalue_real.py main references undefined parser (NameError); requirements.txt under-pins scipy (>=1.9.3) but code needs scipy>=1.11 for false_discovery_control; scripts hardcode --device cuda:0; setup.py entry point points to non-existent src.run. No fabrication indicators for GO:0046717 / R-HSA-8866427 (both reproduce as enriched from the shipped code+data); R-HSA-444821 flagged for human attention only.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 50assessed: 2026-06-14 ⛓ 2614ffc75d5e
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-22
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-15no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetThe paper tests whether gene set enrichment can be reformulated as a per-gene-set phenotype classification problem, such that an intrinsically interpretable, prototype-based deep neural network (DeepGSEA) can detect enriched gene sets in single-cell transcriptomic data more sensitively than existing statistical GSE methods, while remaining explainable unlike black-box DNN approaches.
- ★ DeepGSEA is an explainable deep gene set enrichment analysis method built on interpretable, prototype-based neural networks. method
- ★ Gene set enrichment for C phenotypes can be reformulated as finding a mapping function under which the phenotype-specific gene expression distributions become classifiable in a latent space (Assumption 1). mechanism
- ★ For each gene set, phenotype-specific latent distributions can be modeled as mixtures of Gaussian distributions represented by learnable prototypes (Assumption 2). mechanism
- ★ DeepGSEA uses a shared backbone encoder with gene set-specific heads to efficiently learn common encoding knowledge across many gene sets. method
- ★ Significance tests performed on gene set-specific cell-prototype similarity scores enable screening of enriched gene sets. method
- ★ DeepGSEA demonstrates greater sensitivity and specificity than commonly used GSE analysis methods across four simulation studies. finding
- Ablation studies show that sharing a backbone encoder across gene sets improves the model's ability to mine phenotype information from each individual gene set. finding
- ★ The gene set distribution learned by DeepGSEA can be explicitly visualized via encoded cell and prototype embeddings, providing interpretability of enriched gene sets. resource
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| simulation study (synthetic gene expression modeling) | simulated single-cell gene expression data across multiple phenotypes | simulated differences between phenotype-specific gene expression distributions | sensitivity and specificity of detecting enriched gene sets | — |
| scRNA-seq-based gene set enrichment analysis | three real-world scRNA-seq datasets (not further specified in provided text) | not stated in provided text | identification of enriched gene sets and visualization of learned latent cell/prototype distributions for interpretability | — |
- ▲ DeepGSEA showed better sensitivity and specificity than commonly used GSE analysis methods across four simulation studies.
- ▲ Ablation study indicated that the shared backbone encoder design improves each gene set head's ability to capture phenotype information.
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
DeepGSEA is a prototype-based deep learning framework for gene set enrichment analysis in scRNA-seq data that reformulates enrichment as a phenotype classification task and derives per-gene-set P-values from learned cell-to-prototype similarity scores. Its performance is benchmarked against existing GSE tools by measuring sensitivity and specificity across four simulation studies and three real scRNA-seq datasets. The specific statistical test applied to the similarity scores to generate P-values is described conceptually but the test name is not stated in the provided text. Interpretability is demonstrated via visualization of learned cell embeddings and prototypes.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Significance test on per-gene-set cell-to-phenotype similarity scores (specific test not named in provided text) | Each gene set; comparing distributions of similarity scores across phenotype groups to return a P-value for enrichment | — | not stated |
| Sensitivity and specificity (binary classification metrics with implicit threshold) | Four simulation studies benchmarking enriched vs. non-enriched gene set detection against competing methods | — | not stated |
-
T gene sets are each assigned a P-value and tested simultaneously; no multiple-testing correction procedure is described↳ Could also: Benjamini-Hochberg FDR correction (or Bonferroni for stringent control) across all T tested gene sets could also be applied — When hundreds or thousands of gene sets are tested, FDR control is standard practice in genomics (e.g., as used in GSEA, DESeq2) to bound the expected proportion of false discoveries; reporting adjusted P-values alongside raw ones is a common convention
-
Enrichment significance is derived from learned similarity scores; the specific null distribution and test statistic are not named in the provided text↳ Could also: A permutation test (shuffling phenotype labels and re-computing scores), a Wilcoxon rank-sum test, or a likelihood-ratio test on the classification probabilities could each formally anchor the null distribution — Naming the test and its null distribution supports reproducibility, allows calibration of Type I error, and facilitates comparison with classical statistical GSE methods whose assumptions are well characterized
-
Benchmark performance in simulation studies is summarized at fixed operating points via sensitivity and specificity↳ Could also: Area under the ROC curve (AUROC) or area under the precision-recall curve (AUPRC) could also summarize discrimination across all thresholds — Single-threshold sensitivity/specificity depend on an implicit or chosen cutoff; AUROC and AUPRC aggregate over the full range and are widely used in computational biology benchmarks, which aids comparison with future methods
-
Model training uses cross-entropy loss on phenotype classification, with prototype regularization losses added↳ Could also: A contrastive or triplet loss (common in metric-learning literature) could also be used to directly shape the latent space separation between phenotype classes — Contrastive losses directly optimize inter-class distance in the embedding space, which aligns naturally with the prototype-distance similarity measure the model ultimately uses for significance testing
-
Simulation study parameters (number of cells, genes, phenotypes, noise level, effect size) are not described in the provided excerpt↳ Could also: A systematic parameter sweep varying effect size, cell count, and noise could also characterize the operating range of each method — Varying key simulation parameters reveals the conditions under which each method gains or loses advantage, helping practitioners anticipate expected performance on their own datasets
-
The ablation studies evaluate the shared backbone contribution to per-gene-set classification performance↳ Could also: A paired statistical comparison (e.g., Wilcoxon signed-rank test across simulation replicates) between ablated and full models could also quantify whether backbone sharing significantly improves performance — Formal testing of ablation differences, rather than visual or tabular comparison alone, would allow readers to assess whether observed improvements exceed expected variation across simulation seeds
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
No assessed neighbours yet — the network grows as more papers are assessed.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-38950178 (DeepGSEA)
Paper: DeepGSEA: explainable deep gene set enrichment analysis for single-cell
transcriptomic data. Xiong G et al., Bioinformatics 40(7):btae434, 2024.
PMID 38950178 · PMCID PMC11236288 · DOI 10.1093/bioinformatics/btae434
Code: https://github.com/Teddy-XiongGZ/DeepGSEA (MIT, active; pinned commit
05e7e8aa651905b4853e5b069cdb416a080e051d, 2024-07-13). This is the authors' own
tool (P16 N/A here).
DeepGSEA is a PyTorch deep-learning model that performs gene set enrichment analysis
on single-cell RNA-seq. For each dataset × gene-set database it trains a prototype
model (5-fold CV, seed 0, cudnn-deterministic) and emits, per gene set, an enrichment
auROC + Mann–Whitney/Fisher p-value (Concept_Score.csv); pvalue_real.py then
combines the 5 folds (Pearson) into optimal_pvalue.csv.
In scope (pipeline-derived, attempted)
| Result | Pipeline | Inputs | Status |
|---|---|---|---|
Glioblastoma real-data enrichment (Suppl. Appendix E.2). Authors' run_glioblastoma.sh highlights GO:0046717 (GO concept) and Reactome R-HSA-444821, R-HSA-8866427 (pathway concept) as the reported enriched sets. |
DeepGSEA train (GO + pathway, 5 folds) → pvalue_real.py |
GSE132172 (132 cells, 21209 genes; public GEO) + MSigDB v2023.1.Hs (GO-BP, Hallmark, KEGG, Reactome) + go-basic.obo. ALL public, no login. | PRIMARY — running (SLURM 2175509) |
This is the lowest-hanging, fully-public, smallest-data result → chosen for an honest 1:1 reproduction of the shipped pipeline on the paper's own data.
Out of scope (not attempted — with reason)
| Result | Why not attempted |
|---|---|
| Alzheimer "generation of neurons" P = 1.6×10⁻²⁵ (Fig 4, main text — the clearest reported number) | Data = Broad Single Cell Portal SCP1375 (expression_matrix_normalized.csv + metadata.csv). SCP bulk download requires Google/Terra authentication (session-cookie wall) → effectively data_restricted for an automated agent. Larger (16,692 cells). Not attempted; recorded honestly. |
| Simulation studies (Fig 2/3, e.g. DeepGSEA P = 1.9×10⁻⁹ at DE factor 0) | Requires generating simulated data with Splatter (R/Bioconductor) from GSE212270 (GSE212270_integrated_naive_cd8.rds) per the paper. Data generation is stochastic and the exact simulated matrices are not shipped (only the CV index splits are) → not a deterministic 1:1; heavy R+Splatter setup. Skipped per 80/20. |
| Influenza multi-phenotype analysis (Suppl. E.3) | Larger (18,640 cells); reported numbers also supplement-only. Skipped per 80/20 (glioblastoma already covers the same pipeline). |
Reported-value caveat
Exact glioblastoma p-values live in Supplementary Appendix E.2, which is behind a JS/Cloudflare download wall (PMC viewer PoW; OUP 403; NCBI OA package 404; Europe PMC supplementaryFiles 404). The main-text and full-text XML do not contain glioblastoma numbers. Therefore the glioblastoma claim is graded qualitatively: do the gene sets the authors' own interpretation script hardcodes (GO:0046717, R-HSA-444821, R-HSA-8866427) come out as significant / top-ranked enrichments when the shipped pipeline is rerun on the public data? Exact reported decimals could not be obtained → flagged in AUDIT.md (not a fabrication claim — simply paywalled).
Hard rules honored
- All compute on «our HPC» (SLURM, «infra»); «host» holds only small results + pointers.
- Env built inside the job (conda prefix env on «infra»; CPU torch 1.13.1; scipy bumped
to 1.11.4 because code uses
scipy.stats.false_discovery_control, under-pinned in requirements.txt asscipy>=1.9.3).
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
DeepGSEA's own pipeline reran 1:1 on the authors' own public data (GSE132172 + MSigDB GO-BP, shipped CV splits, seed 0) and the author-hardcoded gene set GO:0046717 came out strongly significant in every fold (BH-adj p ~1e-6), consistent with a genuine finding — no fabrication concern. The limitation is availability, not authors' defect or numeric disagreement: the exact reported glioblastoma decimals are paywalled in Suppl. Appendix E.2 and the main-text Alzheimer number (P=1.6e-25) depends on Terra-gated SCP1375, so the match is qualitative rather than strict-numeric. Minor shipped-code integrity issues (NameError, scipy under-pin, dead entry point) hurt out-of-box reproducibility but don't affect validity. Net: solid, faithful, explainable-partial reproduction → yellow.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.