GeneSetCart: assembling, augmenting, combining, visualizing, and analyzing gene sets.
Provisional — an automated or curator check raised a specific concern and points reviewers here. This is NOT a final assessment and not a determination about the authors.
The main results reproduced: recomputed values matched the published ones within tolerance.
- ✓Same input data as the authors
- ✓Reported values were directly comparable
- ✓Reported values are derivable from the shared data
- ✓The central claim held under reproduction
- 🟡A deviation arose in the data or preprocessing
- 🟡A deviation was attributed to the published material
- 🟡The deviation was non-trivial in magnitude
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
DESCRIBED WELL ENOUGH -> 1:1 on the core pipeline result (re-run after requeue; this version adds the datasets + qc_room blocks the prior ROOM_RESULT lacked). GeneSetCart is a web tool; its gradeable computational results are two case studies + the CFDE library-crossing (Fig 2). Case Study 1 (GTEx-Aging x MoTrPAC Fisher-exact crossing) was reproduced EXACTLY by independently recomputing the crossing from the deposited GMT libraries on «our HPC» (SLURM «job», COMPLETED exit 0): 346 significant pairs (P<0.001), top-2 pairs 35 genes/P=6.518e-38 and 26 genes/P=1.054e-24, 24-gene intersection and 37-gene union -- all matching the paper AND the authors' shipped crossing CSV. Input GMTs/CSV SHA256 are byte-identical to the independent prior run, and the Fisher background N=21000 was recovered from the shipped odds ratios. The 24-gene Enrichr enrichment reproduces the reported coagulation/lipid/immune biology. Fig 2D crossing-percentage methodology reproduced across 8 CFDE libraries: KOMP2 (0.063%) & Metabolomics (0.302%) least-overlapping (exact), same-program GTEx-GTEx high (within-tol). NOT REPRODUCED: (a) Case Study 2 (Alexander disease) consensus up=65/down=20 and its ChEA/WikiPathways/MGI enrichment -- the deposited Enrichr lists are EXPIRED and the upstream limma DE is underspecified (docs_insufficient for this claim); (b) the Case-1 novelty literature statistics -- attributable to underspecified Geneshot usage + GeneRIF/PubMed growth since 2024, NOT fabrication (the underlying 37-gene set is exact and contains all 11 named genes). NO fabrication indicators; one benign paper-prose label swap noted (35 genes attributed to 60-69 Up in text vs 70-79 Up in shipped/recomputed data; numbers identical). NOT ATTEMPTED (out of scope): UI/account/report/Chrome-extension features, GPT-4o hypothesis text (non-deterministic), augmentation/UMAP/heatmap/Venn rendering (no specific numeric claim).
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 87assessed: 2026-06-16 ⛓ 1d977394564b
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-22
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-16no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetThe paper is motivated by the premise that unifying gene set assembly, augmentation, combinatorial operations, visualization, and analysis into a single interactive platform will better support multiomics data integration and hypothesis generation than existing fragmented gene set tools.
- ★ GeneSetCart is a web-based platform that lets users assemble, augment, combine, visualize, and analyze gene sets from multiple sources in one place resource
- ★ Gene sets can be augmented with coexpressed genes, literature comention genes, or PPI-interacting genes via the Geneshot and Genes2Networks APIs method
- ★ GeneSetCart supports set operations (union, intersection, consensus, subtract) to combine selected gene sets into new sets method
- ★ GeneSetCart provides an interactive visualization suite (Venn, SuperVenn, UpSet, hierarchically clustered heatmaps, UMAP) to summarize overlap/similarity among gene sets method
- ★ GeneSetCart integrates 11 external tools (Enrichr, Enrichr-KG, Rummagene, RummaGEO, ChEA3, KEA3, SigCom LINCS, L2S2, CFDE-GSE, PWB, PFOCRummage) for gene set enrichment/analysis method
- ★ For significantly overlapping gene set pairs from the library-crossing feature, a GPT-4o LLM is prompted with set descriptions and top enriched terms to generate a hypothesis explaining the overlap method
- ★ GeneSetCart has a gene set library crossing feature enabling analysis of gene sets from multiple NIH Common Fund programs, demonstrated with two use cases resource
- A Chrome extension lets users extract and load gene sets from Rummagene and RummaGEO while browsing GEO, PubMed, or PMC resource
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| gene set enrichment analysis | user-assembled/CFDE gene sets | none | overrepresented terms/pathways | Enrichr |
| gene-gene coexpression correlation retrieval | ARCHS4 RNA-seq compendium | none | coexpressed genes added to input set | Geneshot API |
| literature comention similarity retrieval | GeneRIF-derived gene-gene comention matrix | none | comentioned genes added to input set | Geneshot API |
| protein-protein interaction subnetwork construction | aggregated PPI data (BioGRID, BioPlex, IntAct, MINT, PPID, iRefWeb, Stelzl et al.) | none | directly interacting genes (max path length 2 between seed genes) | Genes2Networks (G2N) API |
| hierarchical clustering heatmap of Jaccard similarity | gene sets within a user session | none | pairwise set similarity/clustering | Seaborn clustermap |
| UMAP dimensionality reduction | TF-IDF-vectorized gene sets | none | 2D embedding colored by Leiden cluster or user-assigned group | Scikit-learn TF-IDF; Scanpy UMAP/Leiden |
| LLM-based hypothesis generation | overlapping genes from paired NIH Common Fund program gene sets | none | textual hypothesis for high overlap, informed by GO BP, WikiPathways, MGI Mammalian Phenotype, and GWAS Catalog enrichment | GPT-4o (OpenAI) with Enrichr enrichment input |
| transcription factor / kinase enrichment analysis | input gene set | none | ranked TFs or upstream kinases | ChEA3 / KEA3 |
- – GeneSetCart Venn diagrams support visualization of up to 5 gene sets simultaneously 5 sets
- – Default gene set augmentation (coexpression, comention, or PPI) adds up to 200 genes per set 200 genes
- – PPI subnetworks are built using a shortest-path algorithm with a maximum path length of 2 between seed genes path length 2
- – GeneSetCart provides access to 11 external gene set analysis tools via their APIs 11 tools
- – LLM hypothesis prompts incorporate the top 5 enriched terms from each of 4 Enrichr libraries (GO BP, WikiPathways, MGI Mammalian Phenotype, GWAS Catalog) top 5 terms x 4 libraries
- – Gene name validation/mapping to official NCBI symbols is supported across 18 species 18 species
- – Report generation allows selection of up to 5 gene sets plus chosen analysis/visualization modules 5 sets
- – Enrichr, the source for one gene set acquisition route, contains over 500,000 annotated gene sets across more than 530 libraries >500,000 sets; >530 libraries
- count >500,000 gene sets (size of Enrichr's gene set collection usable within GeneSetCart)
- count >530 gene set libraries (number of Enrichr libraries accessible via GeneSetCart)
- count 11 external tools (external enrichment/analysis tools integrated into GeneSetCart)
- count 200 genes (default maximum) (default cap on genes added per augmentation step)
- other maximum path length of 2 (Genes2Networks shortest-path parameter for PPI subnetwork construction)
- count 18 species (species supported for gene name conversion and validation)
- count up to 5 gene sets (maximum sets supported in Venn diagrams and in reports)
- other minDist=0.1, spread=1, nNeighbors=15, randomState=42 (default UMAP embedding parameters)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
GeneSetCart is a software tool paper describing a web platform for assembling, augmenting, combining, visualizing, and analyzing collections of gene sets. No primary experimental statistical analyses with hypothesis testing on biological specimens are reported; instead, the paper describes algorithmic and computational methods implemented in the platform—Jaccard similarity for pairwise set comparison, TF-IDF vectorization with UMAP for dimensionality reduction, and Leiden community detection for clustering. The CFDE gene set crossing feature references 'significant' overlap pairs but does not explicitly state the underlying statistical test or significance threshold.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Jaccard similarity (pairwise, symmetric) | Hierarchically clustered heatmap visualizing overlap among gene sets | — | na |
| TF-IDF vectorization followed by UMAP embedding (Scanpy implementation; minDist=0.1, spread=1, nNeighbors=15, randomState=42) | UMAP scatterplot visualization of gene set similarity structure | — | na |
| Leiden algorithm (modularity-based community detection applied to TF-IDF vectors) | Cluster-coloring of gene sets in the UMAP visualization | — | na |
| Gene set overlap significance test (method not stated in this paper) | CFDE gene set library crossing feature — identifying 'significant' crossing pairs between NIH Common Fund program gene sets | — | not stated |
| Enrichment analysis via external tools (e.g., Enrichr overrepresentation; specific test delegated to each external tool and not described here) | LLM hypothesis generation pipeline — top 5 enriched terms from GO Biological Processes, WikiPathways, MGI Mammalian Phenotype, GWAS Catalog libraries used as LLM context | — | not stated |
-
Gene set pairwise similarity for the heatmap is computed with Jaccard similarity↳ Could also: The overlap coefficient (Szymkiewicz–Simpson), cosine similarity on binary membership vectors, or a matrix of hypergeometric p-values could also quantify pairwise gene set relatedness — Different metrics capture different properties: the overlap coefficient is less penalized by large asymmetries in set size; hypergeometric p-values additionally convey statistical significance of the overlap given a defined gene universe; choice affects which relationships appear prominent in the heatmap
-
Gene sets are embedded into UMAP space using TF-IDF vectorization of gene membership↳ Could also: Binary membership vectors, a Jaccard distance matrix fed directly to UMAP with metric='jaccard', or PCA applied to the binary matrix could also serve as the input representation — TF-IDF down-weights genes that appear in many sets (analogous to stopwords); binary vectors treat all genes equally; Jaccard-metric UMAP operates directly on set-theoretic distance; the choice influences cluster separation and the interpretation of proximity in the embedding
-
The Leiden algorithm is applied to TF-IDF vectors for coloring gene sets by cluster in the UMAP↳ Could also: k-means, agglomerative hierarchical clustering with a dendrogram cut, or DBSCAN could also partition gene sets into groups — Leiden optimizes modularity and its resolution parameter controls granularity; k-means requires a pre-specified number of clusters; hierarchical clustering produces a tree that can aid biological interpretation; DBSCAN identifies noise points; method choice affects cluster boundaries, stability, and number
-
The significance of gene set crossing pairs in the CFDE feature is referenced ('significant' pairs) but the underlying statistical test, universe definition, and threshold are not stated↳ Could also: Fisher's exact test or a hypergeometric test with Benjamini–Hochberg FDR correction is commonly used to assess the significance of gene set overlaps given a defined background gene universe — Explicit reporting of the test, universe size, and correction method would allow readers to reproduce significance calls and interpret what fraction of crossings are expected by chance at a given threshold
-
Tool use cases are presented qualitatively, with no quantitative benchmarking against existing comparable tools↳ Could also: A structured benchmarking comparison against tools such as Intervene, Flame, or Evenn on standardized gene set collections—reporting runtime, scalability with set count or size, or concordance of overlap metrics—could also be included — Quantitative benchmarking would let readers assess relative performance and scalability under controlled conditions, complementing the qualitative feature comparison provided
-
UMAP hyperparameters (minDist=0.1, spread=1, nNeighbors=15, randomState=42) are reported as defaults; user-adjustable sliders are provided but no sensitivity analysis is reported↳ Could also: A brief stability analysis varying nNeighbors and minDist across a grid, or reporting cluster label agreement (e.g., adjusted Rand index) across random seeds, could also characterize layout robustness — UMAP layouts can differ substantially with hyperparameter choices and random seeds; reporting stability metrics would help users judge whether observed clustering patterns reflect true structure or are sensitive to parameter selection
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
Assessed papers, coloured by verdict. Click a node to open it.
- No assessed neighbours yet — the network grows as more papers are assessed.
Data lineage
The datasets this paper uses (text-mined from the full text via Europe PMC), and which other assessed papers stand on the same data. A shared dataset is a factual link — not a judgement.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-40208796 (GeneSetCart)
Paper: Marino GB, Olaiya S, Evangelista JE, Clarke DJB, Ma'ayan A. GeneSetCart:
assembling, augmenting, combining, visualizing, and analyzing gene sets.
GigaScience 14 (2025), giaf025. PMID 40208796 / PMC11984350.
Code: https://github.com/MaayanLab/GeneSetCart (GPL-3.0, commit 7f60d6d).
Data: public MaayanLab MinIO bucket g2sg (GMT libraries, precomputed crossing
CSVs, abstracts) + Enrichr/Geneshot APIs. GEO GSE198817/GSE197044/GSE116327 (Case 2).
GeneSetCart is a web application (Next.js/TypeScript + a small Python Flask backend). Most of the paper describes UI features. The reproducible, pipeline-derived computational results are the two demonstration case studies and the CFDE library-crossing analysis (Fig 2). Set/Fisher/enrichment operations are deterministic.
IN SCOPE (computational, deterministic → attempted)
| # | Result | Pipeline | Status |
|---|---|---|---|
| C1 | Case Study 1: GTEx Aging × MoTrPAC crossing — 346 sig pairs (P<0.001, Fisher exact), top-2 pairs (35/26 genes; P=6.52e-38 / 1.05e-24), 24-gene intersection, 37-gene union | Fisher exact test (SciPy) over two GMT libraries | REPRODUCED 1:1 (exact) |
| C1e | Case Study 1: enrichment of the 24 overlapping genes (KEGG 2021 Human, GO BP) | Enrichr API | REPRODUCED (biology matches) |
| C1n | Case Study 1: novelty assessment — 65% of genes <100 PubMed pubs; HAO1/SLC25A47 <10; 11/37 comentioned with aging/exercise | Geneshot / GeneRIF API | PARTIAL/MISMATCH (gene set exact; literature counts diverge — method underspecified + time-drift) |
| F2D | Fig 2D: % significant crossing pairs per CFDE library pair; KOMP2 & Metabolomics least overlap; same-program (GTEx–GTEx) greatest | Fisher exact crossing across 8 libraries | REPRODUCED (qualitative orderings match) |
| C2 | Case Study 2 (Alexander disease): consensus up=65 / down=20 genes; Enrichr ChEA top-3 RELA/IRF8/STAT3; WikiPathways Spinal Cord Injury WP2432 P=8.234e-7; MGI phenotypes | limma DE (bulk RNA-seq appyter) on 3 GEO studies → consensus → Enrichr | NOT REPRODUCED (blocker, see below) |
OUT OF SCOPE (not a reproducible pipeline output)
- All UI/UX, account/session/sharing, Chrome-extension, report-generation features.
- LLM (GPT-4o) hypothesis text generation — non-deterministic, needs OpenAI key; the paper itself frames these as illustrative, not quantitative.
- Augmentation (Geneshot/G2N), UMAP/heatmap/Venn rendering — interactive, no specific reported numeric claim to grade against.
- API response-time benchmark (Suppl. Fig S1).
Blockers (honest)
- C2: the authors' deposited Enrichr shared lists for both case studies are
expired (
Enrichr/view?dataset=...→{"expired": true}), so the consensus gene lists cannot be retrieved. Regenerating them requires re-running the bulk RNA-seq appyter limma pipeline on GSE198817+GSE197044+GSE116327; the exact per-study sample/contrast selection and DE thresholds that yield consensus-3 up=65/down=20 are not specified in the paper →docs_insufficientfor this claim. - C1n: novelty assessment depends on live GeneRIF/PubMed counts that grow over time and on an unspecified Geneshot endpoint/threshold; reproduction is provisional.
All heavy work ran on «our HPC» (SLURM «job» + front1 light API jobs); data lives on
«infra» «path». Only small result JSONs
copied here.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.