Easy and efficient ensemble gene set testing with EGSEA.
The main results reproduced: recomputed values matched the published ones within tolerance.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Same input data as the authors
- ✓Reported values were directly comparable
- ✓No relevant deviation in data/preprocessing
- ✓No authors-side cause for any deviation
- ✓Reported values are derivable from the shared data
- ✓Any deviation was negligible
- ✓The central claim held under reproduction
- ✓Overall, the reproduction was clean
- Every checked point held up.
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
Described well enough and reproduces 1:1. EGSEA is a Bioconductor workflow article; I ran its published pipeline (EGSEA123 commit cd121b40, EGSEA/EGSEAdata/limma/edgeR) on the paper's own data (GSE63310 RNA-seq + GSE19446 microarray) on «our HPC» SLURM. All 8 deterministic anchors are byte-exact: RNA-seq DGEList dim 14165x9 (C1), all five indexed collection sizes c2/c5BP/c5CC/c5MF/kegg = 4726/4653/584/928/287 (C2), and microarray probe count 7123 (C6) - and show(gsa) independently re-prints MSigDB v5.2 (07-Mar-2017), the same snapshot the 2017 paper used. The ensemble gene-set rankings (C3/C4/C5), which involve permutation-based base methods (safe, padog) run without a seed in the vignette, still reproduce essentially exactly: KEGG top pathways for BasalvsLP (top-5) and comparative (top-4) are identical and in the same order (C4); the LIM ranking table med.ranks (36,37,92,134,180) and vote.ranks (all 5) match exactly with 2 of 5 p.adj values byte-identical (C5); the c2 top-10 contains the same 5 LIM sets the paper highlights with only a single position swap and one 10th-element difference (C3). NOT attempted (the figure/visual ~20%): heatmaps, KEGG pathview maps, MDS/summary/GO-graph plots and the HTML report (no pinnable numbers), the full microarray EGSEA ranking (only its deterministic 7123 anchor reproduced), and byte-exact p.adj for unseeded stochastic methods. No fabrication concern: every reported value is regenerated from the shipped data + published code.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 97assessed: 2026-06-14 ⛓ b7bb422b5909
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-14
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-15no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusThis software tool paper does not test a biological hypothesis; it demonstrates that combining results from multiple gene set enrichment algorithms via the EGSEA package yields a consensus ranking of biologically relevant gene sets, and shows how this workflow extends limma-based differential expression analyses of RNA-seq and microarray data.
- ★ EGSEA combines results from up to 12 prominent gene set testing algorithms to obtain a consensus ranking of biologically relevant gene sets. method
- ★ EGSEA calculates seven statistics to combine the individual gene set statistics of base GSE methods to rank gene sets. method
- ★ The EGSEAdata package provides access to more than 25,000 gene sets for human and mouse organised by database source (MSigDB, GeneSetDB, KEGG). resource
- ★ EGSEA integrates into a limma differential expression workflow and supports both RNA-seq and microarray data for human and mouse. method
- ★ EGSEA results can be reported via heatmaps, KEGG pathway views, GO graphs, scatter plots, bar plots and a combined HTML report. method
- Including more base methods gives better performance, as found via simulation. finding
- The med.rank and vote.rank sorting options efficiently utilize individual method rankings and tend to produce fewer false positives. finding
- Base methods split into competitive (ora, gage, camera, gsva) and self-contained (the remaining eight) null hypothesis methods. method
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| bulk RNA-seq | Basal, Luminal Progenitor and Mature Luminal cell populations sorted from mammary glands of female virgin mice | none (cell population comparison) | gene-level read counts / differential expression and gene set enrichment | Illumina HiSeq 2000, 100 bp single-ended; aligned to mm10 with Rsubread; counts via featureCounts |
| microarray gene expression profiling | Basal (MaSC-enriched), LP and ML cell populations sorted from mouse mammary glands via flow cytometry | none (cell population comparison) | hybridisation intensity / differential expression and gene set enrichment | Illumina MouseWG-6 v2 BeadChips (IDAT/BGX files) |
- – RNA-seq count matrix loaded for analysis after preprocessing 14165 genes x 9 samples
- – Eleven base methods selected for the EGSEA analysis (roast excluded in favour of fry) 11 of 12 methods
- – c2 curated gene set collection indexed against the expression matrix 4726 gene sets
- – EGSEAdata provides large gene set collections for mouse >25,000 gene sets
- count 14165 genes, 9 samples (dimensions of mam.rnaseq.data count matrix)
- count >25,000 (gene sets in EGSEAdata for human and mouse)
- count 4726 (c2 curated gene sets after indexing (Version 5.2))
- count 6166 (c5 GO gene sets for Mus musculus, MSigDB v5.2)
- count 50 (h Hallmark signatures for Mus musculus)
- count 12 (prominent GSE algorithms utilized by EGSEA)
- count 7 (statistics calculated to combine base method gene set statistics)
- count 3 replicates per population (RNA-seq); 5 replicates per population (microarray) (sample replication in the two mammary datasets)
Statistical methods review
Model: opusA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a software/workflow article demonstrating the EGSEA R/Bioconductor package for ensemble gene set enrichment analysis, rather than a hypothesis-testing study. The statistical approach centres on a limma-based differential expression framework (TMM normalization, voom precision weights, a linear model with a design and contrasts matrix) applied to RNA-seq and microarray data from 3 sorted mammary cell populations, followed by gene set testing that runs up to twelve base GSE algorithms (e.g. camera, fry, gage, padog, globaltest, ora, gsva) and combines their per-set p-values into a consensus ranking. Results are reported as ranked gene sets visualized via heatmaps, pathway/GO graphs, scatter and bar plots and an HTML report rather than as point estimates with dispersion.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| limma linear model / empirical Bayes moderated differential expression (with voom precision weights) | differential expression between Basal, LP and ML populations via contrasts (BasalvsLP, BasalvsML, LPvsML) | RNA-seq: triplicate samples per population (9 samples); microarray: 5 replicates per population | not stated |
| camera (competitive gene set test) | base GSE method in the ensemble; comparisons defined by contrasts matrix | — | competitive null hypothesis stated |
| fry (rotation gene set test, fast roast approximation) | base GSE method in the ensemble | — | self-contained null; assumes equal gene-wise variances across samples (stated) |
| ora (over-representation, hypergeometric test via phyper on a 2x2 table) | base GSE method in the ensemble | — | competitive null hypothesis stated |
| gage, gsva (competitive); safe, padog, plage, zscore, ssgsea, globaltest (self-contained) | base GSE methods combined in the ensemble | — | competitive vs self-contained null hypotheses stated |
| p-value combination across base methods (Wilkinson default; also fisher, average, logitp, sump, sumz, votep, median) | merging individual gene-set p-values into a consensus EGSEA score | — | not stated |
-
The workflow combines results from many base GSE algorithms into a consensus ranking and notes that including more methods improved performance in simulation.↳ Could also: A single well-characterised method (e.g. camera alone, which accounts for inter-gene correlation) could also be applied and reported on its own. — A single method yields a simpler, more directly interpretable inferential statement and avoids the question of how heterogeneous null hypotheses (competitive vs self-contained) are merged; the ensemble in turn adds robustness across method-specific assumptions.
-
Individual gene-set p-values are merged using p-value combination functions (Wilkinson default) from the metap package.↳ Could also: Rank aggregation across methods (e.g. the package's own med.rank/vote.rank) or a single FDR-controlled test could also be used as the primary readout. — Rank-based aggregation does not require the combined p-values to retain a calibrated null interpretation and, as the authors note, can produce fewer false positives; a single test keeps a clean type-I error statement.
-
Sorting and significance default to adjusted p-values (p.adj), with the specific multiple-testing method not named in this text.↳ Could also: Explicitly reporting Benjamini-Hochberg FDR q-values per collection per contrast would also be standard. — Naming the correction method and reporting q-values makes the family-wise/false-discovery scope of significance calls transparent across the thousands of gene sets tested.
-
The design uses biological replicates (n=3 RNA-seq, n=5 microarray per group) without a described power or sample-size calculation.↳ Could also: A brief statement of the basis for the replicate number (or its origin in the source datasets) could also be included. — Documenting the rationale for n helps readers gauge the sensitivity of downstream gene set tests, particularly for the smaller RNA-seq groups.
-
Results are presented as ranked gene sets and visualizations rather than with numeric effect sizes and dispersion.↳ Could also: Reporting set-level effect summaries (e.g. avg.logfc) alongside confidence intervals or directional statistics for top sets could also accompany the rankings. — Quantitative effect and uncertainty summaries complement rank-based displays by conveying magnitude and stability of enrichment, not just ordering.
Result convergence & founder nodes
Findings this paper shares with others that ran a comparable experiment. A node’s strength is how many independent papers report it (replication breadth) — not how often it is cited, so a heavily-replicated but under-cited founder still stands out.
-
EGSEA integrates 11 base gene set testing methods (fry substituted for roast) to produce ensemble gene set enrichment scoresother mouse mammary-gland 2017×1papers★ This paper is the founder (earliest)
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
No assessed neighbours yet — the network grows as more papers are assessed.
Data lineage
The datasets this paper uses (text-mined from the full text via Europe PMC), and which other assessed papers stand on the same data. A shared dataset is a factual link — not a judgement.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-29333246 (EGSEA workflow, Alhamdoosh et al. 2017, F1000Research)
What the paper is
A Bioconductor workflow article (tutorial). It demonstrates the EGSEA package (Ensemble of Gene Set Enrichment Analyses) by running an end-to-end gene-set testing analysis on two companion datasets of the same 3 mouse mammary cell populations (Basal, LP, ML):
- RNA-seq — Sheridan et al. 2015, GEO GSE63310, preprocessed into a
DGEList by Law et al. 2016 (RNAseq123). Downloaded as
mam.rnaseq.rdata. - Microarray — Lim et al. 2010, GEO GSE19446 (Illumina MouseWG-6 v2),
distributed as
arraydata.zip(IDAT + BGX).
Code: companion package EGSEA123 (github.com/mritchie/EGSEA123, vignette
EGSEAWorkflow.Rmd). The actual computation is done by the EGSEA +
EGSEAdata + limma/edgeR Bioconductor packages.
Per BRIEF rule P16: reproducing by running these published Bioconductor tools on the paper's own data is fully valid — and is exactly what the workflow itself does.
In scope (pipeline-derived, reproduced)
All results in this workflow are pipeline-derived (every number is printed R output embedded in the article). We reproduce the clearly-specified, low-stochasticity ones first (80/20):
| id | result | determinism |
|---|---|---|
| C1 | RNA-seq DGEList dim 14165x9 | deterministic |
| C2a-e | indexed collection sizes (c2/c5BP/c5CC/c5MF/kegg) | deterministic (EGSEAdata version) |
| C6 | microarray probes retained = 7123 | deterministic |
| C3 | top-10 c2 comparative gene sets (LIM sets) | ensemble; partly stochastic |
| C4 | top KEGG pathways (BasalvsLP, comparison) | ensemble; partly stochastic |
| C5 | LIM gene-set ranking table (Rank/med.rank/vote.rank) | ensemble; partly stochastic |
Out of scope / not attempted (the hard ~20%)
- All figure-only outputs (heatmaps, KEGG pathview maps, MDS plots, summary bubble plots, GO graphs, HTML report) — visual, not pinnable numbers.
- The microarray EGSEA ensemble run (only its deterministic 7123-probe preprocessing anchor is reproduced; the full microarray gene-set ranking is the RNA-seq analysis repeated and adds little marginal evidence).
- Exact med.rank values for stochastic methods: EGSEA combines 11 base methods of which several (safe, padog, and permutation-based steps) draw random permutations and the vignette sets no seed, so byte-exact med.rank/p.adj are not expected. We grade the ranking structure (which sets land in the top-10, vote.rank) rather than chasing identical floats.
Pipelines named per result
- C1: limma/edgeR
calcNormFactors+voomon the shipped DGEList. - C2:
EGSEA::buildIdxover EGSEAdata MSigDB c2/c5 + KEGG. - C3-C5:
EGSEA::egseaensemble (camera, safe, gage, padog, plage, zscore, gsva, ssgsea, globaltest, ora, fry),sort.by="med.rank", thentopSets. - C6: limma
read.idat/neqc/detection-p filter + Entrez dedup.
Environment (run-time resolved)
«our HPC» SLURM, conda prefix env on «infra»: R 4.5.3, limma 3.66.0, edgeR 4.8.2, EGSEA 1.38.0, EGSEAdata 1.38.0 (newer than the 2017 paper's EGSEA>=1.5.2, but the EGSEAdata gene-set snapshot is unchanged — see exact C2 match).
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
Ran the published EGSEA Bioconductor workflow on the authors' own data (GSE63310 RNA-seq + GSE19446 microarray) with the same EGSEAdata MSigDB v5.2 (07-Mar-2017) snapshot. All 8 deterministic anchors are byte-exact (DGEList 14165x9; collection sizes 4726/4653/584/928/287; 7123 probes), the KEGG top pathways match identically and in order, and the LIM ensemble med.ranks (36,37,92,134,180) and vote.ranks reproduce exactly — with 2/5 p.adj values byte-identical. The only deviations are single-position rank swaps from unseeded stochastic base methods (safe/padog), i.e. expected RNG noise. No fabrication concern — a clean 1:1 reproduction.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.