Equivalent change enrichment analysis: assessing equivalent and inverse change in biological pathways between diverse experiments.
Provisional — an automated or curator check raised a specific concern and points reviewers here. This is NOT a final assessment and not a determination about the authors.
The main results reproduced, with only marginal, non-material deviations.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Same input data as the authors
- ✓Reported values were directly comparable
- ✓No authors-side cause for any deviation
- ✓Reported values are derivable from the shared data
- ✓The central claim held under reproduction
- 🟡A deviation arose in the data or preprocessing
- 🟡The deviation was non-trivial in magnitude
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
Described well enough to reproduce. The paper's central computational result (Table 1: KEGG pathways inversely enriched between adipose Glut4 knockout and overexpression, GSE35378) reproduced strongly 1:1 in identity and direction: 4 of 5 reported inverse pathways recovered (Valine/Leu/Ile degradation, Propanoate metabolism, Fatty acid degradation, Neuroactive ligand-receptor) with NES within 0.03-0.12 of the printed values and matching gene-set sizes, starting from the raw CEL files. The 5th (Lysine degradation) came out same-sign (NES<0) but not significant. ECI semantics (+1 equivalent / -1 inverse, bounded [-1,1]) reproduced exactly. Divergences are confined to DB-version-dependent quantities: exact FDR, total pathway counts (KEGG 225->345, Reactome 1647->1815), and the Reactome enriched/inverse counts (27/8 reported vs 92/12 here) all differ because current KEGG/Reactome releases are larger than the 2019 versions and fgsea version/seed differ. No fabrication indicators: all Table-1 values are reconstructable from the shipped data+code. NOT attempted: the antidepressant dataset (Bagot et al., no GEO accession in this RU's brief), the simulation study (generator not shipped), and the GSEA-intersection baseline (not the ECEA pipeline). Notable engineering blocker, worked around honestly: affy::rma and preprocessCore both abort on «our HPC» with pthread_create EINVAL (rc22) even single-CPU and with --disable-threading, so RMA was reimplemented in pure base R (convolution bg-correct + average-rank quantile-normalize + median-polish) = numerically standard RMA. The authors' shipped vignette (GEO series matrix, not the manuscript normalization) was also run verbatim and reproduces deterministically, but as the vignette author explicitly warns, it does not match the paper's numbers (it is dominated by ribosome/translation equivalent changes); that is expected and documented, not a discrepancy in the method.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 62assessed: 2026-06-16 ⛓ c40cae811d76
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-16
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-18no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusThe paper asks whether there is a statistically sound method to identify gene pathways that are equivalently (or inversely) changed across two different experimental conditions or treatments, addressing the lack of robust statistical approaches for comparing functional disruptions between a dataset and reference data.
- ★ The Equivalent Change Index (ECI), a gene-level statistic ranging from -1 to 1, quantifies whether a gene was changed to the same (1) or completely opposite (-1) degree across two experiments relative to their controls. method
- ★ Equivalent Change Enrichment Analysis (ECEA) is a functional genomics approach that identifies pathways with a non-random distribution of equivalently or inversely changed genes using the ECI. method
- ★ ECEA recovers pathways exhibiting equivalent or inverse changes even with complex (non-co-expressed) regulation patterns that GSEA and ORA intersection approaches cannot detect. finding
- ★ On simulated data, ECEA outperformed GSEA and ORA for detecting equivalently and inversely changed pathways across most levels of probability of differential expression and symmetry. finding
- ★ ECEA maintains a low false positive rate for pathways with differential expression but no enforced equivalent/inverse change, unlike GSEA whose FPR increases with probability of differential expression at extreme symmetry. finding
- ★ On biological data (Glut4 knockout/overexpression and antidepressant treatments), ECEA recovered pathways directly connected to the condition being studied. finding
- Existing intersection approaches (GSEA/ORA performed separately then intersected) cannot assess the statistical significance of equivalent/inverse pathway change and assume co-expression of pathway genes. finding
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| Simulation of gene expression / differential expression | simulated data (10 pathways: 1 equivalent, 1 inverse, 1 differentially expressed unrelated, 7 unaffected) | simulated treatment vs control | proportion of pathways detected as equivalently/inversely changed; false positive rate | — |
| Gene expression / RNA microarray differential expression analysis | mouse white adipose tissue (Glut4 data) | Glut4 knockout and Glut4 overexpression (vs respective controls) | log2 fold change, ECI, pathway enrichment (KEGG and Reactome) | limma R package |
| Gene expression differential expression analysis | mouse model of depression | antidepressant drugs ketamine and imipramine (vs control) | log2 fold change, ECI, pathway enrichment for equivalent change (KEGG and Reactome) | limma R package |
- ▼ ECEA identified 8 enriched KEGG pathways in the Glut4 data, 5 of which were enriched for inversely changed genes 8 pathways (5 inverse)
- ▼ ECEA identified 27 significantly enriched Reactome pathways in Glut4 data, 8 inversely enriched 27 pathways (8 inverse)
- ▲ In the antidepressant data, ECEA found 6 KEGG pathways significantly enriched for equivalent change and none for inverse change 6 pathways
- ▲ In the antidepressant data, ECEA found 17 Reactome pathways enriched for equivalent changes and none for inverse changes 17 pathways
- – ECEA was more sensitive than ORA and GSEA in detecting equivalently/inversely changed pathways, except GSEA outperformed ECEA for equivalent change only at low PDE and extreme symmetry
- – Three pathways identified as inversely regulated by ECEA in Glut4 KEGG were also found by the GSEA intersection approach 3 of pathways
- – VEGFR2 mediated vascular permeability pathway (Nos3, Akt2, Calm2, Pdpk1) was identified as inversely changed by ECEA but not GSEA due to mixed up/down regulation
- other FDR cut-off of 0.25 (recommended GSEA threshold used for ECEA enrichment)
- fold_change NES = -2.08 (mmu00280 Valine, leucine and isoleucine degradation, Glut4 KEGG inverse, FDR 1.85e-2)
- count N = 20 (simulation: 5 treatment + 5 control per treatment)
- count 225 KEGG pathways; 1647 Reactome pathways (total pathways available in each database)
- pvalue FDR = 1.01e-2 (mmu03010 Ribosome, antidepressant KEGG equivalent, NES 1.52)
- count 32 (KO) and 29 (overexpressed) significantly enriched GSEA pathways; 11 shared (6 inverse, 5 equivalent) (GSEA intersection approach on Glut4 KEGG)
- other probability of equivalent/inverse change = 0.5 (100 simulations per parameter set)
- fold_change NES = 1.74 (Sulfur amino acid metabolism, antidepressant Reactome equivalent, FDR 8.07e-3)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
The paper introduces ECEA (Equivalent Change Enrichment Analysis) and the Equivalent Change Index (ECI), a novel gene-set enrichment framework for detecting pathways equivalently or inversely changed across two experiments. Performance was benchmarked against GSEA and ORA via simulation (100 runs per parameter set; N=20 per simulation) using sensitivity and false positive rate as evaluation metrics. The method was applied to two biological mouse datasets (Glut4 knockout/overexpression; ketamine vs. imipramine antidepressant treatment), with log2 fold changes estimated using limma and pathway significance reported as FDR-corrected Normalized Enrichment Scores (NES).
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| ECEA permutation-based enrichment test (novel ECI-based statistic, analogous to GSEA machinery; reports NES and FDR) | Primary method: detection of equivalently or inversely changed KEGG and Reactome pathways in Glut4 and antidepressant biological datasets | Not explicitly stated for biological datasets; N=20 (5 samples per treatment + 5 controls across two experiments) for simulations | not stated |
| Gene Set Enrichment Analysis (GSEA/fgsea) | Benchmarking comparator in simulations and biological datasets; intersection approach used to approximate equivalent/inverse pathway detection | N=20 for simulations; not stated for biological datasets | not stated |
| Over-representation analysis (ORA; hypergeometric test implied) | Benchmarking comparator in simulations only | N=20 for simulations | not stated |
| limma moderated t-statistic (log2 fold change estimation) | Pre-processing step for both biological datasets; fold changes serve as input to ECEA and GSEA | not stated | not stated |
-
Simulation sensitivity and FPR were reported as raw proportions across 100 runs with no measure of uncertainty↳ Could also: Report binomial confidence intervals (e.g., Wilson or Clopper-Pearson 95% CIs) around each proportion estimate — Confidence intervals would quantify the Monte Carlo sampling uncertainty in each estimated proportion, making method comparisons more formally interpretable — particularly when sensitivity curves for different methods are close together
-
Method performance was summarized by sensitivity and FPR at a single fixed FDR threshold (0.25) across simulation parameters↳ Could also: Summarize discriminative performance using the area under the ROC curve (AUROC) or precision-recall curve across a range of decision thresholds — AUROC provides a threshold-independent summary of the sensitivity/specificity tradeoff, complementing the fixed-threshold comparisons shown and making performance less dependent on the specific FDR cutoff chosen
-
The GSEA intersection approach used as a comparator has no formal significance test for the overlap of enriched pathways across experiments↳ Could also: Apply a permutation test or Fisher's exact test to assess whether the observed intersection of enriched pathways across two experiments exceeds chance expectation — A formal significance test for the intersection would produce a p-value for co-enrichment, enabling a more direct statistical comparison with ECEA's per-pathway FDR values and reducing reliance on informal interpretation of overlap counts
-
Log2 fold changes for both biological datasets were computed using the limma package without specifying the underlying data type↳ Could also: For RNA-seq count data, also consider limma-voom, DESeq2, or edgeR, which use count-aware variance models — DESeq2 and edgeR apply negative binomial models that explicitly account for count overdispersion; the choice of DE framework can affect fold-change point estimates and their precision, which propagate into ECI values and downstream ECEA results
-
Simulations used a single fixed sample size of N=20 (5 per group per experiment) across all parameter combinations↳ Could also: Vary sample size as an additional simulation factor (e.g., N=10, 20, 40, 80) — Characterizing how sensitivity and FPR scale with sample size would provide practically useful guidance for researchers designing studies who wish to apply ECEA, and would show whether the power advantage over GSEA/ORA is maintained at smaller or larger N
-
Pathway annotations were evaluated using KEGG and Reactome; no GO or curated hallmark gene sets were included↳ Could also: Also apply ECEA to Gene Ontology (GO) Biological Process terms or MSigDB hallmark gene sets — GO and MSigDB hallmark sets are widely used in the field and have different annotation granularity from KEGG/Reactome; including them would broaden demonstration of generalizability and allow practitioners to see expected behavior across the annotation frameworks they most commonly use
Result convergence & founder nodes
Findings this paper shares with others that ran a comparable experiment. A node’s strength is how many independent papers report it (replication breadth) — not how often it is cited, so a heavily-replicated but under-cited founder still stands out.
-
VEGFR2-mediated vascular permeability pathway (NOS3, AKT2, CALM2, PDPK1) is inversely regulated between Slc2a4 knockout and overexpression conditions in mouse WAT; detected by ECEA but missed by GSEA due to mixed intra-pathway gene directionsmicroarray mouse white adipose tissue mixed 2020×1papers★ This paper is the founder (earliest)
-
ECEA identified 27 enriched Reactome pathways comparing Slc2a4 knockout versus overexpression in mouse WAT, 8 of which showed inverse change between conditionsmicroarray mouse white adipose tissue mixed 2020×1papers★ This paper is the founder (earliest)
-
ECEA identified 17 Reactome pathways equivalently enriched between ketamine and imipramine treatment in a mouse depression model, with no pathways showing inverse changemicroarray mouse up 2020×1papers★ This paper is the founder (earliest)
-
ECEA showed greater sensitivity than ORA and GSEA for detecting equivalently and inversely changed pathways in simulation, except GSEA outperformed ECEA for equivalent-change detection at low pathway differential expression with extreme gene-level symmetryother simulated-data mixed 2020×1papers★ This paper is the founder (earliest)
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
No assessed neighbours yet — the network grows as more papers are assessed.
Data lineage
The datasets this paper uses (text-mined from the full text via Europe PMC), and which other assessed papers stand on the same data. A shared dataset is a factual link — not a judgement.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-32093613 (ECEA, Thompson & Koestler 2020, BMC Genomics)
Tool: github.com/jeffreyat/ECEA @ commit 49fae1f47ec6dcb363d27cf3a41805fd94d9fd81 (2019-12-11).
Data: GEO GSE35378 (Affymetrix MG-U74Av2; 12 mouse adipose arrays:
3 Glut4-overexpressor + 3 OE-control + 3 Glut4-knockout + 3 KO-control).
In scope (pipeline-derived; attempted)
| Result | Pipeline | Status |
|---|---|---|
| ECI definition/range ([-1,1], +1 equiv, -1 inverse) | getECI() |
reproduced EXACT (unit test) |
| Glut4 Table 1 — KEGG inversely-enriched pathways (NES, Size) | RMA(CEL)→limma→ECI→doECEA(fgsea, KEGG) |
reproduced 4/5 within-tol |
| Glut4 Reactome — N enriched / N inverse | same, Reactome sets via getReactome('mouse') |
partial (qualitative; counts differ, DB drift) |
| ECEA software runs end-to-end on the paper's data | authors' shipped vignette (GEO series matrix) | reproduced (deterministic) — but author states it ≠ manuscript normalization |
Pipelines named: Affymetrix RMA normalization (oligo/affy in the paper; reimplemented pure-R here), limma differential expression, ECEA = ECI + fgsea (weighted KS permutation enrichment).
Out of scope / not attempted
- Antidepressant dataset (Bagot et al.; ketamine/imipramine RNA-seq, prefrontal cortex) — Tables 5/6. Different dataset, no GEO accession in this RU (brief data = GSE35378 only). Not attempted.
- Simulation study (100 sims/parameter set; ROC-type performance) — synthetic-data generator not shipped in the repo; no specific pinnable reported value. Not attempted.
- GSEA-intersection comparison columns (Tables 2/6) — the paper's alternative ad-hoc baseline, not the ECEA pipeline. Not attempted.
- Wet-lab / phenotype statements — non-computational. Out of scope.
Note on P16 (third-party tool)
ECEA is the authors' own package, applied here to the paper's own data per the described parameters (FDR<0.25, minSize=10, maxSize=500, nperm=1000, p-weighted ECI) — a valid pipeline reproduction.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
This is a methods/tool paper whose central result (Table 1: KEGG pathways inversely enriched between Glut4 knockout and overexpression, GSE35378) reproduced strongly from raw CEL files: 4 of 5 inverse pathways recovered with matching identity, matching sign, and NES within 0.03–0.12 of the printed values, and ECI semantics (+1/-1) exact. All deviations are on technical/version grounds, not the authors' side — current KEGG/Reactome releases are larger than the 2019 versions, inflating total counts (225→345, 1647→1815) and the Reactome enriched/inverse counts (27/8→92/12), and fgsea seed/version shifts FDRs. No fabrication indicators: every Table-1 value is derivable from the shipped data+code. Overall yellow because the count-level mismatches and one non-significant pathway (mmu00310) are real but fully explainable.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.