SMAGEXP: a galaxy tool suite for transcriptomics data meta-analysis.
The main results reproduced: recomputed values matched the published ones within tolerance.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Same input data as the authors
- ✓Reported values were directly comparable
- ✓No authors-side cause for any deviation
- ✓Reported values are derivable from the shared data
- ✓Any deviation was negligible
- ✓The central claim held under reproduction
- 🟡A deviation arose in the data or preprocessing
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
Described well enough to reproduce: SMAGEXP is a Galaxy tool-suite paper (wraps metaMA/metaRNASeq/limma/DESeq2); its only pipeline-derived numbers live in Fig6 (GSE3524 limma top-10) and Fig7 (metaMA indicators for GSE3524+GSE13601), recovered from the repo test-data and the authors' shipped Galaxy history (both identical to the figures). Reproduced by running the AUTHORS' OWN tool R scripts standalone on «our HPC» SLURM with the exact CLI args from the Galaxy provenance; GSE3524 fetched fresh via getGEO, GSE13601 from the 6 shipped CELs (HG-U95Av2). 1:1 RESULT: Claim A (Fig6 limma top-10) reproduced EXACTLY (byte-identical). Claim B (Fig7 metaMA DE/IDD/Loss/IDR/IRR) reproduced within-tol at ~5-8% (515/156/599/30.29/62.53 vs reported 539/169/566/31.35/60.47, 7768 genes in common). metaMA on the authors' own intermediate esets and on our full re-run gave BYTE-IDENTICAL output, proving the upstream limma reproduces faithfully and isolating the residual gap to metaMA's cross-platform mapping via org.Hs.egUNIGENE. That mapping is a genuine reproducibility-decay barrier: NCBI retired UniGene (2019) and current AnnotationDbi no longer builds org.Hs.egUNIGENE for any org.Hs.eg.db version (verified 3.4.1/3.5.0/3.7.0/3.18); we restored it from the archived org.Hs.eg.db 3.7.0 sqlite 'unigene' table, so the authors' exact UniGene snapshot is unrecoverable -> Fig7 cannot be bit-exact on a modern stack. NOT a fabrication signal; every reported value is derivable from the shipped data+code (possible-fabrication note: NONE). Env fix worth noting: rma() failed with pthread_create rc22, fixed by rebuilding preprocessCore --disable-threading. NOT ATTEMPTED (80/20): the RNA-seq use-case (metaRNASeq on recount2 SRP032833/SRP028180/SRP058237 -> DESeq2; no reported numbers) and plot-only outputs. All grades provisional for human review.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 82assessed: 2026-06-14 ⛓ 418ebdd55bf7
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-14
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-15no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusThe paper does not test a scientific hypothesis; it addresses the technical need for an easy-to-use, unified platform to perform meta-analysis of gene expression microarray and RNA-seq data without requiring R coding skills.
- ★ SMAGEXP integrates the metaMA and metaRNASeq R packages into Galaxy to provide a unified tool suite for transcriptomics meta-analysis. resource
- ★ The suite performs microarray meta-analysis (via metaMA) and RNA-seq meta-analysis (via metaRNASeq/DESeq2) using technology-appropriate statistical models. method
- ★ Microarray and NGS data require distinct statistical modeling (Gaussian vs negative binomial) and distinct handling of directional conflicts, so the two packages are not interchangeable. mechanism
- ★ Technology-independent quality indicators (DE, IDD, Loss, IDR, IRR) are reported to judge meta-analysis quality. method
- ★ metaMA combines limma P values via the inverse normal method; metaRNASeq combines DESeq2 P values via inverse normal and Fisher methods. method
- Integration into Galaxy plus a dockerized instance provides reproducibility, accessibility, and easy deployment. resource
- The suite can fetch and process data directly from GEO (microarray) and recount2 (RNA-seq) databases. method
- SMAGEXP is available on the Galaxy main toolshed, Docker Hub, and GitHub under MIT license. resource
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| Gene expression microarray meta-analysis | Human oral squamous cell carcinoma datasets GSE3524 and GSE13601 | none | Differentially expressed genes and meta-analysis indicators (DE, IDD, Loss, IDR, IRR) | metaMA R package within Galaxy |
| Affymetrix microarray quality control and normalization | Human oral SCC GSE13601 (6 .CEL files GSM342582-GSM342587) | none | Normalized expression data, microarray images, box plots, MA plots | QCNormalization tool (rma/quantile/background/log2) |
| limma differential expression analysis (single study) | GEO microarray data (e.g. GSE3524) | two-condition comparison | DE gene table, volcano plot, P value histogram, box plots | limma R package within Galaxy |
| RNA-seq differential expression analysis (single study) | Human lung SCC datasets SRP032833, SRP028180, SRP058237 from recount2 | invasive vs normal (SRP032833); tumor vs normal (SRP028180); tumor vs adjacent (SRP058237) | Differential expression results per study | DESeq2 Galaxy tool |
| RNA-seq meta-analysis | Human lung SCC datasets SRP032833, SRP028180, SRP058237 | none | DE genes, indicators (DE, IDD, Loss, IDR, IRR), conflict detection via signFC column, Venn/UpSet plots | metaRNASeq R package within Galaxy (Fisher and inverse normal) |
- – Microarray meta-analysis of GSE3524 and GSE13601 produced DE gene lists with a Venn diagram and quality indicators (DE, IDD, Loss, IDR, IRR).
- – RNA-seq meta-analysis of three lung SCC datasets produced DE genes, an UpSet plot, and indicators for both Fisher and inverse normal combinations.
- – Potential conflicts between single RNA-seq analyses were flagged by zero values in the signFC column.
- ▲ IDD identifies genes declared DE in meta-analysis that were not significant in any single study, demonstrating integration-driven gains.
- count 2 GEO datasets (GSE3524, GSE13601) (Microarray meta-analysis example datasets)
- count 6 .CEL files (GSM342582 to GSM342587) (Subset of GSE13601 used for QC/normalization)
- count 3 Recount2 datasets (SRP032833, SRP028180, SRP058237) (RNA-seq meta-analysis example datasets)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This technical note describes SMAGEXP, a Galaxy-integrated tool suite for gene expression meta-analysis across microarray and RNA-seq platforms. For microarray data, single-study differential expression is estimated with limma (empirical-Bayes moderated t-statistics) and study-level P-values are combined via the inverse normal method implemented in metaMA. For RNA-seq data, DESeq2 (negative binomial model, Wald test) provides single-study P-values, which are then combined using either the inverse normal or Fisher method in metaRNASeq. Results are reported as ranked lists of differentially expressed genes sorted by Benjamini-Hochberg adjusted P-values, accompanied by meta-analytic quality indicators (IDD, IDR, loss, IRR).
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| limma empirical-Bayes moderated t-test | Single-study differential expression analysis of microarray data (GSE3524, GSE13601 oral SCC datasets) | — | not stated |
| Inverse normal P-value combination (metaMA) | Microarray meta-analysis combining limma-derived P-values across studies | — | not stated |
| DESeq2 Wald test (negative binomial model) | Single-study differential expression analysis of RNA-seq data (SRP032833, SRP028180, SRP058237 lung SCC datasets) | — | not stated |
| Inverse normal P-value combination (metaRNASeq) | RNA-seq meta-analysis combining DESeq2 P-values across studies | — | not stated |
| Fisher P-value combination method (metaRNASeq) | RNA-seq meta-analysis combining DESeq2 P-values across studies (offered as alternative to inverse normal) | — | not stated |
-
Both microarray and RNA-seq meta-analyses use P-value combination methods (inverse normal, Fisher) as the primary approach to pooling evidence across studies↳ Could also: Effect-size-based meta-analysis combining standardized mean differences or log fold-changes under a random-effects model (e.g., DerSimonian-Laird) — Effect-size combination yields a pooled estimate of the magnitude of differential expression and its confidence interval, and directly supports estimation of between-study heterogeneity (τ², I²) — information that P-value combination methods do not provide on their own
-
Between-study heterogeneity is characterised via the tool's custom indicators (IDD, IDR, loss, IRR) and post-hoc conflict flagging in metaRNASeq, rather than a formal heterogeneity statistic↳ Could also: Cochran's Q test and the I² index to quantify and report between-study variability in effect direction and magnitude — Standardised heterogeneity statistics give readers a universally interpretable measure of between-study consistency and are a routine reporting element in clinical and genomic meta-analyses (e.g., PRISMA-P guidelines)
-
Single-study RNA-seq differential expression is performed exclusively with DESeq2 in the implemented pipeline↳ Could also: edgeR (quasi-likelihood F-test or likelihood ratio test) or voom+limma as single-study DE methods feeding into metaRNASeq — edgeR and voom+limma use distinct dispersion estimation strategies; providing these as selectable options would allow assessment of whether meta-analytic findings are robust to the choice of single-study method
-
The microarray meta-analysis pipeline applies a fixed (inverse normal) P-value combination, which assumes homogeneous effects across studies↳ Could also: Adaptively weighted Fisher combination or other random-effects P-value combination methods that allow for between-study variability — Random-effects P-value combination methods can be more appropriate when studies differ in design, microarray platform, or biological population, and may better reflect uncertainty about the true pooled signal
-
Results are visualized with Venn diagrams or UpSet plots showing set overlaps of DE genes across studies and the meta-analysis↳ Could also: Per-gene forest plots displaying study-level and pooled effect sizes (log fold-changes with standard errors) for top-ranked genes — Forest plots allow readers to see the direction, magnitude, and per-study uncertainty of each gene's contribution to the meta-analytic result, providing richer interpretive context than set-overlap diagrams alone
-
No sample size justification or sensitivity analysis is reported for the illustrative examples↳ Could also: Leave-one-out sensitivity analysis or simulation-based power assessment showing how the number of included studies or per-study sample size affects the meta-analytic result — Sensitivity analyses help characterise robustness of combined findings to individual study contributions, which is particularly informative in small meta-analyses (here two or three studies) where a single study can substantially influence results
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
No assessed neighbours yet — the network grows as more papers are assessed.
Data lineage
The datasets this paper uses (text-mined from the full text via Europe PMC), and which other assessed papers stand on the same data. A shared dataset is a factual link — not a judgement.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
A clean reproduction limited only by reproducibility decay. Running the authors' own SMAGEXP R scripts standalone: Fig 6 (GSE3524 limma top-10) reproduces byte-identical (SEL1L3 B=13.33, logFC=3.46, P=2.3e-10), and Fig 7 metaMA indicators reproduce within ~5-8% (515/156/599/30.29/62.53 vs 539/169/566/31.35/60.47). The residual is fully explained and isolated: metaMA's cross-platform mapping needs org.Hs.egUNIGENE, but NCBI retired UniGene (2019) and current AnnotationDbi cannot rebuild that Bimap — so the authors' exact UniGene snapshot is unrecoverable. metaMA on the authors' esets and on the full re-run produced byte-identical output, proving the upstream limma is faithful. No fabrication signal — every value is derivable from shipped data+code.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.