miEAA 2.0: integrating multi-species microRNA enrichment analysis and workflow management systems.
The main results reproduced, with only marginal, non-material deviations.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
- ✓Same input data as the authors
- 🟡Reported values were only indirectly comparable
- 🔴A deviation arose in the data or preprocessing
- 🟡A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🔴The deviation was non-trivial in magnitude
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
miEAA 2.0 is a WEB tool; the shipped repo (Xethic/miEAA-API, commit 2ff258d) is the mieaa REST client, enrichment runs on the Saarland server which is now miEAA 2.1 (paper used v2.0). Reproduced Case Study 2 on the RU's named dataset GSE117000 (Agilent mouse miRNA V19, 36 samples) end-to-end on «our HPC»/«infra»: GEO download -> Agilent FE parse + log2/quantile (R/limma) -> detection filter -> miRBase v19->v22 conversion -> Cohen's-d effect-size ranking (premalignant 6wk vs invasive 24wk) -> GSEA via live API. RESULT = honest PARTIAL: C1 within-tol (214 vs 212; paper threshold unspecified), C2 EXACT (only mmu-miR-6243 drops in v19->v22), C4 PARTIAL (the paper's three named depleted GO/pathway terms - macrophage differentiation, VEGF signaling, vasculature development - all reproduce as depleted in NeuT+ at nominal p, exact p-values differ), C3 MISMATCH (the headline '311 sig categories NeuT+ vs none NeuT-' is NOT reproducible on miEAA 2.1: FDR<0.05 gives 3/2, nominal p gives 2268/2375, and the NeuT+>>NeuT- contrast inverts). The database is NOT depleted (15269-17202 mouse subcategories evaluated), so C3 divergence is attributable to v2.0->v2.1 GSEA/DB changes plus the paper's unspecified effect-size ranking, not to a non-derivable/fabricated value. Data-preprocessing + deterministic conversions reproduce cleanly; server-DB-dependent enrichment counts are heavily version-sensitive. NOT attempted (out of scope): CS1 TCGA-KIRC (different data), CS3 PPMI Parkinson's (access-restricted), TAM 2.0 comparison, GUI/workflow features. No fabrication indicators. Verdicts provisional; human review required (see AUDIT.md).
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 61assessed: 2026-06-16 ⛓ e55c2ae17497
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-16
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-18no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: opusThe paper presents miEAA 2.0, an updated miRNA enrichment analysis and annotation web tool, aiming to extend miRNA set enrichment analysis to multiple species, expand category sets, and integrate API/workflow management; it tests whether the tool can recover biologically meaningful enrichments across diverse case studies.
- ★ miEAA 2.0 supports miRNA/precursor enrichment input from ten frequently investigated organisms, expanding beyond the human-only first release. resource
- ★ The tool provides 134,525 categories from 16 published databases/resources to test against, an order-of-magnitude increase in category sets. resource
- ★ A standardized public API, a Python package, and a CLI tool were implemented to facilitate inclusion of miEAA in modern data analysis workflow systems. method
- ★ Both ORA and GSEA algorithms are offered, with an un-weighted GSEA variant (Kolmogorov-Smirnov test) enabling exact P-value computation without permutations. method
- ★ Novel categories such as annotation confidence level, cellular/sub-cellular localisation, and miRNA-transcription factor interactions were incorporated. resource
- The GeneTrail-based GSEA implementation was profiled and improved to be three times faster on average. method
- New visualizations including interactive heatmaps, word clouds, and GSEA running sum curves with simulated background distributions were added. method
- Six multiple-testing P-value correction methods and a P-value pooling approach were added for customizable stringency. method
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| miRNA-seq (small RNA sequencing) enrichment analysis (ORA and GSEA) | Human kidney renal clear cell carcinoma (KIRC), TCGA samples | none (tumor vs normal tissue comparison) | differentially expressed precursors and enriched/depleted miRNA categories | TCGA miRNA-seq, miRBase v21 precursor counts |
| miRNA microarray enrichment analysis | Mouse breast cancer progression model (NeuT+ vs NeuT- mice), GEO GSE117000, plasma/circulating miRNAs | NeuT oncogene mutation carrier vs wildtype | circulating miRNA enrichment, breast cancer category depletion in GSEA | Agilent microarray, miRBase v19 probes |
| Small non-coding RNA sequencing enrichment analysis | Human whole blood, Parkinson's Progression Markers Initiative (PPMI) cohort | none (Parkinson's disease vs age-matched controls) | miRNA precursor profiles and enrichment results compared to TAM 2.0 | sncRNA-seq, miRBase v22 precursors |
- – Of 1881 miRBase v21 precursors in KIRC, 321 were consistently detected in ≥50% of samples per biogroup and 282 were differentially expressed between primary tumor and normal (FDR P<0.01).
- ▲ ORA of KIRC precursors yielded 541 significantly enriched and seven significantly depleted categories (FDR P<0.05), with top categories associated with cancer including renal cell carcinoma.
- ▲ Strong enrichment of de-regulated KIRC precursors with kidney and other cancers (observed/expected ratio 123/48.6). 123/48.6 ratio
- – GSEA of KIRC detected precursors sorted by effect size revealed 253 enriched and 40 depleted categories; miRNA gene cluster on X chromosome was the most depleted category.
- ▼ KIRC GSEA identified the X-chromosome miRNA gene cluster (147,189,704:147,284,728) as most depleted, consistent with depletion of precursor family hsa-mir-506. P = 8.64 × 10^-10
- – miEAA 2.0 KIRC dataset comprised 591 human miRNA-seq samples: 520 primary tumor and 71 solid tissue normal.
- – PPMI case study compared 2337 Parkinson's samples to 1538 age-matched controls quantified from 4340 sequencing samples across 1600 individuals.
- ▼ Mouse breast cancer dataset comprised 36 samples across premalignant, preinvasive, and invasive stages, with GSEA showing depletion for breast cancer.
- pvalue P = 2.80 × 10^-38 (enrichment of de-regulated KIRC precursors with kidney and other cancers)
- pvalue P = 8.64 × 10^-10 (X-chromosome miRNA gene cluster as most depleted category in KIRC GSEA)
- count 134525 categories from 16 databases/resources (total enrichment categories available in miEAA 2.0)
- count 541 enriched, 7 depleted categories (FDR P<0.05) (KIRC over-representation analysis result)
- count 282 differentially expressed precursors (FDR P<0.01) (KIRC primary tumor vs normal Wilcoxon test)
- count 253 enriched, 40 depleted categories (KIRC GSEA sorted by effect size)
- count 1238 human miRNA categories (TAM 2.0 categories from ~9000 manuscripts (comparison tool))
- count 591 samples (520 PT, 71 STN) (KIRC TCGA miRNA-seq sample composition)
Statistical methods review
Model: opusA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a web-server/tool paper (miEAA 2.0) whose statistical content centers on miRNA set enrichment methods rather than a single hypothesis-driven experiment. The tool offers over-representation analysis (ORA) and an un-weighted gene set enrichment analysis (GSEA) corresponding to a Kolmogorov–Smirnov test that yields exact P-values without permutation, with six selectable multiple-testing correction procedures. In the demonstrative case studies, differential expression between groups was assessed with FDR-adjusted Wilcoxon tests, effect sizes were computed as Cohen's d, and enrichment results were reported as P-values (and observed/expected ratios) after Benjamini–Hochberg FDR adjustment.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| Un-weighted GSEA corresponding to a Kolmogorov–Smirnov test (exact P-value, no permutation) | Core GSEA enrichment algorithm of miEAA; e.g. miRNA set enrichment ordered by effect size in case study 1 and breast-cancer depletion in case study 2 | — | not stated |
| Over-representation analysis (ORA) | Enrichment of de-regulated precursors into categories; case study 1 (KIRC) yielded 541 enriched and 7 depleted categories | — | not stated |
| Wilcoxon test (FDR-adjusted) | Differential expression between Primary tumor (PT) and Solid tissue normal (STN) samples in case study 1; 282 of 321 precursors differentially expressed at P<0.01 | based on 591 KIRC samples (520 PT, 71 STN); 321 consistently detected precursors | not stated |
-
Differential expression between groups in the case studies was assessed with the Wilcoxon (rank-based) test with FDR adjustment.↳ Could also: A dedicated count-based model such as DESeq2 or edgeR (negative-binomial) for the sequencing data, or limma/limma-voom for the microarray and normalized data. — These models explicitly account for the mean–variance relationship of count data and can borrow information across features, which can improve sensitivity and calibration, particularly when group sizes are uneven (e.g. 520 vs 71).
-
An un-weighted GSEA (Kolmogorov–Smirnov-style) producing exact P-values without permutation is provided.↳ Could also: A weighted enrichment statistic with permutation-based null distributions (as in the classic Subramanian et al. GSEA). — Weighting by the ranking metric and permutation nulls can emphasize strongly ranked features and provide an empirical null that reflects inter-feature correlation, offering a complementary view to the closed-form exact test.
-
Effect sizes were summarized using Cohen's d.↳ Could also: Reporting Cohen's d together with its 95% confidence interval, or a rank-based effect size (e.g. Cliff's delta) consistent with the Wilcoxon test used. — A confidence interval conveys the precision of the estimate, and a rank-based effect measure aligns the effect-size metric with the non-parametric test, which can be helpful for non-normal expression distributions.
-
Multiplicity was controlled per category set (database/category-set-wise) by default, with an optional pooling approach.↳ Could also: Applying a single FDR correction pooled across the entire family of tested categories. — A globally pooled correction controls the overall family-wise/false-discovery error across all categories simultaneously, which some workflows prefer for a uniform stringency across the whole analysis.
-
Case-study results were reported primarily as enrichment P-values and observed/expected ratios.↳ Could also: Additionally reporting measures of spread or uncertainty (e.g. confidence intervals on the observed/expected ratio) alongside the P-values. — Interval estimates communicate the magnitude and precision of enrichment, complementing significance thresholds and aiding interpretation across categories.
Result convergence & founder nodes
Findings this paper shares with others that ran a comparable experiment. A node’s strength is how many independent papers report it (replication breadth) — not how often it is cited, so a heavily-replicated but under-cited founder still stands out.
-
GSEA of circulating miRNAs shows depletion of the breast cancer category across mouse mammary tumor progressionmicroarray mouse plasma down 2020×1papers★ This paper is the founder (earliest)
-
De-regulated KIRC precursors strongly enriched for kidney and other cancers (observed/expected 123/48.6)RNA-seq human kidney up 2020×1papers★ This paper is the founder (earliest)
-
X-chromosome miRNA gene cluster (hsa-mir-506 family) is the most depleted category in KIRC GSEARNA-seq human kidney down 2020×1papers★ This paper is the founder (earliest)
-
GSEA of KIRC precursors reveals 253 enriched and 40 depleted miRNA categoriesRNA-seq human kidney mixed 2020×1papers★ This paper is the founder (earliest)
-
282 miRNA precursors differentially expressed between KIRC primary tumor and normal tissueRNA-seq human kidney mixed 2020×1papers★ This paper is the founder (earliest)
-
ORA of de-regulated KIRC miRNA precursors yields significant enrichment of cancer/renal cell carcinoma categoriesRNA-seq human kidney up 2020×1papers★ This paper is the founder (earliest)
Citation network
Where this publication sits in the reproducibility-weighted citation graph — what it is built on, and what is built on it. Citation data from OpenAlex.
Assessed papers, coloured by verdict. Click a node to open it.
- No assessed neighbours yet — the network grows as more papers are assessed.
Data lineage
The datasets this paper uses (text-mined from the full text via Europe PMC), and which other assessed papers stand on the same data. A shared dataset is a factual link — not a judgement.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
scope.md — pmid-32374865 (miEAA 2.0)
Nature of the paper
miEAA 2.0 (Kern et al., NAR 2020) is a web-server tool for microRNA
enrichment analysis. It is a "tools" paper: the contribution is the server +
database, not a single computational study. The shipped "code"
(github.com/Xethic/miEAA-API) is a thin REST client (Python mieaa
package / R package); the enrichment computation itself runs server-side at
https://ccb-compute2.cs.uni-saarland.de/mieaa2/api/.
Per BRIEF rule P16, applying this tool to the paper's own data is a valid, equally-weighted reproduction. We run the tool (and the GEO-data preprocessing pipeline that feeds it) on the paper's named dataset.
Pipeline-derived results (IN SCOPE)
All from Case Study 2 = GSE117000 (MMTV-NeuT breast-cancer mouse model, Agilent-046065 Mouse miRNA V19 microarray, 36 samples), the RU's named accession.
| # | reported result | pipeline | server-dependent? |
|---|---|---|---|
| C1 | 212 miRNAs after detection filtering | Agilent FE parse → quantile-norm/log2 → detection threshold (R/limma) | no |
| C2 | mmu-miR-6243 dropped, miRBase v19→v22.1 | miEAA convert_mirbase |
weakly (uses server mapping tables, deterministic) |
| C3 | NeuT+ GSEA = 311 sig categories; NeuT- = none | rank by effect size (premalig vs invasive) → miEAA run_gsea (mmu, precursor, FDR, 0.05) |
YES (v2.0→v2.1 DB drift) |
| C4 | depleted: Macrophage differentiation 2.54e-5, Vasculature development 1.60e-4, VEGF signaling 0.0016 | same GSEA run | YES |
OUT OF SCOPE (not attempted, with reason)
- CS1 (TCGA-KIRC): different dataset (TCGA, 591 samples); not the RU accession.
- CS3 (PPMI Parkinson's): PPMI data is access-restricted (application required) — cannot fetch; also TAM 2.0 comparison is external-tool/manual.
- Database content claims (134,525 categories / 16 resources): a static
catalog number, partially checkable via
get_enrichment_categoriesbut is the v2.1 catalog, not a pipeline output of CS2. Recorded as context only. - GUI / workflow-management (Snakemake/Galaxy) features: not computational outputs.
Reproduction strategy
Heavy compute (GEO download, Agilent R processing) on «our HPC» SLURM/«infra». miEAA API calls issued from the «our HPC» compute node (has internet) to the live server. Server is miEAA 2.1, not the 2.0 of the paper → category counts (C3) and p-values (C4) are expected to drift; we grade those as provisional and report the qualitative reproduction (NeuT+ strongly enriched/depleted vs NeuT- ~null, and whether the named pathways recur).
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Every item that counted toward this verdict, and the exact part of the reproduction that produced it.
Deterministic, server-independent steps reproduce cleanly — C1 within tolerance (214 vs 212; threshold unspecified) and C2 exact (mmu-miR-6243 the only v19->v22 dropout) — and C4's depleted biology (VEGF/macrophage/vasculature) recurs directionally in NeuT+. The substantive deviation is C3: the headline 311-significant-vs-none contrast is not recovered and even inverts on miEAA 2.1 (FDR 3/2; nominal 2268/2375), a severe direction change. This sits in server-side GSEA computation and is attributable to documented v2.0->v2.1 DB/version drift plus the paper's unspecified ranking/threshold, not to a non-derivable or fabricated value, so overall this is a solid reproduction with explainable, version-driven deviations.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.