EpiDiverse Toolkit: a pipeline suite for the analysis of bisulfite sequencing data in ecological plant epigenetics.
Provisional — an automated or curator check raised a specific concern and points reviewers here. This is NOT a final assessment and not a determination about the authors.
The main results reproduced, with only marginal, non-material deviations.
- ✓Same input data as the authors
- ✓Reported values were directly comparable
- ✓Any deviation was negligible
- 🟡A deviation arose in the data or preprocessing
- 🟡A deviation was attributed to the published material
- 🟡Reported values were not (fully) derivable from the shared data
- 🟡The central claim did not (fully) hold under reproduction
- 🟡Overall, the reproduction showed a material discrepancy
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
PARTIAL reproduction (described well enough to reproduce the per-sample pipeline; the paper is a well-documented software paper). EpiDiverse 'direct chain' = the exact pipeline tools (erne-bs5 align, MethylDackel extract, change_sam_queries.py bisulfite masking, freebayes) run by hand on Populus nigra sample LT_06, because the EpiDiverse Nextflow would not run cleanly on «our HPC». S3 (Table S3) GRADED: coverage 37.54x vs 39.30x (-4.5%), insert 294.8 vs 288 (+2.4%), GC 18.25% vs 19.56% (-6.7%) all WITHIN-TOL; mapping 85.58% vs 80.27% PARTIAL (erne-bs5-only all-mapped vs paper's erne-bs5+segemehl merged+dedup; unique-only ~73.5% brackets the paper value); MAPQ 51.6 vs 31.3 MISMATCH (raw erne-bs5 MAPQ scale differs from final-BAM Qualimap; tool-stage artifact, flagged). MethylDackel (the brief-pinned tool) runs and emits correctly-formatted per-context CpG/CHG/CHH bedGraphs (confirmed). S4 (Table S4) MISMATCH with full root-cause: the EpiDiverse/snp chain (params verified IDENTICAL to lib/snp.nf) runs end-to-end but over-calls ~4.9x (27.1M SNPs/15.3M Q30 vs 5.58M/1.69M). I directly TESTED the leading hypothesis (missing Picard dedup) by re-running the whole chain on a deduplicated BAM (16.47% dups removed): counts moved <2% -> dedup hypothesis REFUTED; the over-call comes from the upstream single-aligner BAM + bisulfite strand-masking on broken pair-flags, not duplicates. S5 DMR(4300) / S6 EWAS(93) NOT ATTEMPTED by design (whole 23-sample cohort = full-study re-run; stretch-only per scope.md). Compute genuinely ran across 4 completed «our HPC» jobs; honest partial with a tested negative result, no fabrication. PROVISIONAL — must be independently checked.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
Assessment versions
Every reproduction run is kept as an immutable version — anchored to the data as it stood, with a tamper-evident chain hash. A rerun (e.g. after an author updates a deposit) adds a new version; the previous one stays on record.
-
v1 current initial assessment Score 50assessed: 2026-06-18 ⛓ a943f4b0d8e3
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-26
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-19no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetThe paper presents the EpiDiverse Toolkit to test whether a standardized, FAIR-compliant pipeline suite can address the computational challenges (cost, poor reference genomes, multi-context methylation) of applying whole genome bisulfite sequencing analysis to non-model plant species at population and community scale.
- ★ EpiDiverse Toolkit provides Nextflow-based pipelines for WGBS mapping, methylation calling, variant calling, differential methylation, and EWAS tailored to non-model plant ecology resource
- ★ A novel masking procedure enables conventional SNP calling and k-mer-based sample clustering directly from bisulfite sequencing data, avoiding need for separate DNA-seq method
- ★ High-sensitivity mapping mode gives demonstrable improvement in precision-recall for non-model plant methylation analysis compared to high-throughput mode finding
- metilene shows higher sensitivity for finding DMRs compared to other tools per a recent benchmark finding
- ★ Region-level EWAS analysis using DMRs is more robust and statistically powerful than position-level analysis of all methylated sites due to reduced multiple-testing burden finding
- ★ Incorporating SNP data derived from the same bisulfite data into the EWAS G-model can resolve the confounding genetic component driving epigenetic associations mechanism
- ★ kWIP k-mer diversity-based clustering can indicate DNA methylation-based sample grouping and may be more robust than site-by-site methylation clustering finding
- Toolkit is built on Nextflow following nf-core best practices, containerized (Bioconda/Docker/Singularity) for portability and reproducibility across systems resource
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| Whole genome bisulfite sequencing (WGBS) mapping and methylation calling | Populus nigra leaf tissue, 23 libraries (German and Lithuanian clone populations) | none | mapping rate, genome-wide methylation level (CG/CHG/CHH) | MethylDackel; samtools stats/plot-bamstats |
| Variant calling (SNP) from masked bisulfite alignments | P. nigra WGBS BAM files | none | SNPs/short variants in VCF/BCF format | Freebayes; bcftools stats/plot-vcfstats |
| Sample clustering via k-mer diversity | P. nigra WGBS BAM files (variant-masked) | none | distance matrix / hierarchical clustering tree | kWIP |
| Differential methylation region/position analysis | P. nigra bedGraph methylation files, German vs Lithuanian population groups | none (population/group comparison) | significant DMRs/DMPs (q<0.05), hyper/hypo-methylation heatmaps | metilene |
| Epigenome-wide association study (EWAS) | P. nigra methylation sites/DMRs, SNPs, and leaf flavonol content phenotype (parent generation) | none (association/QTL mapping) | quantitative trait loci (QTLs), position- and region-level P/q-values | GEM suite |
- – WGBS mapping rates ranged from 78.38% to 80.44% across 23 P. nigra libraries under default parameters 78.38%-80.44%
- – Hierarchical clustering of methylated sites (all contexts) cut at ~5.25×10^6 formed two discrete groups plus one outlier (LT_02) 5.25e6 cut height
- – Significant DMRs (q<0.05) were identified from pairwise comparison of German and Lithuanian population groups
- – Position-level EWAS (E-model, all methylated CG sites) found no QTLs significant at common (P<1e-8) or suggestive (P<1e-6) thresholds
- ▲ Region-level EWAS restricted to significant DMRs found 92 QTLs significant at q<0.25, with one at q<0.05 92 QTLs (q<0.25)
- – kWIP-based clustering showed greater congruency with DMR heatmap grouping than site-by-site methylation clustering in some samples (e.g. LT_10, DE_41, DE_44)
- – Functional annotation of significant QTL regions identified genes homologous to an ascorbate-specific transmembrane electron transporter, a caspase family protein, a mechanosensitive ion channel protein 3, and methyltransferases PMT2/PMT24 potentially relevant to flavonol content
- other 78.38%-80.44% (WGBS mapping rate range across P. nigra libraries)
- other 5.25 x 10^6 (hierarchical clustering tree cut height forming two groups plus one outlier)
- pvalue q < 0.05 (significance threshold for differential methylated regions (DMRs))
- pvalue P < 1 x 10^-8 (common (Bonferroni) significance threshold for position-level EWAS)
- pvalue P < 1 x 10^-6 (suggestive significance threshold for position-level EWAS)
- count 92 significant QTLs at q < 0.25 (one at q < 0.05) (region-level EWAS using significant DMRs as markers)
- count 23 independent WGBS libraries (test dataset from P. nigra German and Lithuanian populations)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This paper presents the EpiDiverse Toolkit, a bioinformatics pipeline suite for WGBS analysis in ecological plant epigenetics, demonstrated on 23 P. nigra libraries from two clone populations. The toolkit applies non-parametric differential methylation testing with metilene (with multiple comparison correction yielding q-values), hierarchical and k-mer-based sample clustering for group definition, and epigenome-wide association studies via the GEM E-model at both genome-wide position-level and pre-filtered region-level. Results are primarily reported as counts of significant DMRs (q < 0.05) and QTLs (q < 0.25 or q < 0.05) rather than effect-size estimates.
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| metilene non-parametric test (MWU/KS-based) for differential methylation | Pairwise comparison of methylated sites between hierarchically-defined groups of P. nigra samples; Figure 2B, Supplementary Table S5 | 23 WGBS libraries | stated |
| GEM E-model linear regression (EWAS) | Association of CG methylation (position-level and DMR-level) with leaf flavonol content; Figure 3, Supplementary Table S6 | 23 WGBS libraries | not stated |
| Hierarchical clustering with manual tree cut at 5.25×10⁶ | Unsupervised grouping of P. nigra samples by shared methylated sites prior to DMR analysis; Figure 2A | 23 WGBS libraries | not stated |
| kWIP k-mer weighted inner product distance matrix with hierarchical clustering | Alternative sample clustering based on k-mer diversity after SNP masking; Supplementary Figure S2b | 23 WGBS libraries | not stated |
| Principal component analysis (PCA) | Unsupervised visualisation of sample variation at shared methylated sites; Supplementary Figure S1 | 23 WGBS libraries | na |
-
Sample groups for DMR analysis were defined by manually cutting the hierarchical clustering dendrogram at a single chosen threshold (5.25×10⁶)↳ Could also: Dynamic tree cut algorithms (e.g., dynamicTreeCut R package) or model-based clustering (e.g., Gaussian mixture models via mclust) could also be used — Data-driven cut methods determine the number of clusters from the data itself rather than requiring a manually chosen threshold, which may be informative when the expected number of biological groups is not known a priori
-
Position-level EWAS significance was evaluated against the heuristic threshold P < 1×10⁻⁸, which the paper itself notes may be less than ideal for epigenetic data↳ Could also: Permutation-based significance thresholds or genomic-control lambda inflation adjustment could also be applied — Permutation preserves the actual correlation structure of the tested positions and can yield thresholds calibrated to the specific number and dependence structure of tests, rather than a fixed genome-wide heuristic borrowed from GWAS
-
Differential methylation was identified using metilene, a pairwise-comparison approach, with groups defined by global hierarchical clustering↳ Could also: Beta-binomial models implemented in DSS or MethylKit could also be used for DMR/DMP analysis — Beta-binomial approaches explicitly model overdispersion in per-site methylation read counts across replicates, which may improve sensitivity and specificity when within-group variability is high or replicate counts are modest
-
The EWAS E-model was run without adjusting for the genetic component; the G-model (incorporating SNP data) is described as optional and available in the pipeline↳ Could also: A linear mixed model incorporating a genetic relatedness matrix (GRM) as a random effect could also be used to control for population structure — Accounting for shared genetic background is especially relevant when comparing populations from distinct geographic origins, as unmeasured genetic similarity can confound methylation–phenotype associations
-
Region-level EWAS significance was reported at q < 0.25 as the primary threshold, with one region also meeting q < 0.05↳ Could also: Reporting results at a stricter primary threshold (e.g., q < 0.05 or q < 0.10) or applying a two-stage discovery/replication design could also be used — Simultaneously reporting findings at multiple FDR thresholds gives readers information to assess the likely proportion of false positives among reported associations, particularly relevant when sample sizes are small
-
PCA was used alongside hierarchical clustering for unsupervised exploration of methylation profiles↳ Could also: UMAP or t-SNE could also be used for non-linear dimensionality reduction of methylation data — Non-linear methods can reveal local neighbourhood structure and sample sub-groupings that are compressed or distorted in PCA, and may be informative when methylation variation reflects multiple interacting biological axes
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-34805989 (EpiDiverse Toolkit, NAR Genom Bioinform 2021)
Paper: Nunn et al. 2021. EpiDiverse Toolkit: a pipeline suite for the analysis of bisulfite sequencing data in ecological plant epigenetics. PMCID PMC8598301, DOI 10.1093/nargab/lqab106.
Nature of the paper: A software/pipeline paper. The EpiDiverse Toolkit is a Nextflow suite of 4 pipelines: wgbs (alignment + methylation calling), snp (masked variant calling), dmr (metilene differential methylation), ewas (GEM association). It is demonstrated on a Populus nigra WGBS test cohort.
Code:
- Authors' pipelines: https://github.com/EpiDiverse/{wgbs,snp,dmr,ewas} (v1.0, Sep 2020).
- The brief pins the third-party tool MethylDackel (github.com/dpryan79/MethylDackel),
which is the methylation-extractor invoked inside EpiDiverse/wgbs (Suppl Table S1
lists a
MethylDackelprocess). Per P16, running the authors' own pipeline (which embeds MethylDackel) is the faithful and equally-valid reproduction.
Data:
- Reads: ENA PRJEB44879 — Populus nigra var. italica WGBS, 150 bp paired-end. Paper uses 23 libraries (13 DE + 10 LT, Suppl Table S2). The full deposit contains 205 samples / 421 runs (4 populations CZ/DE/LT/PL) — the paper uses a subset. Each used sample = 2 ENA runs (2 lanes), 21–37 Gbp, ~40–68× genome coverage.
- Reference: ENA PRJEB44889 = assembly GCA_950102115 (Pnigra-var-italica, 417 Mbp).
In scope (pipeline-derived, attempted)
| # | Result | Pipeline | Reported (loc) | Plan |
|---|---|---|---|---|
| 1 | Per-sample mapping rate % | wgbs (erne-bs5 default) | 78.38–80.44%, per-sample in Suppl Table S3 | Run EpiDiverse/wgbs on individual sample(s), read mapping rate from bam_statistics; compare to S3 row |
| 2 | Per-sample coverage mean, GC%, MAPQ mean, insert-size median | wgbs | Suppl Table S3 | Same run, samtools stats |
| 3 | Per-sample variant counts (total SNPs, SNPs>Q30) | snp | Suppl Table S4 | Run EpiDiverse/snp on the same BAM(s); compare to S4 |
| 4 | MethylDackel methylation calls (bedGraph) | wgbs / MethylDackel | (deposited as part of pipeline output; the tool the brief pins) | Confirm MethylDackel runs and emits per-context calls |
Harder / aggregate results (attempt only if all-sample compute is feasible)
| # | Result | Pipeline | Reported (loc) |
|---|---|---|---|
| 5 | 4300 significant CG DMRs (q<0.05) | dmr (metilene) | Suppl Table S5 (4300 rows) — requires ALL 23 samples mapped + grouped |
| 6 | 93 region-level EWAS QTLs (q<0.25; 1 at q<0.05) | ewas (GEM) | Suppl Table S6 (text says "92") — requires full WGBS+SNP+DMR on 23 samples |
Results 5–6 require the entire 23-sample WGBS run (~617 Gbp) plus SNP+DMR+EWAS — a full re-run of the study. Treated as stretch goals; if compute is not feasible they are honestly recorded as not attempted, not as a mismatch.
Out of scope (not pipeline-derived / not attempted)
- Wet-lab flavonol measurements (RAU, Suppl Table S2) — assay, not pipeline.
- Resource/runtime configuration values (Suppl Table S1) — infra config, not a result.
- Hierarchical clustering tree-cut threshold (~5.25e6) — depends on full cohort.
Honesty notes
- Paper text says "92 significant QTLs (q<0.25)"; Suppl Table S6 actually lists 93 rows with q<0.25. Minor reported-vs-shipped discrepancy, flagged in claims.tsv.
- The default aligner is erne-bs5; whether S3 rates are erne-bs5-only or erne-bs5+segemehl merged is resolved at run time by matching the pipeline's default behaviour.
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.