msPIPE: a pipeline for the analysis and visualization of whole-genome bisulfite sequencing data.
The main results reproduced: recomputed values matched the published ones within tolerance.
- ✓Same input data as the authors
- ✓Reported values were directly comparable
- ✓No relevant deviation in data/preprocessing
- ✓No authors-side cause for any deviation
- ✓Reported values are derivable from the shared data
- ✓Any deviation was negligible
- ✓The central claim held under reproduction
- ✓Overall, the reproduction was clean
- Every checked point held up.
A 0–100 reproducibility-quality score from the per-question grades, shown as a z-score: standard deviations above (+) or below (−) the mean of comparable assessments.
▸Reproduction agent’s raw note
REPRODUCED (headline, exact 1:1). Ran the authors' own msPIPE (Docker image jkimlab/mspipe:latest via Apptainer; Bismark v0.23.1dev, methylKit 1.20.0) on the paper's own data PRJEB28044 (4 pooled human WGBS runs -> hg38) with documented default parameters (q<=0.5). BOTH reported human DMC counts reproduce EXACTLY: C1 blood old-vs-young = 244 (paper 244) and C2 sperm old-vs-young = 34514 (paper 34514). All 8 fastq MD5-verified; Bismark mapping efficiency 89.5%; ~50M CpGs covered/sample; methylKit united 9.44M (blood)/8.17M (sperm) CpGs. Found and worked around a real msPIPE bug: it samtools-sort -n's the Bismark BAM and feeds the sorted BAM to the methylation extractor, which (samtools 1.6) breaks mate adjacency -> empty methylation calls while the job exits rc=0; fix = extract on the unsorted bam. Compute was staged across chained <=12h SLURM jobs («infra» assoc cap) and shifted to node-local scratch to survive a shared-account storage-quota exhaustion. NOT completed (honest scope): C3 (393 promoter genes) blocked by msPIPE's GMA failing to emit promoter.bed; C5/C6 (Fig2 CHG/CHH genome-wide context) skipped (CX report too slow + quota); C7 (Table2 GO) g:Profiler errored + depends on C3; C4 mouse not attempted. Verdict: the paper's two headline pipeline-derived results are faithfully and EXACTLY reproducible; secondary annotation/visualization outputs were not completed in this environment.
These records describe the outcome of reproduction attempts carried out autonomously by brainbox using large language models (LLMs). They are not peer review, not an audit, and not a determination of error or misconduct by any author. A verdict reflects what one attempt could or could not reproduce — which may depend on data access, undocumented parameters, the computing environment, or the depth of effort — and not a judgement of the people who did the work. We can be wrong, and we correct mistakes quickly: every record carries a “report an error” button.
✎ I am an author of this paper
Updated or fixed a deposit, or is there an erratum? Ask us to re-run the metrics. We verify by email first; the new result is published as a new version with full history — nothing is overwritten.
Provenance — full disclosure
When this reproduction was carried out, which methodology version was used, and by whom — so the record can be audited and checked independently.
- Reproduced
- 2026-06-23
- Rubric version
- v1.0
- Assessed by
-
🤖 AI curator · claude (ai-curator room) · v1.0 · run #1 2026-06-23no human curator yet
- Last updated
- 2026-08-05
Provisional, curator- or AI-assessed, and independently checkable. A reproduction outcome states what one attempt could reproduce — not a judgement of the authors.
Deep full-text extraction
Model: sonnetExisting computational pipelines for whole-genome bisulfite sequencing (WGBS) analysis are incomplete (often lacking downstream differentially methylated region and hypomethylated region analyses) and difficult to apply beyond human data, so an easy-to-use, end-to-end pipeline covering pre-processing through downstream methylation analysis and visualization is needed.
- ★ msPIPE is a new end-to-end pipeline integrating pre-processing, alignment & methylation calling, and downstream DNA methylation analysis (profiling, HMR, DMR, functional enrichment) with publication-quality figure output for WGBS data method
- ★ msPIPE uniquely supports both DMR and hypomethylated region (HMR) analyses along with gene functional enrichment analysis, unlike most compared pipelines finding
- ★ msPIPE automatically prepares and sets reference genome sequences and annotations from the UCSC Genome Browser database for any BSgenome-supported species resource
- msPIPE supports two alternative alignment/methylation-calling tools (Bismark and BS-Seeker2), increasing flexibility method
- ★ msPIPE was successfully applied to human sperm/blood and mouse rod photoreceptor WGBS datasets to generate methylation profiles and identify DMRs with downstream functional findings finding
- msPIPE is distributed as a Docker image containing all dependent packages/software, simplifying installation resource
| Assay | System | Perturbation | Readout | Platform |
|---|---|---|---|---|
| WGBS read QC and trimming | human blood and sperm (6 young men 18-24y, 6 old men 61-71y) | aging (young vs old) | read quality metrics after trimming | TrimGalore!/FastQC/MultiQC |
| WGBS read QC and trimming | mouse rod photoreceptors (3 young 3-month-old, 3 old 24-month-old male mice) | aging (young vs old) | read quality metrics after trimming | TrimGalore!/FastQC/MultiQC |
| WGBS alignment and methylation calling | human sperm and blood samples | aging (young vs old) | CpG/CHG/CHH methylation calls | Bismark |
| WGBS alignment and methylation calling | mouse rod photoreceptor samples | aging (young vs old) | CpG/CHG/CHH methylation calls | Bismark |
| hypomethylated region (HMR) analysis | human old sperm sample | none | coordinates/counts of UMRs and LMRs, including promoter-overlapping UMRs | MethylSeekR |
| differential methylation (DMC/DMR) analysis | human blood/sperm, old vs young | aging (case vs control) | differentially methylated cytosines/regions, hyper- vs hypomethylation | methylKit |
| differential methylation (DMC/DMR) analysis | mouse rod photoreceptors, old vs young | aging (case vs control) | differentially methylated cytosines/regions | methylKit |
| functional enrichment analysis | genes with DMC/DMR-containing promoters (human and mouse) | none | enriched functional terms | g:Profiler |
- – In a feature comparison against 9 existing pipelines, msPIPE was the only one supporting two optional alignment/methylation-calling tools, additional downstream analyses, and automatic reference preparation
- – msPIPE and wg-blimp were the only compared pipelines supporting both DMR and HMR downstream analyses; only msPIPE additionally supported gene function analysis
- – msPIPE and Nextflow methylseq were the only pipelines that could automatically prepare and set required reference genome/annotation data
- – msPIPE generated methylation profiles (CpG/CHG/CHH context methylation levels across promoter, gene, exon, intron, intergenic regions and Circos plots) for the human old sperm sample
- – msPIPE was applied end-to-end to human (PRJEB28044) and mouse (PRJNA556668) WGBS datasets using Bismark for alignment/methylation calling and methylKit for DMC analysis, followed by g:Profiler functional enrichment of DMC-associated genes
- count 6 young men (18-24 years), 6 old men (61-71 years) (human WGBS dataset (blood and sperm) sample composition)
- count 3 young (3-month-old), 3 old (24-month-old) male mice (mouse WGBS dataset (rod photoreceptor) sample composition)
- other 70-90% (background: proportion of CpG dinucleotides methylated in normal human tissue cells)
- other methylKit DMR: >=2 DMCs within max 500 bp distance; BSmooth DMR: >=3 DMCs, >=10% methylation difference, >=70 CpGs, min length 1 Kbp (DMR definition criteria used by msPIPE's two supported DMR tools)
- other bin size 100 Kbp (genome-wide average methylation level bin size for Circos plot outer track)
- other window size 500 bp, step size 100 bp, -1500 to +1500 bp around TSS (sliding window used for per-gene TSS-centered methylation profiling)
- other -c 5 -q 0.5 (non-default msPIPE options (coverage and q-value cutoffs) used in the human/mouse dataset application)
Statistical methods review
Model: sonnetA neutral, descriptive read of the statistical approach — what was done, and (for shared learning, not as criticism) what could also have been done.
This is a software/methods paper describing msPIPE, an end-to-end computational pipeline for whole-genome bisulfite sequencing (WGBS) analysis; it is demonstrated on published human (6 young vs 6 old men, pooled blood/sperm libraries) and mouse (3 young vs 3 old male mice, rod photoreceptors) datasets. Differential methylation is identified using existing R packages (methylKit or BSmooth), which assign a q-value to each methyl-C position, and gene-level results are passed to the g:Profiler tool for functional enrichment. No classical inferential tests (e.g., t-test, ANOVA) for group comparison are described in the pipeline itself; statistical calling of differential/hypomethylated regions is delegated to the underlying published tools with pipeline-level threshold filtering (e.g., q-value cutoff, minimum methylation difference).
| Test | Applied to | n | Assumptions |
|---|---|---|---|
| methylKit differential methylation calling (per-cytosine test with q-value), aggregated into DMRs (≥2 DMCs within 500 bp) | pairwise case-vs-control comparison of methylation levels (DMC/DMR analysis), applied to human young-vs-old sperm and mouse young-vs-old rod comparisons | human: 6 young men vs 6 old men (pooled libraries); mouse: 3 young mice vs 3 old mice | not stated |
| BSmooth-based differential methylation calling (alternative to methylKit; DMR requires ≥3 DMCs, ≥10% methylation difference, ≥70 CpGs, ≥1 Kbp length) | pairwise case-vs-control DMR analysis (alternative option to methylKit within the pipeline) | same as above (per comparison set) | not stated |
| g:Profiler functional enrichment analysis | gene sets whose promoters overlap DMCs/DMRs, for human and mouse applications | not stated (depends on number of DMC/DMR-related genes identified) | not stated |
-
Differential methylation calling relies on q-values produced internally by methylKit or BSmooth, without the manuscript specifying the underlying per-cytosine test (e.g., Fisher's exact test, logistic regression, or beta-binomial model) or the multiple-testing correction algorithm.↳ Could also: Explicitly reporting the statistical model used for per-cytosine testing (e.g., logistic regression with overdispersion, or a beta-binomial model as in DSS/dmrseq) and the specific FDR method (e.g., Benjamini-Hochberg) applied. — Naming the exact test and correction method would let readers evaluate assumptions such as handling of biological variability between replicates and control of the false discovery rate across the genome-wide set of tested cytosines.
-
Human and mouse comparisons use small group sizes (6 vs 6 pooled libraries; 3 vs 3 mice) without a stated power calculation.↳ Could also: Reporting a power/sample-size justification, or supplementing q-value-based DMR calls with a resampling/permutation-based significance estimate. — This can help quantify how sample size affects the sensitivity and stability of DMR calls, particularly for pooled or low-replicate designs.
-
Methylation levels are presented as summary ratios/percentages and visualized in bar plots, histograms, and Circos plots without stated measures of variability (e.g., SD, SEM, or CI) across replicates.↳ Could also: Reporting a measure of dispersion (e.g., SD/CI across sample replicates) alongside average methylation levels in the profiling plots. — Adding a dispersion measure would convey how consistent methylation levels are across replicates within each sample, complementing the point-estimate bar/histogram plots.
-
Two alternative DMR-calling tools (methylKit and BSmooth) with different DMR definitions (e.g., minimum DMC count, distance thresholds, or minimum methylation difference) are offered as pipeline options, and results may differ depending on which is chosen.↳ Could also: Running both tools on the same dataset and reporting the overlap/concordance of DMRs identified, or using a consensus/ensemble approach across methods. — Cross-tool concordance checks are a standard way to characterize how sensitive DMR calls are to the choice of underlying statistical model.
-
Functional enrichment of DMC/DMR-associated genes is performed with g:Profiler without describing which specific statistical enrichment test (e.g., hypergeometric or Fisher's exact test) or background gene set was used.↳ Could also: Specifying the enrichment statistic and background/reference gene set, or comparing results across multiple enrichment tools (e.g., g:Profiler, DAVID, clusterProfiler). — Making the background set and test explicit clarifies how enrichment significance was computed and supports reproducibility of the functional analysis.
What was reproduced
The exact results taken into scope, with each reported value next to the value our attempt produced.
Scope — pmid-36123620 (msPIPE)
Paper: Kim et al. 2022, msPIPE: a pipeline for the analysis and visualization of whole-genome bisulfite sequencing data. BMC Bioinformatics 23:382. DOI 10.1186/s12859-022-04925-2 · PMCID PMC9487059.
Nature of the paper: This is a software/tool paper. msPIPE is the authors'
own pipeline (GitHub jkimlab/msPIPE) that wraps standard WGBS tools
(Trim Galore, FastQC/MultiQC, Bismark or BS-Seeker2, samtools, methylKit,
MethylSeekR, g:Profiler, circlize/ggplot2) into one config-driven workflow. The
"results" are demonstrations of the pipeline on two public WGBS datasets, so
essentially everything quantitative in the Results is pipeline-derived and in
scope — there is no wet-lab work by the authors; the sequencing was done by the
original data depositors.
Pinned repo commit: 65a63da2b23f70f52b50d39b75e0f73b9dd21467 (resolve/confirm on clone).
Datasets the paper runs the pipeline on
- Human WGBS — PRJEB28044 (RU's primary data accession). 4 paired-end runs,
reference hg38:
- ERR2722068 = pooled blood, old men · ERR2722070 = pooled blood, young men
- ERR2722069 = pooled sperm, old men · ERR2722071 = pooled sperm, young men
- Mouse WGBS — PRJNA556668 (rod photoreceptors), reference mm10. Paper uses 6 of the 11 deposited runs: 3M rep1-3 (SRR9833662/663/664, young) vs 24M rep1-3 (SRR9833670/671/672, old).
IN SCOPE (pipeline-derived results we attempt)
| # | Result | Pipeline path | Where reported |
|---|---|---|---|
| C1 | 244 DMCs (q≤0.5), human blood old-vs-young | Bismark→methylKit DMC | Results, human section |
| C2 | 34,514 DMCs (q≤0.5), human sperm old-vs-young | Bismark→methylKit DMC | Results, human section |
| C3 | 393 differentially methylated genes with ≥1 DMC in promoter (sperm) | DMC→GMA/promoter annotation | Results; Table 2 input |
| C4 | 274 DMCs, mouse rod 3M-vs-24M | Bismark→methylKit DMC | Results, mouse section |
| C5 | CpG/CHG/CHH methylation context distributions; CHG/CHH "hardly methylated" in human, "clearly detected" in mouse (Fig 2c-f) | methylation calling | Fig 2 |
| C6 | Avg CpG methylation by genomic context (promoter/gene/exon/intron/intergenic) (Fig 2b) | GMA | Fig 2b |
| C7 | GO/functional enrichment of the 393 genes (Table 2) | g:Profiler | Table 2 |
Primary target (quick-minimum / floor): install + run msPIPE end-to-end on the human dataset and reproduce C1, C2 (the two headline DMC counts). One msPIPE run over the 4 human samples with two DMR comparisons yields C1+C2 and feeds C3/C5/C6/C7. Stretch: C3–C7, then the mouse dataset (C4).
OUT OF SCOPE / not attempted
- Table 1 (qualitative feature comparison of 10 pipelines): editorial/manual literature comparison, not a computed result.
- Fig 1 (workflow schematic): a diagram.
- Original wet-lab sequencing (done by the upstream depositors, not the authors).
- Exact runtime/benchmark wall-times: hardware-dependent, not 1:1 reproducible; will report our observed runtimes as context only, not graded.
Reproduction approach
Run the authors' own tool (msPIPE) on the paper's own data with the documented default parameters (Bismark mapping, q-value 0.5). This is a faithful 1:1 attempt. Heavy compute on «our HPC»/SLURM; data + intermediates on «infra».
Known risk / feasibility notes
- Human WGBS is large: ~1.4B read pairs total across 4 runs (~180 GB gz), aligned to
hg38 with Bismark — hundreds of core-hours; will run per-sample alignment jobs and
resume via msPIPE
--skip_*flags. - DMC counts (esp. C1=244) are sensitive to methylKit version, coverage filters, q-value handling, and Bismark version — exact match is not guaranteed; within-order / same-direction agreement is a meaningful positive result. Will pin tool versions from the paper where stated (Trim Galore 0.6.0, Bismark 0.20.0, MethylSeekR 1.34.0).
Assessments & scoring basis
Each contributor’s verdict, the per-question basis, and the auditable, itemised worksheet behind it.
An automated assessment. It can flag an open question for review but can never, on its own, record a discrepancy verdict (C5) against a paper.
Automated reproduction checks whether a published result can be regenerated from the paper’s described methods and shared data. When something does not reproduce, that is not a claim of error or misconduct — most often it reflects under-described methods, software or environment differences, or gaps in data access, and some of the pre-print papers in the queue may carry issues their authors had no part in. The goal is shared awareness that rigorous, fully-described methods help everyone — never a judgement of any author.
Are you an author? We would genuinely like to hear from you — to clarify the record, add data or code, re-run the pipeline after an accession update, and publish your response right next to the assessment. Everything here is open and auditable.
🚩 Report an error in this record
Spotted something wrong — a verdict you’d contest, a data or value error, or a private detail that slipped through? Tell us, with a short justification. Authors and readers are equally welcome to write in; we review every report.
Prefer email, or the form below not working? Contact us at support@doesitreproduce.com.
Reproduction footprint
claude-opus-4-8Measured resources invested to assess this paper — sanitised (machine class only, no job ids/paths). Compute = HPC accounting (SLURM); tokens = the AI agent's session.